Irish Computer Science Leaving Certificate Curriculum Consultation Update

Last Tuesday I attended a consultation session for the Leaving Certificate Computer Science Curriculum. This is Ireland’s shot at putting CS on the pre-university curriculum, specifically the Irish Senior Cycle – which leads right up to where secondary school and university meet. I am particularly interested in this as I teach, research, and am pretty much obsessed with CS1 – the first programming course that CS majors take at university. I am also teaching this year on a new programme at my university, University College Dublin (with support from Microsoft), that is one of the first (if not the first) teacher training programmes specifically for this new curriculum.

The event was hosted by the National Council for Curriculum and Assessment, and was addressed by Irish Minister for Education and Skills, Richard Bruton. It was an engaging and lively day of discussion and it was really good to see so many different stakeholders in attendance. I was in one of I believe 6 (or more) focus groups, and we had university professors, industry leaders (including Apple and Microsoft), current (and former) school teachers, and a member of the curriculum development team in the room, (and I am missing a few people here).

There is another consultation event on September 16 at Maynooth University, hosted by the Computers in Education Society Ireland (CESI). The consultation officially closes on September 22, and a final draft of the curriculum is expected soon thereafter.

Introducing programming to the Irish primary and secondary curricula

follow 3 pngYesterday the Irish government anounced more strategy for their plan to introduce programming to the primary and secondary curricula. As currently set out, this will take place through the mathematics curriculum (at least at the primary level). As part of the wider plan, Computer Science will be introduced as a secondary level (high school) subject, a development I have also been following recently.

The Digital Strategy for Schools 2015-2020 plan for 2017, which was launched by Minister for Education Richard Bruton yesterday, seeks to increase the use of ICT (information and communications technology) in Irish schools and includes the introduction of a benchmarking system from this September to allow teachers to track their progress in using digital technologies when teaching.

I have read the Department of Education press release, and the Digital Strategy for Schools 2015-2020 Action Plan 2017 (both cited by The Irish Times yesterday) as well as the Digital Strategy for Schools 2015-2020 Action Plan cited by The Times (London) yesterday. It seems at this point there are plenty of  “details” but not many details.

So I dug into the references of the Digital Strategy for Schools 2015-2020 Action Plan and found Programming and Coding -Draft Specification for Junior Cycle Short Course. This (although not specifically part of the 2015-2020 Action Plan, or the Action Plan 2017) had some interesting details. Note: this dates from October 2013 and was published by the National Council for Curriculum and Assessment, one of the many chefs in a busy kitchen. The Junior Cycle is the first three years of second-level education, set at level 3 of the National Framework of Qualifications. The closest analogue in the US would be Junior High School. Under Strand 1 Computer science introduction it is stated that students should learn about My digital world: The importance of computers in modern society and my life Being a coder – step by step: How to start programming and develop basic algorithms. The following learning outcomes are listed:

1.1 present and share examples of what computers are used for and discuss their importance in modern society and in their lives
1.2 describe the main components of a computer system (CPU, memory, main storage, I/O devices, buses)
1.3 explain how computers are devices for executing programs via the use of programming languages
1.4 write code to implement algorithms
1.5 test the code
1.6 develop appropriate algorithms using pseudo-code and/or flow charts
1.7 discuss and implement core features of structured programming languages, such as variables, operators, loops, decisions, assignment and modules
1.8 evaluate the results in groups of two or three

Under Strand 2 Let’s get connected it is stated that students should learn about Making connections: Computers are communication devices and Bits and bytes: How computers store data. The following learning outcomes are listed:

2.1 discuss the basic concepts underlying computer networks
2.2 describe how data is transported on the Internet and how computers communicate and cooperate through protocols such as HTTP
2.3 build web pages using HTML and CSS
2.4 explain how search engines deliver results
2.5 explain how computers represent data using 1’s and 0’s
2.6 investigate how drawings and photos are represented in computing devices

Under Strand 3 Coding at the next level it is stated that students should learn about More advanced concepts in programming and computational thinking and
Documentation and code analysis. The following learning outcomes are listed:

3.1 creatively design and write code for short programming tasks to demonstrate the use of operators for assignment, arithmetic, comparison, and Boolean combinations
3.2 complete short programming tasks using basic linear data structures (e.g. array or list)
3.3 demonstrate how functions and procedures (definition and call) capture abstractions
3.4 describe program flow control e.g. parallel or sequential flow of control – language dependent
3.5 document programs to explain how they work
3.6 present the documented code to each other in small groups
3.7 analyse code to determine its function and identify errors or potential errors

Under Strand 4 Problem solving in the real world it is stated that students should learn about Real world problems: Computer Science inspiring me and computational thinking and Putting the pieces together: Build a final software project that incorporates concepts learnt in the previous strands. The following learning outcomes are listed:

4.1 identify a topic or a challenge in computer science that inspires them
4.2 conduct research on the topic/challenge
4.3 work in teams of two or three and decide on a topic or challenge on which to build a final software project
4.4 brainstorm ideas in the requirements-gathering phase
4.5 discuss aspects of user-interaction design for the project
4.6 design, implement and test a solution
4.7 document team contributions and document the code
4.8 present to peers for feedback
4.9 assess the feedback
4.10 based on feedback, complete the software project and present a convincing argument for the final proposal to their peers

Now, that’s meaty! I have to admit I found this most intriguing: 3.4 describe program flow control e.g. parallel or sequential flow of control – language dependent. The report goes on to discuss assessment also. I won’t analyze that here, but I’ll copy the ‘Features of Quality’ for Strand 3 Coding at the next level:

Achieved with Distinction (90-100%): There is evidence that the programming tasks are executed with complete confidence and there is a very high level of creativity demonstrated. The tasks demonstrate an excellent understanding and comprehensive knowledge of the advanced concepts of programming and computational thinking. There is evidence that very good connections are made between team members and effective and communicative team working is demonstrated.

If this could be accomplished by the age of 14 or so we would be cooking with gas! Let’s not forget though, this is a 2013 draft specification and afor now we have no idea if any of this will become reality.

Back to the Digital Strategy for Schools 2015-2020 Action Plan 2017. Below I have included ‘Key Elements of the Plan’ and ‘What does success look like’. Of these, I found this to be particularly interesting: All students [will] have a digital portfolio with self-created content across the entire curriculum and a recognised capacity in discerning the ethical use of digital technologies.

Key elements of the plan include:

  • A new clustering programme, through which schools across the country will collaborate with each other on innovative projects for using digital technology in teaching and learning. This will be an important means of encouraging schools to innovate in this area, and also for the system as a whole to develop new method
  • A programme of curriculum reforms will see ICT embedded in all emerging curricular specifications and intense preparation for the phased introduction of Computer Science as a Leaving Certificate subject option from 2018 and coding as part of the primary school maths curriculum
  • A new Digital Learning Framework will be trialled in the new school year and will allow schools to evaluate their progress and measure how they stand against benchmarks of highly effective practice of using digital technologies in teaching and learning. Examples of good practice will continue to be captured and shared amongst the teaching community.
  • Provision of a range of professional learning programmes for teachers and school leaders to enable them engage effectively in whole school planning and self-evaluation to support them to embed digital technologies in teaching, learning and assessment
  • A full suite of content and exemplars of good practice available through an online portal which will also facilitate the sharing of good practice between teachers.
  • The continuing rollout of a €210million capital investment programme backed by the dissemination of research on best practice in equipment selection, collaboration and technical support
  • A progressive programme of high-speed broadband connectivity
  • New ways of business and industry supporting schools in embedding digital technologies in all aspects of their activities
  • Reviewing policy on the ethical use of the internet and online safety for young people

What does success look like?

1.   All functions of teaching and learning in  schools are fully digitally supported and enabled, with full engagement across the entire school community

2.   All subject specifications support a constructivist learning model and reflect the role of digital technologies in facilitating this model

3.   All students have a digital portfolio with self-created content across the entire curriculum and a recognised capacity in discerning the ethical use of digital technologies

4.   All schools can demonstrate effective or highly effective practice as described in the Digital Learning Framework, underpinned by a whole-school approach to e-planning

5.   All teachers have upskilled and embed digital technologies in their teaching practice

6.   Our Providers of Initial Teacher Education have become recognised leaders in innovative learning for quality outcomes

7.   An increasing number of schools participating in clusters each year leading innovation in the use of digital technologies that can be disseminated to all schools

8.   Good protocols are in place to assist schools in managing their digital resources with robust relationship with industry, business and higher education institutions

9.   All schools have high speed broadband connectivity

10. Schools use a variety of ICT equipment and delivery models for supporting their learning activities with demonstrable cost effectiveness and shared learning

Sources: Irish Times, The Times (London), the Department of Education (Ireland), Digital Strategy for Schools 2015-2020 Action Plan 2017, Digital Strategy for Schools 2015-2020 Action Plan, and Programming and Coding -Draft Specification for Junior Cycle Short Course.

The enhancing compiler error messages saga: the saga continues

I was sorry to have missed Raymond Pettit, John Homer and Roger Gee presenting the latest installment of what is becoming the enhanced compiler error messages saga at SIGCSE earlier this month. Their paper “Do Enhanced Compiler Error Messages Help Students?: Results Inconclusive” [1] was two-pronged. It contrasted the work of Denny et al. [2] (which provided evidence that compiler error enhancement does not make a difference to students) and my SIGCSE 2016 paper [3] (which provided evidence that it does). It also provided fresh evidence that supports the work of Denny et al. I must say that I like Pettit et al.’s subtitle: “Results Inconclusive”. I don’t think that this is the final chapter in the saga. We need to do a lot more work on this.

Early studies on this often didn’t include much quantifiable data. More recent studies haven’t really been measuring the same things – and they have been measuring these things in different ways. In other words, the metrics, and the methodologies differ. It’s great to see work like that of Pettit et al. that is more comparable to previous work like that of Denny et al.

One of the biggest differences between my editor, Decaf, and Pettit et al.’s tool, Athene, is that Decaf was used by students for all of their programming – practicing, working on assignments, programming for fun, even programming in despair. For most of my students it was the only compiler they used – so they made a lot of errors, and they all were logged. Unlike Denny et al., my students did not receive skeleton code – they were writing programs, often from scratch. On the other hand, Athene was often utilized by students after developing their code on their own local (un-monitored) compilers. Thus, many errors generated by the students in the Pettit et al. study were not captured. Often, the code submitted to Athene was already fairly refined. Pettit et al. even have evidence from some of their students that at times the code submitted to Athene only contained those errors that the students absolutely could not rectify without help.

As outlined in this post, Denny et al. and I were working towards the same goal but measuring different things. This may not be super apparent at first read, but under the hood comparing studies like these is often a little more complicated than it first looks. Of course these differences have big implications when trying to compare results. I’m afraid that the same is true comparing my work with Pettit et al. – we are trying to answer the same question, but measuring different things (in different ways) in order to do so.

Specifically, Pettit el al. measured:

  1. the number of non-compiling submissions; similar to did Denny et al., but unlike me
  2. the number of successive non-compiling submissions that produced the same error message; Denny et al. measured the number of consecutive non-compiling submissions regardless of why the submission didn’t compile, and I measured the number of consecutive errors generating the same error message, on the same line of the same file
  3. the number of submission attempts (in an effort to measure student progress)
  4. time between submissions; neither Denny et al. nor I measured time-based metrics

I also did a fairly detailed comparison between my work and Denny et al. in [4] (page 10). In that study we directly compared some effects of enhanced and non-enhanced error messages:

In this study we directly distinguish between two sets of compiler error messages (CEMs), the 30 that are enhanced by Decaf and those that are not. We then explore if the control and intervention groups respond differently when they are presented with these. For CEMs enhanced by Decaf the control and intervention groups experience different output. The intervention group, using Decaf in enhanced mode, see the enhanced and raw javac CEMs. The control group, using Decaf in pass-through mode, only see the raw javac CEMs. Thus for CEMs not enhanced by Decaf, both groups see the same raw CEMs. This provides us with an important subgroup within the intervention group, namely when the intervention group experiences errors generating CEMs not enhanced by Decaf. We hypothesized that there would be no significant difference between the control and intervention groups when looking at these cases for which both groups receive the same raw CEMs. On the other hand, if enhancing CEMs has an effect on student behavior, we would see a significant difference between the two groups when looking at errors generating the 30 enhanced CEMs (due to the intervention group receiving enhanced CEMs and the control group receiving raw CEMs).

As mentioned, the metrics used by Pettit et al. and Denny et al. are more common to each other than to mine. Pettit et al. and Denny et al. both used metrics based on submissions (that is, programs) submitted by students, or the number of submission attempts. This certainly makes comparing their studies more straight-forward. However it is possible that these metrics are too ‘far from the data’ to be significantly influenced by enhanced error messages. It is possible that metrics simply based on the programming errors committed by students, and the error messages generated by these errors are more ‘basic’ and more sensitive.

Another consideration when measuring submissions is that just because a submission compiles, does not mean that it is correct or does what was intended. It is possible that some students continue to edit (and possibly generate errors) after their first compiling version, or after they submit an assignment. These errors should also be analyzed. I think that in order to measure if enhancing error messages makes a difference to students we should focus on all programming activity. I’m afraid that otherwise, the results may say more about the tool (that enhances error messages) and the way that tool was used by students, than about the effects of enhanced error messages themselves. I am sure that in some of my research this is also true – after all my students were using a tool also, and this tool has its own workings which must generate effects. Isolating the effects of the tool from the effects of the messages is challenging.

I am very glad to see more work in this area. I think it is important, and I don’t think it is even close to being settled. I have to say I really feel that the community is working together to do this. It’s great! In addition there may be more to do than determine if enhanced compiler errors make a difference to students. We have overwhelming evidence that syntax poses barriers to students. We have a good amount of evidence that students think that enhancing compiler error messages makes a positive difference. Some researchers think it should too. If enhancing compiler error messages doesn’t make a difference, we need to find out why, and we need to explain the contradiction this would pose. On the other hand, if enhancing compiler error messages does make a difference we need to figure out how to do it best, which would also be a significant challenge.

I hope to present some new evidence on this soon. I haven’t analyzed the data yet, and I don’t know which way this study is going to go. The idea for this study came from holding my previous results up to the light and looking at them from quite a different angle. I feel that one of the biggest weaknesses in my previous work was that the control and treatment groups were separated by a year – so that is what I eliminated. The new control and treatment groups were taking the same class, on the same day – separated only by lunch break. Fortuitously, due to a large intake CP1 was split into two groups for the study semester, but was taught by the same lecturer in the exact same way – sometimes things just work out!

I will be at ITiCSE 2017 and SIGCSE 2018 (and 2019 for that matter – I am happy to be serving a two year term as workshop co-chair). I hope to attend some other conferences also but haven’t committed yet. I look forward to continuing the discussion on the saga of enhancing compiler error messages with anyone who cares to listen! In the meantime here are a few more posts where I discuss enhancing compiler error messages – comments are welcome…

[1] Raymond S. Pettit, John Homer, and Roger Gee. 2017. Do Enhanced Compiler Error Messages Help Students?: Results Inconclusive.. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE ’17). ACM, New York, NY, USA, 465-470. DOI: https://doi.org/10.1145/3017680.3017768

[2] Paul Denny, Andrew Luxton-Reilly, and Dave Carpenter. 2014. Enhancing syntax error messages appears ineffectual. In Proceedings of the 2014 conference on Innovation & technology in computer science education (ITiCSE ’14). ACM, New York, NY, USA, 273-278. DOI: http://dx.doi.org/10.1145/2591708.2591748

[3] Brett A. Becker. 2016. An Effective Approach to Enhancing Compiler Error Messages. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education (SIGCSE ’16). ACM, New York, NY, USA, 126-131. DOI: https://doi.org/10.1145/2839509.2844584

full-text available to all with link available at www.brettbecker.com/publications

[4] Brett A. Becker, Graham Glanville, Ricardo Iwashima, Claire McDonnell, Kyle Goslin, Catherine Mooney. 2106. Effective Compiler Error Message Enhancement for Novice Programming Students, Computer Science Education 26(2-3), pp. 148-175; http://dx.doi.org/10.1080/08993408.2016.1225464

full-text available to all at www.brettbecker.com/publications

Computer Science to be offered in Irish secondary curriculum by 2019

According to RTE news, the Irish Minister for Education has said that Computer Science, including programming, will be a Leaving Certificate subject by 2019, paving the way for Irish students to have the option to study the subject at secondary school. However the nature of the curriculum has not been settled, nor is not known how many schools will offer the subject. The rules in Irish schools vary, but most require students to take seven subjects for the Leaving Certificate. Some students may take fewer, for example, those who are not taking Irish. Other students may choose to take more. There is no specific rule about how many subjects a student should take, however students must pass six subjects in the Leaving Certificate in order to be eligible for NFQ Level 8 (honours) undergraduate degrees, and the CAO uses a student’s best six subjects to determine CAO points (which determine eligibility to study at the undergraduate level). Every student must take English, mathematics and Irish, unless they have an exemption from Irish. Students will normally choose another four subjects [1].

The UK’s computing curriculum, as part of the UK National Curriculum, has been running since September 2014 for students in primary and secondary school. The road to this curriculum was long and winding, which Neil C. C. Brown and colleagues document in this ACM Transactions on Computing Education paper.  In the US, the CS4all initiative is helping organise efforts to bring computer science to primary and secondary schools, where the landscape is complicated due to primary and secondary schooling being overseen largely at the state level. According to Code.org, only 33 states allow students to count Computer Science courses toward their high school graduation requirements, leaving them to take classes, if they’re offered, as electives. However, Computer Science classes still aren’t even an option for high school students in many districts. See the last link for an interactive map that summarises the status of Computer Science efforts in each state. This week, as part of Computer Science Education Week, the White House announced efforts in two federal agencies to expand access to and quality of computer science education in US K-12 schools [2].

 

China tops list of best programming nations

A new study has given insight into the world’s developers and how they score in various software development aspects. According to HackerRank‘s data, which includes over 1.5 million users, China and Russia score as having the most talented developers. The developers are scored based on a combination of skill and speed. Chinese programmers outscore all other countries in mathematics, functional programming, and data structures challenges, while Russians dominate in algorithms, the most popular and most competitive arena. Although the United States and India provide the majority of competitors on HackerRank, they only manage to rank 28th and 31st.

The folks at HackerRank looked at each country’s average score (50 countries in total) across the following 15 domains: Algorithms, Java, Data Structures, C++, Tutorials, Mathematics, Python, SQL, Shell, Artificial Intelligence, Functional Programming, Databases, Ruby, Distributed Systems, and Security. They then standardized the scores for each domain (by subtracting the mean from each score and then dividing by the standard deviation; also known as a z-score) before finding the average. They then converted these z-scores into a 1-100 scale for easy interpretation.

Below is the top 20:

  1. China – 100
  2. Russia – 99.9
  3. Poland – 98.0
  4. Switzerland – 97.9
  5. Hungary – 93.9
  6. Japan – 92.1
  7. Taiwan – 91.2
  8. France – 91.2
  9. Czech Republic – 90.7
  10. Italy – 90.2
  11. Ukraine – 88.7
  12. Bulgaria – 88.2
  13. Singapore – 87.1
  14. Germany – 84.3
  15. Finland – 84.3
  16. Belgium – 84.1
  17. Hong Kong – 83.6
  18. Spain – 83.4
  19. Australia – 83.2
  20. Romania – 81.9

Do you notice anything that most of these countries have in common? They are almost all non-English speaking countries! Astounding. Singapore (13th place) has four official languages, English, Malay, Mandarin and Tamil, but English is the language of instruction in all public schools. Hong Kong (17th place) has English and Chinese as official languages and English is used varyingly in education. Australia (19th place) is the highest-ranked country where English is the only national language.

The big shock here is that the US is ranked 28th with a score of 78.0, and India ranks 31st with a score of 76. Also, the UK is 29th with a score of 77.7. That’s right. According to HackerRank, the homeland of Don Knuth, Stanford, MIT, etc. is 28th in the world in terms of software development. By the way, the US and India also provide the largest numbers of developers on HackerRank – and they don’t feature in the top 5 of any of the 15 domains tested.

At this point I should give a nod to Ireland, placing 5th in Artificial Intelligence, and 32nd overall with a score of 75.9.

China also topped the list in three domains: mathematics, functional programming, and data structures, and were in the top 5 in seven domains. China is also the second-least likely country to choose Java, next to Taiwan. China also has the largest proportion of developers focusing on Python.

I find the following results particularly astonishing:

  1. The US is ranked 28th
  2. India is ranked 31st
  3. All but three of the top 20 are non-English speaking countries

This is related to my interest in non-native English speakers learning computer science, which will be the topic of an upcoming post, and is the topic of one of Mark Guzdial’s recent posts.

See HackerRank’s blog post here for more.

HackerRank is a California based company that ranks programmers based on their coding skills and helps connect those programmers to employers.

Learning from instantaneous feedback designed for experts, provided by a machine

I remember reading Matt Jadud‘s thesis and being struck by a paragraph on punched cards, repeated here with kind permission:

In 1967, the Univac 1107 system running at Case University (later to become Case Western Reserve University) in Cleveland, Ohio, boasted an impressive turnaround-time of 12 hours on most jobs. Lynch reports that “A user can submit a deck at 8:00 a.m., have it on the input tape by noon, his job completed by 12:35, and the output returned to him by 5:00 p.m. in the evening.”[Lyn67] Compared to students today (who work in laboratories equipped with many machines, each thousands of times more powerful than the Univac 1107), the 9-hour turnaround time seems to approach infinity; but Lynch goes on to say that “It should be noted that only 10-15 percent of the runs are the first of the day, 85-90 percent are repeats, and about a third of the runs have a circulation time of less than 5 minutes. It is often possible to debug a moderate size program in less than an hour by gaining frequent access to the computer.”

This early report on programmer behaviour seems to imply that having access to rapid feedback from a computer regarding the syntax and semantics of a program was a valued interaction style. Not only is this behaviour exhibited by students today, but tools like Eclipse continuously re-compile the programmer’s code, highlighting errors as the programmer develops their code in real-time; this is the “rapid compilation cycle” taken to its natural limit.

It wasn’t that the above contains anything I didn’t pretty much know, but I had never thought about it in quite that way. Imagine having to wait hours or days to find out if a program compiled successfully! Perhaps some readers can remember those days. Anyway, this got me to thinking: what if compilers didn’t return error messages (or an indication that the compilation was successful) immediately? How would programming, and teaching programming, be different? This quickly led to thinking about how learning, and teaching, programming is different to other disciplines – and what that means.

In 1986, Perkins, et al. noted that under normal instructional circumstances some students learn programming much better than others. Investigations of novice programmer behavior suggest that this happens in part because different students bring different patterns of learning to the programming context. Students often fall into varying combinations of disengaging from the task whenever trouble occurs, neglecting to track closely what their programs do by reading back the code as they write it, trying to repair buggy programs by haphazardly tinkering with the code, or having difficulty breaking problems down into parts suitable for separate chunks of code. The authors categorized programming students into groups: stoppers, who quickly gave up when faced with errors; movers, who would work their way through, around or away from errors; and tinkerers, who poke, tweak, and otherwise manipulate their code in a variety of small ways, sometimes making progress towards a working program, sometimes not. Either way, the fact that the compiler provides instant feedback is what allows these behaviors to exist.

Today’s students must write code differently than their counterparts decades ago due to this instant feedback. the behaviors described by Perkins et al. wouldn’t be possible (or would be very different) if students had to wait hours or days for compiler feedback. I have witnessed students use the compiler to tell them where their errors are. I have done the same myself. Why bother pouring over your code with spectacularly aching scrutiny when the compiler will instantly, and with little concrete penalty, tell you what you did wrong (syntactically)? All you have to do is click this little button! Note that when I say instantaneous feedback, I am not addressing Eclipse-like red squiggly line behavior. That is of course ‘more instantaneous’ than having to click a button. But traditional push-the-button feedback is more or less instantaneous compared to the way it used to happen. As for the red squiggly lines, watch this space, there will be something to come there soon. Also, in this post I am not addressing the point that most compiler feedback is poor. See this post for an example.

Is there something lost in the present scenario? Was there value to thinking like a compiler, going over your code to ensure that when the real compiler does, there are no errors? Is there something that students are not learning, not experiencing, or not taking away from their  programming practice in not doing so?

I will leave that question where it is, lest we disappear down the rabbit hole. It is however worthy of future thought.

Before this post ends however, what about the feedback that compilers provide? Feedback is of course regarded as one of the most important influences on learning and motivation, noted by Watson et al. in the context of programming. Unique to computer programming is the fact that much of the feedback that programming students receive comes:

Edit Jan 30 2017: Also see this post for a discussion and an example of what can be done about this.

Clearly, programming students are in a different world when they are in CS1, compared to students in Phys1, Med1, or many (all?) other 101/introductory courses where feedback comes:

  • from a human
  • often delayed
  • designed for them, and (hopefully) in terms that they can fully understand

What does this mean for those that are learning to program? They get instant, difficult to understand feedback from a machine! For one thing, as they are in a different world to so many other students, their expectations, their outcomes, and the way that they are taught should probably be different too.

I’ll avoid the second rabbit hole by leaving that thought there… for now.

Becker, B.A. (2016) An Effective Approach to Enhancing Compiler Error Messages. In Proceedings of the 47th ACM Technical Symposium on Computer Science Education (SIGCSE 2016), (pp. 126-131). ACM

Jadud, M. C. (2006). An exploration of novice compilation behaviour in BlueJ. PhD Thesis. University of Kent

Perkins, D. N., Hancock, C., Hobbs, R., Martin, F., & Simmons, R. (1986). Conditions of learning in novice programmers. Journal of Educational Computing Research, 2(1), 37-55

Traver, V. J. (2010). On compiler error messages: what they say and what they mean. Advances in Human-Computer Interaction, 2010

Watson, C., Li, F. W., & Godwin, J. L. (2012). BlueFix: using crowd-sourced feedback to support programming students in error diagnosis and repair. In International Conference on Web-Based Learning (pp. 228-239). Springer Berlin Heidelberg

Misleading, cascading Java error messages

I have been working with enhancing Java error messages for a while now, and I have stared at a lot of them. Today I came across one that I don’t think I’ve consciously seen before, and it’s quite a doozy if you are a novice programmer. Below is the code, with a missing bracket on line 2:

public class Hello {
       public static void main(String[] args)  //missing {
              double i;
              i = 1.0;
              System.out.println(i);
       }
}

The standard Java output in this case is:

C:\Users\bbecker\Desktop\Junk\Hello.java:2: error: ';' expected
       public static void main(String[] args)
                                             ^

C:\Users\bbecker\Desktop\Junk\Hello.java:4: error: <identifier> expected
              i = 1.0;
               ^

C:\Users\bbecker\Desktop\Junk\Hello.java:5: error: <identifier> expected
              System.out.println(i);
                                ^

C:\Users\bbecker\Desktop\Junk\Hello.java:5: error: <identifier> expected
              System.out.println(i);
                                  ^

C:\Users\bbecker\Desktop\Junk\Hello.java:7: error: class, interface, or enum expected
}
^

5 errors

Process Terminated ... there were problems.

Amazing. This is telling the student that there were 5 errors (not one), and none of the five reported errors are even close to telling the student that there is a missing bracket on line 2. If the missing bracket is supplied, all five “errors” are resolved.

During my MA in Higher Education I developed an editor that enhances some Java error messages, and I have recently published some of this work at SIGCSE (see brettbecker.com/publications). I hope to get some more work on this front  soon, and in addition I would like to look more deeply at what effects cascading error messages have on novices. I can imagine that if I had no programming experience, was learning Java, and came across the above I would probably be pretty discouraged.

The enhanced error that my editor would provide for the above code, which would be reported side-by-side with the above Java error output is:

Looks like a problem on line number 2.

Class Hello has 1 fewer opening brackets '{' than closing brackets '}'.

A killer novice bug

Recently I had ‘one of those moments’ in my CP1 lab. I am a little embarrassed to say that I spent the last few minutes of this session staring at a C function written by a student that wasn’t behaving the way it should. The students were asked to write a function that returns max value stored in a stack. Below is a reconstruction of the student’s code, slightly simplified:

int max(Stack *sptr) {
    int max = INT_MIN;
    Node popped;
    while (!isEmpty(sptr)){
         popped = pop(sptr);
         if(popped->data > max);{
             max = popped->data;
         }
    }
    return max;
}

The symptom of the bug was that the value of max returned was always the value of the last item popped from the stack.

A few labs later I had a similar situation. This time I spotted the bug right away (and embarrassingly felt slightly proud).

I am going to hope that at this point there are two types of people reading this:

  1. those that see the bug immediately
  2. those that don’t

For selfish reasons I hope that there are at least a few 2’s. At the time I had just written my own solution to this question, and had helped a dozen other students with their versions of their solutions. Needless to say that running up to the 90 minute mark I was suffering from code blindness. At least that’s my excuse for not seeing the error immediately.

So, for all of you 2’s out there, if there are any, the bug is the empty statement after the if condition. I have to say I sensed a mixture of relief and disappointment from the student when I pointed out the bug and her output was now as expected. The relief was coming from obvious places, but the disappointment seemed to stem from a sense of ‘Really? That’s it? You have got to be kidding me.’. I have to say I did feel for her.

A few days later in lecture I wrote the above code on the board (in China we are still using good-old-fashioned chalk and chalkboards), and asked the 100+ students to point out the problem. It took a good two minutes for the first student to do so.

gcc (4.9.3, no options) gives no warning for empty statements. I just checked Java SE 8 update 92, which gives an empty statement after if warning, which despite my unhappiness with most have compiler error messages, is quite nice.

I think next year I’ll spend a little time discussing the empty statement and NOPs, as early as possible, and see if that reduces the troubles experienced this year.

Computer Science: More than just programming and telescopes

James H. Morris, former dean of Carnegie Mellon University’s School of Computer Science had a nice piece in Information Week 13 years ago now, that is as true as if ever was.

In it he argues strongly that CS is more than programming, and that students need a strong sense of empiricism.

 The ability to discern a real phenomenon and distinguish it from myth is vital. Our students learn to measure the performance of people using technology.

I couldn’t agree more. As the former head of a small CS department I oversaw a degree program that was particularly strong on networking – in fact our degree was known for this and popular as a result. (It was technically an IT degree). We had graduates go on to work for Oracle, IBM, and other companies based on their networking experience. I believe it still is the only CS/IT undergrad I know of that has a course on networking in each of all eight semesters. The program is also notable for third year group projects and fourth year individual projects. The fourth year project was equivalent to half of the final year workload! At the time this program had a conservative rank of 17/225 in compliance with the ACM 2008 IT curriculua. This ranking was done using the work of [1] as a guide. The biggest reason for this ranking was having this much project work – the ACM assigns 0.5 points per semester for a capstone experience (out of a possible 4). The inclusion of the fourth year module Principles in Professional Practice and Strategic Business IT also increased the mark relative to many of the other programs.

Morris goes on to state that good CS programs include the liberal arts, mathematics and experimental science. I particularly agree with the first since I enjoyed a liberal arts CS undergrad degree myself. Being at a liberal arts college really allowed me to flex my skills in a diverse environment. As a dual-major (CS and physics) I completed my final year physics project on variable stars, a project that was done almost entirely through a computer – I rarely held my eye to a telescope eyepiece. In fact the stars I was studying normally only became visible on the screen of a computer, too dim to see with the human eye, even through a powerful telescope. But it was the support and interest of my project in the community, many of whom were not in a science program, that made my experience so enjoyable.

I agree with the second point, particularly as I did a MSc in Computational Science where my mathematics skills were really put to the test. I also really put my mathematics skills to use in completing my PhD, which was largely done with a pencil and paper, drawing little squares over and over. My entire thesis including code would at the time fit on a floppy disk (not that I ever tried).

I agree with the third point most in my current research lines of high performance computing and computer science education, both areas where (sometimes slightly different forms of) empiricism is absolutely critical.

James’s final quote was:

As programmed digital devices continue to shrink in size and cost, many of us predict that the computer per se will disappear just as the electric motor disappeared into hundreds of niches in our homes and automobiles. Then we will have a science named after an artifact no one sees. But the essence of the science will still be there, and those who study it will be rewarded not just with riches but with understanding.

This brings me back to astronomy – studying many things we will never see. And of course that brings me to the famous ‘computers and telescopes’ quote (mis?) attributed to Edsger W. Dijkstra:

Computer Science is no more about computers than astronomy is about telescopes.

Dijkstra did say something similar for sure though (EWD 1305):

I don’t need to waste my time with a computer just because I am a computer scientist. [after all, doctors don’t infect themselves with the diseases the study]

[1] B. M., Neupane, B., Hansen, A., & Ofori, R. (2012). Identifying and Evaluating Information Technology Bachelor’s Degree Programs. Proceedings of the 1st Annual conference on Research in information technology (pp. 19-23). ACM.