The enhancing compiler error messages saga: the saga continues

I was sorry to have missed Raymond Pettit, John Homer and Roger Gee presenting the latest installment of what is becoming the enhanced compiler error messages saga at SIGCSE earlier this month. Their paper “Do Enhanced Compiler Error Messages Help Students?: Results Inconclusive” [1] was two-pronged. It contrasted the work of Denny et al. [2] (which provided evidence that compiler error enhancement does not make a difference to students) and my SIGCSE 2016 paper [3] (which provided evidence that it does). It also provided fresh evidence that supports the work of Denny et al. I must say that I like Pettit et al.’s subtitle: “Results Inconclusive”. I don’t think that this is the final chapter in the saga. We need to do a lot more work on this.

Early studies on this often didn’t include much quantifiable data. More recent studies haven’t really been measuring the same things – and they have been measuring these things in different ways. In other words, the metrics, and the methodologies differ. It’s great to see work like that of Pettit et al. that is more comparable to previous work like that of Denny et al.

One of the biggest differences between my editor, Decaf, and Pettit et al.’s tool, Athene, is that Decaf was used by students for all of their programming – practicing, working on assignments, programming for fun, even programming in despair. For most of my students it was the only compiler they used – so they made a lot of errors, and they all were logged. Unlike Denny et al., my students did not receive skeleton code – they were writing programs, often from scratch. On the other hand, Athene was often utilized by students after developing their code on their own local (un-monitored) compilers. Thus, many errors generated by the students in the Pettit et al. study were not captured. Often, the code submitted to Athene was already fairly refined. Pettit et al. even have evidence from some of their students that at times the code submitted to Athene only contained those errors that the students absolutely could not rectify without help.

As outlined in this post, Denny et al. and I were working towards the same goal but measuring different things. This may not be super apparent at first read, but under the hood comparing studies like these is often a little more complicated than it first looks. Of course these differences have big implications when trying to compare results. I’m afraid that the same is true comparing my work with Pettit et al. – we are trying to answer the same question, but measuring different things (in different ways) in order to do so.

Specifically, Pettit el al. measured:

  1. the number of non-compiling submissions; similar to did Denny et al., but unlike me
  2. the number of successive non-compiling submissions that produced the same error message; Denny et al. measured the number of consecutive non-compiling submissions regardless of why the submission didn’t compile, and I measured the number of consecutive errors generating the same error message, on the same line of the same file
  3. the number of submission attempts (in an effort to measure student progress)
  4. time between submissions; neither Denny et al. nor I measured time-based metrics

I also did a fairly detailed comparison between my work and Denny et al. in [4] (page 10). In that study we directly compared some effects of enhanced and non-enhanced error messages:

In this study we directly distinguish between two sets of compiler error messages (CEMs), the 30 that are enhanced by Decaf and those that are not. We then explore if the control and intervention groups respond differently when they are presented with these. For CEMs enhanced by Decaf the control and intervention groups experience different output. The intervention group, using Decaf in enhanced mode, see the enhanced and raw javac CEMs. The control group, using Decaf in pass-through mode, only see the raw javac CEMs. Thus for CEMs not enhanced by Decaf, both groups see the same raw CEMs. This provides us with an important subgroup within the intervention group, namely when the intervention group experiences errors generating CEMs not enhanced by Decaf. We hypothesized that there would be no significant difference between the control and intervention groups when looking at these cases for which both groups receive the same raw CEMs. On the other hand, if enhancing CEMs has an effect on student behavior, we would see a significant difference between the two groups when looking at errors generating the 30 enhanced CEMs (due to the intervention group receiving enhanced CEMs and the control group receiving raw CEMs).

As mentioned, the metrics used by Pettit et al. and Denny et al. are more common to each other than to mine. Pettit et al. and Denny et al. both used metrics based on submissions (that is, programs) submitted by students, or the number of submission attempts. This certainly makes comparing their studies more straight-forward. However it is possible that these metrics are too ‘far from the data’ to be significantly influenced by enhanced error messages. It is possible that metrics simply based on the programming errors committed by students, and the error messages generated by these errors are more ‘basic’ and more sensitive.

Another consideration when measuring submissions is that just because a submission compiles, does not mean that it is correct or does what was intended. It is possible that some students continue to edit (and possibly generate errors) after their first compiling version, or after they submit an assignment. These errors should also be analyzed. I think that in order to measure if enhancing error messages makes a difference to students we should focus on all programming activity. I’m afraid that otherwise, the results may say more about the tool (that enhances error messages) and the way that tool was used by students, than about the effects of enhanced error messages themselves. I am sure that in some of my research this is also true – after all my students were using a tool also, and this tool has its own workings which must generate effects. Isolating the effects of the tool from the effects of the messages is challenging.

I am very glad to see more work in this area. I think it is important, and I don’t think it is even close to being settled. I have to say I really feel that the community is working together to do this. It’s great! In addition there may be more to do than determine if enhanced compiler errors make a difference to students. We have overwhelming evidence that syntax poses barriers to students. We have a good amount of evidence that students think that enhancing compiler error messages makes a positive difference. Some researchers think it should too. If enhancing compiler error messages doesn’t make a difference, we need to find out why, and we need to explain the contradiction this would pose. On the other hand, if enhancing compiler error messages does make a difference we need to figure out how to do it best, which would also be a significant challenge.

I hope to present some new evidence on this soon. I haven’t analyzed the data yet, and I don’t know which way this study is going to go. The idea for this study came from holding my previous results up to the light and looking at them from quite a different angle. I feel that one of the biggest weaknesses in my previous work was that the control and treatment groups were separated by a year – so that is what I eliminated. The new control and treatment groups were taking the same class, on the same day – separated only by lunch break. Fortuitously, due to a large intake CP1 was split into two groups for the study semester, but was taught by the same lecturer in the exact same way – sometimes things just work out!

I will be at ITiCSE 2017 and SIGCSE 2018 (and 2019 for that matter – I am happy to be serving a two year term as workshop co-chair). I hope to attend some other conferences also but haven’t committed yet. I look forward to continuing the discussion on the saga of enhancing compiler error messages with anyone who cares to listen! In the meantime here are a few more posts where I discuss enhancing compiler error messages – comments are welcome…

[1] Raymond S. Pettit, John Homer, and Roger Gee. 2017. Do Enhanced Compiler Error Messages Help Students?: Results Inconclusive.. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE ’17). ACM, New York, NY, USA, 465-470. DOI: https://doi.org/10.1145/3017680.3017768

[2] Paul Denny, Andrew Luxton-Reilly, and Dave Carpenter. 2014. Enhancing syntax error messages appears ineffectual. In Proceedings of the 2014 conference on Innovation & technology in computer science education (ITiCSE ’14). ACM, New York, NY, USA, 273-278. DOI: http://dx.doi.org/10.1145/2591708.2591748

[3] Brett A. Becker. 2016. An Effective Approach to Enhancing Compiler Error Messages. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education (SIGCSE ’16). ACM, New York, NY, USA, 126-131. DOI: https://doi.org/10.1145/2839509.2844584

full-text available to all with link available at www.brettbecker.com/publications

[4] Brett A. Becker, Graham Glanville, Ricardo Iwashima, Claire McDonnell, Kyle Goslin, Catherine Mooney. 2106. Effective Compiler Error Message Enhancement for Novice Programming Students, Computer Science Education 26(2-3), pp. 148-175; http://dx.doi.org/10.1080/08993408.2016.1225464

full-text available to all at www.brettbecker.com/publications

Party like it’s 2018: Computer Science to be offered in Irish secondary curriculum in 2018, not 2019

Two months ago I wrote about news reporting that Computer Science was to be offered in Irish secondary curriculum by 2019. So I was a bit surprised (but after living in Ireland for the best part of 17 years not entirely surprised) when I read this week that Computer Science is to be offered in Irish Secondary schools next year!

According to the Irish Times, this decision is being driven in order to address the skills gap (that has been around for a long time), and that coding* is to be on new primary mathematics curriculum.

The Irish Government Action Plan for Education 2017 published on February 6 states under the heading Technology in Education:

Implementation of computer science leaving cert subject brought forward 12 months to September 2018; development of computational thinking and coding through new maths curriculum at primary; new advisory group led by industry and experts to develop new plans for greater use of technology in education.

* I really prefer the word programming, in just about every context…

Computer Science to be offered in Irish secondary curriculum by 2019

According to RTE news, the Irish Minister for Education has said that Computer Science, including programming, will be a Leaving Certificate subject by 2019, paving the way for Irish students to have the option to study the subject at secondary school. However the nature of the curriculum has not been settled, nor is not known how many schools will offer the subject. The rules in Irish schools vary, but most require students to take seven subjects for the Leaving Certificate. Some students may take fewer, for example, those who are not taking Irish. Other students may choose to take more. There is no specific rule about how many subjects a student should take, however students must pass six subjects in the Leaving Certificate in order to be eligible for NFQ Level 8 (honours) undergraduate degrees, and the CAO uses a student’s best six subjects to determine CAO points (which determine eligibility to study at the undergraduate level). Every student must take English, mathematics and Irish, unless they have an exemption from Irish. Students will normally choose another four subjects [1].

The UK’s computing curriculum, as part of the UK National Curriculum, has been running since September 2014 for students in primary and secondary school. The road to this curriculum was long and winding, which Neil C. C. Brown and colleagues document in this ACM Transactions on Computing Education paper.  In the US, the CS4all initiative is helping organise efforts to bring computer science to primary and secondary schools, where the landscape is complicated due to primary and secondary schooling being overseen largely at the state level. According to Code.org, only 33 states allow students to count Computer Science courses toward their high school graduation requirements, leaving them to take classes, if they’re offered, as electives. However, Computer Science classes still aren’t even an option for high school students in many districts. See the last link for an interactive map that summarises the status of Computer Science efforts in each state. This week, as part of Computer Science Education Week, the White House announced efforts in two federal agencies to expand access to and quality of computer science education in US K-12 schools [2].

 

You are what you measure: Enhancing compiler error messages effectively

semicolonCompiler Error Messages (CEMs) play a particularly essential role for programming students as they often have little experience to draw upon, leaving CEMs as their primary guidance on error correction. Further, they provide immediate feedback, with implications discussed in this post. In the absence of an instructor, the compiler and its messages are the only source of feedback on what the student is doing correctly, and incorrectly. There is another issue at hand however – CEMs are frequently inadequate, present a barrier to progress, and are often a source of discouragement.

At SIGCSE 2016 I presented a paper which showed that enhancing compiler error messages can be effective, referred to here as Becker (2016). I also led a more in-depth study with a more focused comparison approach that was recently published in Computer Science Education (see my publications page for details on both). In 2014 Denny, Luxton-Reilly and Carpenter published a study providing evidence that enhancing CEMs was not effective, generating a bit of discussion on Mark Guzdial’s Blog. Although these papers came up with opposing conclusions, there are a ton of variables involved in studies like this, and two things in particular are really important. These might sound really obvious, but bear with me. These two things are:

  1. What is measured
  2. How these things are measured

Another important factor is the language used – as in the English terminology – not programming language. That will come up here soon enough.

In Becker (2016) I measured four things:

  1. number of errors per compiler error message
  2. number of errors per student
  3. number of errors per student per compiler error message
  4. number of repeated errors per compiler error message

Denny et al. measured three things:

  1. number of consecutive non-compiling submissions
  2. total number of non-compiling submissions
  3. number of attempts needed to resolve three errors: Cannot resolve identifier, type mismatch, missing semicolon

Getting back to my fairly obvious point that what is measured (and how) is of critical importance, let me dig into my four metrics for some of the not so obvious stuff. For starters, all four of my metrics involve student errors. Additionally, although I was measuring errors, for three of my metrics I was measuring some flavor of errors per CEM. This is important, and the wording is intentional. As I was investigating the effect of enhancing CEMs, the ‘per CEM’ part is by design. However it is also required for another reason – there is often not a one-to-one mapping of student committed errors to CEMs in Java – so I don’t know (from looking at the CEM) exactly what error caused that CEM. I could look at the source code to see, but the point is that from a CEM point of view, all I can know is how many times that CEM occurred – in other words, how many (student-committed) errors (of any type/kind/etc.) generated that CEM. See work by Altadmri & Brown (2015) and my MA thesis for more on this lack of a one-to-one mapping of errors to CEMs in Java. This makes things tricky. Finally, each metric warrants some discussion on its own:

  1. The number of errors per CEM was measured for all errors encountered during the study (generating 74 CEMs in total) and for errors generating the top 15 CEMs, representing 86.3% of all errors. Results indicated that enhancing CEMs reduced both.
  2. The number of errors per student was not significantly reduced when taking all 74 CEMs, but it was for errors generating the top 15 CEMs.
  3. The number of errors per student per CEM was significantly reduced for 9 of the top 15 CEMs (of which only 8 had enhanced CEMs). The odd-one-out was .class expected. Sometime I’ll write more on this – it’s a really interesting case.
  4. The number of repeated errors per CEM is dependent on the definition of a repeated error. I defined a repeated error similarly to Matt Jadud – two successive compilations that generate the same CEM on the same line of code. Also, this was for the top 15 CEMs.

If we now look at the metrics of Denny et al., the first two involve student submissions, which may have contained errors, but errors are not being measured directly (well, we know that the compiling submissions don’t have any errors, and that the non-compiling submissions do, but that’s about it). Only the third involves errors directly, and at that, only three particular types. What was really measured here was the average number of compiles that it takes a student to resolve each type of error, where a submission is said to have a syntax error of a particular type when the error is first reported in response to compilation, and the error is said to have been resolved when the syntax error is no longer reported to students in the feedback for that submission.

So, comparing the results of these two studies, if this post were trying to reach a conclusion of its own, the best we can do is to compare the following result from Denny et al.:

  • D1. Enhancing compiler error messages does not reduce the number of attempts needed to resolve three errors (really, CEMs): Cannot resolve identifier, type mismatch, missing semicolon.

and the following from Becker (2016):

  • B1. Enhancing compiler error messages does reduce the number of errors that generate the CEMs: expected, incompatible types, ; expected, and many other CEMs.
  • B2. Enhancing compiler error messages does reduce the number of errors per student that generate the CEMs: expected, incompatible types, and many other CEMs*
  • B3. Enhancing compiler error messages does reduce the number of repeated errors generating the CEMs: expected, incompatible types, and many other CEMs.*

These are the only four results (across both papers) that measure the same thing – student errors. Further, we can only specifically compare the results involving the three CEMs that Denny et al. investigated. Becker (2016) investigated 74, including these three.

* The number of errors (per student, and repeated) generating the CEM ; expected was not reduced in these cases.

So, despite the differing general conclusions (Denny et al. indicate that enhanced CEMs are not effective, while Becker (2016) indicates that enhanced CEMs can be effective) if we synthesize the most common results from each paper, we end up with what the two studies agree on (sometimes), which is ; expected:

  • D1. Enhancing compiler error messages does not reduce the number of attempts needed to resolve missing semicolon (Denny et al.).
  • B2. Enhancing compiler error messages does not reduce the number of errors per student that generate the CEM ; expected (Becker 2016).
  • B3. Enhancing compiler error messages does not reduce the number of repeated errors per student that generate the CEM ; expected (Becker 2016).

I find this to be particularly unsurprising as ; expected is one of the most common CEMs (in my study the third most common, representing ~10% of all errors) and the actual CEM itself is one of the most straightforward of all Java CEMs. However, Becker (2016) had one result (B1) which showed that the number of errors generating ; expected CEMs was reduced. So for this CEM, maybe the jury is still out.

It may seem that the two studies didn’t agree on much, which technically is true. However I hope that any readers that have persevered this long can appreciate the nuances of what is measured (and how) in these types of study, particularly when comparing studies. It is very challenging because the nuances really matter. Further, they can really complicate the language used. If you try and make the language easy, you miss important details, and get ambiguous. Incorporating those details into the language affects readability.

Finally, I think that this post demonstrates the important need for studies that attempt to repeat the results of others, particularly in an area where results are contested. Comparing two different studies poses several other problems (apart from what is measured and how), and I won’t go into them here as most are well known and well discussed, but I do think that the difficulties that come about to the use of different language is an often overlooked one.

Either way, I believe that the results in Becker (2016), and the recent Computer Science Education article are robust. These studies provide many results do indicate that enhanced CEMs can be effective.

China tops list of best programming nations

A new study has given insight into the world’s developers and how they score in various software development aspects. According to HackerRank‘s data, which includes over 1.5 million users, China and Russia score as having the most talented developers. The developers are scored based on a combination of skill and speed. Chinese programmers outscore all other countries in mathematics, functional programming, and data structures challenges, while Russians dominate in algorithms, the most popular and most competitive arena. Although the United States and India provide the majority of competitors on HackerRank, they only manage to rank 28th and 31st.

The folks at HackerRank looked at each country’s average score (50 countries in total) across the following 15 domains: Algorithms, Java, Data Structures, C++, Tutorials, Mathematics, Python, SQL, Shell, Artificial Intelligence, Functional Programming, Databases, Ruby, Distributed Systems, and Security. They then standardized the scores for each domain (by subtracting the mean from each score and then dividing by the standard deviation; also known as a z-score) before finding the average. They then converted these z-scores into a 1-100 scale for easy interpretation.

Below is the top 20:

  1. China – 100
  2. Russia – 99.9
  3. Poland – 98.0
  4. Switzerland – 97.9
  5. Hungary – 93.9
  6. Japan – 92.1
  7. Taiwan – 91.2
  8. France – 91.2
  9. Czech Republic – 90.7
  10. Italy – 90.2
  11. Ukraine – 88.7
  12. Bulgaria – 88.2
  13. Singapore – 87.1
  14. Germany – 84.3
  15. Finland – 84.3
  16. Belgium – 84.1
  17. Hong Kong – 83.6
  18. Spain – 83.4
  19. Australia – 83.2
  20. Romania – 81.9

Do you notice anything that most of these countries have in common? They are almost all non-English speaking countries! Astounding. Singapore (13th place) has four official languages, English, Malay, Mandarin and Tamil, but English is the language of instruction in all public schools. Hong Kong (17th place) has English and Chinese as official languages and English is used varyingly in education. Australia (19th place) is the highest-ranked country where English is the only national language.

The big shock here is that the US is ranked 28th with a score of 78.0, and India ranks 31st with a score of 76. Also, the UK is 29th with a score of 77.7. That’s right. According to HackerRank, the homeland of Don Knuth, Stanford, MIT, etc. is 28th in the world in terms of software development. By the way, the US and India also provide the largest numbers of developers on HackerRank – and they don’t feature in the top 5 of any of the 15 domains tested.

At this point I should give a nod to Ireland, placing 5th in Artificial Intelligence, and 32nd overall with a score of 75.9.

China also topped the list in three domains: mathematics, functional programming, and data structures, and were in the top 5 in seven domains. China is also the second-least likely country to choose Java, next to Taiwan. China also has the largest proportion of developers focusing on Python.

I find the following results particularly astonishing:

  1. The US is ranked 28th
  2. India is ranked 31st
  3. All but three of the top 20 are non-English speaking countries

This is related to my interest in non-native English speakers learning computer science, which will be the topic of an upcoming post, and is the topic of one of Mark Guzdial’s recent posts.

See HackerRank’s blog post here for more.

HackerRank is a California based company that ranks programmers based on their coding skills and helps connect those programmers to employers.

Learning from instantaneous feedback designed for experts, provided by a machine

I remember reading Matt Jadud‘s thesis and being struck by a paragraph on punched cards, repeated here with kind permission:

In 1967, the Univac 1107 system running at Case University (later to become Case Western Reserve University) in Cleveland, Ohio, boasted an impressive turnaround-time of 12 hours on most jobs. Lynch reports that “A user can submit a deck at 8:00 a.m., have it on the input tape by noon, his job completed by 12:35, and the output returned to him by 5:00 p.m. in the evening.”[Lyn67] Compared to students today (who work in laboratories equipped with many machines, each thousands of times more powerful than the Univac 1107), the 9-hour turnaround time seems to approach infinity; but Lynch goes on to say that “It should be noted that only 10-15 percent of the runs are the first of the day, 85-90 percent are repeats, and about a third of the runs have a circulation time of less than 5 minutes. It is often possible to debug a moderate size program in less than an hour by gaining frequent access to the computer.”

This early report on programmer behaviour seems to imply that having access to rapid feedback from a computer regarding the syntax and semantics of a program was a valued interaction style. Not only is this behaviour exhibited by students today, but tools like Eclipse continuously re-compile the programmer’s code, highlighting errors as the programmer develops their code in real-time; this is the “rapid compilation cycle” taken to its natural limit.

It wasn’t that the above contains anything I didn’t pretty much know, but I had never thought about it in quite that way. Imagine having to wait hours or days to find out if a program compiled successfully! Perhaps some readers can remember those days. Anyway, this got me to thinking: what if compilers didn’t return error messages (or an indication that the compilation was successful) immediately? How would programming, and teaching programming, be different? This quickly led to thinking about how learning, and teaching, programming is different to other disciplines – and what that means.

In 1986, Perkins, et al. noted that under normal instructional circumstances some students learn programming much better than others. Investigations of novice programmer behavior suggest that this happens in part because different students bring different patterns of learning to the programming context. Students often fall into varying combinations of disengaging from the task whenever trouble occurs, neglecting to track closely what their programs do by reading back the code as they write it, trying to repair buggy programs by haphazardly tinkering with the code, or having difficulty breaking problems down into parts suitable for separate chunks of code. The authors categorized programming students into groups: stoppers, who quickly gave up when faced with errors; movers, who would work their way through, around or away from errors; and tinkerers, who poke, tweak, and otherwise manipulate their code in a variety of small ways, sometimes making progress towards a working program, sometimes not. Either way, the fact that the compiler provides instant feedback is what allows these behaviors to exist.

Today’s students must write code differently than their counterparts decades ago due to this instant feedback. the behaviors described by Perkins et al. wouldn’t be possible (or would be very different) if students had to wait hours or days for compiler feedback. I have witnessed students use the compiler to tell them where their errors are. I have done the same myself. Why bother pouring over your code with spectacularly aching scrutiny when the compiler will instantly, and with little concrete penalty, tell you what you did wrong (syntactically)? All you have to do is click this little button! Note that when I say instantaneous feedback, I am not addressing Eclipse-like red squiggly line behavior. That is of course ‘more instantaneous’ than having to click a button. But traditional push-the-button feedback is more or less instantaneous compared to the way it used to happen. As for the red squiggly lines, watch this space, there will be something to come there soon. Also, in this post I am not addressing the point that most compiler feedback is poor. See this post for an example.

Is there something lost in the present scenario? Was there value to thinking like a compiler, going over your code to ensure that when the real compiler does, there are no errors? Is there something that students are not learning, not experiencing, or not taking away from their  programming practice in not doing so?

I will leave that question where it is, lest we disappear down the rabbit hole. It is however worthy of future thought.

Before this post ends however, what about the feedback that compilers provide? Feedback is of course regarded as one of the most important influences on learning and motivation, noted by Watson et al. in the context of programming. Unique to computer programming is the fact that much of the feedback that programming students receive comes:

Edit Jan 30 2017: Also see this post for a discussion and an example of what can be done about this.

Clearly, programming students are in a different world when they are in CS1, compared to students in Phys1, Med1, or many (all?) other 101/introductory courses where feedback comes:

  • from a human
  • often delayed
  • designed for them, and (hopefully) in terms that they can fully understand

What does this mean for those that are learning to program? They get instant, difficult to understand feedback from a machine! For one thing, as they are in a different world to so many other students, their expectations, their outcomes, and the way that they are taught should probably be different too.

I’ll avoid the second rabbit hole by leaving that thought there… for now.

Becker, B.A. (2016) An Effective Approach to Enhancing Compiler Error Messages. In Proceedings of the 47th ACM Technical Symposium on Computer Science Education (SIGCSE 2016), (pp. 126-131). ACM

Jadud, M. C. (2006). An exploration of novice compilation behaviour in BlueJ. PhD Thesis. University of Kent

Perkins, D. N., Hancock, C., Hobbs, R., Martin, F., & Simmons, R. (1986). Conditions of learning in novice programmers. Journal of Educational Computing Research, 2(1), 37-55

Traver, V. J. (2010). On compiler error messages: what they say and what they mean. Advances in Human-Computer Interaction, 2010

Watson, C., Li, F. W., & Godwin, J. L. (2012). BlueFix: using crowd-sourced feedback to support programming students in error diagnosis and repair. In International Conference on Web-Based Learning (pp. 228-239). Springer Berlin Heidelberg

‘Supercomputing’ in the curriculum

A recent article on ComputerWeekly.com is calling for supercomputing to be put ‘in the curriculum’. In it, Tim Stitt, head of scientific computing at the Earlham Institute, a life science institute in Norwich, UK, says children should be learning supercomputing and data analysis concepts from a young age.

Although I agree in principle, the article doesn’t specify a particular curricula although it does seem to be aimed at pre-university ages. In the article, Stitt claims that current initiatives such as the new computing curriculum introduced in the UK in 2014 which makes it mandatory for children between the ages of five and 16 to be taught computational thinking, may “compound the issue”, as children will be taught serial rather than parallel programming skills, making supercomputing concepts harder to learn later on. Again, I can agree in principle, but the extent to which learning parallel programming after learning ‘normal’ sequential programming is debatable, and will certainly vary considerably from student to student.

I have mixed feeling about the word supercomputing. I can imagine someone saying “Really? You are going to teach supercomputing to kids? Don’t you think that’s a bit much?” I couldn’t blame them for being skeptical. The word itself sounds, well, super. Personally I think that High Performance Computing (HPC) is more down to earth, but I also concede that that may still sound a little ‘super’. I have some experience with this. I am one of many that maintain the Irish Supercomputer List. That project didn’t start off as the Irish Supercomputer List, but we changed the name in order to, quite frankly, be more media ‘friendly’. (Side note – interesting discussion on disseminating scientific work to the media here).  Additionally, the Indian and Russian lists also have the word supercomputing in their names and/or URLs. The Top500 list also used the word supercomputing before they rebranded a few years back. Anyway…

So, what we are really talking about is putting Parallel Computing (or parallel programming) in the curriculum, and therefore opening the door to supercomputing, as almost all HPC installations require parallel programming. In fact the current Top500 Supercomputer List is composed entirely of clusters (86.2%) or Massively Parallel Processors (MPPs – 13.8%). Clusters are parallel computer systems comprising an integrated collection of independent nodes, each of which is a system in its own right, capable of independent operation and derived from products developed and marketed for other stand-alone purposes [1]. MPPs (such as the IBM Blue Gene) on the other hand, are more tightly-integrated. Individual nodes cannot run on their own and they are frequently connected by custom high-performance networks. They key here is that in both cases memory is distributed (as are the cores), thus requiring parallel algorithms (and therefore parallel programming).  Before switching gears I would like to return to the point I opened this paragraph with – we are talking about parallel programming – not necessarily supercomputing – although learning parallel computing is indeed the essential requirement to eventually program supercomputers.

At the university level, there is more than an awareness of the issues that form the core of the argument which is the focus of the article that I started this post with. In particular there are two conferences/workshops that directly address HPC education at university level:

  1. Workshop on Education for High-Performance Computing (EduHPC-16), held in conjunction with SC-16: The International Conference on High Performance Computing, Networking, Storage, and Analysis
  2. The Parallel and Distributed Computing Education for Undergraduate Students Workshop (Euro-EDUPAR 2016), held in conjunction with Euro-Par 2016, the 22nd International European Conference on Parallel and Distributed Computing.

[1] Dongarra, J., Sterling, T., Simon, H. and Strohmaier, E., 2005. High-performance computing: clusters, constellations, MPPs, and future directions. Computing in Science and Engineering, 7(2), pp.51-59

ITiCSE 2016 mini report & photos

It’s the final day of the 21st Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) here in Arequipa, Peru. It has been a really informative conference in a stunning location. There was a great amount of diversity this year with attendees from 37 countries, and the first time that ITiCSE was held outside Europe. I met a lot of Latin/South American delegates who normally can’t make it to ITiCSE in Europe which was great. Alison, Ernesto, and the rest of the committee really pulled out all the stops. From top to bottom – venue, organization, amazingly thought out excursions, and the spectacular conference dinner – they left covered every detail. And of course the program was packed full of great CSEd research.

The venue is the Universidad Católica San Pablo, who have also put a load of pictures up on their Facebook page, and have a nice article on their webpage (in Espanol) – Headline translation: Experts from more than 35 countries meet in Arequipa to analyze education in Computer Science.

For most of the conference I have been occupied with my working group (WG6) – see here for abstract. I also presented a paper A New Metric to Quantify Repeated Compiler Errors for Novice Programmers. Being pretty busy I haven’t been able to compile much in terms of an in person report of many papers, but here I present my take on three before we get to the photos. All three of these talks really impacted (and at times challenged) my perceptions of some CSED topics.

Mehran Sahami‘s keynote Statistical Modeling to Better Understand CS Students (abstract here) was very insightful. It considered developing statistical models to give insight into the dynamics of student populations. The first case study focused on gender balance and demonstrated that focusing on simple metrics such as percentages can be misleading, and that there are better ways to capture how program changes are impacting the dynamics of gender balance. The second looked at the performance of populations that are experiencing rapid growth. This case study showed one answer to the common statement/observation “the number of weak students is increasing”, and the answer was somewhat surprising – performance during a stage of unprecedented growth was quite stable.

Andrew Luxton-Reilly gave an excellent talk on his paper Learning to Program is Easy. This really challenges the notion that ‘programming is hard’ which is upheld by much of the literature and he argues, the community. This talk really caused me to reconsider my own beliefs about teaching programming, and to question my own expectations, assessment, module learning outcomes, and even program learning outcomes.

Mark Zarb presented a paper he authored with Roger McDermott, Mats Daniels, Asa Cajander and Tony Clear titled Motivation, Optimal Experience and Flow in First Year Computing Science. The authors examined motivation from the perspective of Self Determinism Theory and also considered the optimal state known as Flow – also colloquially known as being in the zone. After discussion how these concepts can be measured they presented preliminary results looking at motivation and flow in a first year computing class. The results were extremely encouraging and made me realize that there is a lot we don’t know about how students think ‘unconsciously’ while learning.

And now, the promised pictures. Don’t forget that there are many more here. I also included some photos from the conference bus tour which to say was all-encompassing is an understatement. We stopped about a half-dozen times at places like town squares, miniature farms (with alpacas of course) and a charming restored colonial-era mansion.

See you in Bolognia for ITiCSE 2017!

A1
Welcome
A8
Mehran Sahami’s keynote
WG6
Working Group 6!
A7
Main auditorium
A2
Lunch tent
A3
Lunch tent
A4
Lunch tent
A6
View from outside the lunch tent with the help of a little zoom
A5
View from outside the lunch tent with the help of a little zoom
C1
Bus tour
C2
Bus tour
C3
Bus tour
C4
Bus tour
C5
Bus tour
C6
Bus tour – La Mansion del Fundador
C7
Bus tour – La Mansion del Fundador
C8
Llama? Alpaca? I really should know by now…
D1
Tour of Santa Catalina Monastery
D2
Tour of Santa Catalina Monastery
D3
Conference Dinner and Peruvian entertainment
E1
Basílica Catedral de Arequipa
E2
Basílica Catedral de Arequipa
E3
Plaza de Armas de Arequipa
E4
Plaza de Armas de Arequipa
E5
Alley off Plaza de Armas de Arequipa
F1
Conference closing – more Peruvian entertainment!
F2
Conference closing – the conference committee joins in the dancing
F3
Adios! Off to Cusco and Machu Picchu!

Update – I couldn’t resist adding a few photos I took in Cusco today:

DSCN0325

DSCN0391

DSCN0554

DSCN0587

DSCN0419

DSCN0686

DSCN0790

And a parting shot from Machu Picchu Mountain…

ITiCSE Machu

‘Computer Science is Future-Proof’

In the end I chose to put quotes around the title of this post. Initially I didn’t, but I gave in to my reluctance of stating that anything, even my beloved profession, is future-proof.

U.S. News & World report didn’t use the quotes, but they had a subtitle to put a little more context to such a bold claim:


Computer Science is Future-Proof

No matter how technology transforms the jobs market, computer science expertise will be crucial.

Lisette Partelow, theauthor of the above article used a quote (below) which was said to her recently by an educator to exemplify skepticism surrounding all of the attention that CS has been getting lately:

uncertainty about the future job market means that giving students opportunities to learn computer science, while trendy, is essentially pointless. Whatever students learn now will be as out of date as MS-DOS and car phones by the time they can put it to use

Of course Lisette went on to back up her headline, but before I go over that, I would like to put forward two responses to the educator quoted above.

1. A quote from this article in Forbes should do to begin:

 Just because computer languages have a way of becoming obsolete, that doesn’t mean that we don’t need to know how to code. There is an underlying logic to the digital world and we must be capable of operating within that logic in order to function in it.

2. I learned to navigate the MS DOS command prompt around 25 years ago. This included basic commands, navigating the file system, and using system files like autoexec.bat and config.sys. This experience served me well. When I started learning linux, I moved straight in. In fact autoexec.bat and config.sys are very similar to .bashrc and other startup scripts, which I occasionally still use today. This is a great example of a technology becoming outdated, but the skill not, and reflects Lisette’s position below. 

So, back to Lisette, who notes that the U.S. Bureau of Labor Statistics has reported an estimate that there will be three jobs available for every new college graduate from a computer science program in 2016. She also states that “the programming languages my classmates learned in high school and college are probably defunct by now. Yet those who pursued computer science back in those dark ages still managed to get jobs at Google and other prestigious tech firms and kept these jobs as technology changed. Like the rest of us, they learned to adapt on the job as their field shifted.”

I would argue that almost all people who work in or with technology, are not working with the same tools or languages that they first learned. However, they did learn technology. Isn’t this how it is: Learn a technology (then abandon), learn another (then abandon), etc. Nothing puts you in a good position to learn in a fast-paced, constantly changing environment like learning something fast-paced, and constantly changing. In fact, the College Board has found that If a student takes AP computer science in high school, that student is eight times more likely to major in computer science in college.

The above makes the point that exposure is the key here. Exposing students to technology, even fast-paced, constantly-changing, going-to-be-obsolete-tomorrow technology.

I just love how Lisette wraps up her article. This is almost priceless:

Maybe the naysayers are right that the jobs of the future will be super-strange and that many of them won’t require coding skills that look anything like what we are teaching students now. However, computers and computing are taking over nearly every aspect of our lives – Americans look at their smartphones an average of 46 times per day, for example – so it’s likely that some basic understanding of how these systems work and can be leveraged will be an asset, even for intergalactic pilots and teleporter engineers.

I’ll have a go at my own wrap-up:

Technologies (almost by definition) become obsolete. People who can learn, cope, and change with these inevitable obsolescences don’t.

 

 

Misleading, cascading Java error messages

I have been working with enhancing Java error messages for a while now, and I have stared at a lot of them. Today I came across one that I don’t think I’ve consciously seen before, and it’s quite a doozy if you are a novice programmer. Below is the code, with a missing bracket on line 2:

public class Hello {
       public static void main(String[] args)  //missing {
              double i;
              i = 1.0;
              System.out.println(i);
       }
}

The standard Java output in this case is:

C:\Users\bbecker\Desktop\Junk\Hello.java:2: error: ';' expected
       public static void main(String[] args)
                                             ^

C:\Users\bbecker\Desktop\Junk\Hello.java:4: error: <identifier> expected
              i = 1.0;
               ^

C:\Users\bbecker\Desktop\Junk\Hello.java:5: error: <identifier> expected
              System.out.println(i);
                                ^

C:\Users\bbecker\Desktop\Junk\Hello.java:5: error: <identifier> expected
              System.out.println(i);
                                  ^

C:\Users\bbecker\Desktop\Junk\Hello.java:7: error: class, interface, or enum expected
}
^

5 errors

Process Terminated ... there were problems.

Amazing. This is telling the student that there were 5 errors (not one), and none of the five reported errors are even close to telling the student that there is a missing bracket on line 2. If the missing bracket is supplied, all five “errors” are resolved.

During my MA in Higher Education I developed an editor that enhances some Java error messages, and I have recently published some of this work at SIGCSE (see brettbecker.com/publications). I hope to get some more work on this front  soon, and in addition I would like to look more deeply at what effects cascading error messages have on novices. I can imagine that if I had no programming experience, was learning Java, and came across the above I would probably be pretty discouraged.

The enhanced error that my editor would provide for the above code, which would be reported side-by-side with the above Java error output is:

Looks like a problem on line number 2.

Class Hello has 1 fewer opening brackets '{' than closing brackets '}'.