I was thrilled to learn that a paper I co-authored, First Things First: Providing Metacognitive Scaffolding for Interpreting Problem Prompts, was selected for the best Computer Science Education Research paper award at the 2019 ACM SIGCSE Technical Symposium on Computer Science Education.
I had a great time working on this project with my co-authors: James Prather, Ray Pettit, Paul Denny, Dastyni Loksa, Alani Peters, Zachary Albrecht and Krista Masci, and I look forward to future work with them in this area. The paper will be presented on Friday, March 1 at SIGCSE 2019 in Minneapolis, Minnesota, at 11:10AM. The paper is available on the ACM Digital Library via www.brettbecker.com/publications.
Brief synopsis below:
We found that simple modifications to an automated assessment tool can improve metacognitive awareness in novice programmers. Specifically, students that used the modified tool showed a higher degree of understanding problem prompts and were more likely to complete programming tasks successfully.
When solving programming problems, novices are often not aware of where they are in the problem-solving process. For instance, students who misinterpret the problem prompt will most likely not form a valid conceptual model of the task and fail to make progress towards a working solution. Avoiding errors, and recovering from them once they occur, requires metacognitive skills that enable students to reflect on their problem-solving processes. For these reasons, developing metacognitive awareness is crucially important for novice students. Previous research has shown that explicitly teaching key steps of programming problem-solving, and having students reflect on where they are in the problem-solving process, can help students complete future programming assignments. Such metacognitive awareness training can be done through personal tutoring, but can be difficult to implement without a high ratio of instructors to students.
We explored a more scalable approach than personal tutoring, making use of an automated assessment tool, and conducted a controlled experiment to see whether scaffolding the problem-solving process would increase metacognitive awareness and improve student performance. We collected all code submissions by students in both control and experimental groups, as well as data from direct observation using a think-aloud protocol. We used pre- and post-tests to measure the shift in participants’ growth mindsets.
We investigated whether providing an explicit metacognitive prompt – one that requires solving a test case immediately after reading the problem prompt in an automated assessment tool – would assist novice programmers in overcoming the metacognitive difficulties identified by prior research. We conducted think-aloud studies with participants, and participant observation allowed us to record the participants’ actions, apparent thought and problem-solving process, and external reactions to error messages and other feedback.
There is limited research on scaffolding students’ metacognitive awareness, and none that we are aware of that utilise a scalable solution, in the form of an automated assessment tool. We also investigated if the use of this tool resulted in a shift in the students’ growth mindset. We achieved this through a mix of qualitative and quantitative analyses.
The primary contribution of this work is two-fold. First, our results indicate that this line of research, although not conclusive, warrants future work. Second, we have identified two specific questions for future work:
- How do randomly generated test cases impact studies like this?
- Is there a correlation between the number of times a student re-reads a problem prompt and the number and severity of metacognitive difficulties faced?
Our future work includes replicating this work at a larger scale, with greater numbers of participants from multiple diverse institutions. We feel this is required as the results of this study are promising, but we would like to see if there is more conclusive evidence and more generalisable results. In addition we aim to work towards answering the two questions set out above.