KO Chiu Yu
Department of Economics
C.Y. Ko (2016). Using formative feedback, peer review, and online video-recorded presentations to enhance students’ research reports. CDTL Brief, 19(1), 3-9.
This article describes how timely formative feedback was provided to students reading the elective module EC5322R/EC5322 “Industrial Organization” at various milestones in their research projects. The provision of timely formative feedback was used as an intervention to enhance their reports for such projects. Formative feedback was provided by the instructor and peers following a peer review of students’ online video recorded presentations. The intervention was carried out over two rounds of teaching EC5322R/EC5322, with some fine-tuning of the approach after the first round. Some preliminary findings from the pilot study is also shared.
EC5322 ‘’Industrial Organization’’ is listed as both a graduate elective and an undergraduate module, EC5322R, which is open to undergraduate students pursuing an Honours Degree in Economics. The purpose of this module is for students to be able to understand and make in-depth analysis of various theoretical issues related to modern industrial organization. The module aims to enable students to develop their own economic models to explain real-world phenomena, and communicate these models well in both oral and written form. Hence, 70% of the module’s assessment is based on an independent research project, while the remaining 30% is based on the final exam.
In my first year of teaching EC5322R/EC5322 (n=15), students were asked to present their findings in class before submitting their written reports. However, the main problem we discovered about this instructional strategy was that in the course of doing their projects, many students were too focused on the technical details and failed to see the “big picture”, that is, understanding the key economic concepts that underpin a lot of these real-world phenomena. For example, when students attempted to apply the economic models discussed in class to explain the outcomes of particular markets, some of them failed to appreciate that while the assumptions from different models may seem innocuous when taken separately, they may not be plausible when applied wholesale in certain markets. Had they received timely feedback on the preliminary work they had done for their projects and refined it accordingly, they would have written papers of better quality.
Interventions introduced: First cycle of interventions
As a result, in the second year, I refined the initial instructional strategy by replacing the in-class presentation with an online video recorded presentation. We felt that incorporating peer review through online video recorded presentations would benefit students’ learning in a few ways. First, it would facilitate timely feedback and enable closer monitoring of students’ progress throughout the project. According to Nicol and Macfarlane-Dick (2006), providing timely feedback is one of the strategies that could be adopted to give quality feedback to students (p. 210). Secondly, through peer feedback, “students learn a great deal by explaining their ideas to others and by participating in activities in which they can learn from their peers” (Boud, 2001, p. 3). Another motivating factor was that the online mode would alleviate logistic problems. For instance, if each student were to do a 10-minute classroom presentation of their project, it could take several weeks of classroom time even for such a small class. With online video recorded presentations, we could set aside more time for in-class post-presentation debriefs and discussions. Finally, submission of multiple drafts would be hard to apply in this case as building economic models would require iterative development. When it comes to building a good economic model, one should follow the advice found in a phrase attributed to Albert Einstein: “Make everything as simple as possible, but not simpler.” A good economic model requires abstraction of complicated real-world problems into a few key assumptions. They should not only be simple enough to be tractable for analysis but also flexible enough to accommodate different possible extensions. Refining an economic model usually takes much more time than developing the first prototype.
The quality of the written reports appeared to improve with this modification to the instructional strategy. This could be due to students’ participation in the online discussions following each presentation, which generated constructive comments. In fact, one student’s paper was so well-written that it was later developed into an integrated honours thesis, and that particular student went on to achieve a double first-class honours.
Moreover, as the post-presentation discussions generated constructive comments that helped improve their written reports, I asked students to comment on their classmates’ presentations using the forum function of the website on which the presentation videos had been uploaded. Although we incentivised the peer evaluation by making it count towards 5% of their final grade, it was observed that the comments collected were less thoughtful and constructive than expected. One possible reason could be that students may encounter difficulty in evaluating a working idea from the video presentation without a written draft.
Interventions introduced: Second cycle of interventions
Therefore, in my third year of teaching EC5322R/EC5322, I refined the instructional strategy further by including an additional online video-recorded presentation to allow students to provide feedback in the early stages of their projects. A 3-page written proposal was also included to facilitate the peer review process for the second online video recorded presentation. Based on my subjective evaluation (as well as feedback from colleagues and students who compared the submissions with those from previous cohorts), the quality of the peer reviews, written reports and in-class presentations all improved significantly with this further modification. Informal interaction with students and a survey confirmed the success of this experiment. The following sections detail the implementation of these additional learning activities.
Project Milestones Provided for Students
The revised instructional strategy, with its additional learning activities, was divided into three key phases over the semester:
- The screening phase. To kick-start the assignment, students were asked to share initial ideas of their respective independent research projects via 5-minute online video-recorded presentations in Week 5. The post-presentation discussions I had with the class were useful in helping to filter out less promising ideas and for me to provide input to improve the more promising ones. Prior to these discussions, students were given guidelines on how to evaluate their peers’ work. To complete the peer evaluation, they are required to answer qualitative questions in a form created on Google Docs.
- The feedback phase. Based on the feedback received from their peers, each student would prepare and submit a short written proposal (3 pages) in Week 8, in which they had to formulate the research question. This was followed by a second extended online video-recorded presentation (10 minutes) in Week 9 which focussed on their methodologies, and a peer review in Week 10.
- The reporting phase. Finally, students presented their results during the last two classes before submitting their final reports in Week 13.
Thus, there were five tasks within the three phases where students could receive critical and constructive feedback before submitting their written reports. Table 1 lists the project milestones for the research project. The tasks were arranged within these phases to scaffold their learning; it also gave them ample time to revise their ideas, or even switch to other topics. To implement such an elaborate feedback mechanism, I gave out a clear roadmap during the first class with detailed guidelines for each part of the assessment.
Screening phase: 5-minute online video recorded presentation
During this initial phase, students were given a list of topics to explore. These topics had been attempted by previous cohorts and were useful in helping the current batch decide whether they were able and willing to conduct original academic research within 13 weeks.
What was the rationale for including the screening phase? I felt that it mentally prepared the students as they embarked on the research project, which was crucial for a successful paper. It minimised their tendency to procrastinate, an easy trap to fall into unless they really enjoyed working on their projects. Since the willingness to do research was hard to write down as a prerequisite, the screening phase allowed them to do a self-check at an early stage of the project. However, I did emphasise that they were free to explore topics beyond the list, especially since the class composition differs from year to year.
In Week 5, each student would upload a 5-minute presentation of their chosen research topic to Youtube, which allowed for easy viewing and file-sharing during the peer-review component of the project. Students could also choose to put their videos on a restricted channel to ensure their presentations were not searchable online. While students were free to choose their presentation styles, they were also given minute-by-minute guidelines on how to structure their presentations. In the guidelines, it was recommended that they spend one minute each to do the following in their presentation: describe their research idea in non-technical terms, justify its importance, spell out deficiencies in previous research, explain what can be done, and summarise the expected outcomes. Having students record their presentations reduced their anxieties over presenting their preliminary ideas. To further relieve students of this pressure, they were awarded full marks for their effort in completing this first online video recorded presentation. Each student would subsequently receive my feedback via email with emphasis on the feasibility of the ideas they proposed. After this screening phase, three students shifted their research focus or changed research topics.
Feedback phase: 10-minute online video recorded presentation, research proposal, and peer review
As part of the milestone for the project, each student was to submit a 3-page research proposal in Week 8. In the proposal, they had to highlight the importance of their proposed research question, conduct a thorough literature review, and describe the methodology. The page limit was set to ensure students produce a concise report. This would not only compel them to think more deeply about the outcomes they wanted to achieve for this research project, it would also help reduce the number of pages to comb through for the subsequent peer review. It was interesting to note that students strictly adhere to the criteria listed for the proposal. In fact, two students sent me an email to request a topic change, even though my approval was actually not required.
In Week 9, each student submitted a second 10-minute online video recorded presentation in which they gave further elaboration of their analysis. One important purpose of the second presentation was for students to provide verbal and graphical illustrations, especially given that a formal mathematical model may not be ready at this stage. Once the entire class had submitted their presentations and proposals, each student evaluated the presentations of two randomly assigned classmates on Week 10. For the peer review, I issued a standardised feedback form for students to record their qualitative comments on potential deficiencies as well as suggestions for improvement. The comments were then posted online to inspire further discussion. I emailed my input to each student only after the entire class had completed the peer review. During this phase, only one student switched topics.
Reporting phase: In-class presentation and written report
Students presented their research findings via 15-minute in-class presentations in Weeks 12 and 13. We incentivised the post-presentation discussions by getting students to provide qualitative feedback to their peers individually, with evaluation forms provided. The feedback formed part of the overall assessment. We hoped that this would encourage active class participation amongst students and garner higher quality feedback. A surprising thing we observed was that having to do the two online video-recorded presentations may have substantially improved their time-management skills. Most of students from the current cohort were able to complete their presentations within the time limit, unlike those from the previous two years.
Based on the feedback received from their peers after the second online video-recorded presentations, students finalised and submitted their 20-page written reports at the end of Week 13.
Findings and Discussion
At the end of the research project, I conducted an informal survey to collect students’ feedback about the whole process. Ten out of a total of sixteen students in the class responded.
According to the quantitative feedback collected, 100% of the respondents found that comments from myself or their classmates were useful in helping them improve their research project. On a scale from 1 to 5 (best), overall usefulness of the learning activities received an average score 4.6. More details are shown in Table 2.
In terms of the 20-page written reports submitted, I found most of them to be of high quality. The survey results also indicated that students were happy with the quality of their papers, with 50% of the respondents planning to further develop their written reports into research articles.
The majority of qualitative comments were positive but there were two suggestions worth mentioning.
First, one student expressed disappointment over the peer review, noting that “comments from [the] professor are useful, but comments from one of two peers may not be so useful because some peers may have not taken peer review work seriously.” I believe that this might be due to the fact that students might be lacking in experience when it comes to critically analysing research projects. To minimise such occurrences in the future, I may provide students with guidelines and examples of good peer feedback, or even explore other peer feedback formats such as voice feedback. In addition, I may consider getting future cohorts to write short reflection journal entries about their learning experiences throughout the process. Such documentation would serve as a good source of data in helping me to further refine and develop the learning activities.
Second, some students suggested that the second online video-recorded presentation could be conducted earlier in the semester to ensure that “[they] have more time to adjust the model according to the comments from others.” One way to do this would be to have students submit both the 3-page proposal and the online video-recorded presentation in the same week to allow more time for them to provide peer feedback.
One of the biggest challenges I faced when it came to teaching students to conduct original research in EC5322R/EC5322 was helping them to overcome the tendency to procrastinate. Most of the time, they had poor time management skills and had the tendency to underestimate the time required to complete the analysis. It was observed that some spent several weeks on gathering information and looking for a topic, which left them with very little time to complete their paper. In fact, some believed that they could complete their project by cramming all their tasks in the last few weeks! Also, while the conventional approach of using interim presentations and submission of drafts can help students set project milestones, they are not ideal for research projects such as the ones for EC5322R/EC5322, which require frequent feedback. This is because for such projects which require iterative development, it is very difficult to transform a budding research idea into a full-fledged research project without taking multiple detours. This is the case even for established researchers. Although the current learning activities implemented in EC5322R/EC5322 seem to be helpful in providing some form of scaffolding to students in carrying out their projects, it would be crucial, as a complementary measure, to include additional rounds of feedback in future and even different channels of feedback.
Meanwhile, online video recorded presentations have several advantages over the submission of drafts and interim in-class presentations. First, they do not occupy any classroom time since students can watch them outside of curriculum hours. As such, multiple online video-recorded presentations can be used as a mode to scaffold students’ learning through peer feedback. Second, students may feel more at ease presenting their ideas online instead of speaking in front of the whole class. Third, students can evaluate their own presentations and improve their exposition through multiple recordings. Fourth, online video-recorded presentations allow students time to think through their responses before sending in their feedback. Fifth, the online platform ensures that the exchange of ideas and comments between peers is not constrained by time limits, which it would be if the presentations were conducted in the classroom. Finally, going through two rounds of online video-recorded presentations means that students learn how to manage their time more efficiently in a presentation; they would have had more practice learning how to plan their presentations within the time limit. Thus, based on preliminary findings of the pilot study, it seems that the use of formative feedback through peer review leveraging on online video-recorded presentations enhances the quality of students’ written research reports.
Boud, D. (2001). Introduction: Making the move to peer learning. In D. Boud, R. Cohen, & J. Sampson (Eds.). Peer learning in higher education: Learning from & with each other (pp. 1-17). London: Kogan Page Ltd.
Nicol, D. & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199-218. http://dx.doi.org/10.1080/03075070600572090.
About the Author
Dr Ko Chiu Yu teaches EC4344 “Financial Market Microstructure” and EC5322/EC5322R “Industrial Organization”. He subscribes to the philosophy that effective learning requires understanding of multiple facets in both depth and breadth. He also believes that giving students continuous feedback during the course is crucial as it means they learn from their mistakes,which consolidates the knowledge they have acquired, and gives them the motivation to explore further and deeper.