TAN Hong Ming
Department of Analytics and Operations
NUS Business School
Hong Ming describes his experience of developing and using question pools to design fair quizzes. He shares the results and reflects on the challenges of the design process.
Tan H. M. (2023, April 26). Designing fair quizzes with a question pool. Teaching Connections. https://blog.nus.edu.sg/teachingconnections/2023/04/26/designing-fair-quizzes-with-a-question-pool/
Quizzes are widely recognised as a useful educational tool, allowing educators to evaluate students’ comprehension of material and provide feedback to improve learning outcomes. However, implementing quizzes can present challenges such as difficulty in creating effective quiz questions, and potential negative effects on student motivation and engagement. Multiple-choice testing is a popular quizzing approach that enhances material retention, but it can also lead to false knowledge acquisition due to exposure to misinformation (Roediger III & Marsh, 2005). To minimise these negative effects, educators should provide feedback during multiple-choice testing (Butler & Roediger, 2008). Additionally, concerns about cheating during quizzes present another obstacle to effective implementation.
On top of proctoring, one way to further mitigate unwanted collaboration during quizzes is to generate the quiz from a question pool. This pooled approach ensures that the individual quiz each student receives belongs to one of several possible combinations of quiz questions instead of the same permutation, without the lecturer expending additional effort to customise quizzes for each student. However, this approach has a weakness in terms of ensuring fairness, which we illustrate with a real in-class example1 below.
DAO1704 “Decision Analytics Using Spreadsheets” is a compulsory course for all NUS Business School undergraduates. Typically taken in their first year, students are taught the theory and skills needed to capture business insights from data for decision-making using spreadsheets. DAO1704’s objective is for students to become proficient in the extensive use of Microsoft Excel in the business environment.
In DAO1704’s previous run, each student was given a quiz comprising three questions generated from a 10-question pool. The problem with this pooled approach was that the questions would vary in their difficulty levels. As seen from the breakdown of quiz results in Figure 1, if a student got a quiz comprising Questions 1, 2, and 4, she would have an extremely difficult time, compared to one who got Questions 3, 6, and 9.
Figure 1. Breakdown of quiz results.
This raises an issue of fairness. It is not enough that on average, students get the same difficulty level for an assessment. We need to ensure that each student receives the same level of difficulty. Being unlucky cannot be the reason a student receives a more difficult test.
Suggested Solution: Separate Pools
One suggestion is to design the quiz so that each question has its own question pool to generate from. Each pool should have questions that are similar in difficulty level. The easiest way is to have slightly different versions of the same question, e.g. changing the numbers. For multiple-choice questions (MCQs), the options can remain the same, while the question can be changed so that different versions have different options as the correct answer. An added advantage of this approach (with the same options) is that when students discuss their solutions after the quiz (and they tend to do so), they may discover that they had chosen different options and investigate further. This occurred for previous runs, and students have reached out to gain a better understanding of who got the correct answer, only to find out that both were right. This peer discussion, which occurs together with instructor explanation, increases student learning (Smith et al., 2011).
We see the results of the suggested solution in Figures 2 and 3. In the implementation, the quiz is such that each question had a pool with only two variations. For example, for a two-question quiz, the Question 1 in the quiz is either Question 1 or Question 2 in Figure 2, and Question 2 in the quiz is either Question 3 or Question 4. As evident from Figures 2 and 3, the pairs of Questions 1-2 and Questions 3-4 have similar difficulty levels.
Figure 2. Breakdown of results with two questions each from separate question pools.
Similarly, for a three-question quiz, the question pairs in Figure 3⸻Questions 1-2, Questions 3-4, and Questions 5-6 -all have similar results, reflecting the similarity in difficulty levels.
Figure 3. Breakdown of results with three questions with separate question pools.
An Interesting Finding
Figure 4. “True” and “False” variants of the same question.
One interesting finding was that the “True/False questions may not be as straightforward to design. One quiz had a question where respondents had to identify all statements that were “True” (or “False” in the other variant). Figure 4 shows the two variants, the only difference being that one asks for all the “True” statements, and one asks for the “False” statements.
One would expect there to be no difference in the difficulty level and hence the results. Surprisingly, as Figure 5 illustrates, Question 3, the true variant, has significantly more students answering it wrongly than Question 4, the false variant.
Figure 5. Results of the “True” (Question 3) and “False” (Question 4) variants.
On further research, this outcome is known in pedagogy research (Cronbach, 1941; Peterson & Peterson, 1976). We should be mindful when designing “True”/”False” questions, as the level of difficulty may not be what we think.
The results suggest that designing quizzes with separate question pools can effectively reduce cheating and promote collaboration among students. By varying the questions within each pool, educators can create a more diverse range of questions testing the same concepts and increasing the quiz’s validity. Furthermore, the finding that students are more likely to investigate and discuss different answers when they have the same options suggests that providing consistency in answer options encourages more post-quiz collaboration among students.
The surprising results observed in the “True/False” questions demonstrate the importance of careful question design. While one expects the two variants to have similar difficulty levels, the significantly higher number of incorrect answers for the “True” variant suggests that the question’s phrasing may have influenced students’ understanding of what was asked. This highlights the need for educators to not only vary the quiz content, but also carefully consider each question’s wording and phrasing to ensure it accurately tests the desired concept.
Overall, these results suggest that by carefully designing quizzes with separate question pools and using consistent answer options, educators can improve the effectiveness of quizzes in promoting learning outcomes while mitigating risks of cheating. However, it is important to remember the need for careful question design to ensure that quizzes accurately assess students’ understanding of course material.
- The quiz questions as well as the analyses of the questions and their results (see Figures 1-5) were done on LumiNUS.
Butler, A. C., & Roediger, H. L. (2008). Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing. Memory & Cognition, 36(3), 604-616. https://doi.org/10.3758/MC.36.3.604
Cronbach, L. J. (1941). An experimental comparison of the multiple true-false and multiple multiple-choice tests. Journal of Educational Psychology, 32(7), 533-543. https://psycnet.apa.org/doi/10.1037/h0058518
Peterson, C. C., & Peterson, J. L. (1976). Linguistic determinants of the difficulty of true-false test items. Educational and Psychological Measurement, 36(1), 161-164. https://doi.org/10.1177/001316447603600115
Roediger III, H. L., & Marsh, E. J. (2005). The positive and negative consequences of multiple-choice testing. Journal of Experimental Psychology: Learning, Memory, & Cognition, 31(5), 1155-9. https://doi.org/10.1037/0278-73220.127.116.115
Smith, M. K., Wood, W. B., Krauter, K., & Knight, J. K. (2011). Combining peer discussion with instructor explanation increases student learning from in-class concept questions. CBE—Life Sciences Education, 10(1), 55—63. https://doi.org/10.1187/cbe.10-08-0101
TAN Hong Ming is a lecturer at the Department of Analytics and Operations, NUS Business School and a research fellow in the Institute of Operations Research and Analytics at the National University of Singapore (NUS). His research includes decision-making under uncertainty, information economics, sports analytics, and causal inference. He has multiple industry collaborations, mainly with airlines and in healthcare. Hong Ming teaches undergraduate- and graduate-level analytics courses, and in analytics executive education programmes at NUS. He was previously a lecturer at Temasek Polytechnic, and has a PhD in Operations and Analytics.
Hong Ming can be reached at email@example.com