I chuckled when I read this article: Desperate undergrads pray to ‘bell curve god’
Superstitions aside, students correctly know that the bell curve does affect them in some way or other. However, I hope that no one is feeling haunted by the bell curve.
What is this ‘bell curve’ all about?
In probability theory, the normal distribution is a continuous probability distribution that has a bell-shaped probability density function, known as the Gaussian function, or informally, the bell curve. The normal distribution is the most prominent probability distribution, because many large sets of data are approximately normally distributed.
For example, the heights of all students in NUS are likely to be normally distributed. The weights of NUS students probably follow a normal distribution too. Likewise, if I set exams targeted at the average competency of a group of students, and if the class is large enough, the exam scores are likely to follow a normal distribution curve. Setting such an exam is, by no means, easy. Pitch it tough, most students will fail. Set it too easy, and many will score very high grades, and the resulting scores are hardly differentiated.
Grading and Moderation
Differentiation is necessary for CAP purposes, and for Honours classification, and these are here to stay for the foreseeable future. Most if not all major universities have variants of degree classes or GPA scores. And because of the need for differentiation, many institutions from North America to Asia, use the bell curve as a mechanism to moderate marks.
Module requirements may encompass different modes of assessment such as tutorial presentations, laboratory reports, projects, essays, as well as mid-term and final examinations. Grading may be based on absolute performance, relative performance, or a combination of the two. Higher-level modules with small enrolments typically grade a student based on his absolute performance; larger lower-level modules take into account a student’s performance vis-à-vis the other students in the same module. Where necessary, the final grade which a student receives for a module may be subject to moderation.
One important reason for grade moderation is that examiners come from diverse academic backgrounds and may be accustomed to different marking regimes. While we do make every effort to make sure modules are designed with clear learning outcomes, and professors are responsible to ensure their exams are pitched at the right level, grade moderation will prevent grade inflation or deflation, and helps to achieve consistency in assessment grading across modules.
How NUS Applies the Bell Curve
At the end of the semester, a student is awarded a grade (and not specific marks) for each course taken. As such, NUS adopts a ‘recommended grade distribution’ in the following manner. I shall illustrate with an example – bear in mind that this is meant for illustrative purposes and is not the actual distribution that we are currently practising.
Example of a Recommended Grade Distribution:
| Grade | A+, A, A- | B+,B, B- | C+,C | D, F |
| Proportion | Not more than 25% | Not more than 40% | About 30% | Not more than 5% |
One could compute the average grade point:
(5 x 0.17 + 4.5 x 0.08 + 4 x 0.14 + 3.5 x 0.13 + 3 x 0.13 + 2.5 x 0.15 + 2 x 0.15 + 1 x 0.05)
which is approximately 3.34, or roughly B grade.
And here are 3 possible course grade profile scenarios after moderation. For the first module, the test/exam may have been too easy, resulting in high cut-off marks for each grade. The average mark was 83 for a class of 630. The department will also look at the paper before moderation.
In the next scenario, the resulting cut-off marks for each grade after moderation, are moderate.
In this final scenario, the resulting cut-off marks for each grade after moderation are rather low.
I must emphasize that the recommended grade distribution is not applied blindly, and there are ample opportunities, within reason, for discretion and flexibility.
- First, the class size must be large enough, preferably above 30. For smaller classes, Professors are given discretion on an appropriate grade distribution because small sets of data may not be normally distributed.
- Second, we are not looking for a perfect fit, i.e., we usually ignore small deviations.
- Third, if Professors have strong reasons to deviate from the recommended grade distribution, we are usually amenable to acceding to their requests. For example, some modules belonging to special programs in the various Faculties/Schools have their own distributions.
- Fourth, we sometimes also look at the CAP profiles of a class, and tweak the grade distributions appropriately. For example, if many students with high CAP choose a particular course, it will not be fair to apply the recommended grade distribution to this class. Another example: for Honours classes comprising students with an average CAP of at least 3.5, the grade distribution will be skewed higher.
Additionally, much goes on post-exams, before the grades are finalized and released to students. First, the grade profiles for individual modules are examined and compared at the Department level, and then across Departments at the Faculty level. All grades are carefully scrutinized by Department and Faculty Boards of Examiners before they are submitted to the Board of Undergraduate Studies and the Board of Graduate Studies for approval. Further checks are conducted at the University level by the Board of Undergraduate Studies and Board of Graduate Studies to ensure that there is consistency of assessment across Faculties/Schools.
In conclusion, I hope that this post has given a better picture of how the bell curve works at the NUS, and hopefully this helps to alleviate some bell curve anxiety. The bell curve is used primarily as a tool to moderate grades, and as a guide to prevent grade inflation or deflation. We do not apply the bell curve mindlessly or excessively. Students are sometimes worried about falling on a ‘wrong side’ of the bell curve. Do not worry too much, more often than not, we err on helping students along.
Here’s also wishing everyone a joyous Lunar New Year!
Dear Sir,
You mentioned “Differentiation is necessary for CAP purposes, and for Honours classification, and these are here to stay for the foreseeable future.”
Lets say for a module where every student is good at the paper(Perhaps it is too easy ?) and everyone is scoring , a very high scores, is there really a need to differentiate in such scenario ? If one forces a a bell curve on such scenario , one will end up differentiating students based possibly on human errors instead of differences on the understanding of the subject.
For such scenario , why is it not possible that the school gives everyone the same grade since everyone is good at the paper? I believe such scenario is quite common for some of the modules in NUS.
I think we singaporeans also need to be less critical of everything and be able to adapt to all sort of systems. There is no perfect system, for everything. Be it political or school grading. Creating a perfect system is one thing, eradicating old system is another. Even with successful implementation of the best system thought of, who knows really how good is it? So why not learn to be adaptive? The cold truth is the world will never going to change for a person, to survive, you have to adapt to the environment.
It may be true that we need to learn to adapt to the environment but with the environment promoting the wrong mentality in the students. Is it justifiable for us to change for the environment or do you think that it is better for the environment to change for its students? I believe most if not all students have been affected by it one way or another. Do you think that Kiasuism is valid for meritocracy?
In a ‘meritocracy’ ‘merit’ is determined by education, which requires some form of discrimination between students. Without the bell curve, an absolute grading system wouldn’t completely get rid of kiasuism either. Kiasuism is possible as long as it’s possible to lose.
… it would seem that if you want to eradicate that you will need to abolish the idea of grading entirely. For then, there will be no loser. But then without that system, employers would also have an even greater headache sorting through potential applicants for positions. And students lose one channel of feedback on their results.
Whether:
1. Meritocracy is a good system
2. Grading in universities should serve the needs of employers in society
3. Sorting through applicants by honours class is a good system
4. Kiasuism is necessarily a bad thing
5. It is possible to change the environment at all, given NUS doesn’t have control over society at large
6. Meritocracy must necessarily lead to kiasuism
Are all debatable points that are somewhat beyond the scope of this discussion, and which merit their own discussions.
http://www.facebook.com/photo.php?fbid=10150422574527734&set=a.98301942733.92939.541152733&type=1&ref=nf
Political agenda aside, this story is very interesting. In Singapore, the only reason for bell curve to be so powerful is kiasu mentality, otherwise this will happen:
http://www.phdcomics.com/comics/archive.php?comicid=1410
(Am I the only undergrad reading PhD comics?)
I forgot to add, most likely your final exam papers are graded by the grad students (esp Science, Com and Engineering course with class >100), and not by professors themselves. Do you really believe PhD candidates are much better than us, given that they may only receive up to 4 more years of education? Hell, I don’t.
NUS need transparency for the exam and grading system. An A may be an A, but 80% for an A or 100% for an A make a different to students’ perspective. If transparency of personal level is that difficult to show, it is easier to show the 10th and 90th percentile in the results slips.
You could say in that case that no one is qualified to mark exam papers. I mean NUS professors and lecturers from other universities don’t need any kind of educational qualification in order to teach and usually only have their PhD’s.
I can assure you that postgraduates are more than capable to mark exam papers and markers are given strict marking schemes that they must stick to. So in answer to your question, yes PhD candidates are good enough to mark exam papers although in my case, we only marked lab reports from practical classes.
For modules which have some form of objective detail, or at least a high consensus in a particular way of thinking (particularly Mathematics modules), the marker has no effect on the paper. I could get a Secondary 3 student to grade it; if she follows the given key closely enough, I find no issue in letting her mark me. For papers along the lines of a more subjective basis like history or philosophy, even a full professor may never be fully qualified to mark the paper, since ultimately an objective, or highly consensus based view may not be present.
Generally speaking, someone who has already passed through the Bachelor’s system should be ok for marking. But if you want to be picky, for some papers indeed nobody is truly qualified to mark the papers.
And yet someone has to do the job.
Besides, what does the credentials of the marker have to do with your latter point on transparency? You can have a transparent system with an incapable marker, an opaque system with Nobel laureate markers, transparent systems with Nobel laureate markers… etc.
In terms of the impact the marker has on a bell curve grading, it is ok as long as the same marker marks the same question across all papers, or a highly standardised key is used. The only problem arises when controversial topics are marked by different people – a situation I do not see occurring often.
Hi, there is an error in the computed GPA. The B-, C+, C and D grades are worth 3, 2.5, 2 and 1.5 points respectively. The current error implies that there are 55% of the students obtaining a B+, B or B- grade.
Dear Moi,
Thanks for alerting us on the computation and we’ve adjusted it! The table is for illustrative purposes 🙂 Cheers.\
mod, Rachel
Damn right, Michael, spot on!
One of my second year core modules had an average grade of C, with most of the class doing badly and almost nobody getting A. The professor (it was said) rigorously demanded these grades because the cohort did not know the material and he wanted them to know it.
In the second half of the year, when the other half of the cohort took it under a different professor, almost everybody did well; possibly due to lenient grading by the professor, but also with great deviation from bell curve.
Which module do you think people will take? Should I try to learn under the fierce professor, or the lenient one?
My friends call me foolish, because I take a variety of modules across faculties. To them, it doesn’t make sense to work harder on my core modules and harder on cross-faculty modules; for them, they do enough for core modules and try to ace unrestricted cap-pullers, knowing they can score higher in the non-core modules and using it to boost/boot-strap their CAP.
I am a lover of learning. And I would like to sample many introductory and interesting modules; yet I restrict myself because I daren’t do too much overall damage to my CAP. I feel like it’s a frivolous luxury – I do well enough, but I can’t gamble away my hard work on something I’m interesting and passionate about because it will hurt my CAP. I have to take safe – as far as possible – modules, and I have been punished for exploring, and trying things.
I suppose this is another hidden lesson from NUS; there will be no rewards for exploring, trying new things, and daring to go outside your major.
And don’t get me started on S/U.
I seriously hope that it is not true. I know at least one module where a lot of lecturers from the same department shared the marking of all the final examination papers because it was a huge class. And I hope we don’t use those kind of words; they are not nice. But certainly NUS should follow the policy that other good universities practise, which is to allow students to review their examination papers the next semester, so that they can learn from their mistakes. It is certainly not wise to end learning on handing in some scribbled script that becomes scrap paper after the grades are finalised. Moreover, allowing students to view the marking will ensure that the grades are indeed correct and not subject to the graders’ moods and fancies. Of course we hope that there is no such biased marking under the current system but if we can prevent the whole problem, why not? Furthermore, the grade or even the exact marks are hardly important. Should we simply care about crossing the line between one grade and the next? I hope not. But until the NUS administration changes, this is going to be the natural outcome of the grade game: put in just enough effort to get the grade; does it matter if I understand what I am doing or learn anything at all? I personally would rather spend more time exploring each subject itself than spend time practising producing examination-smart (marker-friendly) answers but, as it is, I don’t have much choice but to waste a little time to do just that.
True of course. But if the benefits of something new outweigh the costs of changing from the old, we should not dismiss it just like that. We have been engineering our environment for thousands of years, nowhere near to simply adapting to it. As long as we continue to do it carefully to help others, there is no reason to stop and just accept the status quo. If we want fairness, we have to start somewhere. We cannot simply say that because the whole world is full of unfairness, therefore we should not bother to be fair. And for the NUS examination system to be fair, allowing students to see their graded papers is indeed necessary, even if they are not returned to them but kept by NUS after the reviewing.
Thanks very much for all your comments. I am heartened to see students posting lengthy and some well-considered commentaries. There are many interesting suggestions, and I briefly highlight brief comments as follows:
#1: Thanks, Jon, the CAP for graduate students is more to get them to take course work seriously. Most do, but for the others who don’t, they cannot continue.
#4: Thanks, Koala, for empathizing with me! We do monitor teaching performance. If there are cases of bad teaching, do alert us. We will take immediate action to counsel and guide our colleagues appropriately.
#5: I sense that you may have taken the GEM Living with Math, and found that you did not learn anything valuable. Would you mind elaborating why so?
#6: Dear Michael, grading can be very much dependent on the professor. The level of difficulty of the paper and the relatively leniency in grading are two important factors. If he/she chose to set a difficult paper and grade strictly, you have grade deflation, and if he/she chose to set an easy paper and grade very leniently, you will have grade inflation. There is thus a need to remove such biases when you are treating 25,000 students! You also objected to the “reward” system arising from moderation – there is some idealism here. First, grade moderation already incorporates some elements of absolute grading, i.e., you need to meet a minimum requirement for an A, or for that matter, any grade. Thus, if one sets a low mark for an A, it is pretty much the absolute system. Imagine everyone graduating with first class honours at NUS – What will employers think of our graduates? Will students be motivated to learn more? Refer to the story on the Economics professor highlighted in Kim’s comments (#3 on page 2) – there is really no incentive for anyone to excel, and the average will degenerate.
#7: Exams are blunt instruments for measuring learning effectiveness. I do think that quantitative modules, e.g., Science and Technology modules, are easier to handle through exams. Even grading is less subjective.
#8: David speaks of an idealized situation, perhaps achievable in a small setting, for instance, in a liberal arts university (i.e., about 4,000 to 5,000 students in total). For a university with the scale of NUS (i.e., 25,000 undergraduate students), it is impractical to speak of such an idealized situation. I like your suggestion that our Deans and Heads attend classes to get a sense on the teaching. Some do, and many of our Deans and Heads also teach. In addition, one component of teaching evaluation is based on peer reviews. Once every three years, a pair of colleagues will sit in on classes and provide an assessment of the teaching. For promotions, a candidate will need two sets of independent peer review assessments.
#9: Thanks, Jack, for arguing very convincingly why moderation is required to smoothen the effects of exams, despite the fact that it is itself an imperfect tool for testing learning effectiveness. However, while it was mentioned that the “societal function for grading” was to “help employers decide who to hire”, I would like to clarify that it was never the intention for educators to facilitate this function. It was primarily meant to differentiate the performance of students. Jack also proposed shifting the module evaluation (by students) till after the exams. This was discussed, and one of our concerns was that our students may base their evaluation on the perceived level of difficulty of the exam or their perceived performance during the exam.
#10: Alex thought that it is unfair to subject exchange students to our high standards. Well, that is something exchange students will have to accept, as it will not be fair if there were two systems. Foreign universities accepting NUS students do not make exams more difficult for them because they are used to a more stringent system. Further, most exchange students only get the Pass/Fail grade, and frankly, it is rather difficult to fail a module if you put in enough effort. My students told me of professors who contact the students who are potentially going to fail the module, and some even counsel them individually and suggest that they ‘withdraw’ in order to protect the students’ CAP.
#11: Dear Douglas, I am not convinced that it is easier to score an A in an arts module compared with a Science and Technology module. I will do a check.
#13: I hope Jack is not saying that it had happened in NUS. I will be very disappointed to have students pull stunts such as “teaching” wrong things to classmates to get themselves on the right side of the bell curve. NUS has no place for such selfish students! I like your suggestions on leveraging on the IVLE Forum, and I will be alerting my colleagues.
#16: Thanks, Douglas, for sharing the story of Steve Smale. It gives hope for many students!
#17: Dear Patrick, I do not agree that Computer Engineering students are disadvantaged. Computer Engineering (CE) is a multi-disciplinary program, and this requires deep knowledge in both computer science and electrical engineering. There are no two ways about it but to ensure that our CE students are competent in both.
#18: It is true that NUS has expanded (in student enrolment) quite rapidly over the last 10 years. We are at equilibrium right now, and working towards more effective teaching through smaller groups. Our problem is always one of scale. While it is easier to have a program like the USP for 200 students, our challenge is to scale up such experiences 30 fold. The University Town’s Resident Colleges is an attempt to scale up about 10 fold.
#19: I want to assure Herbert that the Department looks at the exam papers both before and after the exam. Even with these checks, it may be difficult to ascertain with certainty the level of difficulty as perceived by students. For instance, a professor may set what seemed to be a difficult paper, but he may have posed similar questions during tutorials or discussions. McGill’s practice is good and warrants a close study. Our students do well in SEP and in Masters or PhD programs overseas (even at the very top universities) – this surely is testimony that the quality of degrees awarded by NUS is high. The stringency in grading contributes to this to some extent. If it were otherwise, top universities would shy away from accepting our NUS students and graduates!
#20: I think Patrick’s view of how a university should be run is overly simplistic.
#21: The difficulty level of a module, as perceived by students, depends on several factors: teaching effectiveness of the professor, materials presented, grading, etc. Posters, oral presentations, interviews are all continuous assessments used more and more by our professors. Frankly, CA can be more demanding than exams! In the next blog, I shall share some details on these.
#22: To NUS Graduate: I might do more disservice should I give everyone a CAP of 5!
#24: Dear Douglas, thanks for your proposal. Your suggestion, despite being crafted in mathematical language, is essentially to allow a high proportion of As for smaller classes. As I have emphasized in my blog article, we are not fixated in enforcing the grade distribution for every module (those with enrolment of more than 30). There can be deviations, even big ones, if professors provide good reasons.
#25: I felt Garrett could have been more diplomatic in putting his points across. Jack in his various responses had articulated the reasons for moderation and the need for differentiation. I do agree that giving back the exam papers is a good thing to do, but that is another of my battles.
#27: Thanks, Zhi Ping, I agree that some general information on general performance (alluded to in Comment #19) is good.
#28: Dear Samuel, I am glad you chose Finance because of your interest, and not because it provides an easier route to a First Class Honours. Many students like Jack, Douglas, Herbert, etc., have commented that exams are really crude instruments for evaluating learning effectiveness. We need to complement exams with other instruments. However, there is no perfect instrument, especially, when we need to apply to thousands of students.
#33: I dislike the analogy that “students are the university’s customers” – this is a totally commercial way of looking at education. In several of my blogs, I have highlighted areas or skills which graduates would need to increase their employability, but students have not shown much interest in these areas. This suggests that students may not always know what they need. A university that panders to the likes of students is not a university.
#35: Dear Jack, we have exclusion criteria for GEM and SS modules. If I am not mistaken, as an example, Math majors cannot take the GEM module Living with Math. For LSM1401, Chemical Engineering, Pharmacy and Chemistry students all have similar A level qualifications, and it is fair to put them all in one group for teaching and for grading. Perhaps a handful of these students may have H3 Biochemistry, but it would not be fair to expect higher standards from these students.
Page 2:
#1: As mentioned, we allow deviations from the grade distribution. Even for large classes, if the professors provide good reasons and convince us that students have learnt, we would be happy to accede to their request.
#2: There is wisdom in what D had written, but sometimes, a person can change the world. Perhaps, some time in the future, an NUS graduate can change the world!
#3: Thanks, Kim, for the nice story about the Economics professor. This is a very effective way to learn the real lesson. Of course, in this story, the tradeoff is painful.
#4: We assure Kim that we normally do not allow graduate students to grade exams. Most graduate students grade only homeworks, lab reports, and in some special situations, tests.
#6: Grading aside, one of the best way to learn is to teach. I am trying to experiment with getting our Honours students to do a practicum – teaching Year I students in the same major. Each Honours student can be a tutor to a small group of 5 to 6 freshmen, and he/she must prepare all the materials and anticipate what these mentees would ask in class.
#7: Your attitude is right! We want to encourage more students like you, to explore the bountiful learning opportunities and to enjoy them. The S/U option is one such mechanism, and we have plans to enhance it. Our current concern with regards to the S/U option is that too many of our students see this option as a mechanism to “remove” their low grades (and hence to increase their CAP).
#10: Let me explain how grading for a large-enrolment module is done. Departments have 3 to 5 days after the exams to send the grades. Take an example of a module enrolling 1,500 students, e.g., a foundational maths module for engineering students, co-taught in 5 sections (of about 300) by 5 professors. The professors come together to set the exam, and prepare a very elaborate marking scheme. The Department assigns markers, and for purposes of fairness, a question is graded by one single person – this ensures consistency. It is a delicate logistics problem because the scripts are passed around the various markers. One missing script can raise hell! The scripts are then checked for unmarked pages and the marks are also carefully checked before they are entered into a mark sheet. The time constraint is to ensure that all students get their results (yes, moderated by the Department, Faculty and University) within 2 weeks after the last day of exams.
Thanks for reading through all of the comments posted prior. I have a few things to say after reading your thoughts on them though (as usual)
#P1:5 I actually took Living With Mathematics as well, as I was interested to find out more about the application of mathematics to daily living (since Mathematics, by definition, will produce relations that can be said to be absolutely true, which is hard, even impossible, for empirical disciplines). I found the module more or less did deliver what it promised, but the thing is that part of it had already been covered in H2 Mathematics. Thus, perhaps students who have taken it might find it of less applicability. That said, the topics on graphs, logic and cryptography were undoubtedly interesting, and fresh.
#P1:9 What was the outcome of the discussion on the evaluation? I do see that opinions of the difficulty of the paper could be skewed, but some types of comments are factual in nature and could thus be relatively objective.
Such as:
1. This paper tests applications of concepts extensively/this paper is mainly an exercise in regurgitation.
2. XYZ MCQ questions are too vague as their phrasing could lead to completely different interpretations by candidates.
3. These questions are identical to questions found in XYZ past year paper.
Also, subjective comments on difficulty could be valuable in themselves. For instance, if a lot of students report that they found the exam easy but for some odd reason their scores prior to grade assignment were low, that could signal a serious conceptual error that must be addressed.
Also, may I ask what would be the practical drawbacks of delaying the evaluation till after the examinations are complete? As this only provides new information, whether reliable or not, there is probably some reason why there would be hesitation in implementing the change.
#P1:13 I have, thankfully, not yet encountered such a situation personally. I hear about it on the grapevine every now and then, but I am unable to objectively verify the claims. After all, the grapevine is the grapevine. However, the necessary rivalry that would allow this to happen does exist. I find it easier to get along with people from completely different majors, in my CCAs, for example, even though I share a lot more things in common with people from my faculty. Not that everyone is necessarily “hostile” per se; “coldness” would be more appropriate a description.
#P1:35 There are differences in the A level subjects we took to take the certificate, though, even though we all get an equivalent level certificate. A lot of the content covered in LSM1401 is really H2 Biology material, and we not only took that subject (that would have been bad enough for balance), but in addition, almost every single one of us aced it, because it is a matriculation requirement for us. I don’t have anything personally to lose from this arrangement, but I don’t find it particularly equitable for them. In reverse, If we were to consider a scenario where someone who has taken H2 Physics but is majoring in Mathematics were to take a Physics related GEM, say, on the applications of Newtonian physics on orbital mechanics, I’m quite willing to bet that I would not have a chance beating said student in that subject, since my Physics training stopped at H1. My coursemates, who largely didn’t take Physics at all, would fare even worse. The number of students with H3 Biochemistry taking LSM1401 is indeed negligibly small, but the number who have taken H2 Biology are very cleanly divided between the majors, and its impact is significant. Also, we took another Pharmacy module coded PR1101 PPDA, which went through some pretty detailed chemistry that was, again, covered in LSM1401 1 module later.
P2:6 Do these honours students cover tutorials for said modules, or are these outside of formal class instruction time? This is something I would very much like to see happening in a more widespread way.
P2:12 Openly engaging the institution you are ostensibly trying to change in a hostile manner is unlikely to produce the desired result of change. Diplomacy isn’t a matter of just being nice. In terms of practical effectiveness, you should attempt to maximise the chance of being understood, or having a change made, if you wish to do those things. If you do not wish to do either, then there are probably more entertaining places on the Internet where you can spend your time.
You do have points on a current lack of transparency that I agree with however.
Still, the current trend is an increase in transparency, which is markedly different from what is the case in the country at large.
If the current NUS administration did not value the feedback of students at all, the 2nd highest ranked member of the administration wouldn’t be wasting time on this site. It is better P.R. to deny communication entirely than to say you are listening and then completely ignore the suggestions given – as we can clearly see from the complete fail of the ‘feedback’ regarding the integrated resorts, and the resultant public furore, particularly among the religious.
The concrete steps already taken by the Provost’s Team regarding earlier posts is testament that they do not operate in a similar fashion as the people involved in that PR fail. (Specifically, adjustments made in the assessment of the Managerial Economics paper after in response to feedback after some admin staff failed to print sufficient examination papers for the candidates in time for the examination.)
Our colleagues have just completed an in-depth study on student feedback, and I could share some of the key ingredients and recommendations with you. As to honours students conducting tutorial classes for freshmen, I am thinking of a practicum module in which honours students earn academic credits for conducting these tutorials. The honours students must have a deep understanding of the foundation knowledge, and must know how to convey them effectively, in order to be able to conduct tutorials well.
This sounds interesting, sir. =P
To distill my earlier points, the problem with the NUS grading system is (1) lack of transparency and arguably (2) too much bureaucracy in the 3 levels of moderation.
And while my earlier points may not have been too “diplomatic”, the points raised are nevertheless relevant, but were unfortunately glossed over. In addition, the fact that the NUS Administration “does not like” the idea that students are paying customers, does not change the fact that they are. As such, they are entitled to certain basic expectations that one would expect from any educational institution, such as transparency in the grading process. In fact, I would argue that NUS practices is contrary to the norm adopted by other “world-class” universities out there. The attitude of the NUS Administration’s “we know what’s best for you” only serves to reinforce the closed-minded and un-transparent manner policies are made, which is rather reminiscent of how things are run in Singapore in general.
Contrast this to overseas universities, where even the syllabus planning committee do have representatives from senior students, to solicit feedback from the student’s point of view. I would clarify that I am not suggesting that the NUS Administration cater to the every whim of the students, but the call for transparency is so basic/fundamental that it would indeed be ludicrous for an educational institution to deny it.
Alas, one must always temper one’s expectation, after all this is NUS.
Topic: Humble perspective on problems surfacing if exam papers are released.
Hi, I am going to graduate soon already, but let me tell from my experience as student in Singapore for over 15 years, what are the potential problems if exam papers are released.
1) From my experience, there exists some “kiasu” students who are rather marks-oriented, who will attempt to “squeeze” out more marks (as much as possible) from the examiner. This is highly subjective, and I wouldn’t say it is wrong, but it does negatively impact the rest of the students who don’t do the same, as comparatively their marks will be lower.
So yes, even though it would be theoretically “fairer” to students if exam were released, the presence of the above type of students totally negate and reverse the “fairness” of releasing the exam.
2) If the exam papers are released to students (individually), this can potentially over-benefit students with “seniors” who have taken the course. The benefit of having a “senior” who has taken the same course under the same lecturer, and passing his material to you, is tremendous, as I can personally attest. Due to the fact that some lecturers “recycle” their midterm/exam/homework questions, having the solution of an exam passed down by a senior is extremely beneficial to a “junior”, to the extent that I can say it is “almost unfair” to someone who has no “senior”.
——
This are just some analysis of the effects of returning exams, from a students point of view, to provide a balanced argument.
What I would think is 100% beneficial, is for lecturers to at least release a partial solution of selected questions, so that we learn how to solve the problems. The solution should be posted online, so that everyone (including future students of the module) is able to access it, so as to not over-benefit “students with seniors”, as mentioned in point 2.
Thank you very much.
Thanks, Douglas. These are some of the issues which we had debated on. The first scenario on the actions of “kiasu” students is very likely, and can complicate our already complex exam processing system. I think we can control the second scenario. I think this is what we can do – have each lecturer highlight what the students have done right or gone wrong in the exam, and have this posted on the IVLE. Further, the lecturer can also post the answers to the exam on the IVLE.
This would be a significant improvement on the transparency (or lack thereof) of the marking process of finals already, and it serves much of the purpose for which many of us want to know the answers in the first place.
In terms of a cost/benefit kind of ratio, this approach is superior to the current system from our (students’) point of view, and doesn’t bring with it as many drawbacks for the lecturers as full result disclosure might bring.
Dear Garrett
You had raised relevant points in your previous comment, including the return of exam papers which does have merit to it, but as I have said, it is another one of my battles as Provost. You may also be pleased to know that we had already started including student representation in the University Committee on Educational Policy two years ago, and a revision to the S/U policy had been made thanks to student feedback to NUSSU.
Nevertheless, my response about the “undiplomatic” nature of your comment refers to the way you generalized about the “bureaucratic, non-transparent and selfish nature of Singapore and NUS” and how this was a “lousy system”. In your latest comment you ended in a despairing, almost derogatory manner – “Alas, one must always temper one’s expectation, after all this is NUS”.
In the first place, this blog post is an attempt to shed light on the grading system, an attempt to increase transparency. It is ironic that while you are demanding transparency, you spurn an effort from us, via this post, to provide this veritable transparency you so desire. The NUS administration also does not operate in a “we know what’s best for you” way; this blog project is in fact a portal through which I can interact directly with students and get direct feedback on what’s best for you!
The perpetuation of transparent systems requires responsibility from both sides – we, as people who run, maintain, and reform the system need to communicate to the students the rationale for certain traits that exist in the system and justify them, but the students, if sincere about introducing meaningful reform, need to be responsible in their use of language as well. Carelessly branding all of Singapore as “selfish” just signifies to the interlocutor that the person speaking is emotional and not in a constructive frame of mind. How can meaningful progress be made in such a situation?
In your comments you spoke positively about many overseas universities – rest assured that we reference the best practices from many universities in the world, and I myself have had the privilege that you also had to experience these systems first-hand. Yet I’m sure you are well aware that even these systems have their own flaws. Top universities around the world share NUS’ problems, for instance, with finding a right balance in grading, to avoid grade deflation or inflation – http://www.nytimes.com/2010/01/31/education/31princeton.html
For me the meaningful challenge is to make NUS a better place for current and future students. I hope you will step away from emotional criticisms of our alma mater and adopt a more optimistic outlook on NUS’ future. Rest assured that the return of exam scripts, among other things, are issues that I have been and will continue to look into.
(Just want to point out first that I’m not currently an NUS student, though I did study there many years ago.)
What do you think about scoring students against the bell curve from the previous year’s results? This would have the effect of reducing competition between students, reducing kiasu-ism.
You would have problems ensuring that tests in both years have the same level of difficulty, but surely it is worth looking into?
For example NYU determines honours based on GPA compared against the previous year’s GPAs.
http://www.nyu.edu/registrar/graduation/latin-honors.html
Effective with the September 2008 graduating class, the GPA cutoffs for each category are determined by the combined GPA distribution from the preceding academic year, all graduation moments included. The cutoff for summa cum laude is the GPA included within the top 5 percent of the previous year’s graduating class.
I know that some universities practise this. If the intake each year is roughly homogeneous in ability, it is possible. NYU, for instance, is a private college and takes in a small group of very good students. Further, as you have pointed out, there must be parity in the level of exams. For a larger university, with many different disciplines, there could be parity issues, and this may shortchange some students.
Hi guys,
At least you can S/U after the results are out.
-Frustrated NTU Student
True enough, the NTU system of S/U before result release just seems pointless to me, giving the lucky (or the skilled bell curve readers) an advantage, without really helping to make it more flexible.
That said, it would be nice if we can emulate the SMU system of additional credit for A+ scored. As of this moment it makes no difference to CAP, but I’m sure you’ll agree with me that a student with 9 A+ and 1 B+ deserves a better appraisal based on results alone than another with 9 A and 1 A-; at the moment there is no academic incentive for students for either NTU or NUS to go for A+.
I agree – the SMU system of giving more credit to A+, is worthy of consideration.
While we are on the topic of grades, I have an honest question which I hope the provost can shed some light on.
I am from the business school and would like to understand why the classification of first class honours is dependent on not just CAP but also a thesis component. Furthermore, why is it necessary to get a A- in order to get a first class honours? As far as I know, NTU and SMU do not have that requirement. I fully understand the rationale between having a thesis or FYP component, to train the students in analytical thinking and research. Unfortunately, I cannot understand the need for an A- grade. Does it mean that if I have a cap well above 4.5 but a B+ for thesis/fyp, i am not considered a first class student? The university should understand that not all students are capable in research work. Given that most students probably would not go into research after their graduation, i find that requirement irrelevant. Perhaps, the university should consider an option for students to decide if they want to go into academia in the future and not tie the degree classification to research work so as to force students to do research.
Given that majority of the graduates will not go into academia and would enter the society to work, I think it is pertinent for the university to understand that most students enter university in search of a degree as a pathway to better employment and not academia. As such, it would be best if the university helps to market its students to potential employers by giving its students better grades and classification.
I believe just last year, only 22 students graduated from NUS business school with a first class honours whereas a greater proportion did so in NTU. I believe this is setting students back in the working world, where employers may not understand the technicalities between the degree classifications in the various university.
Lastly, I do not understand the rationale behind having a bellcurve for level 4000 modules in Business School. In the business school, where there is no direct honours, most students who are weaker would graduate in three years. Therefore, the students who choose to stay for a fourth year are usually the better ones. To impose a bellcurve on them would be to further segregate the students and reduce the numbers who can graduate with a good honours degree (typically classified as at least 2nd upper). I think this is unfair to those who have worked so hard for three years only to be disadvantaged by the bellcurve in their fourth year.
I agree with MJ on reviewing why the classification of first class honours is dependent on not just CAP but also a thesis component.
What many students are worried about is the bad luck of getting an extremely strict professor would eliminiate any chances of getting an A- or above for the FYP thesis.
I am from the science faculty, and have heard “90%-true rumors” from seniors on which professors to try to avoid, because getting an A- or above from those professors is close to impossible.
On the contrary, having a lenient professor would be as lucky as striking a lottery, getting an A for a whooping 12-16MC.
The thesis component is often viewed as the capstone of an honours programme, and that is why good performance in the honours thesis is required for the award of a first class honours. That was the thinking many years ago, and things may have changed. Perhaps it is good for us to review this criterion.
Dear Sir, I think there is a need for NUS to define what is learning. Based on my understanding from the past 18 years of education, learning is the understanding of a subject, which will be used to complement future work or more understanding of relevant subject. You have mentioned before that exam is brunt instrument to measure the effectiveness of learning. I agree to this statement. To my understanding, effectiveness of learning is how much you can apply that understanding you have learned to the subject or work that needed this understanding. So the end result of education, is to ensure that a person understood the subject? Do you agree? In that case, I advocate that bell curve is unnecessary in grading system, because bell curve system do not complement the effectiveness of learning. I have come out with 4 points that show why bell curve is unnecessary.
Firstly, you have mentioned that it gave motivation for student to study a subject, but so does the percentile system. In normal cases, no one will satisfy with a low grade. He/she will work hard towards 100% of learning effectiveness, percentile system is more than enough to motivate student to study hard, and this means that bell curve system is not necessary to provide motivation to the student to study a subject. Based on my experience, what it really does is it forces student to be top in the class, and not increase to his/her effectiveness of learning.
Secondly, you have mentioned that bell curve is needed for honour and CAP system, but I have mentioned that learning is to have understanding in a subject, so as long as everyone retain enough understanding to his or her discipline, so what if everyone got first honour in that discipline? Isn’t that as long as the student has full understanding on that subject, he/she is deserve to get first honour grade? So this again shows that bell curve is unnecessary to be used for honour and CAP system.
Thirdly, you said that there will be biasness in how an exam is set without bell curve system. In the first place, the in-charge of the subject has the responsibility to set the exam paper so that it is able to test his/her students whether they are fully understand that subject. If a certain subject is too easy compare to the other subjects, that subject should not exist in the university scope in the first place. If that subject is marginally easy compare to other subject, then the subject can have less weight compare to other subject, this is reflect to the MC system that we currently used. So biasness in exam should not be the reason why bell curve is used.
The fourth point, you have mentioned that employer will be skeptical in recruiting graduate if all graduates are getting first honour. To me, honour system complements on how to differentiate the effectiveness of how the person understands his/her subject, people who retain the highest level of understanding can get first honour grade. So if everyone is able to retain the highest understanding in his field, what is wrong for everyone to obtain first honour? But in reality, not everyone will have the capability to score the full understanding in his/her subject. Therefore, even without bell curve system, not everyone would obtain first honour in the same time. And in the event that everyone is first honour graduate and is able to show their outstanding capability at work, that will greatly improve the reputation of the university.
I have listed all the points why bell curve is unnecessary, if you do not agree to any of them, please justify them anytime. I am currently studying my last year in engineering. Personally, I am really disappointed that I have to abide the bell curve system; it makes my learning too competitive and makes my learning not enjoyable. In order that my future junior can enjoy learning, I do wish that the university can consider revamping the system. Thanks.
Dear Jensen – thanks for sharing your thoughts. The exam is a blunt instrument to evaluate learning, and the bell curve mitigates some of the disparity introduced by exams. Unfortunately, with or without the bell curve, it is never ideal. You should read the comments in this blog, particularly those of Jack’s and Douglas’, as they compare the strengths and weaknesses of the bell curve.
NUS should have a letter explaining its bell curve, to help NUS students applying for graduate schools in the US/UK.
Hi,
I humbly propose that NUS should have a letter explaining its bell curve, to help NUS students applying for graduate schools in the US/UK.
For example, Princeton has a grading policy letter (http://www.princeton.edu/career/undergrads/resumes/grading-policy-letter/gradingpolicyletter.pdf), explaining:
“In 2004, the Princeton University faculty adopted new institutional grading expectations to address locally the national problem of grade inflation. This policy applies common standards to all academic departments and programs: A-range grades (A+, A, A-) shall comprise no more than 35 percent of grades earned in undergraduate courses and no more than 55 percent of grades for junior and senior independent work. My purpose in writing to you now is to enable you to better evaluate our students’ credentials by placing their academic records in context.”
I humbly suggest that NUS should come up with a short one-page letter, that would help NUS students be more competitive when going for further studies. The latest situation in the US and UK is that universities are short of funds and hence cannot offer as many scholarships as before. Coupled with intense competition for the Asian applicants category from China and India, the Singaporean student heading for graduate studies overseas (especially with tuition waver or teaching assistantships) will find himself in very very stiff competition. Many students from US schools will have close to perfect GPAs, graduating summa cum laude, be in Phi Beta Kappa, or something like that. The NUS student may find it hard to compete, even though his capabilities are comparable to peers overseas.
I hope a Grade Policy Letter will help NUS students be more competitive when applying for overseas graduate schools.
Sincere thanks,
From a concerned NUS student
This is a good suggestion – we could probably put a commentary about the distribution of NUS grades on the transcripts.
“It is not a symbol of rigor to have grades fall into a ‘normal’ distribution; rather, it is a symbol of failure — failure to teach well, failure to test well, and failure to have any influence at all on the intellectual lives of students.”
(Making Sense of College Grades (Jossey-Bass, 1986), Ohmer Milton, Howard Pollio, and James Eison)
Thus, in their important study, Making Sense of College Grades (Jossey-Bass, 1986), Ohmer Milton, Howard Pollio, and James Eison write, “It is not a symbol of rigor to have grades fall into a ‘normal’ distribution; rather, it is a symbol of failure — failure to teach well, failure to test well, and failure to have any influence at all on the intellectual lives of students.” Making sure that students are continually re-sorted, with excellence turned into an artificially scarce commodity, is almost perverse.
http://www.alfiekohn.org/teaching/gi.htm
Same comments as in #21. Do we have a better alternative?
“The most destructive form of grading by far is that which is done “on a curve,” such that the number of top grades is artificially limited: no matter how well all the students do, not all of them can get an A. Apart from the intrinsic unfairness of this arrangement, its practical effect is to teach students that others are potential obstacles to their own success. The kind of collaboration that can help all students to learn more effectively doesn’t stand a chance in such an environment.
http://www.alfiekohn.org/teaching/fdtd-g.htm
However, in a real-world scenario, the chance that all the students can get an A in the first place, while the paper is not set at such an easy level that As are easy to get, is virtually nil.
The fact is that people do have differing capabilities. This is not to say that there will be people who are the best in everything all the time; generally speaking the top people per module change with the module. Whether or not the bell curve is instituted, not everyone would get an A anyway. In order for the counterargument to take hold, it requires acceptance of the premiss that “there will be a chance that everyone in a batch for a module will be so good that they will all deserve to get an A”.
Slim as those chances are, their moderating system would probably be able to detect this kind of situation earlier on from CAP scores and more significantly, CA scores. While an unmodified bell curve system might very rarely have this as a weakness relative to an absolute system, the system here isn’t unmodified, unlike what I previously thought.
However, I do empathise with the sentiment of the second part, that some students will interpret the placement of a bell curve as a reason for them to view others as competitors. CA components may somewhat reduce this tendency if they have to work with others often enough that cooperation has an effect on the end goal, and randomisation of group assignments can prevent this from degenerating into excessive clique formation. However, there is no way to ensure this perception will not be formed.
On the other hand, if an absolute marking system were to be used, one could argue that that might also have a practical effect of teaching students that the only people they can rely on to ensure their own success is themselves, which is somewhat counterproductive to teamwork as well, albeit not as severely as the competition scenario.
Ultimately though it depends on each student to form their own perceptions on why they are studying.
I have been thinking about this – why are universities (and our post-secondary schools) giving us alphabetical grades instead of returning us our examination scripts? How could we then know where we’ve gone right or wrong?
Examinations are supposed to act as assessments of learning. I don’t see how it’s being done here…
Just seeking your views on this, though it may not be related to this topic at all.
P.S. I’m a teacher-to-be so this really bothers me.
Thanks in advance for your time!
Dear Sir,
I chanced upon this blog while looking for an official NUS policy statement about bell curves, and was very pleasantly surprised with the open discussions about many sensitive issues.
Thank you sir, for the effort to reach out to us and being patient with all our comments. I do feel your consistent and steadfast manner of replying to the multitude of constructive and sometimes destructive comments is an inspiring example of how PR should be done over the blogosphere.
Cheers!
I just looking forward to your new works!I have really enjoyed reading your blog posts.I’m actually enjoying the theme/design of your blog.Hope you have another good post!
Hey Dr Tan, I just came across a very interesting article on how a few professors in MIT decided to reform their grading system.
http://web.mit.edu/fnl/volume/204/winston.html
I have one more question about bell curves for Level 4000 modules with a significant number of graduate students.
will they be graded on the same bell curve?
I don’t think it is fair for undergrads to be graded on the same curve as say, masters students or even phd students.
such modules are common in the math department, many level 4000 modules have a significant number of graduate students. the bell curve will be badly skewed for the poor undergrads. from personal experience, an undergrad who is usually the top 25% in the class can easily drop to bottom 40% in a class full of master students.
on the same note, how about exchange students? are they graded on the same curve as the others too?
I agree with Douglas and MJ on their views regarding the classification of first class honours. If the requirement of at least A- is imposed for the thesis, then some students might just focus on their FYP, and neglect their other regular modules. I feel that we can somehow follow the rules and regulations of the Law Faculty, i.e. if students
(a) Finish in the top 5% of their class based on numerical average for all subjects taken at NUS; and
(b) Have obtained a grade of A- or better in at least 40% of the subjects taken at NUS.
Dear Sir,
I would like to know how large a sample size should be for the normal distribution to be considered applicable.
My ME3101 groupmates and I recently found out that a bell curve is applied within each group (consisting of 5 to 7 members). We feel that doing so is, in a way, forcing a curve upon an exceedingly small number of people, and that makes no sense statistically. To use your example, it is tantamount to picking 6 people taller than 1.8 metres and saying that the shortest of the 6 is short (despite being 1.8 metres tall) and that 2 metres (let’s say this is the median height within the group of 6) is the norm.
We currently feel quite wronged and would appreciate any information on how such a grading system is deemed appropriate in this case.
Yours Sincerely,
Timothy Chua Yi-Neng
Dear Sir,
Would it be possible not to reveal the weightage/percentage of the assignments, projects and examinations for a particular module? In my humble opinion, this could allow for a more straightforward in the analysis of consistency throughout the duration of the course.
I graduated from NUS in 2009. I was in engineering and did not do well in my core modules. I am having a mental condition that affected my cognition and focus. In 2 semesters I failed 3 modules, got 4 D’s, 1 D+, 1 C and 1 B-. During my time, the assessment was mainly quizzes, and exams, which were in my opinion difficult as I have difficulties in retaining knowledge and applying them in a test based assessment. I also could not understand the content of the lectures because they were taught in a short span over a few lectures and I have short attention span.
Can there be special consideration for those with learning difficulties, like a special one to one consultation with the lecturers and more assessment time for quizzes and exams?
Thank you.
Thanks a lot .. thats Great
To me CAP system has no meaning except in creating a competition among students and actually not reflecting the students knowledge turning to percentage. Also not revealing the scores in indvidual sub assessment of a module will result in insufficient information with student there by unable to realize what went wrong in modules where student not performed. The university should provide break up marks such as Lab or assigment or theory etc to ascertain weak area so that student can start pay attention to weak are
Just read this upon graduation. Seems very interesting. Was aware that for some modules in NTU they do not really adopt the Bell Curve system, and students scored better, and had better chances (or plausibly) during employment. Upon realization that my faculty only granted 1 first class honours student, and around 10+ second upper honours students among the whole cohort of 150 students, I wonder how the normal distribution of bell curve works here, as I was trying to figure out my percentile in the cohort. 🙂
thank you for sharing knowlegable information with us