Monthly Archives: June 2013

Online Assessments

5 Replies

Technology in Pedagogy, No. 17, May 2013
Written by Kiruthika Ragupathi

Online Learning and Educational Apps seem to be the new buzzwords in education. The advent of MOOCs, educational applications (apps) and online lectures delivered via iTunesU, Coursera and TED, look set to bring about a paradigm shift in modern pedagogy. Yet, it is always important to be mindful of the educational principles that underpin good (and sound) pedagogy says Erle Lim, an Associate Professor at the Department of medicine, Yong Loo Lin School of Medicine from the National University of Singapore. As educators, it is important to ask, “Are we engaging our students?”, and more importantly, “Are we teaching students to curate knowledge rather than just acquire lots of meaningless facts?”

Assessments and high-stakes examinations are therefore important to determine if students are learning (and applying what they learn). Despite the healthy skepticism about these new-fangled ideas, we need to ask ourselves if we should embrace technology and better utilize smart devices and online tools to fully engage our students and test their ability to apply what they have learned, rather than just regurgitate “rote” knowledge. In this session, A/P Lim discussed the potential for online assessments – how to use them, and when not to.

A/P Lim started the session with a brief introduction to online assessments, and then highlighted the benefits and problems associated with using online assessments.

 Assessment + computer + network= Online assessment.

Benefits of online assessments

  • Instant and detailed feedback – how students perform, how the top-end students in relation of bottom-end students
  • Flexibility of location and time – log on and take on the exam at any time – important to differentiate to formative and summative assessments.
  • Multimedia – makes it more lively when these multimedia objects are incorporated
  • Enables Interactivity – blogs, forums
  • Academic dishonesty – essay questions can be automatically submitted to platforms like  Turnitin, iThenticate to check for plagiarism
  • Lower long-term costs
  • Instant feedback/instant marking
  • Reliability (machine vs. human marking) – scoring is impartial
  • Impartiality (machine vs. human)
  • Greater storage efficiency – digital versus hard-copy exam scripts
  • Able to distribute multiple versions of the exam 
  • Evaluate individual vs. group performance – how has one individual scored vs. the cohort
  • The report generating capability allows to identify learning problem areas.
  • Allows to mix and match question styles in the exams

Disadvantages of online assessments

  • Online assessments can be expensive to establish
  • They are not suitable for all assessment types
  • Cool is not necessarily good. Just because something is new and easily available may not be the best. Sometimes established old things are better.
  • There is potential for academic dishonesty and plagiarism, even with Turnitin, it is possible to tweak the answer to be not detected.
  • The online assessments gives only the “right” and “wrong” answers, and not necessarily on not necessarily on how students arrived at the answers.
  • Potential for glitches, and therefore every problem has to be envisaged

The Modified Essay Question – an evolving scenario

The questions in a written examination can be constructed in different ways, e.g. short answer questions (SAQ) or essay questions. However, the use of short answer questions (SAQ) AND essays for online assessments make it difficult to mark online. Therefore the essay questions were modified to a “Modified Essay Question (MEQ)” which replicates the clinical encounter and assesses clinical problem- solving skills. The clinical case is presented in a chronological sequence of items in an evolving case scenario. After each item a decision is required, and the student is not allowed to preview the subsequent item until the decision has been made. The MEQs test higher order cognitive skills, problem-solving and reasoning ability, rather than factual recall and rote learning, and is generally context-dependent.

How useful is the MEQ

  • Measures all levels of Buckwalter’s cognitive abilities: recall or recognition of isolated information, data interpretation, and problem solving;
  • Measures all of Bloom’s 5 levels of Cognitive Processing: Knowledge, Comprehension, Analysis, Synthesis, and Evaluation;
  • Construct and content validity;
  • Dependable reliability coefficients;
  • Correlate well with subsequent clinical performance
  • Allows students to think completely in a new way with firm pedagogical underpinnings

Challenges and limitations of using MEQs

  • Recall of knowledge and the questions
  • Structurally flawed compared with MCQs.
  • MEQ re-marking: lower scores than were awarded by the original, discipline-based expert markers.
  • Failed to achieve its primary purpose of assessing higher cognitive skills.

Points to consider when planning a good test/examination

  • Valid: The test measures what it is supposed to measure
  • Reliable:  (a) At any given time, the same student should be able to score the same mark, even if he/she had taken the test at a different time (b) Score what you are supposed to score
  • Objective: Different markers should mark the same script with the same standard, and award the same mark
  • Comprehensive: tests what one needs to know
  • Simple and fair: (a) language clear, unambiguous questions (b) Tests appropriate level of knowledge
  • Scoreable: Mark distribution fair

How to set Good MEQs?

  • Understand the objectives of the assessment and be familiar with curriculum materials that relate to learning outcomes.
  • Determine expected standards: what do you expect the candidate to know? It is important that there is a clear alignment between what students have learned and what they are being tested on. Always test what is taught to them, not to test beyond students’ level of understanding.
  • It is also a good idea to involve peers when setting the MEQs and getting colleagues to try the questions out. This will also enable you to determine if the timing allotted is adequate and will also allow you to assess the adequacy of mark distribution. Get comments and criticisms from your peers.
  • Do not set questions in silo. The formation of MEQ committees will be advisable, and ensure a good distribution of specialists in the committee (e.g., paediatrics and adult, subspecialty groups).
  • Provide sufficient realistic clinical and contextual information, thereby creating authenticity in the cases.
  • The components of the online assessment in order to increase discriminant value of examination.
  • The design of the assessment should be contextual, sequential, with enough time to think. Due to the use of sequentially expanding data, students should not be allowed to return to their previous responses in order to change answers.

How to set FAIR MEQs

  • Good quality images
  • Data for interpretation:
  • Information must be fairly presented: don’t overwhelm the candidates
  • Choose relevant/reasonable tests: no esoteric tests (if possible), don’t give unnecessary data to interpret
  • If tests essential to the MEQ but students not expected to know how to interpret: can use to teach – i.e. give them the report, but leave to them to interpret eg CT brain image showing ICH
  • Keep to the curriculum

Q & A Session

Following the presentation by A/P Erle Lim, a lively discussion ensued and listed below are some questions from the subsequent Q & A session.

Q:  Why do you find essay questions difficult to mark online? I have a very opposite experience. Maybe your system is different. If your question is sufficiently clear, students will be able to cope and I don’t find it difficult to mark.
EL: There are advantages and disadvantages. One is you don’t have to read bad handwriting.
Q:  You talked doing the assessment anytime and anywhere. How do we know if they are doing it in groups or doing it by themselves?
EL: I am sure there are some settings to be able to control the assessment.
Q:  How do you go about the bell curve?
EL: We do get a decent bell curve. It is not necessarily even.  We accept the marks as they are and for the School of Medicine, we do not do a bell curve.  Every individual exam is not tweaked, it is only done at an overall level.

Social Media in Education

5 Replies

Technology in Pedagogy, No. 16, April 2013
Written by Charina Ong (cdtclo@nus.edu.sg) based on the presentation notes of John Larkin

What is Social Media Anyway?

John Larkin started the session by asking how “connected” are you as a teacher? John Seely Brown posits, “The Internet is not simply providing information, but access to people”. A common definition of social media may thus be: “a blending of technology and social interaction for the co-creation of value.”

In the session, John Larkin showcased several social media exemplars as used in NUS and various Singapore organizations. He then presented various social media tools and shared his experiences on how he used Blogs to teach History.

Why is Social Media Important?

Social media provides rich learning opportunities to foster students’ digital skills and talents. Employers value employees with technology skills that can be readily utilized to create value, and be able to contribute and communicate well. They are not looking for exam results anymore. The challenge for educators therefore, is to be able to teach students on using social media effectively:  to benefit the community and the society; to deepen their skillset and widen their experience; how to connect with their peers and more importantly be able to develop a skillset that employers are looking for.

How is Social Media Being Used?

There are various social media tools available, the list is endless. As educators, we need to foster developing mature competencies within the students so that they can use these tools to achieve meaningful outcomes.

John showcased examples from various disciplines on how social media is used in education. For example, faculty from Business schools and Language Communication often use Facebook and Twitter to facilitate communication and collaboration. Others use Flickr to exhibit their work. Blogs are used for writing collaboration, design, and sometimes even for programming. Line allows student teams to quickly collaborate during field trips exchanging images, video, audio media messages and make free voice calls. Linkedin is another great and sophisticated tool to connect to people related to one’s own field, and as John put it, it is the ‘thinking person’s Facebook’. Google plus is a multilingual social networking site, similar to Facebook, and lastly, Evernote, an excellent note taking tool that can be used to collaborate with peers.

With this great variety of tools readily available, educators have to be mindful of what they hope to achieve when using a particular tool. In their article on Social software for learning: what is it, why use it? Leslie and Landon write “The adoption of social software is not synonymous with the effective delivery and assessment of quality teaching and learning.” It is therefore essential to balance and plan ahead while carefully considering why and how to use the tools appropriately.

The following below are some examples on how social media is used effectively: 

NUS Exemplar

Sivasothi, a lecturer from Department of Biological Sciences, has been using blogs to communicate with his students and the general public for the past ten years and facilitates and encourages connection and communication between his students and the public.

Originally students submitted their assignments, research, tasks, etc to the lecturer allowing only the teacher and student to read. To maximise the impact and to create greater student ownership, Siva asks students to write about their research and use social media to communicate their findings, publish the result of their field trips, and photographs in Biodiversity class blog.

Students take ownership over their blog entries enabling them to develop a sense of responsibility over the information they upload as it is not only for the eyes of the instructor but their peers and external community. These entries also allow them to hone their communication skills when they express their thoughts, skills, knowledge, and attitude from a framework that the public can understand. This also led Siva into organizing guided talks around the mangroves (Kent Ridge, Pasir Ris, etc.) with his students. This gave his students the opportunity to discuss on their research with the general and no longer merely articulating their work in terms of writing a dissertation, an assignment, or task that only the lecturer would understand. They are now articulating their research, findings, knowledge, and passion in a way that the general public would understand. This is a good example of using social media at its best with the student establishing connection not only with the teacher but also their peers and the general public. This also enhances the employability skills that are necessary in their future workplace.

John Larkin’s Experience

John shared prime resource materials in his history class using WordPress on how he uses social media and the way he encourages his students to use these social media tools to collaborate and communicate. “I choose an area that I’m passionate about and have knowledge of and start publishing about it using WordPress as a blogging platform. My knowledge and passion eliminate a layer of stress and the students utilize these resources.

I get my secondary 2 students to write their stories, imagining they live in the past and I ask them to publish it online. These students vary in skills- I have students with severe learning disability, they mix with other students who are capable. I tell them to take a moment from history and write a story. They were using WordPress as a publishing tool to express their views about Black day. Students don’t just skim the surface but they are thinking about the subject matter, writing and publishing. Their peers, as well as their parents and the community see this. They think deep and reflect on how they can utilize this fully.” says John.

How Do I Get Started: Next Steps

John Larkin offered these tips to get you started with your social media journey:

  • Take small steps and choose part of your curriculum particularly you’re passionate about;
  • Select the tools equipped for you, the one you can use most effectively;
  • Work with small cohorts (30-40 Post graduate students);
  • Collaborate with your colleagues; and
  • Delegate to the students.

Q&A Session

Following the presentation by John Larkin, a lively discussion ensued and listed below are some questions from the subsequent Q & A session.

Q: A month ago, a person on social media challenged the court decision that the judge had made. The person was prosecuted by the court because he is not allowed to express his opinion and challenge the court. Social media in my perspective, in order to change the world, you need to have freedom to express your views and not be punished. What is your opinion about this?

JL: I don’t have a direct answer to your question. People in Australia have more freedom to express their views as compared to Singapore which is completely a different environment. In classroom context, I teach students how to use social media responsibly. I tell my students to think deeply about the things they write and publish. We have to teach our students the right way of using social media. We need to teach them to think deep and think about the tools that they want to use.

Q: You mentioned that your students publish their own stories online. Have you ever thought of getting some historians to come in to critique or provide feedback to the students?

JL: Yes, at the JC1 level, history class that I’m teaching, students have communicated with a group of Archeologist blogging about Pompeii. Students get to interact with them, getting comments, etc. I believe that bringing experts into the classroom is important.

Q: Social media is public and people can follow you. If student posts something, it may affect people’s view about the way we educate our students. How can we manage this and how can educators be exemplars?

JL: You can apply demerit points if students go off track. Students in general are quite responsible. I was first reluctant to have my students to publish their work, but I realized that they were as good as me. We often forget that we need to give them some responsibilities as part of their learning. What I will suggest is that whatever you choose to set up for your class – make it a closed group discussion. For example, Facebook, consider using a page or group, and then only open it to the immediate group.