This blog post aims to share the discussions and reflections of the webinar presented by Dr. Shannon Johnston (refer to Webinar about Evaluating Design-Your-Own-Module (DYOM)). In her discussion on how to evaluate a DYOM, she shares her experience of redesigning a third year Indonesian language curriculum as a case study.
Framework
The following considerations can be used to guide evaluation of DYOM.
Firstly, one should identify the Purpose(s) of the module or curriculum. Examples of a purpose could look like ‘to prepare graduates for futures with the language’ and ‘to be interesting, different, and to motivate’.
Secondly, Data has to be collected. Data could come in many forms: from observation of student behaviour to collected feedback.
Thirdly, Evaluation, judged based on whether the Purpose(s) are achieved, and what the Data suggests.
On Analysing Data
Questions when faced with collected data are as follows:
- What might the data tell us?
- Whose story does the data tell? (i.e., who is included and who is left out)
Taking a step back, it is also important to examine not only the data, but what data was collected, why that data, whose voice is privileged by the data, and what other data should be collected.
On Collecting Evaluative Feedback
There is not one way to collect feedback and nothing dictates that feedback can only be collated through one method. For example, students can take part in a reflection session in which they discuss the role they play or what they learnt. Leading questions can be prepared.
Other ways of collecting feedback include:
- student daily blogs
- student final assessment item
- student focus group interviews
On Design Thinking Methodology
The Design Thinking methodology can be used in evaluating Design-Your-Own-Module. For example, referring to Stanford’s Design Thinking, the following 5 main steps of (1) Empathise, (2) Define, (3) Ideate, (4) Prototype, and (5) Test can be used. The progress may not be linear and one may find oneself going back to the drawing board or a few steps back in order to come up with a better design. For instance, the prototype may not work in a way that one expected or the ideation process would have to be used again in order to find a better solution.
Design Thinking also emphasises 3 fundamental values – empathise, observe and identify. With reference to the field of teaching and learning, staff should be student-centred. Questions like ‘what was it like for the student?’ and ‘what was the students’ experience?’ are crucial.
Identifying latent needs is another important lesson from Design Thinking. This links to a previously discussed goal of DYOM, which is to tap on the intrinsic motivation of students (refer to Webinar about DYOM via Group Work with Supervision). This gets staff to probe deeper into students’ core motivators outside of ‘completing the module’ or ‘getting a good grade for the module’.
Parting Questions for Evaluating DYOM
- What worked well?
- What did not work so well?
- What did we not do (facilitators or educators) that we should have done?
- What will we do now?