Reviewing courses is an essential part of preparing for future runs and measuring impact on the effort put in so far. We recognise how much energy it takes to plan, design, build and run a course, so doing even more can be daunting. To stop and take the time to reflect can easily be overlooked, especially if other pressures are demanding attention. But there is a lot of value in reviewing, evaluating, measuring – and these pages should help guide you through this part of the process.
Review is an essential part of the overall course development process. It helps internalise our actions and inform our next moves by measuring what worked well, what needs to be improved and leads towards fulfilling reporting requirements for stakeholders. It provides a chance to gain insight and create measures for comparison by summarising numerous complex interactions.
Measuring impact is a motivator for review; project and initiatives will benefit, or be required, to report back on aspects such as outcomes, reach, demographics or demonstrable change from the resources invested.
Your reason to review will vary as a variety of stakeholders, aims and intentions will modify the type of review performed. It is recommended to define motivations or objectives from the outset as these can then be measured during a review phase. Ideally every course / partner will know what it’s being measured against from before it’s designed. However these can also often be omitted from the start, or priorities change (or are introduced) during the life of a course, or the partnership. This does have an impact but it should not prevent a review to take place – or prohibit review altogether!
It can be daunting to start a review – what do you focus on? What can be left out? No two reviews may look alike due to the variety of reasons, diversity of stakeholders or plethora of ‘stuff’ that goes into each course – few are ever really that alike, even from the same educator.
We have compiled a list of areas that are common place in reviews, or that we know from experience would fit well into a review or evaluation.
Who are you consulting / reviewing or who are you reviewing you? You may fit into one, or more, of these stakeholder categories. The intention is to consider their needs and ensure your evaluation is suitable for their requirements.
- Learners (on-campus, widening participation, lifelong and more)
- Project Lead
- Mentors / Hosts
- Learning Technologists / Distance Education specialists
- Instructional Designers
- Curriculum Enhancement
- Management and Senior Leadership
- External Partners
A stakeholder needs assessment drives the questions which can help guide a review. As you identify stakeholders you’ll want to ask:
- What are their objectives for this activity?
- What information do they need to know?
This is to ensure the review caters for stakeholder’s needs and provides the reviewer a focus / objective which must become a constituent of the review.
Setting objectives will assist with your review. One approach would be to set SMART Objectives:
- S – specific, significant, stretching
- M – measurable, meaningful, motivational
- A – agreed upon, attainable, achievable, acceptable, action-oriented
- R – realistic, relevant, reasonable, rewarding, results-oriented
- T – time-based, time-bound, timely, tangible, trackable
Once objectives have been set, you’ll want to consider which areas will be evaluated and to what depth.
Below is a set of measures that could be worth considering in a review or evaluation. In brackets there are some example sources for data or evidence which may be collected for analysis. You will want to consider how these measures fit into the wider evaluation and their overall relevance (i.e. mitigating misleading headliners and instead linking data sources to build a robust claim to achieving a set of objectives).
- Retention and learner participation (active learners, learner course usage data, comments, completion, demographics)
- Openness and widening participation (demographics, learner’s profile data, open step content / step openness)
- Income and cost (course development and running costs, academic and facilitator time, statement/certificate or upgrade revenue, private courses)
- Marketing and brand development (marketing analytics, campaigns, adverts, editorials)
- Learner satisfaction (course surveys, comments, social media, feedback)
- Student recruitment, credit-bearing courses (lead generation, student cohorts, marketing campaigns)
- Pedagogical innovation (new approaches to teaching, integrating practice back to other teaching activities)
- Research dissemination (demographics, global reach, page views, open steps)
While evaluating lots of courses Reka Budai, Strategy Analyst at FutureLearn, has found eight things to consider during course revision:
- Think beyond English – while most courses are in English we know it’s not the first language for many of our learners.
- Keep it short and simple – readability makes a large difference, consider the readability of your content and your target audience.
- Practical examples from around the world – you’ve got a global community and learners appreciate a breadth of examples and case studies.
- Schedule with your learners, not against – there are good and bad times to schedule a course throughout the year, if you’re unsure – ask us. You can also check the course schedule.
- Fit around your learner’s regular lives – learners will have competing demands for their time and attention, consider learning time and course duration.
- Know your learners – you want to attract the right audience so be sure to include prerequisites and make the course outcomes clear and suitable.
- Challenge and teach – learners taking quizzes should receive positive and encouraging feedback to support their learning, not just be tested.
- Interact, engage and have fun – courses are not about content but about a connected community of learners – make your course fun!
For more information on reviewing your course read our recommended review areas.