By Brent Gregory, Lecturer, University of New England Business School

As our higher education environment becomes increasingly competitive, especially from non-traditional players, we face pressure to both improve the student experience and do this for a lower utilisation of resources. Scalable Personalisation is a way of achieving these seemingly conflicting outcomes because it is where we tailor the learning journey to better align with student needs and we do this in a way that is highly sensitive to effective resource use. My journey to scalable personalisation has been a long and challenging road littered with disappointments, detours, dead ends and unfulfilled premises.  The implementation of On-Demand exams has become a major enabler to support scalable personalisation.

This is a complex story, not only exploring challenges and benefits of On-Demand Assessment while acknowledging the thinking and support structures that are necessary for this model to survive, but also understanding the need for the holistic thinking.  In considering the role that on-demand exams could play for you, I would like to direct your attention to three thoughts:

1. Capturing student share of mind is a crucial task for academics
2. On-demand assessment is integrated into the student’s learning environment and is not an add on
3. What do you need to know before you embark on the on-demand exams journey

Firstly some definitions.  On-Demand assessment enables students to undertake an assessment task at a time of their choosing during the study period, provided they have completed all gateway tasks. An On-Demand exam is typically the most significant task in the on-demand assessment package.  The Progression Pipeline is the package of scaffolded activities that students must pass through to complete all on-demand assessment activities.  Figure 3 provides an example.

Capturing student share of mind is a crucial task for academics

In order to support students to acquire a particular skill, we may set them an activity to enable them to achieve our desired outcome (i.e., acquire the desired skill). Unfortunately, developing brilliant material does not ensure that students will undertake what you set for them. As humans, we can only process a miniscule proportion of the information that is presented to us and so have a number of inbuilt filters that Delete, Distort and Generalise (James & Woodsmall, 1988) about the outside world so that we can sufficiently simplify this and can internally represent the outside world inside our heads.    In Figure 1, the HELP model is built around four questions that need to be addressed to increase the chances of students undertaking the activities that you set for them. HELP identifies ways in which the message can navigate the inbuilt filters that we all possess. The four questions are:

• What will Hinder the student from starting the activity (e.g., the instructions to start are not sufficiently clear)
• What will Encourage them to start the activity (e.g., student received an email reminding them to start the activity)
• What will cause Leakage from progress on the activity (e.g., outside responsibilities are absorbing time)
• What will Promote progress on the activity. Promoting student progress is where on-demand assessment comes into play.

A useful model for considering how you may support students to progress through your material is the concept of Flow. According to Csikszentmihalyi (1991). This is the state in which people exhibit the highest states of intrinsic motivation and he identified a set of characteristics that needed to be present to reach this state of flow. This included, having a challenging, but achievable task that was defined by clear goals and supported by frequent immediate feedback to confirm the progress toward the goal; having the ability to concentrate on the task; and having the paradox of control.

It is in the Paradox of control that on-demand exams play such a major role.  A students’ ability to exert control on when they their final exam provides, for many students, a much greater sense of control.  When you build an environment with some of the other elements of flow (clear goals, feedback) you are giving students the opportunity to become more engaged in the whole learning process.  It is likely you will consider the design characteristics of your learning environment and it is in the design of your learning environment where on-demand assessment can be such a valuable ingredient.

On-demand assessment is integrated into your learning environment, it is not an add on

To maximise the benefit of on-demand assessment, we pay particular attention to developing well signposted paths through scaffolded activities supported by progress reports. Gateway tasks are set to verify that each student has understood the content, before they are permitted to progress to the new materials.  If they don’t have sufficient knowledge they need to cycle back to gain that knowledge.  The map in Figure 2 is a visual representation of our approach. For example the journey to master the accounting equation starts with the basic accounting equation and then the expanded accounting equation, followed by the accounting elements. Students then need to successfully negotiate a gateway task before they move into the area of understanding Debits and Credits.  That will involve definitions, the normal balance and some historical context. The students have another gateway task before they cross the bridge to explore the relationship between the accounting equation and the financial statements.  They continue this journey, with the associated gateway tasks for the complete island and then they are tested to ensure they have sufficient skills to graduate to the Accounting Cycle arena.

The Assessment Flow in Figure 3 provides the opportunity to answer some of the questions most often asked about on-demand assessment. When you look at the first block, ‘Fundamentals’ you will notice that there are two tutorial quizzes followed by a gateway activity (GW1) and the Assessable Quiz (AQ1).  AQ1 is purely a summative task and students are only provided with a mark.  This quiz is also randomly generated so that no two students will get the same quiz.  However, every quiz tests the same learning objectives and has the same structure.  The gateway activities are formative tasks with have two major roles.  They are a major source of feedback to the student on their progress and attention areas.  This is necessary because we do not provide feedback for assessable quizzes. All feedback is automated.  The gateway tasks stop students from progressing until they have gained at least the base level of knowledge.

The assessment flow depicted in Figure 3 is part of our progression pipeline. The progression pipeline enables us to determine how progressed each student is on any one day and also provides us with a summary of progress for all students.  It is exceedingly helpful in identifying areas where students are having problems

What do you need to know before you embark on the on-demand exams journey?

My advice is not to jump straight into it.  There are significant advantages in starting small to develop a basic understanding of the process in a low stakes environment.  Starting with one low value assessment task is a safe starting point and will provide you with significant learning, in particular, it will give you a strong sense of what is involved.

On-Demand assessment implies automation. It is also likely that there will be a component of invigilation somewhere in the assessment chain. This will require attention to student privacy concerns.

Ultimately, on-demand assessment is likely to require a significant capital investment, and this will result in a very low marginal cost.  When setting an on-demand exam, a significant amount of planning is required and this needs to be supported by a large and well-structured question databank. You will require mechanisms to ensure equity in exams delivered and mechanisms to cope with inequities.  Fortunately, online exams produce sufficient data to enable efficient procedures to be developed.

It is likely that a majority of students will respond positively to this and experience so far is that the bulk of students are likely to travel at a roughly similar space. In particular there will be a cohort of students that require deadlines and require support to keep up with the deadlines.

As mentioned earlier On-Demand exams is just a part of your learning environment and the learning environment is just a part of your overall approach to supporting student learning. To combine an improved experience that is delivered in a scalable way, it is necessary to have a holistic view. The business world provides ways of address such situations.  My larger framework for developing the learning environment has been the RENT framework and RENT is an acronym for four key areas we need to address – Recording, learning Environment, Needs and Team.

The thought of using business principles in education tends to make many academic cringe. However, Scalable Personalisation is coming in a massive way. My sense is that this is something we need to cooperate on so that Scalable Personalisation is shaped for community not for corporations.  It is something that we will do best when working together.

Brent will be presenting this work in the upcoming BE-SIG webinar on Tuesday 29 September at 10:30am AEST. Please join us to hear about this ASCILITE award winning initiative.  Details >>

References

Csikszentmihalyi, M. (1991). Flow: the psychology of optimal experience. New York: Harper Collins.

James, T., & Woodsmall, W. (1988). Timeline therapy and the basis of personality. Capitola, CA: Meta Publications.