Andrew Cram1, Lynne Harris1, Corina Raduescu1, Elaine Huber1, Sandris Zeivots1, Andrew Brodzeli1, Sue Wright2 and Amanda White2
1 The University of Sydney, 2 University of Technology Sydney
In recent years, our assessment practices have been disrupted by two significant global issues: firstly, the need to conduct assessment online due to government regulations introduced in response to COVID-19; and more recently, the impact of widespread use of generative AI, which is dramatically changing the nature of assessment designs that can be considered high quality.
While these two events are comprised of different issues, we may gain useful insights into our current assessment design challenges by examining and reflecting on the challenges we faced with assessment design during the emergency online teaching phase of the COVID-19 pandemic.
In our 2022 ASCILITE conference paper (Cram et al., 2022) we reported a snapshot of the use and challenges of online assessment in Australian higher education business schools, using survey data collected from 97 educators and educational developers in 2021-22. The shift to fully online assessment was particularly disruptive for many Australian business schools, as they had traditionally prioritised on-campus delivery to cater for their relatively high intake of international students.
The data clearly highlighted the dominant use of some forms of assessment, with widespread use of written assessments, online exams and quizzes, and live/recorded presentations. There is very little use of other forms of assessment. Each of the other forms of assessment we asked about (i.e. participation, reflection, self/peer assessment, simulations/games, or design/creative works) had been used by fewer than 35% of respondents. Moreover, higher weighted assessments (>30%) were more often to be using exams/quizzes or written assessments, while there was significantly greater diversity in assessment types for assessments with lower weightings and smaller cohorts (<100 students).
Alongside this, we found that over half of the respondents had experience with between 2-4 different forms of online assessment, while another third of respondents had experience with between 5-8 different forms of online assessment.
The challenges faced by respondents in designing and delivering online assessment were numerous and varied. We identified thirteen distinct categories of challenges for educators, covering themes of ensuring academic integrity, changes to grading and feedback, technology, logistics and timing, student perspectives and support, institutional policy and support, and additional time and effort required.
Our findings support the notions that assessment design and delivery are multi-faceted and quite involved processes for educators, and that assessment change and innovation are gradual and more likely to occur within lower weighted assessments with smaller cohorts. This highlights some clear tensions for the higher education sector relating to the recalibration of our assessment designs and practices to cope with the emergence of generative AI.
To assist educators in the efficient (re)design of high-quality assessments, and as part of a broader research project, we have developed a framework for the design and evaluation of quality online assessments (Huber et al., 2023).
Figure 1: Framework to design and evaluate quality online assessments
This framework is intended to not only assist academics/stakeholders evaluate and design quality online assessments, but also to facilitate conversations and a deeper understanding of the trade-offs between different considerations and the relations with contextual factors such as policy, accreditation, resourcing and scale. More information about the framework can be found at https://bizonlineassessment.com.
What’s next for our research?
As is the case with most research projects, there is always more work to be done. We are currently investigating the impact of AI on our framework through surveys and focus groups with educators and students across three countries. In addition, we are further validating our framework with a range of other stakeholders including industry partners and accrediting bodies. Once we incorporate their feedback, we will be conducting another national survey to finalise the dimensions and contextual factors of our framework. Watch this space (and your email inbox)!
Acknowledgement
This research was commissioned by the Australian Business Deans Council (ABDC).
References
Cram, A., Harris, L., Raduescu, C., Huber, E., Zeivots, S., Brodzeli, A., Wright, S., & White, A. (2022). Online Assessment in Australian University Business Schools: A Snapshot of Usage and Challenges. In S. Wilson, N. Arthars, D. Wardak, P. Yeoman, E. Kalman & D. Y. T. Liu (Eds.), Reconnecting relationships through technology. Proceedings of the 39th International Conference on Innovation, Practice and Research in the Use of Educational Technologies in Tertiary Education, ASCILITE 2022 in Sydney, https://doi.org/10.14742/apubs.2022.181
Huber, E., Harris, L., Wright, S., White, A., Raduescu, C., Zeivots, S., Cram, A., & Brodzeli, A. (2023). Towards a framework for designing and evaluating online assessments in business education. Assessment & Evaluation in Higher Education, 1–15. https://doi.org/10.1080/02602938.2023.2183487
Frameworks are of value, but only as a supplement to the formal training, and qualification of teaching staff. The shift to online assessment during the recent SARS-CoV-2 pandemic was most disruptive for institutions which had no contingency plan for this known risk. In contrast, some Singapore educational institutions, notably NTU, prepared for e-learning after their their SARS-CoV-1 experience in the early 2000s. Will history be repeated, with universities reverting to outdated education techniques, delivered unqualified staff, until SARS-CoV-3, or some other emergency, again keeps students from campus?