Before setting up an online summative assessment, it is imperative that you have approval from your Deputy Head of School Education and AQSC to ensure that the risks are being mitigated effectively and adequate provision for the support of all students taking the assessment are in place.
The Centre for Higher Education Practice has produced an online assessment guide which includes alternatives to short duration tests or exams. We strongly recommend that you investigate these options and use these in preference where possible. You can also contact CHEP for guidance about designing Open Book exams.
In uncontrolled environments, where students are completing computer-based assessments outside of an invigilated University computer-based exam location, there is no way to assure that problems will not occur. Students may be adversely affected as a result.
Lockdown browsers and online proctoring are measures that are not currently available at UoS, whether you are running essay, short answer format, or defined-response questions.
Creating robust, online, defined-response tests (such as MCQs) requires front-loading development time. Typically, such tests tend to focus on the ability to recall and organise information. It is better practice to design test questions that focus more on higher-order and critical thinking and that require the application, analysis, and evaluation of learning, rather than simple knowledge recall.
In terms of security, be aware that students can take a picture of questions on their phone, or write down the questions. While random selection can help to ensure students do not see the same questions it will not result in an even distribution of questions. In a defined response test, students may still see the same questions as their peers, even with randomisation selected.
In terms of support, ServiceLine is available to resolve technical queries in developing assessments, but at this time we cannot guarantee dedicated and scalable support to “in flight” Short Duration Online Assessments. Blackboard tests had been recommended only for formative use previously and OnDemand for summative use. Blackboard tests have not been tested for suitability for summative use.
Tools to support short duration online assessments
| Turnitin via Blackboard | Blackboard Tests | Blackboard Assignments | OnDemand | |
| Recommended for | Long essay and typed short essay/text answers | Short answer computer-based tests, bubble sheets, or defined response. | Short answers, uploading handwritten answers & images | Only if you have used it before |
| Features | Plagiarism check | No plagiarism check. Variety of question types inc multiple-choice, fill in the blanks & short answer Randomised questions Math editor | No plagiarism check | Variety of questions types inc multiple-choice, short answer, essay & file response |
| Marking tools available | Rubric Quick marks Inline marking | Automated marking Yes – except for File Response, Essay, and Short Answer. | Direct annotation Inline marking Quick marks Rubric | Automated marking – except for File Response, Essay, and Short Answer. |
| Set up | Multiple assignments need to be created if more than one document is to be uploaded | Time needed to develop robust, potentially single-use, defined-response questions. | Only to be used if you have used this before. Contact ServiceLine for technical checking and scheduling. | |
| How will the assessment be completed? | Students advised to download the question paper, work offline in Office 365 using autosave. Save and upload a Word document.** | Online via Blackboard | Students advised to download the question paper, work offline in Office 365 using autosave. Save and upload a document. For handwritten answers, scan paper to PDF using a scanner or the Office Lens or OneDrive app on a mobile** | Online |
| Risks | Low. Users’connections issues – mitigated if they use O365 with autosave | High* – dependent on users’ internet connection. See Risks and Mitigation section | Low. Users’connections issues. No access/knowledge of tools needed for uploads such as smartphone or scanner | High* – dependent on users’ internet connection. See Risks and Mitigation section |
Risks and possible mitigation
The risks outlined in this section apply to all forms of short duration online assessment. The assessment recipes contain additional, specific risks.
Practice makes perfect
In all circumstances, it is very beneficial to run a short formative assessment using the same methods and options that would be used in the summative assessment. This allows students to get used to the interface and process and allows staff to ensure that the results they receive are what they expect.
Be aware that cloud-based services may change quickly
Blackboard, Turnitin, and OnDemand are all “cloud based services”. We do not maintain and upgrade these services ourselves, instead, they are hosted externally. This means that the services may change from week to week – usually for the better, but sometimes with unintended consequences.
As an example, in November a new text editor was introduced into Blackboard. A bug was found where response feedback for the Grade Questions feature of the Grade Centre was missing the new content editor, and text entered into the box there wouldn’t save. Blackboard will be delivering a fix for this issue in December.
The lesson is, outside of a controlled environment of campus-based exams, there is potential for new issues to be introduced in our cloud-based services.
Have you considered students in different time zones?
Be mindful of those students outside of the UK time zone when selecting certain times for tests to be available. Allowing the assessment to be taken at any time of day will be more inclusive for those outside of the UK.
Do students have the appropriate equipment or connectivity to complete an assessment?
Part of the approval process for Short Duration Online Assessments is conditional on this risk being mitigated.
Errors found in the assessment after it has been made available.
Ensure that the questions, rubric and all settings have been thoroughly proof read and reviewed for errors by yourself and relevant colleagues in good time before the assessment is due to take place.
Where a mistake is found after students have started the test please follow the appropriate AQSC Policy.
The IT service (e.g Blackboard) becomes unavailable during the assessment window.
Create a backup plan; for example, students should send an email to the school student office with their submission attached, and ensure your students and school are aware of the plan you agree.
Guidance for students
- ALL questions about the assessment, including technical queries, should be directed to a named contact in the faculty*, such as an instructor of the course.
- This contact will then answer questions about the assessment and, in the case of technical questions they are unable to answer, they should use the “Raise Serviceline Ticket” link in the course’s Control Panel.
| *this is based on on this section of Section IV: General Information and Regulations of the University Calendar that states that “The nominee(s) of the Associate Dean (Education) must be available during the whole of the examination period in case of queries”, to keep the Short Duration Online Assessment process aligned with this exams process that it replaces, the nominee (usually a course instructor) should be available for student queries during their working hours. |
Assessment Recipes
If you:
- Have obtained the necessary approvals from your DHoSE and AQSC,
- Have taken steps to ensure risks are mitigated (including, but not limited to, the information above and in the assessment recipes),
- Have satisfied yourself that the necessary functionality can be achieved using the tools above.
Then review the following recipes which have been designed to support the following assessment types:
- My assessment is essay based:
- My assessment contains many short answer questions requiring typed answers:
- My assessment uses computer-based tests, bubble sheets, or defined response:
- Â Assessment Recipe: Blackboard Tests for Multiple Choice / Defined Response / Computer-based tests.
- If you already were using OnDemand, contact ServiceLine to discuss continuing using OnDemand.
- My assessment requires handwritten answers; for example maths calculations and diagrams.
- My assessment is made up of multiple different types of the above.
- See the individual assessment recipes.
Checking a short duration online assessment
Certain staff will be responsible for checking and verifying assessment set up.