What should you be aware of when considering using a Short Duration Online Assessment in your module?
- What to consider first
- Frequently asked questions
- Minimising Risk
- Blackboard / OnDemand Comparison table
- Current Recommended Approach for Deployment of a Blackboard Test
- Case Studies
What to consider first
Is there any other way of assessing your learning outcomes?
Schools that wish to offer one or more on-line traditional exams and/or one or more on-line open book exams in AY 2020/21 are asked to request approval from AQSC. Approval is conditional on the proposing School providing assurance that it has made or assured adequate provision for the support of all students taking the assessment. Therefore, it is worth considering whether your learning outcomes can be assessed by a means other than a Short Duration Online Assessment. It’s worth seeking advice and guidance from CHEP, the Faculty’s Associate Dean for Education, and your Deputy Head of School for Education.
It’s also worth keeping in mind that the University has little experience with Short Duration Online Assessments delivered remotely to students’ own devices; computer-based exams run centrally in the past have focussed on the use of controlled environments such as Workstation computers and Faculty laptops.
In terms of support, while ServiceLine is available to resolve technical queries there are no current plans to provide dedicated and scalable support to “in flight” Short Duration Online Assessments.
Has your request to run a Short Duration Online Assessment been approved by AQSC?
Contact your Deputy Head of School Education to find out more about this process.
Have you considered students in different time zones?
Be mindful of those students outside of the UK time zone when selecting certain times for tests to be available . Allowing the assessment to be taken at any time of day will be more inclusive for those outside of the UK.
Frequently Asked Questions
What is ExamStart and can I use it?
ExamStart is a tool used to secure computer-based, invigilated exams. It only works on University workstations within the University network. It can only be used by students who are physically attending exams on campus.
What is OnDemand and can I use it?
OnDemand is the new name for Questionmark Perception. It is a multiple choice / defined response system that integrates with our ExamStart system.
A limited number of licences for OnDemand are available to the University. To manage this demand those who have already used OnDemand in the past may continue to do so. New authors should only be recommended OnDemand if no other tool can serve the specific requirement needed (such as Review and Record if purchased).
What is the lead in time for additional features like Review and Record?
At least four weeks from a University mandate and the provision of a charge code for purchase.
What are Blackboard tests and can I use them?
Blackboard allows for tests to be built within a Blackboard course. A number of question types are available and tests can be deployed with a variety of options. Tests reside within their own course and can be delivered only to students on that same course. Tests may be copied from one course to another by instructors. Blackboard tests had been recommended only for formative use previously and OnDemand for summative use. As such iSolutions does not have experience supporting the use of Blackboard tests for summative use.
When making a Blackboard test, instructors are completely in control of all aspects of the setup and deployment of the test. Conversely with OnDemand there has been a technical checking process because the scheduling of the assessment is done by iSolutions.
What’s the word limit for text entries?
In Blackboard there are two main question types for receiving text entry.
Essay: Blackboard state that there is no word limit.
Short Answer: Blackboard state that there is no word limit. You can set how many lines the answer box should have, up to 6 lines. This set an expectation of how long the answer is expected to be.
In OnDemand there is no word limit.
Our strong recommendation is to avoid using short answer and essay question types in Blackboard or OnDemand assessments. It is much more efficient to use Blackboard Assignments, eassignment, or Turnitin if you require students to write short answers or essays. There is also a current issue when writing feedback for short answer or essay questions in Blackboard tests: the text editor does not show the buttons for formatting text, so only plain text can be entered when writing feedback for these question types.
Refer to the comparison table below for further information on question types.
What’s possible in the way of automated marking?
In Blackboard and OnDemand tests all question types except for Short Answer, Essay, and File Response are marked automatically. It is much more efficient to use Blackboard Assignments, eassignment or Turnitin if you require students to write short answers, essays or upload files.
Refer to the comparison table below for further information on automatic marking of different question types.
Presentation / set-up
I have a PDF file of an exam paper, can I upload it to Blackboard and it automatically become a test that will be marked automatically?
No. When creating a test using a file import process, only a spreadsheet based file can be used. You may find it more practical to use the PDF exam paper as the basis for a Blackboard assignment, where students could upload a scan of their answer which could then be marked on line using Bb Annotate, which includes pen based annotation. The use of Blackboard tests tends to require a bit of thinking first about which question type is most appropriate, and whether the question should be changed in any way to better suit the potential of using a Blackboard test. Creating computer-based assessments like these tends to be front loaded in terms of effort, with the return on investment coming over time from less work required for marking.
Can tests be divided into sections, with several questions in one section?
Not in Blackboard tests.
In OnDemand you can divide assessments into sections, using question blocks, but students cannot move between those question blocks once a block has been submitted.
Can questions include images?
In Blackboard and OnDemand tests, yes.
Can options include images?
In Blackboard and OnDemand tests, yes.
Can questions include video?
In Blackboard tests, yes. In Blackboard you can upload a video file up to 2500 MB into the question text box, but without using Panopto students may not be able to play that video. However due to the recurrent issues that students report of problems where they have to “click to play” videos hosted on Panopto, it is not recommended to use videos in questions without thorough testing before hand. The use of video increases the internet bandwidth required to complete a test.
Videos are not recommended in OnDemand. Videos are normally hosted externally and we do not recommend embedding external content.
Can I randomise the display of questions?
In Blackboard tests both the questions and the order of answers to questions (where applicable) may be randomised.
Can I build pools of questions, from which random questions can be automatically selected in an assessment?
Blackboard: You can build pools of questions, these are contained within the course in which you are building them. You can then build tests that take for example 5 questions at random from pool A, 10 questions from pool B, and 2 questions from pool C.
The use of question pools is for confident advanced users because diagnosing issues with assessments based on question pools is by its nature more difficult.
OnDemand: Question banks are supported, but normal advice is to use the pool to make a static exam per sitting so that if a question is incorrect none or all of the students are disadvantaged.
Is there support available to get the assessment set up?
Step by step guides are available on elearn.southampton.ac.uk
How do I hide results from students?
Blackboard: Leave the “show test results and feedback to students” options empty in the Test Options. Set the ‘hide from students’ setting for column in Grade Centre and make sure the score is not included in the running total. See https://elearn.southampton.ac.uk/blackboard/assessment/hiding-or-showing-marks-to-students/
Can I import questions in a spreadsheet?
Blackboard: You can create questions in excel, using a specific format specified by Blackboard, save it as a tab separated text file and then upload to Blackboard.
How do I export results and analyse data?
OnDemand: view this guide
How can I prevent cheating?
Typically tests tend to focus on the ability to recall and organise information. Design test questions to focus more on higher order and critical thinking that requires the application, analysis, and evaluation of learning, rather than simple knowledge recall.
Blackboard: with the current functionality and features available to us it is very difficult to prevent cheating.
Measures that would require additional cost and are not currently available at UoS:
Lockdown the browser: A commercial add on is available which will lock down the test taker’s computer. This still does not prevent test takers from photographing the screen during the test, or working with others or accessing resources using another device. The use of lockdown browsers tends to introduce increases in support calls from test takers and tends to restrict the type of device they may use to take the test.
Online proctoring: While there are many commercial online proctoring services there are just as many guides on how to bypass them. The use of online proctoring may also have privacy implications and would not necessarily have been part of what test takers believe they “signed up for”.
Mitigation techniques for Blackboard tests:
1. Use question pools and randomisation.
While building question pools is much more effective in OnDemand, you can build them in Blackboard. Question pools can be used to make it more likely that test takers will receive different questions when taking the test.
When building the questions, you can select for the answers to be displayed in a random order, for example in multiple choice and multiple answer questions.
When deploying the test you can select that the questions are shown in a randomised order.
2. Ensure no feedback is shown once the test is complete.
When setting the deployment options set not to show any feedback to students, such as correct answers or what the possible answers were.
3. Make the test available during a tighter time period.
The less time that the test is available, either through using the availability criteria or by setting a timer, the less time there is for the questions to be shared between test takers. The other side of this coin is that the use of timers or limited availability restrictions can have a negative impact, see the risk minimisation table.
4. Set questions to appear one at time.
If questions are shown all at once it is much simpler for a test taker to take a photo of the whole test and share it quickly onwards.
5. Create a “pre-test” that has to be completed before the real test becomes available.
You could create a “pre-test” with questions such as “I will follow the University’s academic integrity policy” and “I will ensure I have read all the guidance before taking the test” where using adaptive release the real test will not become available until this pre-test has been satisfactorily completed.
6. Use the Blackboard test access log
After a test has been taken you may use the Blackboard test access log to review a detailed list of all the interactions a student had with the test.
How can I protect my questions so I can use them again next year?
You can’t. Students can take a picture on their phone, or write down the questions, or video themselves taking the assessment.
Can questions be screenshotted?
Can I observe/record students while they’re taking the test?
To our knowledge, online proctoring is not available at UoS.
Can I send students messages while they’re doing the test, without them having access to other people/info through the messaging tool?
Has web accessibility testing been performed on Blackboard / OnDemand Tests.
Blackboard: not specifically for tests, but you can read our Blackboard accessibility statement for information about Blackboard accessibility.
Can individual students be allocated longer to complete the test?
Blackboard: Yes, go to https://elearn.southampton.ac.uk/blackboard/deploytest/ and scroll to the “Exceptions” section.
OnDemand: Yes, although all wording and guidance is to not have time limits as the system is designed for invigilators to time students
When Assessments Go Wrong
How can I guarantee it won’t go wrong?
In uncontrolled environments such as where students are completing computer-based assessments outside of an invigilated University computer based exam location there is no way to assure that problems will not occur. However, a number of mitigation strategies are suggested below.
Can students do it offline?
Does it save as they are going along?
Blackboard: In most circumstances yes, although there are Blackboard Knowledge Base issues that state that sometimes this does not happen. We recommend students click the Save Answer button manually as they answer questions.
Will ServiceLine offer dedicated support while the assessment is live?
To our knowledge this is not planned.
In all circumstances it is very beneficial to run a short formative assessment using the same question types and options that would be used in a summative assessment.
This allows students to get used to the interface and process and allows staff to ensure that the results they receive are what they expect.
What to tell students
Further recommendations to make to students are:
- Ensure you are using a computer (not mobile or tablet) on a supported operating system with a supported web browser.
- Ensure your computer is plugged in to a power source.
- Ensure your computer is connected to the internet using a wired (Ethernet) connection.
- Using an incognito window, access Blackboard via https://blackboard.soton.ac.uk direct, not via Sussed. Use only this one tab to access Blackboard. Do not open Blackboard in multiple tabs.
- Select the Save Answer boxes, if available, as you progress through the test.
- When using a mouse, do not double click on any buttons or boxes.
- Do not use the browser “back button”, navigate the test using Blackboard’s own navigation interface.
- When the test has been completed, take a full screen screenshot of the page that states your assignment has been submitted, and ensure the data and time is shown.
Expand to reveal the Scenario-based mitigations table
|What could go wrong?||Blackboard Mitigation|
|Error found in question/answers after assessment made available.||If students have already started the assessment you may not be able to change a question that students have already attempted or if they are currently attempting the assessment.
It may be best to wait for the assessment to have been completed by all students, then delete the question with the issue and regrade the answers.
|Student is disconnected and cannot continue assessment.||Allow at least two attempts on the assessment when deploying the assessment.
Do not select Force Completion. For details of these and other settings view this page.
|A student requires extra time to complete the assessment but has not been given this extra time.||Exceptions can be added when the test is deployed, but if this is reported after a test start time has passed it may be a question of an appeal to go via the School’s standard process.|
|An assessment has been scheduled to be available for the wrong time, or for too little time.||Scheduling can be adjusted when the test is deployed, but if this is reported after a test start time has passed it may be a question of an appeal to go via the School’s standard process.|
|Blackboard becomes unavailable during an assessment.||Allow at least two attempts on the assessment when deploying the assessment.
Do not select Force Completion. For details of these and other settings view this page.
|A student does not have the appropriate equipment or connectivity to complete an assessment.||Part of the approval process for Short Duration Online Assessments is conditional on this risk being mitigated.|
|A student inadvertently changes their answer by scrolling the mouse wheel.||With ordering, matching, and jumbled sentence questions, answers are selected with a drop down menu. When the drop down is selected, using the mouse wheel will cycle through the options within the drop down. It is possible for students to be unaware that a drop down is still selected as they try to scroll down a page using a mouse wheel.
Mitigations for this are:
|A student sees a score or correct answer feedback when they should not.||Leave the “show test results and feedback to students” options empty in the Test Options. Set the ‘hide from students’ setting for the Grade Centre column and make sure the score is not included in the running total.|
|A student was still doing a test when the deadline was reached and did not submit on time, leaving their attempt marked “in progress”.||You can submit a test on behalf of a student if required.|
Blackboard / OnDemand Comparison table
Expand to reveal the table
|Fill in the blank||Yes||Yes|
|Fill in multiple blanks||Yes||Yes|
|Short answer||Yes not recommended, better to use Bb assignment, eassignment or Turnitin . There is a current issue when writing feedback for short answer or essay questions in Blackboard tests: the text editor does not show the buttons for formatting text, so only plain text can be entered when writing feedback for these question types.||Yes, but they are not marked, so better to use an assignment tool|
|Essay||Yes not recommended, it’s better to use Bb assignment, eassignment or Turnitin. There is a current issue when writing feedback for short answer or essay questions in Blackboard tests: the text editor does not show the buttons for formatting text, so only plain text can be entered when writing feedback for these question types.||Yes, but not recommended|
Note: Specify answer should use decimal points rather than significant figures to avoid confusion.
|Either / Or||Yes||Yes|
|File Response||Yes not recommended, it’s better to use Bb assignment, eassignment or Turnitin .||Not Recommended|
|Hot Spot||Yes (there have been issues with this in the past but they appear to have been resolved.)||Yes|
|True / False||Yes||Yes|
|Multiple Answer||Yes but it is recommended to state how many answers should be selected in the question and not allow partial credit.||Yes|
|Fill in the blank||Yes||Yes|
|Fill in multiple blanks||Yes||Yes|
|Either / Or||Yes||Yes|
|True / False||Yes||Yes|
|Can it be divided into sections, with several questions in one section?||No||Yes|
|Can questions include images?||Yes||Yes|
|Can options include images?||Yes||Yes|
|Can questions include video?||Yes?||Not recommended|
|Can random questions be allocated to each student?||Yes||Yes|
|Can questions be imported in a spreadsheet?||Yes, if in specific format||Yes if just text and no multiline questions.|
|Can marks be hidden from students?||Yes||Yes|
|How do I export results and analyse data?||See this guide, and this guide.||See this guide|
|Does it send results to Banner?||No||No|
|Can questions be screenshotted?||Yes||Yes|
|Can I observe / record students while they’re taking the test?||No||Only once the university commits to purchasing Review and Record|
|Can I send students messages while they’re doing the test, without them having access to other people/info through the messaging tool?||No||No|
|Can individual students be allocated longer time to complete the test?||Yes, follow this guide and scroll to the “Exceptions” section.||Yes, although all wording and guidance is to not have time limits as the system is designed for invigilators to time students|
|When it all goes wrong|
|Can students do it offline?||No||No|
|Does it save as they are going along?||Automatically, but recommended that students use the Save Answer button as they go to be certain.||Yes|
|Will ServiceLine offer dedicated support while the assessment is live?||No||No|
Current Recommended Approach for Deployment of a Blackboard Test
These steps assume that the test has already been created but not yet deployed to a content area for students to access.
1. Prepare groups to organise AER students
Identify any students with additional exam requirements (AER) are expected to take the test. There may be some who require more time than others.
Using the Blackboard groups tool, create a manual enrol group for each AER group. Name each group so it will be clear which AER group it refers to, e.g. “AER + 30 mins extra”. Set “Group is visible to students” to No and tick “Create Smart View for this group” add the corresponding students to each group.
2. Deploy the test
Follow the steps in the guide to deploying the test, settings that are particularly relevant for a Short Duration Online Assessment are listed below.
Open test in new window: No – this reduces the possibility of students accidentally logging out of Blackboard in a different window and being locked out of the test
Make available to students: Yes – students will not be able to access the test unless this option is set.
Multiple Attempts: tick and set a least two attempts. If a student is locked out of their test they can start at least one more attempt before requiring technical support or raising an appeal.
Score attempts using: highest mark. If more than one attempt is taken, the mark of the highest attempt is used.
Force Completion: Do not tick. If you tick this box, then once participants have begun the Test, they may not exit and continue working on it at a later date. This means if students lose their internet connection or have a power outage while taking the test they will not be able to complete it.
Set timer: If you do use a timer be aware that students who lose their connection to the quiz (e.g. network or power issue, closing tab accidentally.
Display after/until: tick the boxes and set the dates and times to correspond with dates and times that the assessment should be accessible to students.
Password: to avoid increased support requests, do not set a password.
Exceptions: add exceptions for AER student groups, you can allow more attempts, extra time if using a timer, and allow for extended display until dates.
Due Date: do not set a due date, use the display until settings to set when the test is no longer accessible. We recommend this because if you allow exceptions for AER students they will not affect the due date, and the due date is not a setting which can be added to exceptions.
Include this Test in Grade Centre Score Calculations: Do not tick – this will prevent the score of the test affecting the running total in the Grade Centre, potentially giving students an indication of their score even if results are hidden.
Hide results for this test completely from the instructor and the Grade Centre: Do not tick: ticking this box will mean the results will never be recorded, once students start taking this assessment this option can never be undone.
Show test results and feedback to students: untick everything. In most circumstances for summative assessment the marks should not be shown to students after they have completed the assessment. This can be adjusted later to reveal feedback at a set date if desired…
Test presentation: One at a Time – this keeps the screen space small, and keeps a connection with Blackboard when students move to the next question.
Prohibit Backtracking: Do not tick. If this is ticked students will not be able return to an earlier question.
Randomise Questions: Tick if you wish for the order of questions in the test to be presented randomly. This affects the question order. Note that other areas of randomisation are:
- When building a test question you may specify if possible answers (where applicable) should be presented in a random order.
- When building a test you can state that there should X number of questions chosen at random from a question pool. Multiple pools can be used.
3. Ensure results are not visible to students
Follow the steps in this guide to ensure that the results will not be visible to students.
Case studies will appear here once available.