Whatever stage we’re at on our life’s journey, it’s likely we remember the last time we took an exam. Almost one fifth of current University of Southampton (UoS) students take their exams on a computer. The University of Southampton is leading the way in the uptake of computer-based examinations (CBE).
At the heart of this innovation is the Managed Learning Environment (MLE) Team who have been developing and supporting the computer-based exams experience at UoS since the early 2000s. During the Semester 1 exam period, University of Southampton President and Vice-Chancellor Professor Mark E. Smith visited with the MLE team to find out about their vital work in supporting several key pillars of the University’s strategy.
Computer-based exams (CBE)
With almost two decades’ experience of running computer-based exams, as well as developing the related software and infrastructure, the MLE team have made the current exam experience as smooth and pain-free for students as possible. The MLE team took the Vice-Chancellor through the CBE process.
Starting the exam
Any University public workstation may be used to take an exam. The exams office block-book workstation rooms and arrange for invigilators to be present in each room. The invigilators, exams office, and the MLE team communicate using two-way radios. Once the student sitting the exam enters their ID number, their student card and id photo appears, allowing the invigilator to ensure the student sitting the exam is the same person who appears on their ID card.
For many of us, taking an exam is a nerve-wracking experience. In earlier years, students may forget their password and have to wait for IT support in order to start their exam. The latest improvement to the ExamStart system, developed over the past ten years by Graham Robinson and Dr Pei Zhang, is that students can proceed with their exam even if their password has been reset, their account clamped, or they use a password manager.
Regrettably, our treasured colleague Dr Pei Zhang was unable to attend due to illness.
Exams may use OnDemand which is used for question types such as multiple response and matching questions, or a pared-down version of Word, with no grammar and spelling checks, for essay-based exams, or even more niche tools such as SPSS, Jupyter Notebook, or Authentic World.
A solid and supported experience
As students take their exam, backups are taken every minute. This means that if a computer crashes or fails, the student can be relocated to a vacant computer to continue taking their exam. At the end of the exam, the papers are automatically saved to a secure server.
In their office based on Highfield campus, the MLE team keep a vigilant eye over this essential activity.
Before an exam, workstations run self tests to identify any issues before a student uses the computer. During exams, workstation screens and rooms are monitored remotely from the MLE office in Highfield.
When the Vice-Chancellor arrived the BIOL1020 exam was about to start. Dean Wall, the newest member of the team, took responsibility to be the first point of contact for exam support. This allowed Graham and Louis to take the time to demonstrate to the VC how the team ensures that exams in multiple rooms around our campuses start effectively.
With communication between the Exams Office, Exam Invigilators, and the MLE team by two-way radio, if any issue cannot be resolved remotely, the team attend in person. Invigilators rely on technological reliability, knowing that in extremis, they may call for, and receive, assistance within minutes.
Image 4 – During the exam period the MLE team may monitor activity in the workstation areas using the lecture capture cameras.
Throughout the exam period the MLE team run a Computer-based exams hotline. This allows staff to phone for immediate assistance. The service works in the same way as ServiceLine, using the same call-handling system supported by our colleagues in the University Switchboard. Members of the highly skilled MLE team log into the hotline, available at a moment’s notice to provide expert advice and help. This service is used in particular by colleagues in schools and faculties who run Additional Exam Requirements exams on departmental laptops.
“It’s all about improving the staff and student experience and maximising our effectiveness,” explained Graham. As the exam began, the Vice-Chancellor received an overview of ExamStart and found out what it was like take a Computer-Based Exam himself. With a background in Materials Physics, the VC tried an example exam put together with five questions randomly chosen from a first year physics exam to demonstrate the exam taking experience.
Reflecting on assessment practices
Using computer-based exams improves efficiency and effectiveness. Exams may be generated from a larger pool of questions, and students given questions at random from a pool. Fixed response questions are marked immediately, and statistics generated to help determine whether the question was a “good” question.
The Vice-Chancellor shared with the team his own journey with assessment and how he has used it within his teaching. He reflected with the team on changes he had made in his own assessment practice to improve the retention of students on his modules.
A consistent process
Exams typically take place at the end of each semester, with supplementary exams taken before the start of the next academic year. As such, the MLE team must carefully manage their presence to ensure that the right number of staff are always available to support this practice. Just in the academic year 2018/19, 18,528 exam papers were sat at a computer. This includes those students with “Additional Exam Requirements” who have been given dispensation to use a computer to write an exam that their fellow students are still expected to use pen and paper to complete.
In many ways the process behind a computer-based exam is similar to paper based exams. Academics must submit their final papers to the exams office by about six weeks before the exam, and this is no different for computer-based exams. While the technology would allow changes up to just before an exam is taken, it is crucial the same level of quality checking and assurance as is expected with paper based exams is applied to those taken on a computer.
The Vice-Chancellor was very impressed with what he saw of ExamStart and the processes and procedures that the MLE team has developed and optimised for CBE over many years and said, “It is really interesting to learn about what capability we have developed at Southampton in CBE. In becoming a digitally-enabled university making real use of such an approach can set us apart. It was very impressive to see first-hand what the team have already achieved”.
Toward the end of their meeting with the VC, the MLE team raised some other important aspects of their work.
Continuing on the theme of assessment, one of the key areas of importance in the NSS survey is assessment feedback. The MLE team support Turnitin which is used in Blackboard and eAssignment. In Blackboard, the Turnitin Feedback Studio online marking tool has been used by staff to provide students with 411,197 pieces of feedback in 2018/19. This includes written comments, spoken feedback, and annotations.
Lecture capture is highly valued by University of Southampton students. The MLE team introduced lecture capture technology a decade ago already. The Vice-Chancellor was pleased to hear that 179 teaching spaces are equipped for recording lectures and 104 whiteboards may now be captured. There is no technical reason preventing the further uptake of lecture capture.
The reliability of the cameras and other infrastructure is important here as well. Following continuous improvement practices put in place by the MLE team, cameras used for lecture capture automatically alert the team when faults are detected.
MySouthampton mobile app
The University’s MySouthampton app is going from strength to strength. The team demonstrated to the VC how usage of the app has increased year on year. They outlined the most popular features, which are timetable and email checking, campus maps, and Unilink bus times.
The Vice-Chancellor is keen for students to find available study spaces on campus wherever they are located and the MLE team look forward to taking part in a project to monitor available study space and make it identifiable within the app.
The MLE team work closely with Communications on the app. A successful development has been the use of the app to support Open Days. Prospective students and their parents use the app before the event to identify and set reminders for which sessions in particular they wish to attend, such as workshops on preparing their personal statement. During the Open Day the app reminds attendees when it’s time to attend their preferred sessions.
Blackboard is the best known of the applications that MLE support. In 2018/19 members of UoS spent more than 70 combined years on Blackboard, actively using over 6500 courses. Not only students use Blackboard. As well as running their own modules on Blackboard, staff will find themselves taking health and safety and diversity training, as well as other professional development activities.
This year the MLE team is working closely with Human Resources to help design and implement the “Leading for the Future” line manager training which will be hosted on Blackboard.
The MLE team gave the VC an overview of some key improvements made to Blackboard of late and showed that statistics on usage demonstrate the effectiveness of these improvements.
The MLE team has also been leading the way in the understanding at UoS of The Public Sector Bodies (Websites and Mobile Applications) (No. 2) Accessibility Regulations 2018. These regulations apply to public sector bodies and require that their web-based services, documents, videos and mobile apps meet an accessibility requirement.
What is this requirement? In simple terms it is to follow these four steps:
- Understand the impact of the regulations.
- Check the accessibility of institutional websites, media, and apps, including those provided by third parties
- Make a plan to fix any ways the above do not fulfil the relevant requirements of the European standard on the accessibility requirements suitable for public procurement of ICT products and services in Europe (EN 301 549).
- Publish an accessibility statement for each relevant service.
Toward the end of the Vice-Chancellor’s visit to the MLE office, Matthew Deeprose demonstrated how the MLE team had been making an impact in raising the awareness and importance of these consequential regulations.
“The services we support within the MLE team are among the most used at the University by the widest range of staff and students. Therefore, it is essential not only to understand our responsibilities regarding these regulations, but to align our services with them.”
A key moment in this journey was the work to implement a new responsive theme for the Blackboard VLE. “The responsive theme provides a much better experience, but to implement it we had to configure the theme to use the University brand colour scheme. When we choose colours from the University brand, we must ensure that the colours have enough contrast to ensure their legibility. In the regulations, this means that when text is presented, the colour of the text has a minimum contrast ratio of 4.5:1 with the colour behind it.”
The University colour scheme has 21 colours, making for 441 possible colour combinations, but through testing each one the MLE team found that only 19% of those are at or above the minimum threshold to be used for the presentation of text.
Matt created a “Brand Palette Accessibility Matrix”, where each possible colour combination and its accessibility rating is resolved. It has already been used hundreds of times to ensure that content is both on brand and accessible.
Matt developed an Enhanced ATbar that integrates the ATbar produced by University of Southampton’s Electronics and Computer Science department’s Accessibility Group with Blackboard whilst bringing together accessibility features, built-in to Blackboard but little used and known, into a coherent, congruent, and unified interface. Information about the Assistive Technology Software available at UoS, The Assistive Technology Suite, and Enabling Services is included within a help section, normalising the introduction and promotion of accessibility beyond a self-selecting user population.
The Accessibility Statement allows us to describe the accessibility of services and explain any issues and what we are doing to resolve them. They should also provide further information about how users may make effective use of assistive technologies such as those built into browser and operating systems.
A recent review of Accessibility Statements published on Electronics and Computer Science’s Lexdis site found that in October 2019 less than 4% of organisations covered by the regulations had compliant accessibility statements.
This is another area where the MLE team are leading the way. Back in April 2019 they published the first ever accessibility statement for a VLE, and shared it with the wider community. After sharing practice on the Blackboard community site, and JISC Digital Accessibility Regulations Mailing list, Matt has been invited to write blog posts, and present at conferences on the theme of accessibility.
“This is what is morally right to do and is inline with the University’s agenda for inclusion: to remove barriers and ensure that every member of our community leverages to the maximum extent the services and platforms, that we work hard to develop and maintain, and that they rely upon for their success at Southampton,” concluded Matthew.
The meeting between our Vice-Chancellor and the MLE team was tinged with sadness. Sam Cole, a member of the MLE team since its creation in 2003 is leaving the University to take up a new post at Falmouth University. The team wishes Sam all the best in this next step in his journey.
University of Southampton President and Vice-Chancellor Professor Mark E. Smith was very impressed with the team and their activities, he concluded, “The range of activity covered by the team from the VLE, to lecture capture, to computer-based examinations, to website accessibility is remarkable. The team deals with issues that play centrally into our strategy of making our student experience outstanding. It was hugely encouraging to see the enthusiasm and commitment of the team”.