With a small in-house team we can support a system that can be used by virtually all teachers and students in NSW or elsewhere.
OBOS Scores Top Marks for Onscreen Marking
The Office of the Board of Studies New South Wales is the certifying authority responsible for the content of syllabus materials, registration of non-government schools, and delivery of high school examinations for New South Wales, the most populous state in Australia. LANSA’s development and integration tools have been at the heart of the Board’s IT infrastructure since 1989. Recently the Board chose LANSA to develop and deliver an onscreen marking application as part of its eAssessment program. Onscreen marking allows markers to view and mark onscreen a scanned image of students’ written exam responses, or the students’ responses entered or spoken online.
Mitra Bhar, Manager IT at the Office of the Board of Studies NSW, says “It is important that we continue to investigate how advances in technology can provide improvements in our examination and assessment programs. LANSA’s technology and the IBM i platform help us to provide accurate, highly reliable and secure solutions to meet the requirements of our core business.”
LANSA’s technology and the IBM i platform help us to provide accurate, highly reliable and secure solutions.
- The Logistics of Exams
- Marking Procedures
- Onscreen Marking
- The Benefits
- Conclusion
- 2013 Update: Mobile Apps for Markers
- Company and System Information
The Logistics of Exams
The two main qualifications which the Office of the Board of Studies (OBOS) administers are the School Certificate (SC) and Higher School Certificate (HSC). In recent years over 87,000 Year 10 students took part in the SC examinations and over 68,000 Year 12 students in the HSC examinations. Most of these students take at least six examinations. Recent HSC exams included over 110 different examinations for a wide variety of subject courses. Students worked from 13 million printed pages of questions and submitted their answers in 1.75 million writing booklets.
The setting, conduct and marking of the examinations is a huge logistical task, which involves over 5,000 examination supervisors at 750 exam centers, 7,000 markers assessing practical and written examinations at over 20 marking centers across the state, and more than 750 casual clerical staff.
The appointment system for exam supervisors and markers and the collection of the actual marks has been managed with a LANSA-based solution for many years, but the marking process itself, except for multiple choice exams, was mostly paper-based.
The marking process consumes a significant proportion of OBOS’s annual budget. Accordingly, streamlining the marking process can have a major effect on the OBOS’s cost-effectiveness.
The setting, conduct and marking of the examinations is a huge logistical task.
Marking Procedures
In order to assure a fair outcome for all students, the marking process for written exams is highly structured and sophisticated. The process involves chief examiners, who have been involved in designing the exam, supervisors of Marking, who manage the marking operation in a subject marking center, senior markers, who are typically teachers with extensive marking experience and markers, who are experienced teachers or academics.
After being briefed about marking guidelines by the chief examiner and supervisor of marking, the senior markers will develop a marking kit for the exam section they are responsible for. A section can consist of one or more questions. There are usually at least two senior markers for each exam section. To develop the kit, they first read through a large number of actual scripts (the written responses by students) to get a feeling for the exam at a practical level.
The marking kit needs to include at least one complete set of scoring benchmark examples, such as scripts scoring ‘5 out of 5’, ‘4 out of 5’ and so on. The kit also contains detailed marking guidelines and sample markings. The kit is then tested by other senior markers, again on a representative sample of actual scripts. During this process ‘marking reliability statistics’ are calculated to demonstrate whether a kit is working well, or whether further fine-tuning is needed. None of the marking during this process, which may take several days, is final.
Next the markers arrive at the marking centers to prepare for and do the real marking. Again, each marker is responsible for only a specific question or section of the exam. Markers first need to go through a number of ‘practice scripts’ to get them on the same level of marking as the senior marker has set out. When a marker consistently marks in the expected range on the practice scripts, he or she is ready to start marking real scripts.
The senior marker, who looks after a team of 5 to 10 markers, will randomly insert so called ‘common scripts’ in between the real scripts, which serve to check and make sure that the markers stay fresh and accurate and keep marking within the intended range.
Extended responses and essays are marked independently by two markers and the final mark is the average of the two. If the two marks vary by more than one third of the maximum mark, a third marking will be undertaken, and a senior marker will determine a ‘resolved mark’.
The marking process for written exams is highly structured and sophisticated.
Onscreen Marking
Muir Mathieson, Senior IT Consultant, OBOS, and technical project leader for the Onscreen Marking System, explains, “The onscreen marking process begins with the color scanning of the writing booklets or question and answer booklets the students have submitted and, using LANSA’s integration tool (LANSA Integrator), upload them in PDF format into the IBM i-based marking system.”
“In that process, a custom written LANSA Integrator PDF service takes care that the number of pages is validated, that student and exam center identification details are masked and that the page images are reassembled as required for marking, so that a marker only gets to see the exam section he or she needs to mark. The LANSA Integrator service also sets the PDF properties and saves the documents to their proper file name.”
Authorized markers can access the scanned scripts remotely over the Internet, or from one of the marking centers, with the OBOS’s browser-based Onscreen Marking System, developed with LANSA high level language. The system has specific functionality for each of the main roles associated with the marking process: markers, senior markers and supervisors of markers.
The system allows markers to set preferences for their comfort based on the equipment they are using, such as left or right mark boxes or buttons. Markers can view a list of all the scripts they have been working on that day or previous days. They can add notes for themselves or for immediate attention of their senior marker. They also have the ability to search through all the scripts they have marked using several different search criteria, making it very easy to locate scripts they need to revisit. The system navigates the markers efficiently through the work that needs to be marked, automatically and randomly serving up the next script.
Senior markers have additional functions. They can annotate and upload benchmark examples and practice scripts, and set up a frequency for inserting common scripts in between the real scripts. They have a real-time dashboard view of what their markers are doing and can go to any script to look at the marker’s notes and marks awarded.
“Marking deviations in the practice and common scripts are shown in color, making it easy to discuss variations and interpretation of the marking scheme with that marker. Markers who are consistently ‘on-target’ need not be distracted or taken off the task. Similarly, it’s also easy to supervise new markers and decide whether they need to be given more practice scripts or have them start on the real scripts,” says Mathieson.
The system allows markers and senior markers to exchange messages. This is helpful when a marker needs assistance or wants the opinion of the senior marker about a certain script. Likewise, the senior marker may feel the need to comment on the marking of a certain script and send an instantaneous message to the marker. The messaging system is especially helpful for remote marking, allowing teachers to mark at home.
At any time, the senior markers can view statistics. They can see how the spread of markings for their exam section is evolving, how many scripts are marked and still need to be marked, how much time markers spend on average and on individual scripts.
All internet traffic is encrypted using SSL 128-bit encryption and no marks ever reside on the marker’s desktop. All data (including scripts) are replicated across to a second IBM i server in real-time with each server residing in separate data centers.
Not directly related to the Onscreen Marking System, but part of the same exam system, is the digital recording of oral language exams. This is still at a trial stage for main languages, but already implemented for some language exams. At the point of examination the student sits in front of a computer that is connected to the OBOS’s examination system. After photo identification, the student’s response is digitally recorded and for some exams, marks are entered immediately. Once the exam is over, the examiner submits the digital recording and marks to OBOS’s IBM i-based exam system, where LANSA Integrator takes care of the receiving, naming and storing of the file in MP3 format. The digitally recorded exams can then go through a similar marking process as the written scripts, using exactly the same application with modified terminology.
“Using LANSA Integrator, we have been able to create a highly sophisticated application, where PDF and MP3 files are fully integrated with our IBM i-based core examination system,” says Mathieson.
We have created a highly sophisticated application, where PDF and MP3 files are fully integrated.
The Benefits
Kevin Ford, Manager, Student Support Services Branch at the OBOS, says, “The Onscreen Marking System provides instantaneous marker reliability data and promotes objective and consistent marking. The same strict procedures apply to both paper-based marking and onscreen marking, but the procedures are easier to apply and manage in the onscreen solution.”
“Having a real-time view of marking and having the tools and statistics available to spot deviations, allows senior markers to take immediate corrective action. In the paper-based system it could happen that a large batch of scripts would have to be re-marked, as it could take a while for the senior marker to notice that someone was marking outside the range,” continues Ford.
“Another advantage is that the script and notes can be separated, which is good when a script needs to be objectively re-marked. In that case you don’t want a marker to be influenced by the notes and comments made by someone else. Likewise, the fact that the marker only gets to see the section of the exam that he or she is responsible for, also encourages objective marking.”
The common scripts are more effective in the onscreen marking process, because markers cannot differentiate them from the real scripts. In the paper-based system you could recognize them, because they are photo copies while the real scripts are originals. “Although it should theoretically not make a difference to the markers that they are marking a common script, in reality they may give that script extra attention, because they know it will be monitored. Now, every script is marked entirely on its own merits, making for a fairer and more equitable outcome for all students,” says Ford.
Not being dependent on paper-based procedures allows OBOS to draw on a larger pool of eligible markers. It is also very useful for teachers to gain practical experience with the marking process, so they can better prepare their students for the exams. As traditional paper-based marking requires markers to come to a marking center, the distance made it impractical for many country teachers to participate. “Now we can give them Internet access and support them with online practice scripts, marking kits and messaging. We can even involve teachers who teach the HSC at overseas schools and colleges,” explains Ford.
“The markers are extremely positive about onscreen marking, because of the convenience, the potential for reducing the movement of large quantities of paper and the quick feedback on their marking,” says Ford. “There is also a firm belief that the system is fairer and more equitable for the students.”
“The logistics involved in securely distributing, storing and collecting nearly two million paper-based writing booklets are staggering,” concludes Ford. “It is a far neater and easier-to-manage process to provide secure access to scripts that are kept on our own server. The solution is also more cost effective.”
The Onscreen Marking System provides instantaneous marker reliability data.
Conclusion
The Onscreen Marking System is part of a wider eAssessment initiative. The eAssessment also includes a Computers-in-Exams (CIE) initiative, which allows for exams to be conducted and marked online, eliminating the need for hand-written responses altogether. Some of the HSC computing exams are already conducted online. When given a choice, a significant proportion of students would prefer to use a computer for their exam rather than hand-writing.
OBOS has conducted extensive trials for all eAssessment initiatives, comparing paper-based marking with onscreen marking and handwritten scripts with online exams. Reactions to the CIE initiatives have been very positive from students, teachers and markers.
Bhar, says, “We have tried our Onscreen Marking System with the exam papers from schools in other states and other countries and also with the examinations of universities. The system works equally well in those scenarios.”
“Looking at the international experience, onscreen marking and e-Assessment are the way of the future, It is important that we continue to investigate how technology can provide improvements in the assessment and examination programs.”
“The services we provide are highly visible and critical to the smooth administration of our examinations. We cannot afford to make any mistakes. LANSA’s technology and the IBM i platform help us to provide accurate, highly reliable and secure solutions to meet the requirements of our core business. With a small in-house team and using LANSA’s productive high level development tool, we can support a system that can be used by virtually all teachers and students in NSW or elsewhere,” concludes Bhar.
With a small in-house team we can support a system that can be used by virtually all teachers and students in NSW or elsewhere.
2013 Update: Mobile Apps for Markers
Recently OBOS has provided mobile apps, developed with LANSA’s LongRange, to those markers that need to visit schools for the marking of non-written HSC exam subjects, such as performing arts (drama, music, dance), visual arts, design & technology and industrial technology.
This itinerant marking process, which occurs earlier in the year than the standard HSC exams, involves hundreds of markers travelling by road and air to all corners of the state. The performances and the art and technology items are assessed by at least two markers who provide their marks without consulting each other. Similar to double-blind marking for written examinations, a reconciliation is required if the marks differ by more than a certain allowed variance.
The app allows markers to instantly record their comments and archive photos, audio and video recordings of exam works. Markers also use the app to refer back to recordings of benchmark examples, helping them to apply strict and specific assessment criteria. In addition, the app contains the marker’s travel and accommodation schedule and, using the device’s GPS, shows them how to travel to and from the school to their accommodation.
The mobile solution contributes to a fair and consistent marking process. It also makes the task a lot easier for the markers, not having to take a load of paperwork with them. The app integrates with the OBOS IBM i based exam marking system.
Company and System Information
- On 1 January 2014 the Board of Studies NSW merged with the NSW Institute of Teachers to become the Board of Studies, Teaching and Educational Standards (BOSTES). The Office of the Board of Studies NSW has been using LANSA since 1989. Its IT team has delivered many systems, such as online services to schools, teachers, students and
parents. These include the delivery of HSC results over the web and the collection of over 1.2 million grades and assessments and over 45 million responses and marks per year. - OBOS uses an IBM Power 770 with the IBM i operating system.
- With advice and assistance from LANSA, OBOS has commercialized its Onscreen Marking System, now known as “MarkManager“. For more information
email LANSAemail LANSA or visit http://markmanager.edu.au - For more information on the Board of Studies visit: www.boardofstudies.nsw.edu.au