The ability to monitor the standard of marking as it is occurring is exceptionally important.
Western Australia implements MarkManager for senior secondary schooling examinations
The case study below is about the successful implementation of MarkManager by the Western Australia School Curriculum and Standards Authority (WA SCSA). MarkManager is designed to support on-screen marking of “high stakes” written examinations. The solution is developed using LANSA’s development and integration tools by the Board of Studies NSW, Australia and commercialized with assistance from LANSA. Onscreen marking technology helps to ensure that marking is impartial, consistent and fair to all. This is because of features like real-time monitoring of markers, insertion of quality assurance scripts and easy implementation of best-practice methods, such as double blind marking and segmentation. In addition, onscreen-marking simplifies logistical procedures because written exam responses are color scanned and securely distributed to authorized markers, removing the need to transport boxes of papers.
Following a successful tender, the Office of the Board of Studies NSW (“the Board”) became the supplier of an online marking service to the Western Australia School Curriculum and Standards Authority (“the Authority”). The Board’s solution for the Authority encompasses operation of its MarkManager system on a Software as a Service (SaaS) basis, advice and guidance, support, training, customization, implementation planning and project management. The MarkManager solution is hosted on the Board’s servers in a secure data centre and provided to the Authority through a secure Internet connection.
The Authority uses MarkManager to mark examination responses for the Western Australian Certificate of Education (WACE) examinations, which take place at the end of year 12. The on-screen marking process begins in earnest with the scanning of completed WACE “scripts” (examination response booklets) submitted by candidates. In a controlled online environment, the Authority’s markers access the scripts remotely over the Internet for marking.
Instantaneous marker reliability and activity feedback allows chief markers and team leaders to monitor each marker’s performance and take corrective action immediately if needed. As a mature adopter of information technology to achieve its goals, the Authority has been able to leverage MarkManager to provide an accurate and reliable marking service.
The Challenges
The Western Australia School Curriculum and Standards Authority is an independent statutory authority that is responsible to the Minister for Education. The Authority is responsible for setting standards of student achievement, for the assessment and certification of student achievement according to those standards, for developing an outline of curriculum and assessment in schools, developing and accrediting courses for schools and for maintaining a database of related information.
The WACE examinations are a huge logistical exercise for the Authority. Tasks include hiring and training the markers, printing the examination papers, distributing them to the examination centres, collecting the scripts and getting them securely transported to the markers. Recent statistics show that approximately 17,000 candidates sit for the WACE examinations, each doing on average an exam in 4 courses. Over 68,000 exam booklets are printed and nearly 1,000 markers are involved in marking of the scripts and another 350 markers are involved in scoring the non-written (practical) exams, such as music performance and drama.
One of the major challenges the Authority faces is due to Western Australia’s vast land area. Some of the cities and towns are more than 3,500 klm away from the capital city Perth. This makes it hard for teachers from regional areas to participate as markers in traditional marking, which by its nature requires access to hard-copy scripts and hence a more centralised marking approach. Not only does this impact the number of potential markers available, it also means that teachers do not have the opportunity to improve their curriculum and assessment skills by means of the marking process.
Another challenge of traditional marking is the overhead in catering for double blind marking, which the Authority had found effective in improving reliability and accuracy. In double blind marking, two markers assess a script independently of each other, unaware of the scores awarded by the other marker and unable to view each other’s annotations. When the two scores differ by more than a predefined variance tolerance (for an item, section or total), a reconciliation procedure follows. In the traditional approach, double blind marking requires a separate mark-sheet be manually maintained so that the first marker’s assessment doesn’t influence the second marker. Additionally there is a lot of “paper-handling” in ensuring the scripts are moved securely and quickly between markers.
A final challenge is the ability to monitor the performance of markers in a timely manner. In a traditional marking environment, there is a time delay before a supervisor’s check marking or inspection of “control scripts” highlights a shortfall in accuracy. It would be far preferable to have continuous and automated monitoring of performance, so that corrective action can be taken more promptly.
To overcome these challenges, the Authority was already using on-screen marking technology for some of its WACE exams. Since the accuracy, security and timely progress of marking are essential, the Authority continuously strives for improvement and hence reviews its marking technologies every few years.
In 2011 the Authority put out a request for proposal for a managed services solution for online marking. The solution had to facilitate several key outcomes, including high accuracy marking and an efficient and timely marking process, as well as maintaining the security and confidentiality of all materials. The monitoring and reporting facilities were also important to the Authority. Additionally the solution provider needed to have proven experience and an on-location training capability.
After a comprehensive and detailed tender evaluation process, the Authority selected MarkManager, an online marking solution developed by the Board of Studies NSW, commercialised with advice and assistance from LANSA.
Jenny Morup, Manager, Examinations Logistics for the Authority, explains, “We selected MarkManager because it met our criteria and offered value for money. Value for money is a very important factor for a government agency when a tender is let. We want to achieve the best possible outcome for every dollar spent. That means you need to assess the costs, benefits and risks inherent in an offer carefully, rather than simply selecting the lowest offered price.”
The fact that the solution had proven to work well for the high school examinations in New South Wales (NSW), the most populous state in Australia, and that the Board of Studies NSW would provide implementation assistance were also important factors.
Value for money is a very important factor for a government agency when a tender is let.
The Solution
In its first year of implementation, 2012, the Authority used MarkManager for nine exams: English as an Additional Language and Math (both in several levels), Biological Sciences, Physics and Chemistry. Additional exams might be included in future years, for the duration of the tender.
“The team from the Board of Studies was very knowledgeable, helped us with the planning and gave very good support. Together we set up an operational plan that worked exceptionally well throughout the project,” says Morup.
The Authority’s marking process started with the colour-scanning of the writing booklets the students had completed, which were uploaded in PDF format into MarkManager. During that process the number of pages is validated and the student and exam centre identification details are masked. For some of the exams the pages were reassembled into separate documents, so that markers only got to see the exam section they have to mark.
The Authority doesn’t use marking centres and markers use their own equipment to access the scanned scripts remotely over the Internet, usually from their home. The solution ensures that no marks or scripts ever reside on the marker’s desktop. Markers can set their individual screen preferences, such as to display mark boxes left or right, and they can view a list of the scripts they have been working on. They can add notes for themselves or for the attention of the chief marker.
Chief markers have additional functionality and can pre-mark practice scripts that markers have to complete satisfactorily before they are allowed to start marking real scripts. They can also set up a frequency for inserting control scripts in between the real scripts to check that markers remain on the ball.
Chief markers have a real-time dashboard view of what markers are doing and can go to any script to look at the marker’s notes and marks awarded. They can view the statistics at any time, such as how the spread of markings for a specific exam section is evolving, how many scripts are marked and still need to be marked.
MarkManager’s ability to allow markers and chief markers to exchange messages is considered extremely helpful, especially since the Authority’s markers work remotely. Messaging is used, for example, when a marker wants the opinion of the chief marker about a certain script. Likewise, a chief marker may feel the need to comment on the marking of a certain script.
Melanie Jasper, Senior Consultant Written Examinations (Marking) at the Authority, says that the feedback from the markers was very positive “They found the application easy to use and liked the fact that they could adapt the screens to suit themselves. Access to the scripts was easy and the speed very quick. Markers were able to work on one script after another without experiencing any slow uploads. The system was quick, even in peak periods.”
Access to the scripts was easy and the speed very quick.
The Benefits
“The ability to monitor the standard of marking as it is occurring is exceptionally important,” explains Morup. “In addition to double blind marking, we also have good quality control because of the reference scripts that can be placed throughout the marking process, such as the start-up and control scripts. If a marker starts scoring outside the expected range, the chief marker can take immediate action. The solution helps us to achieve accurate and unbiased marking.”
The Authority aims to select markers that are currently teaching the course. Morup explains why, “It is really good professional development for teachers to experience the high standard of WACE examination marking. They learn to mark using the proper marking keys, which are the explicit statements about what the examiner expects of a candidate. They can take that knowledge back into their schools, back into their classrooms and use it to prepare their students better. It is one of our priorities in choosing markers.”
“Travelling for a training course in marking is do-able for most teachers, but spending a few weeks away from home marking exams is less so. Online marking enables us to take the marking process across the remote regions of Western Australia, while we can still closely monitor the quality and the process. We have international schools that do our courses and the online solution allows us to also include markers that are based overseas. So they can benefit as well, ” continues Jasper.
“An additional benefit is the safety and security of online marking. There is no risk of losing scripts. Having the scripts online is much safer and far more convenient.”
Jasper finds the ability to split an exam into sections very beneficial, “It gives you a lot more choices of how you are going to mark an exam and who is going to mark which section. We can configure the program ourselves to split an exam into sections, for example combine questions 1, 5 and 8 in a section because they all relate to a particular mathematics concept, or separate short answers from extended responses. We can give markers those particular sections that they are specialized in. The examination scripts are still available intact, so we can still look at the whole paper as well.”
The ability to monitor the standard of marking as it is occurring is exceptionally important.
Conclusion
Morup says of the implementation, “It was an absolutely smooth transition and the implementation of the new system went exceptionally well. It worked very successfully right from the beginning. The team from the Board of Studies NSW was very responsive. They listened to us, understood our requirements and adapted the program where needed. It was excellent. The success of any project is the communication.”
“The training and the manuals for the chief markers were extensive and relevant. The additional background information that the Board of Studies provided was also very interesting and helpful. The chief markers particularly commented on the facility of real-time monitoring of markers and the completeness of the reports. The only issue they had was that they didn’t have enough time to try out all of the reports. But as they gain more experience next year, they will be able to explore more of the reporting,” concludes Morup.
It was an absolutely smooth transition and the implementation of the new system went exceptionally well.
Company and System Information
- The Western Australia School Curriculum and Standards Authority develops and regulates school curriculum and assessment, sets standards and reports on those standards. The Authority serves students from kindergarten to year 12 in government and non-government schools in Western Australia and their teachers. For more information see:
www.scsa.wa.edu.au - The Board of Studies NSW develops curriculum and awards secondary school credentials. The Board serves government and non-government schools in the development of NSW school education for years K-12. For more information visit: www.boardofstudies.nsw.edu.au
- For more information about MarkManager visit: http://markmanager.edu.au