Introduction

The Scout Association of Hong Kong (SAHK) offers a variety of air scouting badge courses in Hong Kong to its members in scout section and venture section. Members are required to complete and pass the assessment in the courses in other to obtain the badge. Scout Association of Hong Kong is now looking for solution to tackle pain points identified in the air activities, as listed below:

Dispersed Teaching Resources

Badge courses are held not only in headquarter, but also in its subsidiary units or groups. There is no centralized collection of teaching resources which caused difficulty in aligning teaching quality across units and duplicate effort in preparing teaching materials.

Repeated Use of Question Sets in Assessment

It is found out that within the same syllabus, many of the courses are using identical set of questions which made it possible for participants to easily pass the assessment by obtaining past assessment paper. This had impacted the credibility and the quality of the assessment.

Not Eco-friendly in Using Printed Form of Test Paper

This is against the vision of SAHK as an environmental-friendly organization.

Easy Loss of Paper STA Form

It is common that participants will lose their paper STA form which waste time to validate and to reprocess the participation record of students.

SAHK is looking for an online assessment system to address above pain points identified. The success criteria of the project are listed as below:

  1. System go live within 1 year
  2. Migrate existing paper forms/test into e-form, while reserving the ability to print paper form test
  3. Digitize and automate test paper preparation, marking with flexibility of manual intervention
  4. Online records of student activities

Background Study

The computerization of assessment is no news in the world with many use cases. It is widely adopted in various industries including education, banking etc. For instances, online aptitude test is used to evaluate applicant capabilities for job interview, this has helped organizations to save effort in identifying right candidate. We can see that there are some common intentions in adopting assessment systems across the world including:

Effort Saving in Preparing/Grading of Assessment

System can automatically generate a set of test questions from the question bank. The evaluation of test submission is also automated which participants’ scores and historical results are stored from end to end.This has saved huge amount of manual effort from test preparation, scoring to result input.

Easy Analytic and MIS Reporting

With result of every questions are automatically stored in the database, assessment system has enabled simple and powerful data analytic capability from candidate performance in question level to comparison on candidate level. Cross year analysis is also made possible.

Environmentally-friendly

Reduction in paper printing, learning materials are distributed in a digital way.

Improved User Experience

With assessment system, students can complete the test in anytime, anywhere using mobile device.


Existing Applications

Moodle

A learning tool that provides personalized e-learning environment for students and teachers.

  1. Cross-platform and Open Source
    Moodle is a cross platform application, user can install the application in both Windows and Mac environment. Besides, since Moodle is open sources, it does not require license fees to install the system.
  2. Extensible Features
    One of the features that Moodle provides is quiz module. Users can create their quiz such as inputting question into the question bank, delivering questions in random order, setting the time limit and choosing different kinds of question type. By using this module, user can manage the process of creating and managing quizzes.
  3. Mobile Users Friendly
    Moodle is web-based application, users can access to the learning platform with web browser. As a result, they can use their mobile devices to look up learning materials and attempting quizzes in their home instead of attending to the classroom.
User Interface of Moodle Quiz Module

G Suite for Education

G Suite for Education is a suite of productivity tools for teaching and learning. It contains several modules for teachers and students.

  1. Google Classroom
    Google Classroom is a web application which gives a centralized environment for teachers to upload and distribute the learning materials.

  2. Google Forms
    Google Forms allow user to create a quiz. User can integrate it with Google Classroom to build a learning and assessment platform for students.

UI of Creating Question in Google Form

Comparison

Following is the summary of the discussed applications:

Category Property Moodle G Suite for Education
Overall system Platform Cloud or local server Cloud
Pricing Free Free for eligible
institutions
Extensibility Install plug-in from community No
Mobile compatibility Yes Yes
Learning module Course materials management Yes Yes
Assessment module Question bank Yes Yes
Question in random order Yes Yes
Automated grading Yes Yes
Editing the marking manually Yes Yes
Response of result After quiz submission After quiz submission
or at specific time

Challenges

  1. Synonyms and Typos of Student’s Answer
    In current implementation of the online assessment tools, the auto marking mechanism for fill-in-the-blank and short questions is that to compare whether student’s answer is exactly match with the predefined model answers. Although, student answer is semantically correct, this kind of approach does not tolerate students having wrong spelling or same idea with the answer.
  2. Difficulty in Scoring Open Question
    For open question, students are required to answer in their own words. To score this question type, teachers are required to understand the ideas and keywords of student’s answers. Therefore, online assessment cannot score open question automatically.
  3. Users Unfamiliar with Online Assessment
    For the first time teachers and students are using a new system, they may encounter some difficulties and do not have enough knowledge and experience on the online assessment process.