Case study: VIC162 Mobile
checklists enabling authentic
workplace assessment
Victoria June 2013
Box Hill Institute
VIC162 Mobile checklists enabling authentic workplace assessment
Contents
1.
2.
3.
4.
5.
6.
7.
About Us ......................................................................................................................... 2
1.1
The team .......................................................................................................................... 3
1.2
Knowledge and experience .............................................................................................. 4
The project aim and objective ......................................................................................... 6
2.1
Aim ................................................................................................................................... 6
2.2
Objective .......................................................................................................................... 6
The journey ..................................................................................................................... 8
3.1
Starting the project .......................................................................................................... 8
3.2
Running the project ........................................................................................................ 13
3.3
Wrapping up the project ................................................................................................ 15
The results .................................................................................................................... 19
4.1
Main project outcomes .................................................................................................. 19
4.2
How the outcomes were measured ............................................................................... 20
The lessons learnt ......................................................................................................... 24
5.1
Key successes ................................................................................................................. 25
5.2
Suggested improvements............................................................................................... 26
Passing it forward ......................................................................................................... 27
6.1
Knowledge transfer ........................................................................................................ 27
6.2
Mentorship ..................................................................................................................... 28
More information ......................................................................................................... 29
About Us
National VET E-learning Strategy 2012 – 2015
Page 2
VIC162 Mobile checklists enabling authentic workplace assessment
This case study was developed by Box Hill Institute (BHI) and Air Conditioning and Mechanical
Contractors' Association (AMCA) with funding and support from the national training system’s elearning strategy, the National VET E-learning Strategy (Strategy).
The Strategy provides the VET system with the essential e-learning infrastructure and expertise
needed to respond to the challenges of a modern economy and the training needs of Australian
businesses and workers.
1.1
The team
Box Hill Institute is a leading Victorian vocational and higher education provider known for our
collaborative and creative approach to education in Australia and overseas. The Institute has a long
and diverse history and over the years has been the recipient of many awards and achievements
demonstrating Australia-wide recognition for our efforts.
Our Teaching Centres enable students to achieve their career goals; we add value to business and
deliver innovation and technological change to industry. Our high quality strategic relationships,
facilities and programs make us recognised leaders in a number of vocational areas.
The Air Conditioning and Mechanical Contractors' Association (AMCA) is the voice of the air
conditioning and mechanical services industry and is a nationally accredited Registered Training
Organisation (RTO) offering a range of accredited courses, encompassing post graduate,
apprenticeship, traineeship and pre-apprenticeship levels. (Src: http://www.amca.com.au/)
With a focus on transforming trades from a traditional approach to training and delivery to one that
better integrates workplace learning, encourages input from industry, and adopts mobile learning
and assessment practices congruent with our learners needs and preferences, Box Hill Institute
partnered with AMCA in this project as an opportunity to broaden delivery locations and finding
simple ways to better integrate the workplace in assessment strategies.
The Blended eLearning Solutions Team at Box Hill Institute were integral to this projects success
include the following people:

Project director - Pauline Farrell, Executive Manager, Blended eLearning Solutions

Project manager – Julianne Seaman

Learning Designer - Rowan Peters

Programmer – Gregor McNish

Multimedia Developer – Evan Englezos

Multimedia designer – David Beaumont
National VET E-learning Strategy 2012 – 2015
Page 3
VIC162 Mobile checklists enabling authentic workplace assessment

Professional development officer – Joan Taylor
Key stakeholders included:

Laura Steedman, Executive Office, Training and Development, AMCA

Greg Pringle, General Manager of Trades and Apprenticeships, BHI

Tony Watson, Centre Manager of Building and Furnishing Studies

Petra McLoughlin, Centre Manager for Automotive Transport and Engineering

Sheldon Irving, Centre Manager for Electrical and Refrigeration Trades

Toni Conti, Teacher, Automotive, BHI

Jason Njik, Teacher, Automotive, BHI

Deon Vlotman, Teacher, Refrigeration, BHI
1.2
Knowledge and experience
Comprising of learning designers, multimedia designers/developers, LMS programmers, LMS
support staff, professional development staff and teachers, the Blended eLearning Solutions (BeLS)
team at Box Hill Institute have proven success in developing, delivering and supporting eLearning
programs and systems.
This project extends upon expertise established in undertaking a project supported by the National
VET eLearning Strategy’s Access to Skills funding “AS071Hand Tools for Hand devices”, which
involved the design and development of iBooks for Building and furnishing studies students.
The mobile learning space however is still a relatively new focal point for the Institute however, and
eventhough the team has significant technical and learning design expertise a significant learning
curve was required to increase capability and meet the deliverables of this project resulting in
significant unknowns uncovered in the development phase.
Areas of expertise that our development team needed to build a significant level of knowledge and
skills in across the duration of the project included developing for PhoneGap, creating a user
experience appropriate for mobile devices, scalable mobile interface design, using Moodle web
services, caching on mobile devices, using local mobile storage, compressing media including
photos and video on the mobile devices, accessing local services on mobile devices, creating a twoway link between Moodle and mobile devices, ensuring design aligned as closely as possible to the
e-standards and m-standards, and developing templates that would be available to teachers in
Moodle while having minimal impact on Moodle core code.
National VET E-learning Strategy 2012 – 2015
Page 4
VIC162 Mobile checklists enabling authentic workplace assessment
To ensure that a trial could occur throughout the time of this project, one of the most significant
contingencies was to redesign the assessment templates so that some of the code used to build
working templates could easily be repurposed to suit a different assessment context. This has
meant that as a final deliverable, the four assessment templates that developed include a job safety
analysis, demonstration checklist, procedure-process, and work diary.
There were also some blockers from an infrastructure and access perspective, which required us to
purchase an apple computer and gain internal and external authorisation to gain rights to use an
apple developer’s license and publish in the Apple iTunes Store.
National VET E-learning Strategy 2012 – 2015
Page 5
VIC162 Mobile checklists enabling authentic workplace assessment
2.
The project aim and objective
2.1
Aim
Internal analysis has found that BHI apprentices are predominately kinaesthetic, social and
visual learners, a demographic that is well suited to mobile assessment in workplace
environments. To extend the capacity for authentic workplace learning and assessment, we
wanted to investigate ways mobile devices could be used to record and transfer workplace
assessments (both offline and online) into the Moodle LMS for targeted apprenticeship areas.
To achieve this, we wanted to develop a mobile application where learners could access and
complete their assessments in any location so that it would support learners in remote
locations. To reduce any impact on student data charges, our objective was to allow mobile
submissions to synchronise with Moodle when a Wi-Fi connection was available.
In order to have a tool that would be sustainable, we also wanted to develop a set of generic
mobile assessment templates that teachers could re-contextualise in Moodle for any appropriate
unit of competence. To demonstrate applicability to competency based assessment and trial
these templates, we also aimed to create 3 assessments for each of 4 units of competence
using an appropriate mix of the assessment templates. From a Moodle development
perspective, we aimed to integrate the mobile assessment templates in a way that would align
to existing Moodle functionality ensuring ease of use, and ensuring integrity could be
maintained as much as possible with Moodle upgrades. By aligning as closely as possible to
Moodle assignments, our objective was to allow teachers to view submissions and provide
feedback using the Moodle gradebook.
Furthermore, our team planned to research the range of media that could be integrated into
these assignment submissions, including photos video and audio to strengthen the feasibility for
students to record progress in the workplace.
These objectives were researched, trialled and met with the exception that we weren’t able to
locally compress video so photo and text based submissions is only possible. Learners will be
able to download the project mobile application called “Mobas” from the Apple App Store to
install it on their apple device. For learners who don’t have an apple phone, an Android version
will be made available for download. A “Mobas” plug-in was developed for the Moodle
assignment module and an implementation guide has been written to support Moodle
administrators to install and implement Mobas on their server. As this is part of the assignment
module, learners can also complete their assessment tasks from a personal computer.
2.2
Objective
The key objective of this project was to build a foundation that would support teachers to shift
learning and assessment from traditional classroom based delivery in the trades areas to
workplace based delivery in a National priority training area.
Box Hill Institute (BHI) apprentices are highly visual and kinaesthetic and up to 30% of our
trades students have some degree of learning disability. These learners can be difficult to
engage and retain, and by increasing visual and kinaesthetic learning opportunities we
National VET E-learning Strategy 2012 – 2015
Page 6
VIC162 Mobile checklists enabling authentic workplace assessment
anticipated that we would be able to better align to the needs of an at risk group with significant
learning differences.
Our project team developed a mobile phone application “Mobas” to achieve this, where
assessments that would be typically completed using paper based documents or in a computer
classroom were contextualised into mobile assessment tasks. Once the application is installed
on a learners device, the assessment tasks are accessible and can be completed in any location
regardless of whether a Wi-Fi connection is present. Learners can add text or photos to their
submission, and teachers can view submissions and provide feedback in the Moodle Learning
Management system.
By utilising a blended approach, the project trialled the four assessment types with Automotive,
Electrical and Refrigeration apprentices in a simulated workplace situation and in the classroom.
The learners and teachers both expressed that the application and Moodle side was simple to
use, and the approach would lend itself to workplace based assessment. Furthermore, our
industry partner AMCA has indicated that it would improve assessment processes allowing
learners to demonstrate their level of skills, knowledge and experiences in a contextualised
workplace environment. The teachers who participated in the trial suggested that the
application was a useful way for learners to increase their employability skills in that it allowed
them to use technology for just in time assessment. The outcomes of improved digital literacy
skills harness improve their readiness for work in an environment congruent to their preferred
delivery mode and learning styles.
Since the application requires photos and text based information to be sent to the learning
management system from any location, the expansion of the satellite NBN provides a means for
learners in remote workplace locations to complete and submit assessment tasks that use a
range of media.
National VET E-learning Strategy 2012 – 2015
Page 7
VIC162 Mobile checklists enabling authentic workplace assessment
3.
The journey
3.1
Starting the project
This project required our project team to undertake a significant amount of preliminary research
and upskilling in order to form an appropriate design and delivery strategy from both a quality
pedagogical and infrastructure perspective.
This research involved an investigation into a suitable development platform from which to build
the mobile application. Our investigation analysed three mobile application development
frameworks: PhoneGap, Titanium and Rhodes as per Figure 1:
Platform
Advantages
Disadvantages
PhoneGap
Open source
http://phonegap.com/
Supports HTML5/CSS3/JavaScript
Interface is reliant on the
quality of the web renderer
for the platform
Can integrate existing JavaScript
frameworks such as JQuery
Write once, deploy everywhere
methodology
Cross browser limitations
just like regular desktop
web development.
Existing Moodle app developed on
this platform Common API set to
access phone functionality e.g.
Camera
Supports 8 platforms including
iOS, Android, Windows Phone and
BlackBerry
Titanium
Open source
http://www.appcelerator.com/
Uses JavaScript language for
development
Supports iOS and Android only
Uses native UI - retains device’s
native look& feel
Performance increased as app
does not rely on devices web
National VET E-learning Strategy 2012 – 2015
Steep learning curve for
advanced JavaScript
Framework does not follow
a create once deploy
anywhere mentality
Finished output will not run
in a web browser on a
different platform
Any JavaScript libraries
which reference the DOM
Page 8
VIC162 Mobile checklists enabling authentic workplace assessment
browser
More native functionality
available
Rhodes
Open source
http://rhomobile.com/
Uses native UI - retains device’s
native look& feel
Application structure constraints
allows for rapid development
does not work in this
framework
Built on the ruby
programming language we
are not familiar with.
Smaller community.
Follows the Model View Controller
methodology of programming
Supports 6 mobile platforms
Full IDE for development
Write once, deploy everywhere
methodology
Figure 1: Mobile development platform comparison
Our key considerations were that the chosen development platform would best support usability,
accessibility, sustainability, and be able to leverage existing skills where possible. Out of this
research, we chose the PhoneGap development platform as the code is a cross-platform standard
and can be deployed to a wide range of platforms such as desktop computers and tablets with
minimal changes; it can be integrated with existing JavaScript frameworks such as jQuery; the
system is a free, open source system; and it uses a common Application Programming Interface
(API) set to access phone functionality e.g. Camera.
Aspects of the project’s planning, design and development that worked well include:



Establishing a relationship early on with the key stakeholders of the project was an integral
project to project start up to ensure that their input could be considered prior to any design
or development. This was done through a series of centre presentations and meetings with
key stakeholders. Furthermore, the partnership with the teaching centres and with our
industry partner AMCA ensured that the assessment tasks maintained their integrity from
both a learning and industry perspective.
Research into the different alternatives for Mobile development that would integrate with
Moodle, and collaboration between the instructional designer, multimedia designer,
multimedia developer and LMS Coordinator to come up with the best strategy forwards.
Software and hardware: We purchased an Apple laptop to enable us to develop the
application for apple devices including iTouch, iPhone and iPads. We also purchased
Balsamiq Mockups to establish the user experience with HTML prototypes. This was utilised
after sighting its benefits in the Access to Skills project ‘AS091 E-profiling Smart Phone App’.
National VET E-learning Strategy 2012 – 2015
Page 9
VIC162 Mobile checklists enabling authentic workplace assessment

Our project has not been tested with the National Broadband Network (NBN) however it has
been tested when connected to a Wi-Fi/3G network which proved sufficient for the transfer
of submissions to the Moodle Learning Management System (LMS). The presence of the
NBN will increase the extent of uptake of this application in remote locations where a
standard network connection is insufficient or not available to send media rich assessments
from mobile phones to Moodle.

Conducting preliminary training in December to teaching staff around the mobile
development and how the templates could be used for teachers to create their own
assessment tasks. Positive feedback was received.
Challenges encountered and strategies used include:
 Development of four different assessment tools. The initial assessment tools agreed to by
our key stakeholders and designed through our instructional designer, including the Job
Safety Analysis (JSA), Digital Story, Knowledge Test, and Observation Checklist, proved to be
too dissimilar to develop within the timeframe of this project due where code couldn’t
simply be repurposed once the first assessment tool was developed. The main difficulties
were confronted when considering the knowledge test which needed quiz functionality
separate to the Moodle quiz tools and a final grade to be submitted; and the observation
checklist which would require building a completely different teacher view. As a result, the
project team met to determine what assessment tools could be achieved within the project
timeframe, while still remaining integrity from a pedagogical perspective. The resulting
assessment tools chosen still include the JSA, however the digital story was changed so that
it would be a Procedure/Process, and two new assessment tools were developed including a
Work Diary and Demonstration Checklist. While the demonstration checklist is similar to an
observation checklist, it can be completed on the learner’s device. Before submission to
Moodle however, the teachers need to enter an authentication code on the learners device
as an added layer of verification that the learner had completed the work that is being
submitted.
 Building a mobile application that will dual publish to Android and Apple devices. Initially we
had the understanding the PhoneGap would automatically be available across different
platforms once the initial coding was complete, however we since discovered that we still
needed to do some separate coding for specific mobile platforms. For this reason we are
developing just for Android and Apple devices to capture a large proportion of the smart
phone market that our students use.
 Gaining internal approval to receive access to the Apple Developer License had taken a
significant amount of time however this was eventually approved in May.
 Due to the size of video files we were not able to identify a suitable way to locally compress
and store this information across a range of devices. As a result, the output only allows
learners to complete assessments with photos or text as input.
 Level of support for mobile development – The infrastructure team noted that there are
slight differences in support for HTML5 on Android and IOS.
 A large learning curve required for the infrastructure build. We had previously developed a
number of blocks for Moodle, but hadn't yet attempted developing integrated activities. See
the “Lessons Learnt” section below to view how these challenges were overcome.
National VET E-learning Strategy 2012 – 2015
Page 10
VIC162 Mobile checklists enabling authentic workplace assessment

Resourcing with competing projects and priorities
Many competing priorities for the infrastructure team; including serious issues with our
Moodle LMS; and staff leave impacted timelines severely. The best way this could be
managed was escalating the potential impact on this project and negotiating timelines
internally where possible. Any risks to meeting project deliverables were also escalated on a
few occasions to the strategy.
 The team had found that many of the eStandards/mStandards/WCAG2.0 guidelines did not
apply as we were building a fixed application for iOS and Android devices, not a mobile
website (e.g. fixed dimensions, etc), and for this reason attempted to make use of as many
native features available as possible including the accessibility features. Considerations
from these guidelines that are relevant to our development, including making content
distinguishable (e.g. colour contrast – text on background), being mindful of sizing, layout
and limitations on smart phones (e.g. sending large photos across network – 3G or wifi).
However, it must be stressed that these guidelines cannot be adhered to as closely as a
standard website.
a) We found that WCAG 2.0 compliance. could be addressed in the following areas:
o Distinguishable - adhered to with contrasting colours. This also provided to be an
appropriate level of contrast for learners who wished to use accessibility features on
the iPads and invert the colours being displayed.
o Seizures
o Readable - ability to change font-size in settings
o Input Assistance - validation of fields; only displaying relevant buttons and screens
when necessary
o Compatible – the screen resolution for Android & iOS is identical; build robust
connections using Moodles Web services as much as possible to minimise rework in
future versions of Moodle
b) WCAG 2.0 compliance could not be addressed or is not applicable in the following areas:
o Text Alternatives
o Time-based Media
o Adaptable – the screen width would automatically resize depending on the device
being used however the interface was primarily designed for the iOS and Android
devices.
o Keyboard Accessible - native keyboard functionality was used for text input
o Navigation- this will be different to a standard website due to the nature of the app.
we will be following Moodle’s pathway to access units and checklists
o Predictable
 We had concerns around using HTML5 since it doesn’t fall within the e-standards but are
required because we are specifically targeting iOS and Android devices, and packaging into a
fixed application using PhoneGap, which requires HTML5/CSS3/JS. The HTML however can
be validated for W3C compliance prior to building and packaging into PhoneGap which is
the process we used to be as compliant as possible. Furthermore, it was approved that for
our project the strategy was happy for us to use HTML5.

Lessons learnt include
National VET E-learning Strategy 2012 – 2015
Page 11
VIC162 Mobile checklists enabling authentic workplace assessment
1. Improved estimation of development time required.
Due to a significant level of unknowns and not a large level of expertise in the area of mobile
development, the project team uncovered areas that required a significant amount more
development time than first anticipated, particularly in relation to Web Services for Moodle,
and creating new Moodle resource types for the templates we are creating. By undertaking
this process however our team has built the capability to streamline app design and
development into the future.
2. The approach to Moodle integration needed to be changed a few times. Moodle provides a
lot of functionality, but learning the framework independently is difficult; some aspects of
development are very well documented and accessible, others less so. Approaches that led
to the final approach include:
a) Create a self contained mobile submission module
Our programmers started off writing a self-contained module, which could provide its own
web services the way Moodle mandated. We were able to do this, and get a file uploaded to
the private files area. Moodle file handling from version 2 however is complex due to
security and permissions issues. Moodle only provides a web service to upload to the
private files area, so much additional work would have been required to effectively copy file
submissions into the self-contained module area. Furthermore, even if this was achieved
integration with the gradebook would need to be added.
b) Create a plug-in that modifies the ‘onlinetext’ submission type in the assignment module
We decided to dispense with the self-contained module approach and create an assignment
submission plug-in. The benefit of this is that the gradebook integration was already built in
and the user interface was familiar to the teachers. In effect, the plug-in simply added
functionality to submit an assignment from a mobile device. Instead of using the file upload
functionality, we chose to use the onlinetext area and a simple POST request which allows
images and any other multimedia to be integrated using data Uniform Resource Indicators
(URIs). Data URIs allows a single upload to contain many images obviating the need for
Moodle file handling and additional JavaScript coding. This was trialled with staff and did
work however from a programming perspective it wasn’t ideal. The plug-in had to assume
the presence of an onlinetext submission type for the assignment which assumes that the
material is being entered in an online text editor, not submitted whole via a post request. So
the plug-in needed to violate some of the object oriented constraints of the Moodle
architecture. Also, because the onlinetext view cleans up the editor text-- this removed the
images provided in the data URIs, so this had to be overridden at a higher level. The
approach was not good enough-- Mobas was trying not to duplicate existing functionality,
but it was breaking too many code boundaries.
c) Create an independent “Mobas” assignment plug-in with web services functionality
The final approach was to create a new assignment submission type that operates
independently of all existing assignment types. The new plug-in type was to be modelled on
the online text assignment type to take advantage of the existing code framework, however
would allow for any code changes to remain local to the plug-in. This means that Mobas is
really a way to submit an assignment which can include files and online text. This is ideal
because it is flexible, uses what the teachers already know about assignments, and
integrates cleanly with Moodle. As a submission type, there is no requirement that all
students use it – the teacher can provide online text or file uploads as alternative options.
This thereby makes this also the most accessible option where learners can complete via
National VET E-learning Strategy 2012 – 2015
Page 12
VIC162 Mobile checklists enabling authentic workplace assessment
their mobile phone or via a standard PC through whatever means is most appropriate to
them.
3.2
Running the project
The project and application “Mobas” was trialled with 25 learners and 3 teachers from the areas of
Automotive and Electrical/Refrigeration using iPads and the Moodle LMS. The automotive students
trialled with assessments from the unit: AURC270103A Apply safe work practices. The
refrigeration students trialled the application with assessments from the unit: UEENEEE101A
Apply OHS regulations, codes and practices in the workplace. These units were co-delivered
with a professional development officer from our project team who delivered the learning and
assessment tasks in the context of using the application, while the teacher explained the
assessment tasks and their relevance to the assessment requirements that each learner needs
to fulfil for the associated OH&S units.
Aspects of the project that have worked well included:







Usability and navigation of application: Mobile delivery worked well from iOS devices
including iPads and iPhones. Our learning designer worked closely with the multimedia
developer and programmer to ensure that the user experience would reflect Moodle logic
as much as possible and this was represented in student comments. By aligning the
navigation pathway to Moodle, learner feedback advocated that the user experience was
intuitive and simple to use. Learners also indicated that they were pleased with the
simplicity around attaching and uploading photos where other methods that they had tried
in the past were complicated and time consuming processes.
Reliability of infrastructure: No errors were encountered on the application side in each of
the learner trials and all functionality was working as intended.
Wifi connectivity: All devices connected to the wifi with ease and assessments were
therefore transferred onto the Moodle LMS in the presence of this wifi connection.
Ease of access to application: Learners and teachers could login to the application using
their student username and password, meaning that there were no issues with login on the
day.
Accessibility: One of the learners who would typically use accessibility features on a PC to
cater for his colour blindness accessed the accessibility features on the iPad to invert the
colours. He has indicated that these features were sufficient for his needs and that there
wouldn’t be any need to embed local features within the application to cater for his needs.
Simple to facilitate: Learners received a small induction on how to use the application prior
to completing the assessment tasks. After this induction they were able to add pictures,
complete and submit their assessments with a small amount of guidance.
Small learning curve for teachers: The bulk of the teacher training could be completed in 3045 minutes per teacher due to the small amount of changes evident on the teacher side. For
teachers to create a Mobas assignment type, they still were required to follow the standard
process for setting up an assignment in Moodle and modify a few extra settings. Teachers
were very positive about this and indicated that they would consider changing a large
majority of their existing assessments so that they could be completed either through the
application or through a standard PC. The gradebook simply has an extra column added for
Mobas type assessments, again requiring a minimal level of additional learning for teachers
National VET E-learning Strategy 2012 – 2015
Page 13
VIC162 Mobile checklists enabling authentic workplace assessment


who can mark and provide feedback to students as they usually would for any assignment
type.
Opportunities for the workplace and industry: Learners and teachers both indicated that
they would see this application as an enabler for completing assessments through a just in
time basis in the workplace or simulated workplace environment.
Feedback from trials and usability testing to improve the integrity of the application,
assessment tools, and Moodle assignment plug-in
Challenges encountered and strategies used include:
 Trialling the code while under development
We didn’t want to move the code into production on our live version of Moodle, so we
allowed access to our development server by students and teachers. From a learner and
teacher perspective the interface is nearly identical however the drawback is that the
student submissions would not go into the unit that they are formally enrolled in without
the need to migrate this across to the live server. This thereby meant that teachers were
unable to retain a record of student submissions and teacher feedback for any trials for this
project.
 Arranging trial dates when development was still being done
Trial dates needed to be arranged with the apprenticeship areas and AMCA well in advance
of the foreseeable completion of the development phase of this project. As a result time
was spent briefing stakeholders including learners around trials that in the end weren’t
possible. Within the timeframe of this project we have therefore not yet been able to secure
a date for trial with AMCA however intend to conduct this outside of this project’s scope to
ensure client relations and expectations are maintained.
 Finding trial groups representative of all learners undertaking the proposed units - All
learners found that using the device was exceptionally straightforward and intuitive to use,
and suggested that it would be something they would like to use as a just in time
assessment tool in the workplace. The students indicated that it was simple to understand
what the task required and it was also straightforward to add images. Teachers however
indicated that the groups that this application was trialled with were the ‘good’ groups in
terms of their familiarity in using mobile technologies, and suggested that more contextual
help would be useful for learners with low levels of digital literacy, language barriers, or
other learning differences.
 Difficulty in retaining assessment requirements -Initially the application presented learners
with the assessment description on the screen prior to where the assessment needed to be
answered. While logically this is the way that the information needs to flow and means that
from a design perspective one screen doesn’t present too much information, learners felt as
though there was a degree of cognitive overload in remembering all the elements that had
to be addressed in the response. Our learning designer and multimedia developer worked
together to modify the way that this information is available so that as any stage in the
assessment task a learner can refer to the assessment information icon at the bottom of the
screen to review what the initial instructions were.
Lessons learnt
As a result of the challenges and successes encountered, the key lessons would be as follows:
 Cater for and trial with a mix of learners – This could involve use a mix of random and direct
selection methods for the trial groups to achieve a representative sample of the learner
National VET E-learning Strategy 2012 – 2015
Page 14
VIC162 Mobile checklists enabling authentic workplace assessment


3.3
population. It could also involved conducting a greater level of usability testing from a
broader range of users including teachers, learners, instructional designers, educators,
industry, and employers.
Add more contextual help and instructional text to improve accessibility for all learners
Maintain open communications with the key stakeholders and project team to stay abreast
of project risks in terms of timelines and deliverables.
Wrapping up the project
By participating in this project the Blended eLearning Solutions team has not only increased its
knowledge and skills in mobile application development and use of Moodle, but there is also
greater transparency around the requirements for app development from a learning design,
multimedia design/development, and infrastructure/programming perspective. As the team
worked very closely together to identify where efforts should be placed, a greater cultural integrity
was established including respect for the broad range of skills needed to be accessed across the
team to fulfil the requirements of a project of this proportion.
The team significantly improved their skills in the following areas:

Learning design:
o Re-contextualising standard assessments into mobile assessments
o Designing the user experience and sequencing assessment for mobile interfaces
o Designing assessment templates for mobiles that are achievable from a development
perspective
o Creating mobile storyboards using Balsamiq mockups that can then be converted to
interactive multimedia templates.

Multimedia design:
o Greater understanding around how to design a mobile interface from the
perspectives of screen space, elements required and when these should be
displayed, and to ensure that the key purpose of the information being displayed is
clearly communicated.
o Integrating e-standards/m-standards into app design and development

Development:
o Local caching on mobile devices
o Accessing local services on mobile devices
National VET E-learning Strategy 2012 – 2015
Page 15
VIC162 Mobile checklists enabling authentic workplace assessment
o Using the PhoneGap development platform for iOS and Android app development in
conjunction with HTML5 and JavaScript.
o Accessing Moodle’s web services to develop plugins so information can be sent and
received from mobile phones to Moodle.
o Modifying the assignment plug-in to receive information from mobile phone devices
o Creating new Moodle resource types for each template
o Creating videos with closed captioning using Amara.org

Project management
o Escalating and managing project risks as they occur to the relevant people.
We anticipate that Box Hill Institute can benefit from the following cost-savings as a result of the
following activities:

Internal ongoing staff implementing this project as a part of core business in both the
development and delivery components of our job

Extending the basics of this projects through utilising existing ongoing staff skills and passion
to continue to build around existing funded and Institute wide strategic projects

Embedding the project outcomes into potential new commercial opportunities to share the
new app with our industry partners

Teachers will be able to design and facilitate their own mobile assessment tasks using the
assessment templates for “Mobas” following teacher training. There is a level of
functionality in which teachers can modify existing templates. We anticipate that BeLS will
continue to provide support to our teachers as they develop their online presence. . A
critical aspect of the this development or extension of their online presence will be training
and the ultimate use of the mobile assessment tools and templates to:
o Train teachers on better design and development of assessment tasks
o Training teachers on re-envisioning how the delivery assessments in the workplace
o
Upskill students and change their mindset as to what is an assessment task and how
they can submit this
o Upskill teachers and students with digital literacy

Streamlined communication and feedback between the teacher and learner through the
mobile application and Moodle gradebook.
National VET E-learning Strategy 2012 – 2015
Page 16
VIC162 Mobile checklists enabling authentic workplace assessment

Reduction in printing and associated administrative costs in managing paper based
submissions and audit trails.

Learners can complete assessments outside of the classroom streamlining the assessment
process with workplace based learning.

Greater level of engagement and retention from our learners
Beyond the National VET eLearning Strategy Participation and Skills funding period, this project will
be funded/supported in the following ways:

Beyond the National VET eLearning Strategy Participation and Skills funding period, this
project (MOBAS) will be funded/supported in the following ways:

Internal ongoing staff implementing this project as a part of core business in both the
development and delivery components of our job

Extending the basics of this projects through utilising existing ongoing staff skills and passion
to continue to build around existing funded and Institute wide strategic projects

Embedding the project outcomes into potential new commercial opportunities to share the
new app with our industry partners
As outlined in our original application we anticipate the following mechanisms will support the
sustainability of the projects outcomes to other parts of our organisation and project partners:
o Up-skilling all professional development staff in BeLS in Mobas so that they can conduct
teacher training internally to ensure ongoing use of assessment tools
o Up-skilling all learning designers in BeLS of the capabilities of Mobas so that they can
integrate Mobas assessments where applicable into internal assessment strategies.
o Development of 6 more assessment templates for Box Hill Institute teachers that will enable
teachers to re-use assessment templates for mobile assessment in an ongoing capacity
across any units of competence.
o Testing of the Mobas plug-in with each upgrade of Moodle to ensure its longevity
o Our industry partner AMCA hopes to showcase the possibilities of this application to
relevant industry associations in the belief that it will help improve efficiencies and learner
outcomes from an industry training perspective.
The project will be expanded to other participants, cohorts or jurisdictions in the following ways:
National VET E-learning Strategy 2012 – 2015
Page 17
VIC162 Mobile checklists enabling authentic workplace assessment
o The templates and deliverables for this project will be able to be used / applied to other
units and courses within other trade areas as well as for other industries. This will provide a
mechanism for the quicker adoption of e-learning resources.
o Upskilling of students will encourage the adoption of newer technologies and develop and
understanding of their wider uses in workplace and educational activities.
The capacity of knowledge and skills around mobile learning, mobile development and design, and
synchronisation with Moodle provides opportunities from which Box Hill Institute can extend upon.
This project and deliverables will be of benefit to Box Hill Institute in the following ways:
o Opportunities to further explore and integrate mobile learning and assessment into a range
of courses across the institute.
o Increased feasibility to deliver assessment to remote learners.
o Opportunities to compel external and internal stakeholders to adopt flexible workplace
based models
o Greater engagement and linkages between learners, teachers and employers by enabling
authentic learning and assessment experiences.
o Higher levels of student retention with closer alignment to students preferred modes of
delivery and learning styles
o Capacity to negotiate customisations of the program to meet industry needs.
National VET E-learning Strategy 2012 – 2015
Page 18
VIC162 Mobile checklists enabling authentic workplace assessment
1.
The results
1.1
Main project outcomes
The work undertaken in this project has allowed our team to build capacity in developing mobile
applications for workplace based assessment. We anticipate that this can be extended to provide a
greater number of teachers with the capability to design, develop and deliver their own mobile
workplace based assessments. We also anticipate that this will provide learners targeted through
this project with a greater level of digital literacy skills and engagement as a result of undertaking
assessment in a simulated workplace or workplace based environment.
The following outcomes were intended for learners:





Increased employability skills
An increased understanding of digital literacy and responsible digital citizenship
Development of e-skills to broaden their learning window and increase the amount of
training time in the workplace.
An increased amount of kinaesthetic and social learning that will engage the students
Increased employability and digital literacy skills through opportunities to undertake
workplace learning and assessment in locations that have no access to wifi/3G networks
The following outcomes were intended for employers:




Increased engagement with Box Hill Institute and better capacity to negotiate
customisations of the program to meet their learners needs
Development of a ‘learning culture’ with Industry partners
Increased learner engagement within the student body
Greater flexibility for the students and greater choice around where and how they learn
The following outcomes were intended for Box Hill Institute:



Higher completion rates
Build capacity of knowledge from which to extend upon around use of PhoneGap to better
integrate video and audio with Moodle for a wide range of mobile learning initiatives.
Opportunities to showcase project success to industry partners and compel other Centres
to adopt a more flexible workplace base model of learning
These outcomes were partially met and would be possible with ongoing communications, training,
and use of the Mobas application for assessment. Recent discussions with centre managers and
teaching staff from our three key trades areas indicated that they are positive to embed mobile
assessment using Mobas into their training and assessment plans and have expressed interest in
setting up training for teachers in targeted areas. Outside of the trades areas, our senior educators
have all been introduced to the application and as a result, our Animal Sciences teacher have
expressed interest in the customisation and integration of assessment tasks for their distance and
flexible learners. We anticipate that across the next 5 years these outcomes will be met.
National VET E-learning Strategy 2012 – 2015
Page 19
VIC162 Mobile checklists enabling authentic workplace assessment
1.2
How the outcomes were measured
The outcomes were measured in the following ways:

Learners
o Feedback was gathered from learners through verbal discussion following their trials.
o Observation into the way that learners were operating the application was also
undertaken in conjunction with a final analysis of learner assessment responses.
o The successful transfer of assessment tasks into the Moodle learning management system
demonstrated that the application was working as intended.

Teachers
o Teachers provided feedback in their assessment responses to the Mobas trial with their
students, and also through a verbal discussion after their learner trials.
o Teachers conducted a review around the quality of learner responses to assessment tasks
in Moodle and were satisfied that the responses aligned to the quality that would be
typical from an alternative method of assessment.
o Observation by our professional development officer as to how teachers responded to
using the application and supporting Moodle side was undertaken.

Other stakeholders:
o Discussions with senior educators at the Senior Educator’s network meetings
o Discussions with Laura Steedman from AMCA
o Meetings with teaching centre managers

Usability testing
National VET E-learning Strategy 2012 – 2015
Page 20
VIC162 Mobile checklists enabling authentic workplace assessment
Application interface – Learner examples
Application homepage
Example: Unit home
JSA Template
Example: Learner response
National VET E-learning Strategy 2012 – 2015
Page 21
VIC162 Mobile checklists enabling authentic workplace assessment
Demonstration checklist template
Example: Learner response
Procedure process template
Example: Learner response
National VET E-learning Strategy 2012 – 2015
Page 22
VIC162 Mobile checklists enabling authentic workplace assessment
Work diary template
Example: Learner response
National VET E-learning Strategy 2012 – 2015
Page 23
VIC162 Mobile checklists enabling authentic workplace assessment
2.
The lessons learnt
In addition to the lessons learnt at each stage of the project, Box Hill Institute recognises the
following lessons as a result of this project:
1. The skills of critical path analysis when the scope continues to grow and evolve – how we
can still deliver in the timeframes previously set and within the budget
2. Coming to terms with not being able to produce a “platinum” outcome due to emerging
challenges previously un-scoped
3. This tool has the capacity to be able to quickly change pedagogical approaches. Due to the
cutting edge nature, adoption has been quicker than anticipated due to this project being
developed at the right time, right place and right need reflecting the student and employer
satisfaction feedback results
4. Passionate staff often will add to the project scope through sheer excitement and a
commitment to delivering everything that was scoped without us truly understand the
scope we had designed.
5. The PhoneGap component of the project ended up being less than 10% of the total project
outcomes. The true work was in:
a. Assessment tool design and selection
b. Template design and development
c. Unit of Competence of final selection
d. Access Moodle’s web service to be able transfer information between the
application and Moodle
e. Local caching on mobile devices was quite problematic
f. Accessing local services on mobile devices
g. Upskilling current staff
h. Understanding more about the Moodle framework and how to nest develop without
impacting on Moodle core code.
6. Audit is God and having assessment task in the cloud is an exceptionally risky long term
decision in the current audit environment.
7. The team found it challenging to adhere to the m and e-standards as we are targeting
PhoneGap to create an application for iOS and Android, not a compliant website. Where
possible we have complied with WCAG requirements, and the m and e-standards however
the standards that we have referred to for a greater level of guidance are those supplied by
National VET E-learning Strategy 2012 – 2015
Page 24
VIC162 Mobile checklists enabling authentic workplace assessment
Apple and Android for their respective platforms. As an outcome we would recommend for
the standards to be extended to include specifications for app development.
2.1
Key successes
This project has allowed BHI to report the following key successes:

Robust discussion at the executive level at how this tool can be used to drive pedagogical
changes particularly in the Trades areas to radically improve the percentage of workplace
based assessment

Teaching teachers to design more flexible and innovative assessment tools by stealth.

Promoting the use of assessment tools that live within Moodle to assist the Institute in the
area of audit

Promoting the use of mobile assessment tools based on students feedback so we better
meet their needs

Promote greater communication and transparency between the employers and institute

Promote greater communication and transparency between the employers and their
apprentices/ trainees

Provide a point of differences with both our Industry partners and potential new or existing
clients

Upskill current staff so they had both design and development skills in application
development in this emerging area, opening the possibilities for future app development at
BHI.

Upskill the whole staff into understanding that mobile assessment and delivery is a core
business we move into this emerging era created an understanding that projects of this
magnitude and complexity needed all team members to equally contribute to achieve the
outcomes required

This project delivered better working relationships across teams that often didn’t need to
work closely together and built a trust and respect that will service us well into the future
for other more complex projects.

Further extend our relationship with AMCA and the refrigeration industry

From a design perspective, increased capability to design storyboards for mobile devices
using Balsamiq mockups for wireframe to prototype and aligning the navigation structure to
Moodle LMS.
National VET E-learning Strategy 2012 – 2015
Page 25
VIC162 Mobile checklists enabling authentic workplace assessment

The positivity by teachers around the simplicity of the re-usable templates purporting that
uptake will follow.

Extended opportunities workplace assessment and remote learners
What we would do differently
1. In future, smaller more feasible project deliverables would be designed – even though we
thought we had done that this time
2. Designing a project that is cutting edge can often lead to changing scope. At BHI we have to
learn to be less ambitious
3. As a result of the research undertaken through this project, we are not fully convinced that
a specific app is the best way to go. A web application could potentially service just as well
and prove to be more accessible to learners than an application. Alternatively, as a potential
future project to capitalise on the aforementioned lessons learnt could be to design an
application like Creative Book Builder on iOS to submit files in an ePub format. This would
allow the student to download a template specifying what was required, add multimedia
assets and text to it as required, and upload it via one of Moodle’s repositories, such as
Dropbox. This workflow would be more flexible, and more robust in the long term (as it
requires no Moodle development), but also more complicated for the teacher and students.
2.2
Suggested improvements
Following feedback from learners, teachers, and the design team the following improvements
would help to improve the usability of this application:

Contextual help within question types for learners

The ability to locally compress and attach a range of other media including photo and audio

Teacher view for viewing submissions/providing feedback locally on their own mobile
devices

Notifications to be embedded on the learner side indicating outstanding submissions,
attempted submissions, and those needing to be updated following teacher feedback

Employer/supervisor view to allow an added layer of transparency and authenticity to
assessments completed in the workplace. This can also be of assistance in assessment
validation and moderation processes.

Development of a greater range of assessment template types
National VET E-learning Strategy 2012 – 2015
Page 26
VIC162 Mobile checklists enabling authentic workplace assessment
3.
Passing it forward
3.1
Knowledge transfer
Box Hill Institute will submit to the National Vet eLearning Strategy the following:
1.




12 formative assessment templates for the units:
MSAPMOHS200A Work safely
AURC270103A Apply safe work practices
UEENEE101A Apply OHS regulations, codes and practices in the workplace
VU20874 Maintain essential safety measures
2.




Four customisable assessment tools including:
Job safety analysis
Demonstration checklist
Procedure/process
Work diary
3.



Mobile application “Mobas” for iOS (Apple) and Android enabling:
Learners to complete mobile assessments, attach text and photos
Teachers to provide an authentication code for the demonstration checklist
Teachers can view and grade mobile submissions in the Moodle gradebook
4. Implementation guide for Moodle system administrators detailing:
 how install the Moodle plugin that will update settings within the Moodle assignment
module
 how to integrate the Mobile application with Moodle
 how to provide authentication keys for the demonstration checklist
 how to link to student LDAP accounts
5. Baseline data surveys completed from an organisation/learner and student perspective.
6. Teacher and learner guides
7. Case study documentation
8. Case study video with transcript file
9. Final acquittal information
10. Handover template
National VET E-learning Strategy 2012 – 2015
Page 27
VIC162 Mobile checklists enabling authentic workplace assessment
10.1 Mentorship
The package that we are delivering to the strategy will include the tools and information required
for an organisation to setup their own version of the “Mobas” assessment app and supporting
Moodle interface.
Organisations that will benefit from this package include those:

that already have the Moodle learning management system, and have a high proportion of
learners with Apple /Android phones;

wishing to update assessment strategies so that they include a greater level of mobile
assessment can contact Box Hill Institute for implementation advice and support across the
next two years;

needing to improve workplace based learning and assessment;

needing to rapidly change staff delivery and assessment strategies to survive in this
challenging Victorian environment;

wanting to promote Moodle as a critical audit documentation tool;

wanting to use disruptive technologies to promote innovative that is cost viable and can
change as organisations future direction.
National VET E-learning Strategy 2012 – 2015
Page 28
VIC162 Mobile checklists enabling authentic workplace assessment
More information
For more information on the Box Hill Institute
Project director:
Pauline Farrell
Executive Manager
Blended eLearning Solutions
Phone: 03 9286 9344
Email: p.farrell@bhtafe.edu.au
Project manager:
Julianne Seaman
Student Experience Manager
Blended eLearning Solutions
Phone: 03 9286 9344
Email: j.seaman@bhtafe.edu.au
For more information on the National VET E-learning Strategy:
Email:
Website: flexiblelearning.net.au
With the exception of the Commonwealth Coat of Arms, the Department’s logo, any material protected by a trade mark
and where otherwise noted all material presented in this document is provided under a Creative Commons Attribution 3.0
Australia (http://creativecommons.org/licenses/by/3.0/au/) licence.
National VET E-learning Strategy 2012 – 2015
Page 29