SURF X4L Evaluation Report
Richard Booth / Dave Shearan
August 2004
Table of Contents
Introduction .............................................................................................................. 3
Re-purposed Learning Materials ........................................................................... 4
Use of Virtual Campus ............................................................................................ 5
Questionnaires .......................................................................................................... 6
Stages of Evaluation ................................................................................................. 7
Process for Staff ..................................................................................................... 8
Process for students................................................................................................ 9
Evaluation comments............................................................................................... 9
Staff Responses ...................................................................................................... 9
Student Responses ............................................................................................... 10
Wider implications ................................................................................................. 10
Appendix ................................................................................................................. 12
Completed Staff Evaluations ............................................................................... 12
Completed Student Evaluations ........................................................................... 13
2
Introduction
The purpose of the SURF X4L evaluation is to report findings of the
evaluation process involving staff and students at Shrewsbury College of Arts
& Technology. We wish to document the findings of the on-going work
associated with the Surf X4L project and raise awareness of using and
producing learning materials (re-purposed and new materials).
The learning materials created by the content owners (staff, tutors) and the
end user (the learner, students) will give an insight into the problems,
successes and work involved the creation and development for use by
students via the virtual learning environment (Virtual Campus). The process
includes developing materials, using tools to content package and the transfer
to the VLE.
The results of the completed evaluation forms will feedback views from the
producers of the learning content (tutors, staff) and the end users, (‘the
learners’, students). From this we wished to gauge the learner’s reactions to
using such material. Were they visually exciting, user friendly? Could they be
used as part of a lesson or with tasks set by the tutor? Did they have a place
with the traditional methods of teaching / learning already in use in the
classroom. Did they have any real benefits for the learner?
From the tutor’s point of view, we wished to understand how they felt using
and creating learning materials. Was the process arduous? Did you find it
exciting and did it link in the ways of teaching already in use. It was interesting
to work with tutors and learners to see their perception of the learning
materials, how they should be used and if use of such materials was the way
forward to ‘learning for the future’.
Re-purposed Learning Materials
Ranges of materials and packages have been produced throughout the
course of the project. A selection of such material has been highlighted for
use of evaluation. We assisted in the repurposing of existing materials for staff
in the areas of Construction, Engineering, Hospitality and Catering.
Fig2: courseGenie document
The creation of these materials developed from a number of sources. Existing
course content created by staff, repurposed material from staff content and
including FENC and NLN materials where appropriate. These materials were
used as supplementary use and for quiz and assessment. Use of tools to
produce content and package the learning materials such as Reload and
courseGenie have been developed. We felt that staff found courseGenie easy
to use with most staff being familiar with the Microsoft Word package, which
courseGenie incorporates its use.
We had to account for a slight set back at Scat, as a member of staff in
construction who is involved with the evaluation process had his laptop stolen
with the materials he had produced to use with the students. We hoped that
learning materials could be re-developed in subsequent weeks. Another
member of staff in Media / Photography had damage to his computer and is
awaiting work completed on this machine, so that the photography materials
can be used for evaluation purposes. The construction materials were redeveloped and a good number were available for the evaluation process. We
decided not to use the Photography content, as there were only a small
number of packages created and were out of date at the time the evaluation
commenced.
We have worked closely with key members of staff to generate course /
learning materials including quiz creation. Some staff have felt comfortable
using familiar packages such as Microsoft Word and PowerPoint. We have
helped them to take the material development one stage further by
introducing the courseGenie tool. It has given the packaged content a more
‘complete’ look and feel.
Hospitality & Catering have been keen to develop a complete course
structure, organising work units for each semester with tasks and quizzes built
in at each stage of material development.
In the areas of Construction and Engineering, the tutors creating the content
have produced a series of interactive PowerPoint shows that work very well in
their original format. By re-purposing these packages with courseGenie we
can demonstrate the difference in look and feel of the material and how
integration within a virtual learning environment. The functionality of the
packages, incorporating quizzes and content navigation.
Good use has been made of the courseGenie package to produce learning
content for inclusion on the college’s virtual learning environment. Once all
content, text, images, and quizzes are saved in Word, page headings,
captions and tables can be added to contain text and images, converting the
whole document to an html generated package. Other options include
converting to an IMS object package.
Once the package has been converted, courseGenie automatically places a
zipped and unzipped version, which you can upload to a VLE. The result is a
learning package with a table of contents (which can also be omitted) and full
navigation.
Use of Virtual Campus
Once learning materials had been established, evaluation user accounts were
created to house the material. Some of the content was already in use on the
VLE system with students. Others were using the learning packages for the
first time.
Staff were trained on the use of using learning materials, help with
repurposing and use of the Virtual Learning Environment. We decided to
upload the content for the members of staff but also demonstrated the
processes for their own use of this facility.
Once a folder structure for each area had been created and all the learning
content transferred to the VLE system, we devised a timetable to induct the
specific students needed for the evaluation process. We gave an introduction
regarding the SURF X4L project and why the need for their input. We showed
them where the materials were situated, went through the evaluation form and
what was expected.
5
Fig3: courseGenie package in Virtual Campus
Questionnaires
The questionnaires were produced to collate feedback from staff and students
in different ways:
 How they found the use of the ‘Virtual Campus’ (our virtual learning
environment)
 Production of learning materials / process of creating content (staff)
 Using the learning materials (students)
 Usefulness of such materials / related to course / topic, etc
 Pedagogical implications on teaching and learning
 Other factors governing production of and using created / repurposed
learning materials and packages.
 Wider use of repurposed learning materials / packages in teaching and
learning
 Blended / incorporated use of such materials in traditional lesson
delivery
The results from all the completed evaluation forms were very encouraging.
We only had 5 forms from teaching and support staff, but over 50 completed
from students from the various curriculum areas targeted and who agreed to
take part in the project.
6
Stages of Evaluation
The curriculum areas chosen for the evaluation process were Construction,
Engineering and Hospitality & Catering. A number of students from these
areas have worked through the materials developed for courses and work for
the SURF X4L project and have completed the evaluation form, which we
have produced.
Most of the completed forms have indicated positive feedback and
constructive comments.
“It was all quite easy to use, all of the links were clearly labelled and allowed me to
navigate the program easily.”
“Easy to use with drop down menu of folder and easy to access. Good information on
all different aspects of carpentry, on-site warning signs e.t.c. Found it good to use”
“I found this virtual campus very good and very informative it was quite easy because
I have learnt most of this in the class room with lecturers, but this will be very handy
for exam revision.”
“The presentation on Corrosion was excellent, and suited my style of learning
perfectly, for my current assignment.”
Fig1: Evaluation Form
Members of staff in the chosen curriculum areas have completed the staff
evaluation process detailing their experience of producing and re-purposing
content and using new technology such as authoring tools and the virtual
learning environment (VLE).
The evaluation process indicated how staff found producing such learning
materials, and pedagogical factors with using resources and learning content,
tools, packages, etc. It is crucial to determine the teaching and learning
outcomes from implementing and creating learning materials. From a students
point of view, it will be a very useful exercise to gain how they viewed the
7
materials, their use within the classroom and how useful they found the
content / electronic resources to their course and this new way of learning.
The results have been collated back into a database, from which a report on
the numbers can be produced. The results from the multiple-choice questions
will give a point score. The questions that the participant can give comments
will be listed on a separate form.
Members of staff logged onto the Colleges Intranet so that we have a name to
correspond with their comments and opinions. For students it was not so
crucial and the fact that students do not have separate login to the Academic
Intranet. They imputed their course code or title onto their completed forms.
The final question sets to be used for staff / student evaluation was agreed at
the 11th Surf X4L Project Management meeting on 20/01/04.
Shrewsbury College included the evaluation forms on the college intranet
system so we have an on-line evaluation tool.
A set of interview / peer observation questions will be compiled so that we can
gain further in-depth assessment of the whole evaluation processes, with
particular reference to the pedagogical use of such materials, understanding
and implementation to teaching and learning. This will be aimed at both users
of materials:


The staff that create the learning package materials and use in the
delivery of their lessons
The students who are the end user. Highlighting their new way of
learning, encompassing a wider learning experience.
We originally planned to commence first round evaluation processes by the
end of January 2004. This was re-scheduled and we began during the period
February / March 2004.
Process for Staff
Agreement was sought on which topics, modules and learning packages were
needed. Outcomes were to be re-purposed materials from existing online
resources, off-the-shelf packages and new content and existing course
materials produced by tutors.
Ranges of materials were created to form learning packages for courses in
the areas identified (Construction, Engineering, Hospitality & Catering)
including presentations, module tasks, supported learning materials and
assessment end-tests.
Content folders and structure were devised on the main menu of the Virtual
Campus (VLE) for each area. Once in place, the learning package materials
were uploaded to the subject and package areas of the Virtual Campus.
These were added to the menu structure for access by staff and students.
8
We inducted staff in the use of the evaluation forms available on the intranet
system at the College and how to complete the form.
All tutors including a learning mentor, involved in the process completed the
online response form giving feedback on how they found the process,
successes, failures, and problems throughout development of the materials.
We also sought to gauge findings of perceptions of online materials available
to re-purpose (NLN, FENC, etc).
Process for students
Once learning packages were housed on the Virtual Campus, we inducted
specific student groups from each area to the project and evaluation process.
We worked with groups from Engineering (approx. 12-15 per group)
highlighting why we were carrying out the evaluation and the wider context of
the SURF X4L project. The need for re-purposed learning materials and such
of supplementary online resource materials and the future of learning from
this new way of presenting their course materials.
We asked them to login to the Virtual Campus and browse through the
materials available in their area. Specific logins were created so they would
only see their course area on the system. After working through the materials
and assessment tests we asked them to work through the evaluation form
available on the student academic intranet to give feedback on how they
viewed the materials, their usefulness for their lesson delivery and additional
comments they had.
This process was repeated for Construction and Hospitality & Catering
students. The latter took part by the course tutor within their lesson and
feedback given at the end of the session.
To date we have had approximately 50 students complete the evaluation form
with excellent feedback comments.
Evaluation comments
Staff Responses
“Incorporation of such materials into your scheme of work…. As backup and
support for subject matter and as a form of reinforcement and reference”
“I will utilise the VLE as being a resource which they can use to enhance their
learning and experiences on that subject”
“I will be happy to use this method of teaching for my students well done”.
9
“This would incorporate different learning styles. Very useful for revision and
recap of difficult topics such as the timber technology”.
Student Responses
“I feel the virtual campus is a good idea if you need to research any methods
of work with in your class activities at home or in the LRC”
“I found this virtual campus very good and very informative it was quite easy
because I have learnt most of this in the class room with lecturers, but this will
be very handy for exam revision”.
“I think this type of learning aid should be used in more lessons as it is of
great help and easier than some hand outs we receive”.
“ I found that the materials i used were very useful because it made it easy to
get on with my work easily and efficiently without constantly pestering the
teacher”.
“ Finding the information on the web site was easy. The simple layout of the
site made finding the information easy and enjoyable”
“We should use the virtual campus for more of the subjects that we are doing
on our course”
Wider implications
The evaluation process will form an important part of the final report of the
SURF X4L project. It is from this report that we can gain opinions from the
creators of re-purposed / learning materials and the end user of such
materials, ‘the learner’. These two groups of people are the important groups
that will benefit from these materials and new ways of teaching and learning.
Long-term practical issues can be addressed from the comments and
opinions from those who incorporate and use such materials in their lessons
and the learning outcomes from this.
In a wider context, the development for the College has created great impact
on the ILT element of teaching & learning and this has been highlighted in the
recent Ofsted inspection report (May 2004). Greater understanding and use of
ILT plays an important part in teaching and learning development and has
been evident at Shrewsbury College.
The pedagogical impact of using re-purposed learning materials and creation
of content packages was very well received indeed. Staff saw the virtues
incorporating such materials into their existing teaching methods. The blended
learning approach seemed to be favourable with the majority of staff and this
view was widely held by the students.
10
Students found the materials to be visually exciting and helped significantly
with revision and supporting what they had learned in the lessons. The
materials also gained impact when individual study was needed during
college time and private study at home.
The work completed highlighted the need for new ways of incorporating
teaching and learning, the pedagogical impact that these learning materials
and content packages can achieve. Tutors who assisted with development
work; including use and creation of materials for their courses saw the benefit
of delivering these through the VLE. From the learner’s point of view, it gave a
whole new way of working with new technology. Visually exciting, interactive
materials, ‘all content in one place’ seemed to help students with learning and
their comments from the completed evaluation forms reinforced this view.
11
Appendix
Completed Staff Evaluations
How many people found producing learning materials to be
2
2
1
0
an easy task
quite straight forward, but needed support
quite hard and needed help and guidance
very difficult, needed a lot of support
The incorporation of such materials into your scheme of work? 5 entry
How many people found 're-purposing' learning materials to be
3
2
0
very easy
ok, but needed support
difficult and needed lots of support
How many people found using the tools available to be
2
3
0
easy
ok, but needed support
difficult
They found using the 'Virtual Campus' system was
3
2
0
easy
ok, but needed support
difficult
What aspects of the 'Virtual Campus' did you find easy/hard? 5 entry
The teaching and learning outcomes from using such materials
3
2
0
would be very useful
would be useful
would not be very useful
User: PatrickS
The Incorporation of such materials into your scheme of work?, and How?
Yes - as and when time permits!!! As backup and support for subject matter and as a
form of reinforcement and reference
What aspects of the 'Virtual Campus' did you find easy/hard?
Finding way round initially - not all areas intuitive
Further Comments:
Time consuming - software slow to load up and some students would think it was not
working
User: StuartR
The Incorporation of such materials into your scheme of work?, and How?
12
By integrating the material into resources available, for instance when the students are
studying on a unit that may incorporate safe working practices I will utilise the VLE
as being a resource, which they can use to enhance their learning and experiences on
that subject because of the material that we have placed there.
What aspects of the 'Virtual Campus' did you find easy/hard?
It was quite complicated to log-on to the system.
Further Comments:
Thanks for the support!
User: BillL
The Incorporation of such materials into your scheme of work?, and How?
Yes Good group work available all students were active throughout the session.
What aspects of the 'Virtual Campus' did you find easy/hard?
I could not navigate around the timber yard without assistance due to the arrow key
directing me back to start?
Further Comments:
Very good session. I will be happy to use this method of teaching for my students
well done.
User: ClareC
The Incorporation of such materials into your scheme of work?, and How?
This would incorporate different learning styles. Very useful for revision and recap of
difficult topics such as the timber technology.
What aspects of the 'Virtual Campus' did you find easy/hard?
Good clear pictures/diagrams
User: CameronF
The Incorporation of such materials into your scheme of work?, and How?
Each lesson has PowerPoint, and worksheets. Placed on the VLE in advance or
immediately after allows for better planning and pacing of workload.
What aspects of the 'Virtual Campus' did you find easy/hard?
Sometimes a bit slow. Difficult to upload materials (cumbersome method - needs a
block upload). Easy to get students using it.
Further Comments:
As an extra tool it is a useful thing to have.
Completed Student Evaluations
13
How many people found 'Virtual Campus' to be ..
37
10
1
Very Easy
Ok, but needed help
Hard, needed alot of Help
Aspects of the 'Virtual Campus' found to be easy/hard? 44 entry
How many people found using the learning materials to be
44
6
0
very easy
ok, but needed support
difficult and needed lots of support
What aspects of the materials you liked/disliked? 48 entry
How many people found that the materials used in the lesson were . .
36
8
1
3
easy
ok, but needed help
Hard, needed a lot of Help
of no use
How many people would use similar / same materials again.
44
4
would
would not
How many people found electronic materials useful?
43
3
3
yes
no
Don't know
There are over 31 Comments available.
Surf X4L Student Evaluation form - Comments
The original spelling and formatting has been retained in all case.
All results have been amalgamated.
Overview of questionnaire
Which aspect of the 'Virtual Campus' did you find easy/hard?
o I liked the fact that it got in close to the work so that you could see clearly.
o The start was easy but the rest of it was challenging.
o the pictures were useful to elaberate on the text.
o That you cannot log in from outside the collage.....
o I liked all of it, the layout was good and easy to use.
o i liked them beacuse they were useful and it was easy to understand
o it was easy
o i liked all aspects of the materials
o its good eseay to use
o not alot of detail
o we liked the activities
o i liked the saftey signs programme
o i LIKED THE QUESTIONAIRES
o I liked the diagrams and bold text.
o The learning materials are easy to understand and the information is worded
so you dont get confused by huge words
14
o easy to use with drop down menu of folder and easy to access. good
information on all different aspects of carpentry, on-site warning signs e.t.c.
found it good to use
o needs more information
o I liked the timber process
o it explains everthing
o i liked the various programs that you can use to look at the information
o it was all ok
o It was all good no bad points
o it was all good and i think it has helped me learn something!
o I liked this because it was vry easy to use and understand.
o i like the fact that it actually teaches u somthing unlike most questionares.
o The things i liked the most was on this program was the safety sign questions
i understood and enjoyed but i would of like more questions on all the
subjects in the program.
o Again, well presented. Very useful and informative. Easy to absorb.
o THE RELATED THEORY NOTES WERE VERY HELPFULL IN GIVING
THE BASIC INFORMATION WHICH COULD THEN BE EXTENDED
VIA RESEARCH IN THE PCC TEXTBOOKS. ALSO SOME OF THE
NLN MATERIALS I USED WERE VERY GOOD,INTERESTING TO USE
AND VERY INFORMATIVE
o being able to pull up all the materials easily.
o Having the forms and tables available to me ready made was a great help
instead of making my own.
o all the graphs and things in purchasing costing and control helped me very
much in the subject i even used it to help me with coursework.
o I found that the materails i used were very useful because it made it easy to
get on with my work easily and efficiently without constantly pestering the
teacher.
o I liked the fact that tables and other materials that would have taken a long
time were already waiting for you on the interanet. I liked the set out and
design of the programme it was easy to use. I also liked the fact that it made
it very business like and proffesional.
o also clear to read and gave you alot of information clearly.
o I found then all useful, and the notes were detailed and helpful.
o not having to do things like tables. We saved a lot of time and therefore we
can produce a better more proffesional document
Explain what aspects of the materials you liked/disliked?
o it was easy exsept for finding your way aroundat first.
o The start was easy but the rest of it was challenging.
o getting to the different side shows was easy as well as moving throught the
shots.
o The power point presentation was good and explained the points well...........
o it explains everything and directs you through the program clearly and is easy
to understand
o It was all quite easy to use, all of the links were clearly labelled and allowed
me to navigate the program easily.
o ok beacuse i required little help.
15
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
the sign quetions were easy
it was all quite easy
every thing easy to use
the whole thing was simple to use
useing a pc
it wasnt quick enough and didnt work correctly
bad computers, send more!!
liked go on the tour and questions
understanding the questions was hard
FINDING IT WAS EASY AND USING IT WAS ALSO
I found it easy to access all parts of the program
all of the information sources available
I FOUND THE QUESTIONAIRES EASY
The way to choose a subject from a list and the way information was
presented.
It is easy to locate things and is also easy to access the information that you
are looking for
easy to use with drop down menu of folder and easy to access. good
information on all different aspects of carpentry, on-site warning signs e.t.c.
i found it easy because it was sypull
all of it was easy
the way the information was presented and how to select the info that you
want to look at. doing the tests and the audio
It was easy to use because everthing you needed was on the screen.
reading the words because im a bit stupid and i carnt read.
I found this virtual campus very good and very informative it was quite easy
because I have learnt most of this in the class room with lecturers, but this
will be very handy for exam revision.
safety signs.
it was all well explained and easy to use
easy signs hard timber
Well i found the the Virtual Campus very easy understand to find things and
get to them i needed no help to use the program.
Well Presented. Easy to browse, and fun to use.
all of it was easy.
USING THE OUTLINE TABLES FOR RECORDING AND DISPLAYING
DETAILED COSTING AND CALCULATIONS, ALLOWING LOTS OF
INFORMATION TO BE READ EASILY FOR GENERAL REFERENCE,
RECORD AND MONITORING PURPOSES, THE END RESULTS
LOOKING VERY BUSINESS LIKE AND PROFESSIONAL
Everything was relatively easy to use, but i found it was harder to access
when the intranet was down for two weeks!
surfing about and finding things were very easy because they were well layed
out
I found that the way the lay out made it easy to find information for my use,
also that all the resources in the 'central resources' heading automatically
loaded up in the appropriate microsoft programme.
I found it very helpful with my purchasing and costing unit. It mad it helpful
with my workload having tables already designed, for that choosen piece of
work.
16
o VERY CLEAR AND EASY TO UNDERSTAND, when we had someone to
go though it with us. the layout is very good and clear.i liked how we had the
tables where already produced for us this saved alot of time.
o Easy to know where to find things. because all the work is done on the
computer it looks tider.
o finding the infomation on the web site was easy. The simple layout of the site
made finding the infomation easy and enjoyable
Further Comments:
o OK, but needs alot more information on the web site to make it worth while,
and that you cannot login from outside the collage.....
o i feel the virtual campus is a good idea if you need to research any methods
of work with in your class activites at home or in the LRC but i feel in stead
of explaining how to do certain Procedures on the web, you should be able to
access folder or leaflets in the certain classes to explain how you should do it.
o very good.
o good system
o wouldn't mind more of a challenge in the activities
o SUPERBLY DESIGNED,ESPECIALLY FOR A PERSON LIKE ME,NOT
REALLY THE BRIGHTEST COOKIE IN THE JAR I MOST DEFINITLY
USE AGAIN
o very good system of materials
o I think this is a good sourse of in formation.
o it was very informative and interesting and i would use it again to look up
information for my course.
o it was very good
o A very informative and helpful programme
o it was a good thing to learn about timber and machinery and the lesson was
good.
o A lot of the material (photo's and text) were used in two or three places in the
corrosion sectors. Even though it took a while to load, I think all the
information in one presentation would be sufficient. The presentation on
Corrosion was excellent, and suitedmy style of learning perfectly, for my
current assignment. I did find a small number of spelling mistakes. Overall I
thought this site was excellent.
o AS A PILOT SCHEME I FELT IT WORKED WELL, THERE ONLY
BEING A FEW OCCASIONS WHEN INFORMATION HAD NOT BEEN
SUCCESSFULLY PROOF READ / USED ( E.G. THE CHOCOLATE
QUESTIONNAIRE DESIGN FAULT ONLY CAME TO LIGHT BY
ACTUALLY USING IT ). A BONUS IS STUDENTS BEING ABLE TO
ACCESSS IT FROM HOME.
o I think this type of learning aid should be used in more lessons as it is of
great help and easier than some hand outs we recieve.
o very good helps me very much so
o I think although the programmes on the computers are very useful for my
course. The computers that we are using seriously need updating due to the
fact that they are to slow and are always crashing, and loosing work that we
have been working on.
o this has been a grate help in the lesson and hopefully they can further the use
of the campus in other lessons.
17
o can use it on the internet at home.
o we should use the virtual campus for more of the subjects that we are doing
on our course
18