Developing and Assessing the
Research Skills of Students
Dr Said Al-Sarawi
Research Skill Development and Assessment
ALTC Project Member
Education Research Group of Adelaide (ERGA)
School of Electrical and Electronic Engineering
University of Adelaide
A seminar at Edith Cowan University
Outline










Motivation
Students and staff issues
RSD Framework in Australia
Benefits of using RSDF
What is RSD Framework?
Case Studies – From 1st Year to Master Level
Other Dimensions of RSD Framework
RSD at the Program Level !
RSD in conclusion
Discussion and future work
Motivation (1/2)

PhD completion rate were doubled for students who had
participated in undergraduate research (Bauer &
Bennett, 2003)
 Performance based research funding (old RQF, ERA)
 Performance based learning and teaching support –
Learning and Teaching Performance Fund (DEST, 2006)
 Undergraduate research has been associated with
higher level of student satisfaction and generic skills
development (Kardash, 2000)
 Factors affecting skills implementation are (Lucas et al,
2000)



Scepticism of the message, the messenger and its vocabulary
The skills demanded lack clarity, consistency and recognisable
theoretical base
The skills are dependent on discipline area
Motivation: Student and Staff issues (2/2)
Staff
How can I develop and
assess students’ research
skills?
Is there a systematic,
explicit approach that can
be used?
Student
What is meant by
Research and what skills
are needed?
How I can acquire and
develop these skills?
Is this issue only limited to How are they relevant to
the engineering discipline? my career aspirations?
How can I remove the
“subjectiveness” from the
assessment?
How will my skills be
assessed?
RSDF in Australia
• The University of Adelaide (John Willison)
− Human Biology (Eleanor Peirce & Mario Ricci)
− Electrical Engineering Masters by Coursework (Said Al-Sarawi and
Brian Ng)
− Clinical Nursing (Frank Donnelly)
− Petroleum Engineering (Steve Begg)
− Introductory Academic Program (Richard Warner)
− English (Joy McEntee)
− Dentistry (Vicki Skinner and Leonard Crocombe)
− Oral Health (Sophie Karanicolas and Cathy Snelling)
− Software Engineering (Li Jiang)
− Veterinary Science (Susan Hazel)
RSDF in Australia (Conts.)
• Macquarie University (Psychology: Judi Homewood)
• Monash University (Business Ethics: Jan Schapper;
Sue Mayson: Business; Glen Croy: Tourism)
• University of Melbourne (Business Law: Eu-Jin Teo)
• University of South Australia (Introduction to Tertiary
Learning, 2008: Rowena Harper)
The facets of student research
In researching, students:
1.
2.
3.
4.
5.
6.
embark on an inquiry and so determine a need for
knowledge/understanding
find/generate needed information using appropriate
methodology
critically evaluate information/data and the process to
find/generate
organise information collected/generated
synthesise and analyse and apply new knowledge
communicate knowledge and the processes used to
generate it, with an awareness of ethical, social and
cultural issues.
(Willison & O’Regan, 2007)
Learning to Frame Research
Questions
Level 1
Facet A) Embark on Inquiry
Respond to questions / tasks
arising explicitly from a closed
inquiry.
Facet E) Synthesis, analysis, application
Ask questions of clarification /
curiosity.
Learning to Frame Research
Questions
Level 2
Facet A) Embark on Inquiry
Respond to questions / tasks
required by and implicit in a closed
inquiry.
Facet E) Synthesis, analysis, application
Ask relevant, researchable
questions.
Learning to Frame Research
Questions
Level 3
Facet A) Embark on Inquiry
Respond to questions / tasks
generated from a closed inquiry.
Facet E) Synthesis, analysis, application
Ask rigorous, researchable
questions based on new
understandings.
Why use RSD approaches? (1/2)


Benefits of using the RSD for students
(according to Eleanor Peirce and Mario Ricci,
Medical Sciences, Uni of Adelaide)
Their research skills in our course have
improved.
 They understand much more clearly what is
expected of them.
 They know exactly where they need to
develop, thanks to feedback.
Benefits for Lecturers (2/2)
(according to Eleanor Peirce and Mario Ricci, Medical Sciences, Uni of Adelaide)
We can give feedback on assessment tasks
more accurately and efficiently; we can give
the same quality feedback with less writing,
and faster.
We can get a much better idea of where our
students are from a quick analysis of the RSD
results than from a detailed analysis of
standard marks.
We can easily match assessment tasks with
course objectives, and course objectives with
the University’s Graduate Attributes.
Case Study 1: Human Biology - 1st Year
Read the two short then complete tasks 1 and 2.
O Week RSD
Task 1:
I
II
Integrate the information presented in the two
articles to write your own dot-point notes
A
on the worksheet attached. To do this:
•
Identify 3-4 key ideas from the articles
•
Use these key ideas to formulate
headings and underline each
•
Make bullet-point notes and list them
F
under these headings.
•
After each point, indicate its source, i.e.
whether the idea came from article 1,
article 2, or both
•
Provide a title that embodies the content
of your notes.
Task2:
Which of the two articles do you consider to be the better source? On what
characteristics/features of the article have you based your choice?
Case Study 1: Human Biology - 1st Year
Case Study 1: Human Biology - 1st Year
Literature Research Skill Stream
O Week RSD
I
Lit Week RSD 1
Lit Week RSD 1
II
I
II
I
III
II
Small Group Inquiry
III
I
A
A
A
A
F
F
F
F
II
III
IV
Laboratory Research Skill Stream
Lab RSD 1
Lab RSD 2
I
I
II
A
A
F
F
Individual Open
Inquiry
II
I
II
III
IV
A
F
Field and literature research
Semester 1 Semester 2
Novelty in the engineering case
The differentiation points:
 Clearly state the need to research skill
development
 The students are required to identify a
gap in knowledge for each of the
chosen topics (the unknown)
 Rigorous literature research
Case study 2: Photonic and
Communication – Master Course

Masters (coursework) course at EEE






Demographics




Course run by senior colleague in EEE
2 units of lectures/exams
1 unit of literature research project
Students seek supervisors individually
Topic chosen by student, but requires approval by supervisor
Almost all are international students; usually East Asian background
Technically capable, but generally lack prior experience in
conducting research
Goal: produce a high quality review paper on chosen topic
Mostly negative experiences in 2005



Cohort lacked basic literary research skills
Unstructured approach towards project
Low quality final review papers
Process Details
1.
•
•
•
2.
•
•
•
3.
•
•
Initial diagnostic task
Supervisor supplies two technical papers
Different levels (magazine, journal) for contrast
Students summarise and compare the key points from both
sources in one structured, bulleted list
Detailed supervisor feedback in written form
Knowledge accumulation phase
More articles added to the reading list
Continually add to an organic structured, bulleted list
Strategy on further reading
Student applies critical evaluation on suitability of sources
with supervisor input
Fortnightly workshops for group presentations & discussions
Supervisor supplies feedback; optional: external advice
(CLPD)
Writing phase
Student writes review paper based on list
Supervisor feedback on first draft approx a week before
submission
Outcome and Evaluation

Outcomes from 2006


Total of 6 students plus1 external (industry) student as control
Qualitatively, much improved papers compared to 2005
•

Student perspective
•
•


Coherent structures, logical arguments, conciseness, respect of
referencing practice
struggled to cope with demands of research among the pressures
of regular coursework
English as second language remained a great barrier
Framework “matrix” useful for quantitative assessment
Unexpected benefit – reduction in plagiarism


Turn-it-in software reported major improvements from 2005
Reports tend to be on conservative side.
Case 3: Final Year Project (in Progress)
“We have been concerned about a degree of disconnect between the
desired leaning outcomes and the assessment methods used in these
projects.” HoS of School of EEE, The University of Adelaide.
The set of project deliverables are:
Semester 1
•
•
•
•
Semester 2
•
•
•
Formative literature assessment
(diagnostic value?)
Proposal seminar
design document/interim report
Critical design review (peer review)
final report
final seminar
project log books/wiki/blog/email log
… etc
RSD: Useful First-year to PhD?
Variety of RSD approaches
Five distinct approaches have emerged in the use of the RSD so far:
•
•
•
•
•
Rubrics base to assess the profile of skills for each student, as
demonstrated by Human Biology rubrics. this is the most
common approach, and is used by most disciplines.
A lock-step approach, whereby students are kept 'in formation'
and progressively and corporately develop one level of skills at a
time. This is shown by the Nursing RSD rubrics.
Grading with a specific grading, by incorporation into SOLO
taxonomy to define grading within a specific level set, as used by
Dentistry.
Skill Evaluation, to evaluate the skills and levels required by
existing assessments.
Identification of research skills of higher degree students,
Masters by Research and PhD students and/or their
supervisors/advisors to locate their present skill set and plot
future directions and development needs.
(SOLO: Structured Observed Leaning Outcomes)
Other Dimensions of RSD
•
Degree of Autonomy
•
Degree of Academic Rigour
•
Degree of Conceptual Demand
•
Degree of disciplinary knowledge
required
•
Status of knowledge being pursued
RSD Framework at Program Level !
No studies that consider student outcomes of the explicit
development and assessment of research skills over a
whole undergraduate or masters-by-coursework program
Undergraduate Level
• Bachelor Oral Health, Adelaide Uni, AU
• Bachelor of Media Studies , Adelaide Uni, AU
• Bachelor of Science, Adelaide Uni, AU
• Bachelor of Nursing, Adelaide Uni, AU
• Bachelor of Business, Monash Uni, AU
• Bachelor of Science (Psychology), Macquarie University, AU
Postgraduate Level
• Master of Engineering (Advanced) (Electrical), Adelaide Uni, AU
• Bridging Program for International PhD students, Adelaide Uni, AU
• Masters and PhD (Nursing), PhD (Nursing)
• Trinity College Dublin, Ireland
• Master of Business, Monash University, AU
RSD in conclusion






Provides the Big Picture and relates this to the
assessment details for course coordinators,
lecturers, tutors, and especially students
Informs assessment-first curriculum redesign
Same ‘facets’ for multiple assessments, various
levels
Explicit &Transparent assessment criteria
Coherent & Incremental skill development
Revisited & (potentially) Cyclic Conceptual
Framework
26
Discussion and Future work





How to integrate the framework into other
Programs?
How will the implementation of RSD framework
affect academic workload?
How to monitor and assess student’s progress?
How can this be implemented for larger class
sizes?
How can the framework adopted for non-literary
research skills?
Acknowledgement
This session was funded by an Australian
Learning and Teaching Council Grant
RSD Web Site
http://www.adelaide.edu.au/clpd/rsd
References
1.
2.
3.
4.
Bennett, N., Dune, E. & Carré, C. (2000). Skills Development in Higher Education and
Employment. (Society for Research into Higher Education & Open University Press).
Stevens, C. & Fallows, S.J. (2000). Integrating Key Skills in Higher Education:
employability, transferable skills and learning for life. Routledge, ISBN 0749432659.
Lucas, U., Cox, P., Croudace, C. and Milford, P. (2004). “Who Writes This Stuff?”:
Students’ Perceptions of Their Skills Development. Teaching in Higher Education, 9(1),
55-68.
Willison, J.W. & O’Regan, K. (2007). Commonly known, commonly not known, totally
unknown: A framework for students becoming researchers. Higher Education Research
and Development, 26(3).