Postgraduate Research Methods
Day Conference
Doing Real-World Quantitative Research
Welcome!
Doing real-world Quantitative research
Martin Thirkettle
Overview
• Why do real-world research?
• What is real-world research?
• Example
– Development of radiology expertise
• Using on-line platforms to conduct research
– Qualtrics and cognitive testing
Why do real-world research?
Real-world research
• Provides solutions & insight to problems faced by people outside
academia
– Valuable to industry and large organisations
– Valuable to individuals
• Has immediate impact
• Engages the public with research
• Avoids questions of ‘artificiality’ of lab-based research
– Studying how people actually are, rather than under rarefied conditions
• Can also provide theoretical insight or test
– Tractable questions
What is quantitative real-world research
• Not Psychophysics
– But still rigorous
• Observation of real-world phenomena
• Challenge is to create measures
of performance around behaviour
– Often tasks lend themselves to certain measures

• Often can use real-world variations in the task to investigate factors
– Uncover cognitive abilities and strategies by isolating contributory factors.
• Could lead back to the lab
– Confirmatory experimentation
Development of Radiology Expertise
• Expert radiologists perform visual diagnosis from radiographic
images
– Consistent accuracy is essential
– Performance levels are unattainable to novices
– Expertise is developed
• Classic example of expertise in domain of visual cognition
• Question remains of how this expertise is developed
Previous Cognitive Research into Radiology
• Focussed on differences between experts and novices
– Experts are better in all aspects
• More abnormalities correctly identified
• Fewer false identifications of abnormalities
• Faster identification of abnormalities (Potchen 2006, Norman 1992)
– Experts spend less time scrutinising images
• Can diagnose images from single glance (Drew et al 2012)
– Experts look at less of the image than novices
• Eye tracking shows less area covered and fewer fixations (Manning et al 2006)
Quantifying Radiological expertise
• Task developed to capture visual cognition skills of expert
radiologists.
• Two part task reflecting real-world task demands
– Abnormality Identification
– Abnormality Localisation
• Allows skill to be quantified and titrated
• Performance measured by
– Correct identifications & false alarms
• Signal detection approach
– Error of abnormality localisation
• Raw distance measure
Creating the Radiology Task
Link
Creating the Radiology Task
• Library of 150 radiograph images
– Compiled from Sheffield Children’s Hospital
• Each image has reference answer
– Each image pre-assessed by consultant radiologist & trainee
• Rated difficulty of assessing each image (3 levels)
• Provided ‘absolute’ locations of abnormalities in images
• Library is large enough to allow testing using subsets of the
images
– Allows repeated testing of trainees without repeating images
Piloting the Task
• Task coded in Matlab
• Responses compared to reference set
• Pilot set of 2 consultants and 2 trainees
– Compared to trainees, consultants on average:
• Were faster (14.1sec vs 18.9sec)
• Raised fewer false alarms (0.2 vs 0.5)
– More accurate identifying abnormalities
• Were more accurate localising abnormalities (23.6pix vs 55.9)
Converting task to the web
• Benefits of web-platform
– Dramatically easier data collection
• Consultants are spread across the country
• Trainees and Consultants are very busy
• No-one has to come to the lab!
– Much faster collection of full set of data
– Potentially wider data collection
• Easy to collect data from willing cohorts far away
Qualtrics
• Web platform
– Surveys etc.
• Lots of tools
–
–
–
–
Data collection/securing
Cohort splitting
Metadata collection
Etc
• But can be used for more than
survey studies
Using Qualtrics for cognitive testing
• Qualtrics supports images
– Identify task very simple to
implement
• Qualtrics supports Javascript
– Location task implemented using
custom Javascript
• Records click locations within image
and response times in standard
Qualtrics response box
• Allows full task to be delivered on
Qualtrics platform
Using Qualtrics for cognitive testing
• Qualtrics allows the full breadth of lab-based cognitive testing to
be moved online
– Many experimental designs possible ‘out-the-box’
• Qualtrics natively supports images, multiple choice, records RT, etc.
• Qualtrics has randomisation and counter-balancing tools built in
– Qualtrics support for custom Javascript will allow any task design to
be implemented
• Platform supports it, not Qualtrics Ltd (don’t expect help!)
Using custom code in Qualtrics
Using Javascript in Qualtrics
• Location task more complicated as code
takes responses rather than just displays
image
• Simple Javascript code works fine
– http://www.codecademy.com/ or similar can
show you the basics
• Very easy to import code from elsewhere
– Google, stackoverflow.com
• Online tools also allow you to test your
code
– http://jsfiddle.net/
– Important to test with different browsers
• Difficult bit is getting javascript to talk to
Qualtrics
– Need to load
Qualtrics.SurveyEngine.addOnload
– Allows you to write to Qualtrics
variables – the ‘this’ structure
Using Qualtrics for cognitive testing 3
• Using the survey flow editor
very simple to block and
randomise the presentation of
task components
– 3 Blocks presented in random
order
– 10 images chosen at random
from sets in library
• Qualtrics even ensures all
images presented equal number
of times
Next steps for the Radiology task
• Establish ‘Gold standard’ reference answer set
– Collate answers from a number of consultants
– Use this data to compare images in set against each other
• Recruit trainees to complete task
– Cohort from each of the 5 year training program
• snapshot of development of expertise
– Follow incoming students through their studies
• Needs a grant to support – 5yr+ study
• Return task to the Lab
– Create eye-tracking version of the task to build on behavioural data from
web-based version
Eye-Tracking and Radiology
• Same task as behavioural study
• Add eye-tracking data to established
measures of performance
• Provides implicit measure of
performance and cognitive strategy
• Data produced include:
• Number of saccades
• Fixation duration
• Scan path around image
Real-world outcomes of Radiology study
• Quantify visual diagnosis skill
• Better assessment of radiologists
– Trainees’ progress more tightly monitored
• Better assessment of radiology training
– Success of training interventions can be measured
• Which exercises produce change in performance? When?
• New training interventions can be developed to focus on facets of
expert skill
• Better performance of radiologists
– Improvement in patient care
Quantitative Research in the real-world
• No reason not to take cognitive experimental approaches
beyond the lab
• Challenge is to develop a measure which quantifies behaviour
without interrupting scenario
• Very rewarding
– Instant pay-off
• Only going to become more popular
Thanks
• Dr Amaka Offiah
• Dr Tom Stafford