TDDD89
Lecture 3. Study methods
Method
• Study design, data selection
• What is necessary, what is best?
• Examples of methods
• Methods in Engineering & Science
• Replicability,Validity
• Work in a context
Study design
Method
Results
Observations
Description
Measurements
Quantitative data
Can I trust you?
”Resource use for system-level
tests can be reduced by 50%”
Because ...
Step 1
”I studied how people used
resources through observations
and semi-structured interviews,
which told me that resources
were used 20-30% of booked
time”
Step 2
”I triangulated interview results
with measurements of resource
use and found resources used
approximately 32% of the booked
time”
Step 3
”Based on the measurements, I
devised a system to notify people
of unused booked resources”
Step 4
”Resources were used at 45% of
the booked time during a 3 week
trial period”
Different types of
methods
• Qualitative methods: establish concepts,
describe a phenomenon, find a vocabulary,
create a model
• Quantitative methods: make statistical
analyses, quantify correlations, ..
Human-Centered
methods
• Interviews
• Surveys
• Focus groups/group interviews
• Observations
• Talk-aloud sessions
• ...
Method choice?
•What do you want to find more about?
•Identify the stakeholders (users,
costumers, and purchaser)
Identify their needs
•
Interviews
• Structured or unstructured?
• Group interviews (focus groups) or
individual interviews?
• Telephone interviews
• Use open-ended questions, ex.:
•Do you like your job?
•What do you think about your job?
•Active listening
•Record the interview
•Plan and schedule for that!
Interview analysis
•Transcribe or not?
•Categorize what has been said (encode)
Observations
•Understand the context
•Write down what you see, hear, and feel
•Take pictures
•Combine with interview
•Ask users to use systems if availabe
Usability evaluation
•System Usability Scale (SUS)
•Post-Study System Usability Questionnaire (PSSUQ)
•Heuristic evaluations
•Eye tracking
•First click Testing
•…..
•
System usability scale
(SUS)
Usability performance
measurement
•Task success
•Time (time/task)
•Effectiveness (errors/task)
•Efficiency (operations/task)
•Learnability (performance change)
Describing a method
Don’t write a diary:
• ”To implement a Flux controller, I first
needed to learn about Flux”
Write that which convinces someone
you have done a good job
• ”The Flux controller was evaluated using the
Flux controller evaluation protocol [1]”
Most important
Engineering method vs
scientific method
Method questions
Engineering aspect
Scientific aspect
Have you used
Is it clear that the
Can I trust your
techniques &
evaluation will
work?
methods intended provide the kind of
for the task?
answers we seek?
Are all techniques
Can I build on your
and methods
Can I replicate the
work?
employed described
study?
in sufficient detail?
Case Study
• investigates a phenomenon in a real-world
context,
• with multiple sources of information,
• uses predominantly qualitative methods to
study a phenomenon
Quantitative studies
• Uses statistical analyses of some empirical
data
• Randomization of subjects
• Blocking (grouping) subjects based on
confounding factors
Factors
• That which may correlate with (and
possibly cause) an effect
”How does SCRUM affect product quality as measured by
number of bugs?”
”How is code quality affected by the choice of
programming language?”
”How understandable is a design document when
creating procedural and OO design,
based on good/bad requirements?”
Analysis
• There must be a null hypothesis which we
can test our data against
• One factor, two treatments: t-test, MannWhitney
• One factor, several treatments: ANOVA
• Two factors: ANOVA
Statistics
• There are separate statistics courses, but..
• Separate correlation and causality
• Unless >= 95% confidence, there is no
correlation
• Confidence only part of statistical power
(confidence + effect size + sample size)
Software Engineering
Requirements
Acceptance
Testing
Architecture
Design
Implementation
System Testing
Integration
Testing
Unit Testing
Products
Requirements
Study techniques
Interviews
Surveys
Focus Groups
Architecture
Design
Experiments
Implementation
Proofs
Process
Study techniques
Surveys
Planning
Interviews
Development
Focus Groups
Testing
Case Studies
Integration
Deployment
Maintenance
Other types of theses
Problem
Company/Team
Algorithm
Algorithm
Algorithm
Development
Development
Process
Development
Process
Process
Performance
results
Performance
results
Discussion
Discussion
Theory
Observation
cause-effect
construct
Cause
Treatment
Effect
treatment-outcome
construct
”Do all arrows exist?”
Output
Discussion, example
Does agile development lead to higher quality code?
Fewer
defects
Agile dev
External validity
SCRUM/
No
SCRUM
Bugs Construct
reported
Validity
Internal validity
Work in a context
• Are the authors aware of how this work
will affect people?
Work in a context
• ”Code quality traceability for improved
accountability in agile teams”
Work in a context
• ”Random Indexing to improve article
classification on online media outlets”
Work in a context
• ”Bayesian learning can successfully detect
driving conditions to control NOx emission
profiles”
Work in a context
• ”Visual 3D structure recognition for
automatic targeting systems: A feasibility
study”
Work in a context
• ”Parallel processing for oil reservoir
simulation to improve yield”
Summary
• Use a selection of methods to answer the
types of questions you have
• Consider the possible outcomes, applicability
and requirements for your method
• A method should ideally be internally and
externally valid
• You will need to be aware of ethical aspects,
and your work in a context