Quantifying the Qualitative
Stephen Billett, Education and Professional Studies
Progression
Why quantify the qualitative?
Considerations for research practice and
dissemination
An example ……………………
Gathering quantitative with qualitative
Improving reliability
Tabulation of qualitative data
Why quantify the qualitative?
Qualitative inquiry suffers from claims of:
weak and descriptive analysis, and
analytical procedures that limit its potential to
inform others about the phenomena under
investigation.
Data can be very time and resource intensive to
gather and then not able to be fully used
Risk of qualitative researchers’ audience will only be
peers.
Informing policy or practice often demands greater
certainty, scale and rigour of the kind promised
by quantitative inquiry.
Ways forward
Consider how qualitative-based inquiry can be
undertaken and its data analysed to address these
concerns
This includes:
potentially adding rigour and greater precisions to
quantitative analysis
using informants’, researchers’ and readers’ time
more effectively
Example: Interviews with workers to identify contributions to
their learning
Purpose
Understanding how workers initially learnt their vocational
knowledge and what role workplace experiences play in that
learning.
Attempting to understand how ongoing skill development occurs
through work activities.
Data gathering
Interviews to identify how workers learnt their vocational
knowledge using their experiences
Analysis
Categorisation and frequencies of responses
Interview data
Section 1. Initially acquiring skills
Determine / confirm the nature of their area of skill and inform the respondent
that this will be the focus of the questions.
1a. What does it mean to be a skilled (whatever they do)?
‘what is”
1b. How did you initially acquire the skills for that job?
1c. What was useful about this way of acquiring skills?
1d. What were its limitations?
1e. What do you think is the ideal way of acquiring skills for your type of work?
‘What should be’
Section 2. Continuing development of those skills
Think about the last time you had to solve a problem in your job for which you
did not have the necessary skills or knowledge.
2a.
How did you go about learning what you needed to know?
2b.
Who or what in the organisation helped?
2c.
Did it work, if so why?
2d
When has this type of learning failed to be useful?
Analysis – (e.g. Modes of skill development (1993))
What was useful about the Integrated mode of skill
acquisition?
combined theoretical and practical (theory applied practically
and realistically) (24)
learning from experts on the job (7)
relevant to work skills/ applicable/ specific (7)
on-going development of skills/ upgrading (6)
apprenticeship important initially but o-j-t essential (4)
grounding in basics (transferable skills) (4)
hands-on experience (4)
good educators/ trainers (4)
appropriate background knowledge/ theoretical (2)
up to date methods - first hand experience (1)
practical experience more important than theory (1)
What was useful about the formal pre-employment mode of
skill acquisition?
appropriate background knowledge/ theoretical (10)
combined theoretical and practical (theory applied practically
and realistically) (8)
relevant to work skills/ applicable/ specific (2)
grounding in basics (transferable skills) (1)
good educators/ trainers (1)
share experiences - learn from others (1)
What was useful about on-the-job mode of skill
acquisition?
hands-on experience (79)
relevant to work skills/ applicable/ specific (27)
learning from experts on the job (20)
practical experience more important than theory (7)
up to date methods - first hand experience (4)
you get paid while you learn (3)
on-going development of skills/ upgrading (2)
What was useful about school followed by on-the-job
mode of skill acquisition?
school basics, then o-j-t and learning from others (5)
learning from experts on the job (1)
grounding in basics (transferable skills) (1)
hands-on experience (1)
Frequencies and modes
How could the on-the-job mode of skills acquisition be improved?
• longer o-j-t time and follow-up (24)
Use a question – work based learning could
• no improvement (18)
be better if ….
• more time with expert on the job (16)
• more detailed/ advanced special courses when required (13)
• formalisation of o-j-t and skills development (11)
• more group sharing of skills/ideas (7)
• more up to date theory provided/ background knowledge (7)
• text to support/ complement o-j-t (5)
• better trainers - capable/qualified/ up to date (5)
• on-going refresher courses for staff and trainers (4)
• more specific o-j-t -including vendor training (3)
• better communication between management/staff re staff's training needs (3)
• more motivation and incentive to learn/improve/achieve (2)
• better resources (2)
• more varied work experience (2)
• job training off-site (1)
• improved o-j-t and personnel exchange/rotation (1)
• keep up with modern technological changes (1)
• standardised criteria/ expectations for trade/ profession (1)
• cater for individual differences and capabilities (1)
Example – comparing question 1a and 1e
Integrated
Formal pre- On-the-job School
employment
Actual n=67 (27%) 23 (9%)
148 (59%) 8 (3.2%)
Ideal
N=75 (32%) 19 (8.1%)
138 (59%) 2 (.8%)
Using qualitative responses for a quantitative comparison
Incorporating quantitative procedures within
qualitative method
Example of interview schedule –
Improving reliability
Inter-rater reliability
1. Getting someone else to independently code the data and then
2. Checking for the degree of similarity and difference between the two
raters.
3. Understand difference and then each code another set of data.
4. Then review amend data coding
Table 5.5
Frequency of congruence between raters on all codes
Number of items coded
Total
211
Agreement between researcher and
independent rater (% in parentheses)
51
47 (92)
48
38 (79)
65
53 (82)
47
36 (77)
Total
174 (83)
Tabulation of qualitative data
Table 6.4
Subject attitudes (Nunnaly, 1967)
Sub
A1
A2
F5
F6
Preference for (likes)
satisfying clients - chemical and colour
working with (some) people - cutting
colour
client contact, conversation and cutting
social aspects of work (clients and coworkers)
interaction with clients - cutting
satisfying clients - creative work
C7
C8
C9
O10
O11
helping clients - self-esteem
satisfying and working with clients
different tasks and working with clients
people and clients
meeting people
F3
F4
Preference against (dislikes)
awkward clients and streaking
way some clients treat her
colouring - up-fashion cuts and weddings
overlap of responsibilities between family and
work
perming and chemicals
unhappy clients and those who are rude on the
phone
duties of running a business
unhappy clients
bad days - tints on tints
nothing
styling
Table 6.5
Commonality in categorisation across settings
Settings
Salon A
Salon C
Commonality in categories
Personal backgrounds and concerns
about clients
Concerns about treatments and smooth
running of production line
Salon F
Occupational and interest factors
associated with clients (e.g. freedom
to be creative)
Salon O
Characteristics of clients (age or
frequency of visits)
Differences in categories
A1 - client lifestyle
A2 - openness/ personal confidence
C7 - what they want to spend (just a
trim)
C8 - clients' mood
C9 - all the same
F3 - occupation and age
F4 - personal standing/values
F5 - age and openness
F6 - openness and sort of demands
O10 - age and client status
O11 - status