Level 3: structured observation
質性研究
Qualitative Inquiry in TESOL (P.144-160)
Ma3c0102
甘蘋 ( Victoria Kan)
Level 3 focuses on structured observation.
While recognising the dangers involved in this
approach, it also highlights the valuable part
such observation can play in fieldwork and
provides advice on how to decide whether to
incorporate it. Practical advice is offered on
designing and applying an observation
schedule and potential problems are addressed.
The level also includes a section on how to
calculate inter-observer agreement. (p.105)
Level 3 : Structured observation
•
One of the most important decisions needed to
make is the systematic use of pre-determined
categories will play in the approach.
•
Structured observation is sometimes
referred to as systematic observation , but
this misleadingly implies that participant
observation is not systematic.
•
•
Open observation– the early stages of participant
observation where the observer tries to get a
general sense of the setting and the activities.
Closed observation—observer is strictly coding
behavior on a low-inference schedule, or
instrument.
The hidden dangers of closed observation
potential trouble spots from four angles: perspective,
design, procedure, and interpretation.
1. Perspective
• Strength – to allow researcher to focus on those areas and
characteristics that they wish to know more about.
• Danger—many novice researchers tend to introduce it at
the beginning of their projects and without giving
proper consideration to the implications of these for
the project as a whole.
• Wolcott (1994:159) what concerns him is that
electing for such a system can serve to conceal
more than it reveals…
(p.145)
2. Design
• Apply Wolcott’s point to consider the issue of
labeling.
• Strength— with limited ambitions and with a
view to deepening the researcher’s understanding
of particular features of behavior ,it can serve the
project well
• Danger-- attempt to capture precise teaching
“activities” in a single set of descriptive terms
can produce a plethora of labels and a deal of
confusion.
• Ex: (Adams 1972) took the trouble to survey the
many labels in the area of classroom observation
and produced a daunting list of available terms.
(p.146)
3. Procedure
• Strength—the preparation and administration of an
observation schedule has something for everyone: the
appeal of the systematic for those who enjoy creating
order out of chaos, an opportunity to establish fixed
routines.
• Danger– the range of its attractions makes it a
dangerous instrument:
“There is a tendency … to generate lots of bits of data.
The problem …one of trying to piece them together to
produce an overall picture, or one of trying to find
general themes that link fragments of data together.”
(Bryman 2001 a: 173)
4. Interpretation
•
•
•
•
Bryman’s observation points to the associated
problem of interpretation.
Danger-- accumulative potential of structured
observation can undermine its analytic purpose.
Become swamped with information that has
distorted the shape of the original project.
Comparison –quantitative representations are left
to stand as explanations in themselves; qualitative
researcher needs to remember that description,
whether in narrative or categorical form, is not the
same as interpretation and takes us below the
surface of things.
(p.148)
Participant and Structured Observation
Participant observation
Structured observation
Orientation
Open
Closed
Foundation
Event-based
Category - Based
Form
Narrative
Descriptive
Observer status
Coding
Observer-asinstrument
Post-observation
Observer-throughinstrument
Pre-observation
Recording
Retrospective
Cotemporaneous
Format
Notebook
Observation schedule
Participant observation
Structured observation
Replicability
Non-replicable
Replicable
QI status
Main or
supplementary
method
Supplementary
method
• The two approaches can make a potent combination.
Standard observation schedules
• In practice, most researchers develop their own.
• The groundbreaking contribution to the
structured observation of classrooms was made by
Flanders (1970), who developed a ten-category
observation schedule, the Flanders Interaction
Analysis Categories (FIAC).
• Designed specifically to identify direct and indirect
teacher influence, its coverage and simplicity
broadened its appeal beyond this relatively narrow
focus, so that many subsequent systems have their
roots here.
Presents the main categories in the FIAC system
Teacher
talk:
(a) indirect
influence
1. Accepts feeling
2. Praises or encourages
3. Accepts use uses ideas of pupils
4. Asks questions
(b) direct
influence
5. Lecturing
6. Giving directions
7. Criticising or justifying authority
Pupil
talk:
8. Pupil talk--response
9. Pupil talk -- initiation
10. Silence or confusion
Deciding whether to use structured observation
• From “what do I want to find out?” to “What
schedule do I use?”
• We can use techniques associated with that to
generate new ways of seeing that will in turn prompt
fresh questions, a vitally important preliminary
because the value of the schedule we eventually
devise will depend on the quality of the questions
we ask.
Contribution of an observation schedule (p. 150)
Main
Research
question
Subsidiary
question
Specific
data
needed
Coding
scheme +
schedule
Working up a schedule
• Reasons: Qualitative inquiry generally prefers
inductive reasoning, which is why structured
observation has to be considered in the light of the
wider discovery process.
• Its design represents a creative challenge but also
offers the prospect of a flexible and evolving system
that allows for substantial revision at the trialing stage.
Practical considerations in approaching structured observation (Evertson & Green,1985)
1. Research question(s)
5. Observation instrument(s)
e.g. category system, descriptive system
2. Focus
e.g. particular group(s), event, strategy
6. Observation procedures
e.g. When? How often? Number of
observers?
3. Setting
e.g. classroom, staffroom
7. Analytical procedures
e.g. frequency count, event structure
4. “ Slice of reality”
8. Presenting of findings
Working up a schedule--basic decisions
• Should not be a mechanistic one
• The most basic decision of all is what behaviors will
feature in the observation.
• It is possible to cover all behaviors (FIAC)
• Focus on specific behaviors or events (ex: teacher
questions or group work)
• Four choices are important:
1. Descriptive system or category system—depend
on the extent to which numerical information is
required and whether the relevant behavior can be
broken down into a reasonable number of discrete
categories.
2. Rating scale or non-rating scale—Rating scales
evaluate behavior according to agreed criteria ( ex:
high, medium, low) and rating system are suitable for
purposes of evaluation. Two systems can be
combined.
3. High-inference categories or low-inference
categories—The structured observation involves
one observer and is essentially descriptive, highinference categories are perfectly acceptable, but
if more than one observer is involved lowinference categories are preferable.
4. Event sampling ( event-based coding)or interval
sampling (time-based coding)—In interval
sampling behaviors are coded at predetermined
intervals (e.g. every three seconds).
(p.152)
Working up a schedule--Illustrations
•
Based on rating scale--the first example (Extract 3.12)
which involves interval sampling but no rating, coding takes
place every five seconds.
The best way is not to rely on a watch, but to make a tape on
which you record a click every five seconds.
• Four minutes of coding on a 10-category system (p153)
5
10
15
20
25
30
35
40
45
50
55
60
1
2
2
2
6
6
2
2
6
6
6
6
5
2
2
2
10
2
2
6
2
2
2
10
5
5
3
5
3
3
5
3
3
6
6
4
4
4
6
4
2
2
6
6
6
6
6
5
3
6
6
9
Working up a schedule--Illustrations
•Interval sampling would involve identifying activities
on one axis an time periods on another , as in Extract
3.13. Example of non-rating system ( interval sampling)
5
10
15
TT
ST
\/
\/
20
25
\/
\/
\/
30
\/
35
40
45
50
55
\/
\/
\/
\/
60
\/
Sil
\/
Oth
Extract 3.14 Example of rating system (event sampling) (p154)
Feature
Delivery
Aspects
R
Clarity
4
Loudness
4
Speed
2
Appropriateness
2
Comments
Speed often too fast.
Big ability range in
class, but T made no
allowances when
speaking to weaker Ss.
Working up a schedule--Defining the units
• It is best to think first about higher order categories
and then work out lower order definitions.
• Considerations can be applied to all structured
observation but are designed with category systems.
• Within category:
1. Make sure the category is as clear as possible
2. Check that the category is related to observable
behavior
3. Be as precise as possible.
4. Consider the range of each category. (p.155)
• Among categories:
1. Ensure that all definitions are clear and exclusive
2. Check that there are no gaps in the coverage of
chosen behaviors.
3. The schedule must be practicable.
Essential characteristics of an effective category system
•
•
•
•
Clearly definable categories related to observable
behavior
Mutually exclusive categories—no overlap
The category set is exhaustive
The system can be operationalised
Some practical problems
1. Observer effect:
• You should not underestimate the effect that filling in a
form
• It is possible to hide or disguise the writing or coding
• It is not good to rely on a single observation
2. Expectancy effect:
• Should reflect very carefully on the initial assumptions
• The higher the degree of inference involved in the
observation, the more important it is to be aware of
factors that might influence coding.
3. Observer drift:
• Awareness can help to reduce it and where teams of
coders are involved periodic checks on inter-rater
agreement can direct attention to emerging problems.
4. Central tendency:
• Only to rating scale and refers to the tendency for opt
for something at or near the middle.
• Keeping options to a minimum can help to reduce it.
Calculating inter-observer agreement
• In the case, more than one person is involved
• Problem—observers will choose a particular code
not on the basis of pre-determined criteria
• Straightforward method of making allowance for
chance agreement that does not depend on a
sophisticated grasp of statistics or probability
theory. Known as Cohen’s Kappa (1960), it is
based on records of agreement and disagreement.
(p.157)