Colin Robson, Real World Research Third Edition.
Resources for Chapter 13
 Examples of research using participant observation (p.329)
Andrews, J. P. and Andrews, G. J. (2003) Life in a secure unit: the rehabilitation of young people through
the use of sport. Social Science and Medicine, 56, 531-550. A study using participant observation in a
secure unit, questioning the suitability of sporting activities used as a primary vehicle for, and mechanism
of, rehabilitation.
Gray, D. E. (2006) Coping over time: the parents of children with autism. Journal of Intellectual Disability
Research, 50, 970-976. Reports the results of a longitudinal study of parents coping with autism over a
period of approximately a decade based on ethnographic methods that emphasized in-depth interviews
and participant observation.
Johnson, J. C., Avenarius, C. and Weatherford, J. (2006) The active participant-observer: applying social
role analysis to participant observation. Field Methods, 18, 111-134. DOI: 10.1177/1525822X05285928.
Provides a framework for analyzing the potential benefits of an ethnographer participating in an active
role in a given ethnographic setting. Three case studies are analyzed using the framework; an
ethnographic study of a fish camp in Alaska; a study of the red-light district in Washington, D.C.; and a
study of Chinese and Taiwanese immigrant communities in Southern California.
O’Reilly, C. C. (2006) From drifter to gap year tourist: mainstreaming backpacker travel. Annals of Tourism
Research, 33, 998-1017. Long-haul, long-term independent travel (backpacking) has become increasingly
common over the last few decades. Based on interview and internet material and ethnographic field
research using participant observation, this article considers where this growth in interest has come from,
and how transformations in the perception of backpacking have taken place.
Stevens, C. A. (2006) Being healthy: voices of adolescent women who are parenting. Journal for Specialists
in Pediatric Nursing, 11, 28-40. The purpose of this ethnography was to explore how adolescent women
who are parenting describe what ‘being healthy’ means to them and how they define their health needs.
Traditional ethnographic methods of interviewing and participant observation were supplemented by
‘photovoice’ where participants represented their point of view by taking photographs.
Ussher, J., Kirstena, L., Butow, P. and Sandoval, M. (2006) What do cancer support groups provide which
other supportive relationships do not? The experience of peer support groups for people with cancer.
1
©2011 John Wiley & Sons Ltd.
www.wileyeurope.com/college/robson
Colin Robson, Real World Research Third Edition.
Social Science and Medicine, 62, 2565-2576. This qualitative study examined the questions of what
cancer support groups provide that other supportive relationships do not, and what the self perceived
consequences are of support group attendance. Nine representative Australian cancer peer support groups
took part in participant observation and focus group interviews.
2
©2011 John Wiley & Sons Ltd.
www.wileyeurope.com/college/robson
Colin Robson, Real World Research Third Edition.
 Examples of research using structured observation schedules (p.335)
Barnes, S. (2006) Space, choice and control, and quality of life in care settings for older people. Environment
and Behavior, 38, 589-604. This cross-sectional study aims to establish whether gradation of space and
daytime location are associated with quality of life in care home residents. A well-validated and reliable
observation schedule (Dementia Care Mapping) was used to measure the proportion of time residents
spent in activity and their well-being or ill-being.
Baumfield, V. and Devli, N. (2005) Staying on task: can a thinking skills approach support a productive
pedagogy for inclusion? Journal of Research in Special Educational Needs, 5, 37-42. DOI:
10.1111/j.1471-3802.2005.00037.x. An investigation of the effects of a thinking skills approach on the
on-task rates of pupils with Special Educational Needs (SEN). Pupils were observed by a trainee
educational psychologist using structured observation schedules. Pupils with SEN had significantly
higher on-task rates in thinking skills lessons compared to the non-thinking skills lessons, and pupils with
SEN in the control situation.
Fitzpatrick, P., Gilroy, I., Doherty, K., Corradino, D., Daly, L., Clarke, A. and Kelleher, C. C. (2009)
Implementation of a campus-wide Irish hospital smoking ban in 2009: prevalence and attitudinal trends
among
staff
and
patients
in
lead
up.
Health
Promotion
International,
24,
211-221.
DOI:10.1093/heapro/dap020. A report on the evidence base that supported the decision to implement the
first campus-wide hospital smoking ban in the Republic of Ireland including a one week observational
study to assess smoker behaviour in designated smoking shelters
Kira, M. and Frieling, E. (2007) Bureaucratic boundaries for collective learning in industrial work. Journal
of Workplace Learning, 19, 296-310. DOI: 10.1108/13665620710757851. A study of individual and
collective workplace learning and the connections between them in the contemporary industrial
workplace. Two case studies were carried out in the Finnish package-supplier sector using structured
observations and qualitative interviews.
Moyles, J. (2002) Observation as a research tool. In D. M. Coleman and P. A. Briggs, eds, Research Methods
in Educational Leadership and Management. London: Sage. Includes examples of semi-structured
observation schedules as appendices.
3
©2011 John Wiley & Sons Ltd.
www.wileyeurope.com/college/robson
Colin Robson, Real World Research Third Edition.
Rischel, V., Larsen, K. and Jackson, K. (2008) Embodied dispositions or experience? Identifying new
patterns of professional competence. Journal of Advanced Nursing, 61, 512-521. Report of a study to
explore nurses' competence as revealed during an admission assessment based on structured observations
of admission assessments in an orthopaedic ward by four nurses.
4
©2011 John Wiley & Sons Ltd.
www.wileyeurope.com/college/robson
Colin Robson, Real World Research Third Edition.
 Measuring inter-observer agreement (p.342)
1 Draw up the ‘confusion matrix’ Suppose that the coding schedule has five different categories (A, B,
C, D and E) and that there are 100 occasions when coding has taken place. With two observers, an
agreement takes place when they both use the same code for the occasion. If they use different codes
then that is a disagreement. The pattern of agreements and disagreements can be shown on a twodimensional matrix (often referred to as a ‘confusion’ matrix).
Observer two
Observer one
A
B
C
D
E
Total
A
8
0
0
0
0
8
B
2
21
1
0
0
24
C
1
0
18
2
1
22
D
0
0
0
24
6
30
E
1
0
0
1
14
16
Total
12
21
19
27
21
100
Note that scores on the diagonal from top left to bottom right indicate agreement between the two
observers; scores off this diagonal indicate their disagreement.
2 Calculate the proportion of agreement (P ) This is given by
0
(number of agreements)/(number of agreements + number of disagreements)
(8  21  18  24  14 )
or
which in this case is
0 .85 .
(100 )
NB The index of agreement (or concordance) is simply this proportion expressed as a percentage (i.e. in
this case, 85 per cent).
5
©2011 John Wiley & Sons Ltd.
www.wileyeurope.com/college/robson
Colin Robson, Real World Research Third Edition.
3 Calculate the proportion expected by chance (P ) Probability theory shows that if the probability of the
c
first observer using, say, code A is P ; and the probability of the second observer using the same code is
1A
P , then the probability of them both using the same code by chance is simply the product of these two
2A
separate probabilities (i.e. P x P or 0.08 x 0.12). Hence the total chance proportion for all five codes is
1A
2A
P = (0.08 x 0.12) + (0.24 x 0.21) + (0.22 x 0.19)+ (0.30 x 0.27) + (0.16 x 0.21)
c
= 0.216.
4 Calculate Cohen’s Kappa (K) This is given by the formula
K 
P0  Pc
1  Pc
In the example,
K 
0.850  0.262
1  0.262
 0.797
The value of Kappa, while still quite high, is noticeably smaller than the uncorrected proportion of agreement.
There are ways of assessing the significance of Kappa (see Bakeman and Gottman, 1997). However, as with other
statistics, statistical significance is not everything and, particularly with large samples, it is possible to achieve
statistical significance with proportions which show little agreement between the observers. Fliess, Levin and Paik
(2003) have suggested the following ‘rules of thumb’:
Kappa of 0.40 to 0.60: ‘fair’;
Kappa of 0.60 to 0.75: ‘good’;
Kappa of above 0.75: ‘excellent’. check
Note The software package ‘analyse-it’ includes a simple routine for calculating Kappa – see appendix A.
References
Bakeman, R. and Gottman, J. M. (1997) Observing Interaction: An Introduction to Sequential Analysis. 2nd
edn. Cambridge: Cambridge University Press.
6
©2011 John Wiley & Sons Ltd.
www.wileyeurope.com/college/robson
Colin Robson, Real World Research Third Edition.
Fleiss, J. L., Levin, B. and Paik, M. C. (2003) Statistical Methods for Rates and Proportions. 3rd edn.
Hoboken, NJ: Wiley.
7
©2011 John Wiley & Sons Ltd.
www.wileyeurope.com/college/robson
Colin Robson, Real World Research Third Edition.
 Annotated references to further reading for chapter 13 (p.344)
There is little overlap between the participant observation and structured observation literatures, and hence
this further reading is presented in two sections.
Structured observation
Bakeman, R. and Gottman, J.W. (1997) Observing Interaction: An Introduction to Sequential Analysis, 2nd
edn. Cambridge: Cambridge University Press. Second edition of the classic text on the topic. Strongly
recommended. Provides a straightforward introduction to scientific methods for observing social
behavior, emphasising sequential approaches to its analysis and understanding.
Kerig, P. K. and Baucom, D. H. eds (2004) Couple Observational Coding Systems. Mahwah, NJ: Laurence
Erlbaum. and
Kerig, P. K. and Lindahl, K. M. eds (2001) Family Observational Coding Systems: Resources for Systemic
Research. Mahwah, NJ: Laurence Erlbaum. Comprehensive edited texts discussing the coding systems
that have been developed, as well as the conceptual and methodological issues involved in systematic
observational research with couples and families.
Martin, P. and Bateson, P. (1993) Measuring Behaviour: An Introductory Guide. 2nd edn. Cambridge:
Cambridge University Press. Comes from biology/animal behaviour but of general relevance. Covers
wider range than Bakeman and Gottman in less detail. Excellent annotated reference section.
Sackett, G. P. ed (1978) Observing Behavior, Vol. 2: Data Collection and Analysis Methods. Baltimore, MD:
University Park Press. Slim volume covering wide range of methodological aspects of using structured
observation (e.g. category definition, sampling, analysis, recording and reliability).
Participant observation
Adler, P. A. and Adler, P. (1987) Membership Roles in Field Research. Newbury Park, CA: Sage. Insightful
analysis of the difficulties encountered in different roles (peripheral, active and complete membership).
8
©2011 John Wiley & Sons Ltd.
www.wileyeurope.com/college/robson
Colin Robson, Real World Research Third Edition.
DeWalt, K. M. and DeWalt, B. R. (2002) Participant Observation: A Guide for Fieldworkers. Walnut Creek,
CA: AltaMira. Provides a basic guide to using participant observation in naturalistic settings, and with
communities in many different cultures.
Lofland, J., Snow, D., Anderson, L. and Lofland, L. H. (2006) Analyzing Social Settings: A Guide to
Qualitative Observation and Analysis, 4th edn. Belmont, CA: Wadsworth. Very readable book with
excellent discussion of both methodological issues and practicalities.
Whyte, W. F. (1984) Learning from the Field: A Guide from Experience. Newbury Park, CA: Sage.
Discussion and dissection of issues in field work with focus on participant observation.
9
©2011 John Wiley & Sons Ltd.
www.wileyeurope.com/college/robson