Sampling, Validity and the
Longitudinal Archive
Mike Savage
University of Manchester
Imagining Modern England: popular
identities and the post-war social sciences
How do we use different – even incommensurate qualitative sources to construct accounts of social
change?
• My forthcoming book presents an analysis of social change in
England using a range of archived qualitative social science data.
–
–
–
–
Mass-Observation Directives, 1939-1950
Material from Geoffrey Gorer archives
Elizabeth Bott, Family and Social Networks, 1950-54
John Goldthorpe and David Lockwood, Affluent Worker Studies, 19611964
– Brian Jackson, Working Class Community, 1960-1965
– Richard Brown, Tyneside Shipbuilding Workers, 1969
– Ray Pahl studies on Hertfordshire suburbanites, ‘managers and their
wives’ 1961-1969.
• The use of these diverse studies, even aided by dedicated research
time, poses the challenge of doing justice to the wealth of archived
material whilst presenting a coherent and rigorous account.
Sampling archived qualitative social science data
longitudinally: four issues
• Practical: how do we deal with extensive, often un-hierarchical, data
repositories?
• Methodological: how do we systematically compare archived data
from separate, autonomous archives, in different historical periods
• Historical: How do we see the deployment of research methods
themselves as agents of social change?
• Political: How do we challenge positivist conceptions of sampling yet
in a way which avoids justifying slap-dash research?
1: Political Issues
Challenging orthodox conceptions
of ‘trend analysis’
Conventional survey based longitudinal analysis dominates social
science, but strips out contextual detail to allow strict formal
comparison between aggregate entities (nations, social groups, etc)
over discrete time periods
– Economic indicators (inflation, wealth, unemployment, occupational
structure, etc)
– Psephological studies of voting using British General Election Surveys
– Studies of changing attitudes (e.g. Inglehart, Post-materialism)
• Archived qualitative material does not allow this abstraction from
context and hence cannot deploy ‘representative’ or ‘aggregate’
findings.
• It is however possible to use the messiness of qualitative inquiries
as themselves evidence for hidden processes.
• This offers an historical riposte to Mark Abrams who in 1948
attacked M-O for its unrepresentative samples. 60 years later, we
learn little from opinion poll or survey findings about contextual
historical issues – but the sprawling material in the M-O archive
offers rich pickings indeed.
Example 1: Bott’s Family and Social
Network sample
• Bott and her colleagues intensively interviewed 20
husbands and wives about their marital roles and
relationships.
• Bott emphasises that it is not possible to generalise from
these small numbers, yet develops an influential
analytical distinction between working class families with
role segregation, and middle class families with more
fluid roles. These differences are then explained by
contrasting the social networks.
• The fieldnotes indicate a further revealing feature of the
sample: that nearly all the households were predisposed towards research, and especially the social
sciences, and hence that the project itself reveals the
formation of a new kind of social group committed to
technical and scientific values .
Example 2: Mass-Observation on
social class identities
• Various directives (1939, 1948, 1949, 1990) ask about
class identities, though in different ways.
• It is not possible to use the responses to give an account
of representative trends in class awareness, given the
skewed social composition of M-O
• However, it is possible to explore what they say about
literate middle class awareness
–
–
–
–
Leftist and critical perspectives 1939
Reaction against post-war Labour government 1948
Emergence of a managerial identity, 1949
Individualised and reflexive class identities 1990
• This exercise involves focusing on form as well as
content, and reading the material as accounts of specific,
located, social groups, not ‘the population as a whole’.
2: Practical issues
The problem of sampling
• My project raises the issue of how to deal with such a
wealth of archive material. The modern archive
proliferates data, so preventing comprehensive
knowledge: how does one proceed?
• Much historical archival works takes a hierarchical form
of focusing on key sources, e.g. of powerful
organisations, individuals, decisions, unusual case
studies, etc. The operation involves working with and
through the power biases of the sources
• By contrast, archives such as Mass-Observation and
those with qualitative data on ‘popular’ groups lack an
obvious hierarchy. There are often extensive batches of
observations, interviews, directives, etc. It is not clear
that any particular record is more worthy of examination
than anything else, and one cannot rely on working with
the biases of the archive.
• Should one look at the ‘whole population’ (i.e. all the
sources) or a sample. And if the latter, how is the sample
to be drawn?
Whole populations?
• This initially sounds the most convincing route. It can be practical in
some cases, e.g. Elizabeth Bott, Family and Social Network, where
details of 20 household case studies are kept in one box.
• As soon as the scale of the work increases, this strategy is
problematic. One becomes faced with the need to abstract: e.g.
John Goldthorpe and David Lockwood’s Afffluent Worker project,
with 229 interviews, at home and at work. One strategy here is to
just focus on a particular part of the research. Even so, how does
one take notes?
• Mass-Observation poses another order of magnitude. Its directive
have several hundred responses, often lengthy. It is not possible to
read or analyse them all. Does one reduce the narrative accounts to
simple counts or frequencies? Does one focus on a few ‘exemplary’
cases (in which case, what is the reason for treating them as
exemplary?)
• We are faced with a similar issue to that which
confronted the early social sciences
themselves….. As soon as one studies ‘popular’
groups, who exactly does one talk to?
The politics of sampling
• Consider the following strategic uses of MassObservation material in recent historical work with
different sampling and analytical strategies
–
–
–
–
David Kynaston in Austerity Britain
Tony Kushner in We Europeans
Simon Garfield, Diaries
Nick Hubble, Mass-Observation and Everyday Life
• All indicate the problems of sampling such a vast source.
How does one deal with issues of coding and extracting
relevant material? The danger of the ‘juicy quote’
syndrome.
• I will return to this issue but I want to consider now how
this debate is not just one for us, but resonates with
historical and political concerns. The formation of
sampling methodologies are themselves a key part of
socio-cultural change in the post-war period
3: Historical Issues
Methods and Socio-Cultural Change
• The rise of the post-war social sciences is one of the
most important, yet un-researched, aspect of post-war
change.
– Social scientists are 3% of UK academics 1948 > c. 45% 2001
– Social scientists play pioneering role in elaborating new
‘technologies of the social’ in post-war years
• National sample survey (from 1930s)
• The ‘user questionnaire’ (from 1960s)
• The qualitative interview (from 1950s)
– Social scientists generate ‘epoch descriptions’ which come to
embed change into the social itself
• ‘Affluence’ (1960s)
• Post-industrialism (1970s)
• Globalisation (1990s)
• The social sciences are key agents in the emergence of
‘knowing capitalism’ (Thrift).
Sampling in historical perspective
Let us not consider the history of qualitative research
from the high ground of the national sample survey, but
instead consider the history of the national sample
survey from the standpoint of the situated, qualitative
researcher. In these terms, what does the sample
survey achieve?
1. It constructs ‘public opinion’ from the 1930s (Osborne and
Rose), though there is considerable early resistance and
scepticism.
2. It becomes a key government technology from Second
World War (1942)
3. It provides an inscription device for ‘individual’ development
(Cohort studies).
4. It allows the isolation of ‘social groups’ as definite bounded
entities (notably in 1960s ‘white heat’).‘
5. The survey co-produces the social scientific habitus itself
(through enlisting ‘educated’ research subjects and agents),
1: Surveys and public opinion
• Rose and Osborne (2000) argue that polling creates a new
conception of ‘public opinion’
• This jostles with, and comes to displace, notions of ‘national
character’ (c.f. Mandler) which had predominated beforehand.
• However, until the 1960s, there was considerable suspicion of, and
resistance towards polls (e.g. no one believed their prediction that
Labour would win the 1945 election).
• (Partial) respectability only comes during the 1950s. As late as 1964
it was noted that educational research was
– ‘something of a bandwagon (to which) individuals and bodies with
scanty experience and minimal competence are now turning their
attention to… some of these are commercial interests who skilfully
conceal their origins or convince a reputable educational organisation
that they will finance research without strings. Others are self appointed
pressure groups with innocent and high sounding titles… questionnaires
are a favourite instrument of such groups, since they appear simple to
construct and interesting to fill’
• Consider also Goldthorpe and Lockwood’s critique of ‘tick-box’
research which initiated the ‘Affluent Worker’ project, as well as
Bourdieu’s concerns with survey research.
2. As governmental technology
• The first national survey, under government aegis, in 1942. The
OPCS becomes the main body to conduct national surveys till the
1960s.
• Key devices of post-war government begin to deploy survey
measures, for example the ‘retail price index’, linked to the Family
Expenditure Survey (1957>). Surveys construct notions of the nation
as ‘modern imagined community’.
• The survey becomes the key device for establishing ‘government
departmental’ expertise in the ‘white heat’ of the 1960s Labour
government, defining the client groups of departments (e.g. the
‘poor’, the ‘ill’)
• supplementing the ‘gentlemanly’ Royal Commissions.
– Fulton Commission (Civil Service reform)
– Plowden Report (educational reform)
– Radcliffe-Maud Report (local government reform)
• Surveys were not seen as ‘re-usable’, but as ‘one-off’ inquiries
• Academic social scientists still work at some remove from surveys
Up until the 1950s many teachers, most educationalists,
and nearly all politicians envisaged educational research as
a mildly interesting and marginal activity… the periodic
reports of the Central Advisory Council… formulated their
recommendations by the time honoured means of
canvassing opinion and seeking a consensus. Recent
reports, however have illustrated a revolutionary change.
Crowther, Robins and Plowden were not content with
mainly canvassing opinion… they proceeded to seek out
facts.
(National Foundation of Economic Research, 1967-68)
3. As device for eliciting the ‘individual’
• Until 1970s public surveys rarely focus on national
random samples, but focus on specific ‘problem’ groups,
notably children.
• Cohort Study (1946) and National Child Development
Study (1958) pioneer studies of the ‘developmental
individual’ using innovative panel study design
• During the 1960s, extensive surveys of children and
young people were common.
–
–
–
–
Youth Survey 1961, 1962, 1963
Child Chest Survey, 1966
Buckinghamshire child survey, 1961
Politics and the English child, 1969
• Educational reform and ‘comprehensivisation’ depends
on survey research (e.g. consider the National
Foundation for Educational Research)
4: As elaborating on ‘social groups’
• In contrast to field analysis, surveys permit the sampling
of pre-specified social constituencies, often remarkably
fine-grained, e.g.
– Interviews with the ‘poor’ (numerous)
– interviews with ‘handicapped’ (1968) (N = 12738)
– Attitudes on International Affairs among African Students in
Britain (1963) (N= 291),
– Survey of aircraft noise near Heathrow (1961, 1969, N = 4699)
– Surveys of phone users.
• The national random sample survey only has marginal
position and the impetus comes from political science
(British Election Survey 1964>; Butler and Stokes;
Runciman; Nuffield Mobility Study)
5. As eliciting the social scientific
habitus itself
• Numerous surveys are about students themselves, including
surveys at Cambridge, Essex, Edinburgh, Manchester, Nottingham,
UCL, Exeter
• National Survey of 1960 University graduates, surveys of trainees,
of the ‘impact of schemes’
• Remarkable enthusiasm for surveys of University teachers (1964
Halsey; 1969, Ministry of Labour)
• MENSA plays a role in pioneering user surveys (e.g. 1969 survey to
see if MENSA members are ‘upwardly mobile’)
• New Society pioneers the ‘user questionnaire’.
The social scientific habitus is elicited around
concerns with change, development and
technological modernity.
Lessons for the history of the sample
survey
• We should not take the random, representative
sample as the scientific model from which the
problems of the qualitative archive deviate.
• We should instead see the sample survey as
having their own kinds of agency which it has
the capacity to conceal.
• We should be more confident in developing
strategies for sampling which are appropriate for
qualitative data
4: Methodological issues
Systematic analysis?
• We can be critical of positivist accounts, but still
need to reflect on how best to use the
‘messiness’ of archived data rather than a
simple appeal to genealogical or deconstructive
methods.
–
–
–
–
Sampling based on ‘theoretical saturation’
Sampling based on ‘adequate numbers’
Analysis based on the ‘telling exception’
Analysis based around the formation of coherent
typologies
– Use of content analysis, frame analysis, and narrative
analysis, as a means of emphasising position-taking.
Methodological resources
• Feminist and qualitative arguments about
research methods
• Andrew Abbott’s attention to ‘battles over
jurisdiction’
• Walter Benjamin’s critique of historicism
• Franco Moretti’s mapping of narrative and
plot.
• Pierre Bourdieu’s use of field analysis as a
means of exploring ‘stakes and positions’
Conclusions
• There is huge potential in the use of qualitative
archived data
• The fact that this data cannot be analysed in
similar ways to archived quantitative data is not
an inherent problem so long as researchers
build the biases of the inquiry into the analysis
• We need to develop clear analytical strategies
which allow the rigorous – yet partial examination of such data to avoid the problems
of (a) juicy quote syndrome and (b) false
completeness.