Hynek Jerabek is a professor of sociology at the Faculty of Social Sciences of Charles
University in Prague. He is a national representative of WAPOR for Czech Republic and the
author of numerous works on sociological methodology, including Paul Lazarsfeld’s
Research Methodology. Biography. Methods. Famous Projects. (Prague, Carolinum 2006).
His current research focuses on Paul F. Lazarsfeld’s and Robert K. Merton’s work and he is
head of a research project titled ‘Classic Research Projects as a Source of Inspiration for
Contemporary Social Research’, where he examines the history of empirical social research.
mailing address:
Prof. Dr. Hynek Jerabek
CharlesUniversity,
Faculty of Social Sciences,
Institut of Sociological Studies,
U Krize 8,
158 00 Praha 5 – Jinonice,
CzechRepublic
e-mail address: hynek.jerabek@gmail.com
fax number:: ++ 420 251 080 310
telephone number: ++ 420 251 080 216 ++ 420 251 080 313
Classic Inspirations for Social Research Methodology in the time of Online Access Panels
Hynek Jeřábek, Charles University in Prague, Institut of Sociological Studies, Department of Sociology
Abstract
The plain use of numbers to present the distribution of public opinion is not always the best
strategy. Samples are often not representative. We need to know what research design and
strategy for data analysis can be used in non-standard situations. Some examples of a
methodology for this type of research work can be found in the work of the classic architects
of research methodology in the 1930s and 1940s – Paul F. Lazarsfeld, Robert K. Merton,
Eduard Suchmann, Herta Herzog and William F. Whyte. They frequently used a comparative
design, qualitative methods and small subsamples. And they solved the problem with nonrepresentative data with the help of theory. These examples and innovations could enrich our
current public opinion research.
Keywords: public opinion poll, non-representative samples, comparative research design
1
Classic Inspirations for Social Research Methodology in the time of Online Access Panels
Hynek Jeřábek, Charles University in Prague, Institut of Sociological Studies, Department of Sociology
Regular research paper for WAPOR 64th Annual Conference Public Opinion and the Internet, September 21-23,
2011 Amsterdam, The Netherlands
Introduction
In the age of the Internet we will increasingly be faced with situations where the plain use of
numbers presenting the distributions of attitudes or public opinions is not the right strategy.
Our samples and populations are frequently fuzzy, non-representative, unrecognized, too
small, and often entirely casual in composition. And we need to know what research design
and strategy of data analysis can cope with such an unclear and tricky situation. We need to
know how to arrive at valid analytical conclusions. Although we will be in a relatively
complicated research situation, we will have to be able to reach sufficiently generalizable
conclusions. These results should be valid not just for the ‘internet’ population or for
‘volunteers’ offering their responses to internet access panels. Our goal will be to arrive at
findings that are valid perhaps just under certain, precisely defined conditions, if not
generally. They will apply within certain boundaries, under clearly determined, precisely
defined conditions.
Are there any examples of a methodology for this type of research work? Contemporary
researchers are not yet asking these questions. But the methodological examples do exist.
They are not however linked to current online research or even with the methodology of
online access panels or DIY research. We can find such examples and guidelines for
proceeding in this area by carefully examining scholarly monographs, research reports, and
archives from the first decades of the advancement of empirical social research. We can find
them in the works of the classic architects of research methodology from the 1930s and
1940s. They compared subsamples, examined exceptional circumstances, cooperated across
paradigms, used qualitative methods to create analytical conclusions, or used multi-stage
2
research designs, which substituted large numbers of respondents. These methodological
innovations brought them surprisingly effective analytical results. And contemporary public
opinion research methodology can be enriched by these examples.
1. Comparative design – P.F. Lazarsfeld in RAVAG Study
Example 1: From the differences in the preferences of listener sub-populations Paul
Lazarsfeld revealed findings valid for the listeners of Vienna radio broadcasting in the first
large audience research – the RAVAG study. (1932)
This project used a comparative design and a simple index model that expressed the
difference between the percentage of listeners who wanted to listen to the radio programme
more often and the percentage of listeners who wanted to hear it less often. The index is thus
able theoretically to obtain a value from -100 for cases where all the listeners wanted to hear
the programme less often to +100 for the opposite case where they wanted to hear more such
programmes. Zero could be obtained in various ways: +50% (those who wanted more such
programming) – 50% (those who wanted less) = 0; or + 30% - 30% = 0; or + 10% - 10% = 0.
In standard public opinion research we are interested in the percentage of respondents
who agree with a given response. The outcome we are looking for is information about the
total number of such listeners.
In the RAVAG study, Lazarsfeld used a comparative research design in which the
responses compared either individual groups of listeners or the assessments of individual
programmes by those same listeners. Lazarsfeld was interested in learning about the
programmes and the listeners or, precisely put, in the segmentation of the listener population
for individual types of programmes. To obtain this information it was necessary to obtain
information from the widest variety of listener groups and calculate a series of indexes for the
subsamples for the given programme. The research outcome was not a number but a series of
3
number, a rising or falling series of index values for different groups of listeners in relation to
the same programme. Their comparison showed how the observed characteristics of listeners
influenced the popularity of programmes. The second perspective that could be applies was to
compare programmes with each other in relation to the same group of listeners. These data
revealed which types of programmes the selected group of listeners like most. Over the course
of 1931 and 1932 Lazarsfeld calculated by hand, without the aid of any computation
technology, 7000 indexes for a variety of subsamples of listeners categorised by sex,
occupation, age and region of residence, and for individual categories of programmes ranging
from symphonies, to chamber music, organ concerts, opera, operettas, jazz music, to literary,
and informative programmes and programmes for women and children. These series of
indexes were then compared and interpreted in relation to each other. The research
conclusions took the form of observed patterns and specific findings were portrayed as rising
or following index values for subsamples of listeners arranged according to occupation, age or
region. An example is provided in Table 1.
Table 1: Indexes of interest in classical and light music according to social groups
of listeners – men: RAVAG - Study (Lazarsfeld 1996 : 16)
MUSIC
Classical
light
LABOURERS
- 47
+ 59
TRADES PEOPLE
- 39
+ 53
STATE OFFICIALS
- 28
+ 46
INTELLECTUALS
- 17
+ 22
FARMERS (rural inhabitants)
- 39
+ 37
Note: Indexes indicate for each social group and for each type of music the difference between the % of
those who wish to increase and the % of those who wish to decrease the share of that genre on the radio.
4
Listeners of the Vienna radio (RAVAG) wished to limit the broadcasting of classical
music. However, not all of them to the same degree. The first column in Table 1 shows that
the more complex, autonomous, and cultured a person’s occupation, the lower the resistance
to classical music programming, expressed as a decreasing negative index value.
Conversely, listeners reached differently to the popularity of light, recreational music.
All occupational groups wanted the Vienna RAVAG radio to include more light music in
their programming. And again the outcome was expressed as a series of numbers, a sequence
of index values (in the second column of the table). The more complex, autonomous, and
cultured a person’s occupation, the lower of the value, this time, of the positive index,
indicating the difference between the percentage of listeners who wanted more and those who
wanted less recreational music. More educated listeners were content with a mix of classical
and light music. Less educated favoured popular, recreational music over classical music,
which was expressed by the relevant pair of indexes in the row in Table 1.
Interpreting the values of two indexes of farmers (note: neutral term - rural
inhabitants) in Lazarsfeld’s view it was necessary to take into account that the sample was
made up of a large number of agricultural labourers and also a small number of large farmers
– landowners with relatively high cultural and educational backgrounds. For this reason the
value of the index does not follow a rising or decreasing line but rather expresses a mixture of
the two types of listeners – those with high and those with low cultural and educational
backgrounds.
The significance of the analysis Lazarsfeld conducted eighty years ago to date lies in
the fact that he did not have to have a strictly representative composition of respondents in his
sample. Slight deviations in the share of men or women or inaccuracies in categorisations in
individual occupational groups did not harm the overall informativeness of the results, as the
outcome reported on relations and patterns. The result was the observation of these patterns,
5
tendencies and relations rather than a specific number. So if today we are unsure about the
representativeness of our sample unit, we can concentrate on comparing comprehensively
defined parts of it and the result of our efforts will be information on the relations between
them.
2. In-depth interviews – Herta Herzog in ‘Invasion from Mars’ Study
Example 2: Using in-depth interviews Herta Herzog (1955; [1938]) uncovered the motives for
the mass panic caused by the broadcast of Orson Welles’ dramatization ‘Invasion from Mars’
and thus anticipated the results of a much larger study conducted together with Hadley Cantril
and Hazel Gaudet. (1940)
Herta Herzog began interviewing listeners in New York the very morning after Orson
Welles’ dramatization of „Invasion from Mars” was broadcasting, generating panic, fear, and
spontaneous reactions among listeners. The sci-fi production realistically imitated a radio
broadcast occurring during a planetary catastrophe. Because the threat directly affected the
inhabitants of the east coast of the USA, people fled the cities and save themselves and those
close to them.
What was so extraordinary about the nature of the research design, later called a
‘firehouse project’, was that it: 1) interviewed people directly after an event; 2) the event itself
was a powerfully emotional one; 3) in-depth interviews involved recording the immediate
reactions of those directly affected by the event – the listeners, who had to cope with an
unusual event. But not all the listeners succumbed to panic. Information about the number of
listeners and the share of those who succumbed to panic were obtained around a month later
in a representative survey that used a classic public opinion poll design. The most valuable
findings of the research however were not the percentages and number but rather the findings
on the causes that gave rise to fear, chaos and confusion, and above all the circumstances
6
which made listeners liable to panic and under which they most often succumbed to panic.
Herta Herzog also obtained valuable findings from the interviews of a controlled sample of
individuals who did not succumb to panic. Their critical ability protected them from panic.
However, that ability could be paralysed by the extraordinary circumstances preventing them
from applying this critical ability. This ‘critical ability’ was a characteristic that generally
corresponded with levels of education. But deviations in both directions were apparent. And
even today it is interesting to observe the meticulousness with which Herta Herzog
distinguished between education and critical ability.
The classification of reasons Herta Herzog arrived at after three weeks of in-depth
interviewing was almost the same as the final classification published a year later in the book
Invasion from Mars (Cantril, Herzog and Gaudet 1940). This classification was conducted by
Hadley Cantril, Herta Herzog and Hazel Gaudet together on the basis of the results of two
representative public opinion polls, from an analysis of letters sent to the broadcasting
company and to the broadcasting commission and from a content analysis of the mass media
at the time; and understandably, however, also from Herzog’s in-depth interviews. Even today
it is enough for us to know as that the frame of findings was that the event was extraordinary
in terms of its scale and impact. And if we ask about the basic research findings? It was
mainly about this: identifying and pinpointing the factors that significantly led to sudden,
spontaneous, and powerfully emotional reactions on the part of the listeners. Large
representative studies were only carried out later, several weeks and months after the event,
and thus they were unable to accurately capture the spontaneous and emotional reactions of
the population. Conversely, in-depth interviews were conducted immediately after the event,
when listeners were able to give a realistic description of their feelings and spontaneous
reactions right after the event. At that time listeners realistically described their feelings and
spontaneous reactions to the radio broadcast. The circumstances in which they panicked and
7
reacted suddenly also had to be observed right after the event. The results of the in-depth
interviews by Herta Herzog are highly valid and on the whole exact findings. They are
sufficient to provide a list of significant characteristics and circumstances behind listeners’
panic and reveal the defensive role played by critical ability. What more can we wish for as a
scientific result?
3. 95% response rate in 3 groups – E. Suchman testing Adorno’s Hypothesis
Example 3: At Paul Lazarsfeld’s request Eduard Suchman tested the hypothesis of
Theodor Adorno and thus threw down the gauntlet to all research practitioners, as he achieved
an incredible 95% response rate to the questionnaire he distributed by post. Nevertheless, in
this case the findings did not pertain to listeners of classical music as a whole, on the contrary.
(1940)
In this case Eduard Suchman tested the hypothesis of Theodor Adorno. The sceptical
Adorno’s hypothesis related to the limited capacity of mass media to turn new listeners into
serious aficionados and lovers of classical music. Suchman needed firm (valid and reliable)
data that he could use to compare classical music listeners who had been listening to classical
music all their lives, attending classical music concerts, and often themselves played a
musical instrument or sang, with other types of listeners new to classical music. He divided
the latter into two groups: those who were acquainted with classical music and sometimes
listened to it even before hearing the broadcast, and those who were entirely new to classical
music and had only begun listening to it with the broadcast and did not know it before that.
Suchman needed reliable data in a comparative research design for the three named
groups. It was therefore extremely important to ensure that the return rate of questionnaires
sent to all three groups of listeners was not just high but was comparably high, so that the
information from the respondents in all three groups was equally representative for each. For
8
this reason especially Eduard Suchman saw to it that the questionnaire mailed out to
respondents was returned by as many of them as possible. This meant finding ways of
appealing to respondents which would ensure a high and comparable return rate.
Suchman selected as his basic sample subscribers to the radio programme published
weekly with the programming schedule. Using probability sampling he selected a certain
percentage of subscribers and sent them a questionnaire to determine their relationship to
classical music, its individual genres, composers, performers and musical works. He also
wanted to find out the respondent’s relationship history with classical music. He asked them
about their childhood, whether they played any instruments, attended classical music concerts
earlier or now, and finally whether they listened to classical music on the radio, their listening
habits and their favourite music programmes.
In order to encourage respondents to return the questionnaires he mentioned the
research several times directly during music broadcasts and stated that it was very important
for the listeners who had received questionnaires to complete them and return them by post.
This was followed, as was the custom, with a reminder to complete the questionnaire, and in a
third letter was sent with a paid postage envelope in which to return the completed
questionnaire. Those who did not respond to any of the letters and did not complete the
questionnaire were sent a short questionnaire by Eduard Suchman with just several questions
and an urgent request to complete at least basic information about themselves as classical
music listeners.
The strategy he adopted, requesting the questionnaire be completed, sending
questionnaires three times, issuing urgings and reminders on the radio, helped (along with
choosing a more engaged population and theme they were interested in) attain 95% response
rate, an unprecedented result using mailed questionnaires.
9
Based on their responses Suchman divided the listeners into three groups: 1) those
who had grown up with classical music, played a musical instrument or sang, and attended
concerts; 2) those who were familiar with classical music before the radio broadcast and took
the broadcast as another opportunity to listen to it; and 3) those who first became acquainted
with classical music from the radio broadcast and had not encountered it before and had no
opportunity to listen to it. As a control he also questioned several dozen listeners whose
original responses were ambiguous.
Why did Suchman require such a high return rate? He wanted to obtain representative
information from all three groups of classical music listeners. He was able to confirm
Adorno’s hypothesis that mass media do not build a deep relationship to classical music. And
this finding could not be challenged with the objection that any of the subsamples compared
was not representative. All of them, even those with a tepid relationship to classical music,
responded.
4. Participant observation – W. F. Whyte reveals a hierarchy in a slum neighbourhood
Example 4: William Foote Whyte (1943) revealed the hierarchical structure of an Italian slum
with the aid of participant observation and using qualitative methods reached analytical
conclusions that rank with the results of representative studies.
For his study W.F. Whyte used participant observation as his main method. What did
Whyte use in place of representative data to describe the hierarchical arrangement of the
community of young Italians with whom he repeatedly spent time? Instead of mass
observation he conducted repeat observations. Instead of observing opinions and attitudes he
focused directly on the behaviour and actions of the observed actors. And he complemented
his systematic observations with information drawn from interviews and explanations from
his informer.
10
He observed the actions of one group of young men over a long period. And he found
an activity that because it was a repeated activity was an appropriate substitute for
representative mass data. The activity was bowling tournaments, which took place almost
every Saturday, week after week, the results of which ranked members of the group in a
hierarchy. Whyte described in detail the results of his observations. He also added an
interpretation of the tournament hierarchy, which significantly resembled the hierarchy of
returning favours and subordination in the gang that he observed. The naturalness with which
the group’s members accepted the outcome of the important group match, the annual
tournament was based on their understanding of the situation. (Homans 1950: 165) Whyte, in
the position of a participant observer, obtained his understanding of this natural acceptance
from his informer, the gang leader named ‘Doc’. The fact that the hierarchy of the group was
strikingly similar to the results of the tournament was observed by Whyte himself. And the
theory that corresponded to this observation was described in detail several years later by
G.C. Homans in The Human Group (Homans 1950: 178-181). Whyte’s observations were
described by Homans as an example of his ‘theory of secondary awards’. (Homans 1950: 170
nn.) In a game or contest gang members support their leader and his sidekicks, and by doing
so reward them for looking after and taking care of the group and for properly fulfilling their
roles as leaders, in proportion to their merits and position in the hierarchy. By contrast, they
obstruct and make the game more difficult for those who do not look out for the group and do
not deserve to win and thus be rewarded nor even deserve support to win. The hypothetical
existence of a theory of secondary awards together with the systematic reappearance and
replication of patterns of results in bowling group matches then gave W.F. Whyte a strong
enough argument to substantiate his findings on the mechanisms that gave rise to the
hierarchical structure of a Cornerville gang. (Whyte 1993: 20-23)
11
5. Theory in action – R. K. Merton’s typology of ‘locals’ and ‘cosmopolitans’
Example 5: Using 30 cases R. K. Merton demonstrated the general typology of local and
cosmopolitan influentials with a brilliant analysis of their communication behaviour. (1949)
Merton studied a small group of thirty influentials. Their influence on the formation of
opinions among those around them was robustly substantiated by statements from 86
informers in a small town. When four or more people indicated that an individual was
influential, that was enough evidence to include that individual among his sample. But
Merton did not just make do with finding influentials. First he identified the different reasons
for influence in a community, the heterogeneity in the group of influentials. He found a
hypothetically important difference between two groups of such citizens capable of
influencing goings on and the opinions others around them. This difference was their ‘local’
or conversely ‘cosmopolitan’ orientation, thus whether they focused on people in the town or
on the outside world and relations in it. Merton asked himself how these two subgroups differ
and found that it was by how they attained their influential status and how they applied their
influence. The differences between them are manifested in their methods of communication
and in what things they regard as important.
The strength and persuasiveness of Merton’s discoveries derived from the fact that he
based his considerations on a theoretical explanation for distinct worldviews and thus on
theoretically grounded assumptions about the different ways and means by which each type of
influential influenced those around them. Merton relied on the identification of people with
significant influence in the studied community as provided by the informers. Using projective
questions he identified the basic difference between two groups of influentials. And because
he had something on which to base the hypothetical differences in the ways in which this
influence was gained and applied in the model, he was able to concentrate on the differences
between these two groups. So he was able to go further than Paul Lazarsfeld, who had made
12
do with a general model of the two-step flow of information and applied it irrespective of any
groups of opinions leaders. Merton used a theoretical assumption about differences in the
ways in which local and cosmopolitan influentials acquire influence and the different avenues
by which they apply this influence in their surroundings.
Merton surpassed the public opinion poll model in at least two ways. First, to reveal
influentials he did not just question respondent and thus rely on self-identification; he used
informers. Second, he tried to explain theoretically how personal influence arises in two
different situations and how it is used. The key to surpassing the public opinion poll model in
this case lay in selecting the appropriate theory.
Conclusion
Each of the examples cited above represents one of the many ways in which it is possible to
attain sufficiently significant analytical findings even in situations where the conditions do not
exist for applying the standard approach of the ‘public opinion pool’. These examples present
us with many innovative ways in which to cope with situations when we don’t have
representative data and where the model of the ‘public opinion pool’ cannot be used or would
surely fail.
What do all five examples have in common? In each case the research replaced the
manifest data of mass public opinion polls based on large, representative samples with results
from smaller, more complexly designed studies. The samples of respondents used in these
smaller studies usually were unable to be representative for the whole population. To
overcome this handicap of insufficient representativeness, the researchers used theoretical
findings or hypotheses. The results of smaller studies supported by theory thus serve as an
equal substitute for the transparent numbers provided by public opinion polls. They moreover
have the advantage that they do not relate to accidental differences but are rather open
findings grounded in theory.
13
References:
Cantril, H., Gaudet, Hazel and Herta Herzog (1940): The Invasion from Mars. A Study in the
Psychology of Panic. Princeton, New Jersey, Princeton University Press 1940.
Herzog, Herta (1955): Why Did People Believe in the “Invasion from Mars”? [originally
written in 1938] In: Lazarsfeld-Rosenberg eds. 1955: The Language of Social Research. A
Reader in the Methodology of Social Research. New York, The Free Press 1955, pp. 420 428
Homans, G.C. (1950): The Human Group. Harcourt, New Yourk, Brace & World, Inc.
Lazarsfeld, P.F. (1996): “Hörerbefragung der Ravag.” (manuscript 1932) In: Mark, D. ed.:
Paul Lazarsfelds Wiener RAVAG - Studie 1932 : Der Beginn der modernen
Rundfungforschung. Wien, Guthmann-Peterson, pp. 27-66, numbered and cited
according pp. 1 – 40.
Merton, R.K. (1949): “Patterns of Influence: A Study of Interpersonal Influence and of
Commnunications Behavior in a Local Community.” In: Lazarsfeld, P.F., Stanton, F.N.
eds.: Communications Research 1948 – 1949. New York, Harper & Brothers 1949,
pp.180-219.
Suchman, E.A. (1941): Invitation to Music. A Study of the Creation of New Music Listeners
by the Radio. In: Lazarsfeld, P.F. and F.N. Stanton eds. (1941): Radio Research 1941.
New York, Duel, Sloan and Pearce, pp.140 - 188.
Whyte, W.F. (1993) [ orig. 1943]. Street Corner Society. 4th edition. Chicago: Chicago
University Press.
14