Collecting the Right Data
Dr Stephen Hincks
Department of Urban Studies and Planning
Todays Session…
1. Research Design – recap
2. Generating Data - methods
3. Sampling
4. Using Case Studies in Research
The research design…
?
Methodology vs Method
• Methodology
– Systematic framework that guides the
research
– Ontology – positivist, constructivist…
– Steps and practices
• Methods
• The ‘tactics’ or ‘tools’ used by
researchers to collect data (Robson,
2002)
– Case studies
– Interviews
– Statistical modelling….
Quantitative and Qualitative
Approaches to Data Collection
Data…
“In the developed world, we
often have far more data
than we can ever use. In
most cases, what is lacking
is not data but an
understanding of what is
important ....”
(Lawrence G. 1997 Indicators for sustainable development, in
Dodds F. (ed) The Way Forward : Beyond Agenda 21, London,
Earthscan.)
Data Collection and Analysis
• Collection and analysis are likely to be closely
intertwined缠绕的；错综复杂的
• Iterative 重复的process of collection and analysis
Collection
Analysis
Collection
Analysis
Collection
Analysis
Methods to Generate Data: Some key
decisions…
Will I use:
• Secondary data
collection
(assumptions, design,
collection, presentation,
and sometimes analysis
by others).
• Primary data collection
(your own and original).
• Quantitative data.
• Qualitative data.
• Mixed data and
methods.
–
Philip, L.J. (1998) ‘Combining quantitative
and qualitative approaches to social
research in human geography – an
impossible mixture?’ Environment and
Planning A, 30, 261-276.
Quantitative - Definition
• A quantitative approach is one in which the
investigator primarily uses post-positivist
claims for developing knowledge (i.e. cause
and effect thinking, reduction to specific
variables and hypotheses and questions, use
of measurement and observation, and the test
of theories). (Creswell, 2003, p.19)
Quantitative approaches
• Attempts to explain phenomena by collecting and
analysing numerical data
• Tells you if there is a “difference” but not necessarily
why
• Data collected are always numerical and analysed
using statistical methods – what are statistical
methods?
• Variables are controlled as much as possible
• Randomisation 随机化to reduce subjective bias
Gathering quantitative data
• Data sources include
– Surveys where there are a large number of
respondents (esp where you have used a Likert
scale)
– Observations (counts of numbers and/or coding
data into numbers)
– Secondary data (government data; third party
data etc.)
• Analysis techniques include hypothesis
testing, correlations and regression…
Black swans and falsifiability可证伪性
• Falsifiability or refutability. 可反驳性 of
a statement, hypothesis, or theory is the
inherent与生俱来的，固有的
possibility that it can be proven false
• Karl Popper and the black swan;
https://www.flickr.com/photos/lselibrary/
deductive c.f. inductive reasoning
IMAGELIBRARY/5
• Hypothesis testing
• Start with null hypothesis
i.e. H0 – that there will be no difference
CC BY-SA 3.0,
https://commons.wikimedia.org/w/index.php?curid=1243220
Limitations of quantitative research?
• Some things can’t be measured – or measured
accurately
• Doesn’t tell you why – does it need to?
• Can be impersonal 没有人情味的– no engagement
with human behaviours or individuals – mmmmm.
• Data can be static – snapshots of a point in time – or
not…
• Can tell a version of the truth (or a lie?) – doesn’t all
data?
“Lies, damned lies and statistics” – persuasive power of
numbers – difficult to argue against!
Qualitative - Definition
• A qualitative approach is one in which the inquirer
often makes knowledge claims based primarily on
constructivist perspectives 建构主义观点(i.e. the
multiple meanings of individual experiences,
meanings socially and historically constructed, with
an intent of developing a theory or pattern) or
advocacy/participatory perspectives (i.e. political,
issue-oriented, collaborative or change oriented) or
both. (Creswell, 2003, p.18)
Qualitative approaches
• Any research that doesn’t involve numerical
data
• Instead uses words, pictures, photos, videos,
audio recordings. Field notes, generalities.
Peoples’ own words.
• Tends to start with a broad question rather
than a specific hypothesis
• Develop theory rather than start with one
 inductive rather than deductive
• 归纳而不是演绎
Gathering qualitative data
• Interviews (structured, semi-structured半结构化
的 or unstructured)
• Focus groups
• Questionnaires or surveys
• Secondary data, including diaries, self-reporting,
written accounts of past events/archive把..存档
data and company reports;
• Direct observations – may also be recorded
(video/audio)
• Ethnography人种学
Gathering qualitative data
• Tends to yield产生 rich data to explore how and why
things happened
• Don’t need large sample sizes (in comparison to
quantitative research)
• Some issues may arise, such as
– Respondents providing inaccurate or false information – or
saying what they think the researcher wants to hear
– Ethical issues may be more problematic有疑问的 as the
researcher is usually closer to participants
– Researcher objectivity may be more difficult to achieve
Limitations of qualitative research?
•
•
•
•
It can be very subjective – if done badly
It can’t always be repeated – or is that true?
It can’t always be generalised – or is this true?
It can’t always give you definite answers in the
way that quantitative research can (seemingly
表明上看来)
Perspectives on Secondary Data
Collection
Perspectives on Secondary Data.
It would be wrong to think of social research solely in terms of
the first-hand collection of data by means of, say, observation
or asking questions. A great deal of information is already
available, having been collected by others and often for other
reasons.
BUT
Both the production of official statistics and their secondary
analysis are not unproblematic enterprises.
LIMITS : ACCESSIBILITY and PROVENANCE
The 'Provenance' of secondary data –
How was the data collected and for what purpose? This will
enable an initial assessment as to the likely reliability of
the data, and an initial assessment about its robustness稳
健性.
•
•
•
Questions to consider include:
If the data were collected for a specific reason, what are
the implications of the rationale of data collection for the
coverage and reliability of the data set?
What are the implications of the data collection
methodology for the coverage /degree of detail available in
the data set?
What period does the data relate to? If the information is
not current, but is being used as a proxy for the prevailing
situation, is there any reason to expect that the there have
been substantial changes in the period since data
collection?
Perspectives on Primary Data
Collection
Interviews: Structured, Semistructured and open
• Particularly useful for gathering qualitative data
• Face to face and telephone
• Highly structured questions are good for eliciting诱发，刺
激 quantitative data
– Can be difficult to gain ‘deep’ understanding of a specific issue
• Open or unstructured interviews allow access to
unpredictable information
– Flexible - allows interviewer to probe answers
– Can become overloaded with irrelevant information
• Semi-structured interviews can be designed to cater for
open and closed questions
I’m Going to Generate My Own
Data: Using Surveys…
Survey design
What information is needed?
• Factual/attitudinal; complex/detailed; long/short;
standard/measurement.
Multiple interests…
• Respondent – length / perceived value.
• Interviewer – ease of use, instructions.
• Researcher – hypothesis, research aims, ideas.
• Analyst – coding and processing.
What resources are available?
• Skills, competencies.
• Equipment, funding.
On-line …
• Eg Survey Monkey
Some Basics…
• Questions should be designed so that it is easy for
respondents to understand them and to answer accurately
and clearly.
• Questionnaires should be easy for interviewers to administer.
• Questionnaires should be constructed so that recorded
answers can be easily edited, coded and transferred onto
computer files for analysis.
• Questionnaires should encourage and keep respondents’
interest. Too many or too complex questions can either
generate bad responses or result in the ‘waste bin’ response!
Questionnaire Design
•
•
•
•
•
•
•
Introduction and purpose – why fill it in?
Sequencing and filtering – ease of use?
Question types – open, closed?
Question wording and construction.
Instructions – clear or confusing?
Appearance and layout.
Consent and confidentiality.
Question wording…avoid:
•
•
•
•
•
•
•
•
Ambiguous questions
Imprecise questions
Presuming questions
Questions that rely on memory
Double questions
Leading questions
Hypothetical questions
Overly sensitive questions
OBSERVATION as survey research.
Observation is planned and conducted in a systematic way, rather
than happening spontaneously and haphazardly as it usually does in
every day life. Appropriate techniques are carefully selected for the
purposes at hand. Observations are systematically recorded rather
than stored only in a personal memory, and are carefully
interpreted and analysed, again employing systematic and planned
procedures. Moreover, the data produced by observational
research are subjected to checks on validity so that we can be more
confident about their accuracy than is usually the case with
observational data produced routinely in everyday life.
10 MINUTE BREAK
Sampling….
Sampling
• Units of analysis.
• Selection and sampling of cases or sources of
data in qualitative and quantitative research.
• Random and non-random sampling,
representative samples and statistical
inference.
• Sampling error and bias.
• What to sample, how many, how do I choose ?
What to sample?
• Sample units – people; households; sites; businesses;
planning applications; local governments; professionals;
buildings; neighbourhoods; town villages, cities; etc.
• Do you know the ‘ideal’ population – A finite, total
known population.
• Can you create or have access to a sampling frame – The
collection of units from which sample will be drawn.
Boundaries, definitions, period, accuracy of frame –
inbuilt bias, timing of selection.
• Sample to frame (statistical test) / frame to population
(non-mathematical justification)
Sample size and statistical relevance…
• As a general rule, the larger the sample size, the
more precisely the sample statistics will reflect
the population parameters.
• The standard error is inversely proportional to
the square root of the sample size ie to double
the accuracy means the sample size will need to
be 4 times bigger; to treble it, then the size
needs to be 9 times bigger etc.
• You need big enough samples and ‘cell sizes’ in
statistical analyses eg chi-square method –
filtering, randomness, resources.
Probabilistic Sampling methods…
• Simple random sampling: each case of the
population has an equal chance of selection –
sample errors and over / under
representation.
• Systematic/interval sampling: an interval is
chosen randomly, and every nth case selected
– periodicity.
Probabilistic Sampling methods…
• Stratified sampling分层抽样: population divided
into subsets according to some characteristic which
you know (or suspect) is significant to the study.
Separate random/systematic samples drawn from
each subset
• What do you know, prior to your research, about
your ‘population’ and which of its characteristics are
important to your study or research question,
hypothesis/theories?
RANDOM
SYSTEMATIC
STRATIFIED
1
1
1
2
2
2
3
3
3
4
4
4
5
5
5
6
6
6
7
7
7
8
8
8
9
9
9
10
10
10
11
11
11
12
12
12
13
13
13
14
14
14
15
15
15
16
16
16
17
17
17
18
18
18
19
19
19
20
20
20
21
21
21
22
22
22
23
23
23
24
24
24
Non-probabilistic Sampling…
• In many cases of our research, random sampling
is not possible or appropriate.
• In these situations non-probabilistic methods of
sampling can be used, and justified.
• Typically used in qualitative research, but
consequently no statistical relevance and unable
to reliably say much about representativeness.
Non-probabilistic Sampling…
• Snowball or chain sampling - technique where existing study
subjects recruit future subjects from among their acquaintances.
• Quota sampling – researchers identify characteristics of the
sample (say age and gender) to be selected in advance, then
select desired number of respondents who fit the characteristics
(say 25 old female; 25 young female; 25 old male; 25 young
male).
• Volunteer sampling – participates in the study volunteer to do
so, in response say to an advertisement or general request to
play a role in research – self-selecting.
• Judgement or purposive sampling – researcher selects a
‘typical’ or ‘interesting’ case or cases for study – might be based
on theoretical or practical justification.
Case Study Approaches
Case study research
• A research design where the case(s) exhibit the operation
of some identified general theoretical principle – theory
led.
• Case studies use multiple methods of data collection or
generation to examine a phenomena in detail – a choice of
methods.
• Simply ‘cases’ referring to situations/ behaviours/ events of
interest to your topic, research question or area of study.
• It HAS to be more than the setting(s) of your research –
“my research was done in Sheffield”.
Main ‘case based’ research study
designs…
Single-case
designs
Multiple case
designs
Holistic (single e.g. Sheffield city e.g. Nottingham,
unit of analysis) centre.
Derby and
Leicester city
centres.
Embedded
e.g specific office e.g. specific
(multiple units
developments in office
of analysis)
Sheffield city
developments in
centre.
the above.
Why use case based studies ?
A Single case study:
• Critical case, testing well-formulated theory.
• Unique/extreme case of a rare or specific
phenomenon.
• Revelatory of previously inaccessible phenomenon
although commonly present.
• In conjunction with other research approaches.
Why use case based studies ?
Multiple case based studies:
• Evidence is often more compelling.
• Study offers a more robust analysis.
• Comparative studies.
• Replication of similarity or contrary results
rather than sampling
• But extensive resources and time.
Comparative Studies
“An aim of comparative research is to understand and explain
the ways in which different societies and cultures experience and
act upon social, economic and political changes, plus how these
views relate to more general changes and thus shared
experiences and actions in the face of similar concerns and
pressures.”
May T. (2001) Social research, 3rd ed. Milton Keynes, OUP.
SEE:
Healey P. & Upton R. (eds) (2010) Crossing borders: international exchange and planning
practices. Routledge: London.
Sanyal B. (ed) (2005) Comparative planning cultures. Routledge: London.
Some Reading…
Bryman A. (1988) Quantity and quality in social science. London, Routledge.
Black T. (1999) Doing quantitative research in the social sciences. London, Sage.
De Vaus D. (1991) Surveys in social research. London, UCL Press.
Fink A. (1995) How to design surveys. London, Sage.
Fink A. (1995) How to ask survey questions. London, Sage.
Fink A. (1995) How to sample surveys. London, Sage.
Fink A. (1995) How to analyse survey data. London, Sage.
Gillham B. (2000) Case study research methods. London, Contiuum.
Gorard S. (2003) Quantitative methods in social science. London, Continuuum.
Gray D. (2004) Doing research in the real world. London, Sage.
Hakim C. (2000) Research design : successful designs for social and economic research. 2nd ed.
London,Sage.
Hedrick T. et al (1993) Applied research design : a practicalguide. CA, Sage.
Kumar R. (1996) Research methodology. Sage, London.
Litwin M. (1995) How to measure survey reliability and validity. London, Sage.
May T. (1997) Social research. Milton Keynes, OUP.
Moser C. & Kalton G. (1985-) Survey methods in social investigation. London, Heinemann.
Oppenheim A. (1996) Questionnaire design and attitude measurement. London, Heinemann.
Punch K. (1998) Introduction to social research. London, Sage.
Sapsford R. & Jupp V. (1996) Data collection and analysis. London , Sage.
Yates S. (2004) Doing social science research. Open University/Sage.
Yin R. (1984 -) Case study research design. London, Sage.
Further links
• Some content borrowed from SkillsYouNeed website
(http://www.skillsyouneed.com/learn/research-methods.html)
Other useful links:
• Introduction to Quantitative and Qualitative Research Models (William
Bardebes). PDF at http://tinyurl.com/qq-models
• Methods Map: http://www.methodsmap.org
• Ready To Research: http://readytoresearch.ac.uk
• Methods@Manchester:
http://www.methods.manchester.ac.uk/resources/categories
• Research Data Management training: http://datalib.edina.ac.uk/mantra/