Key Variables: Social Science
Measurement and Functional Form
Presentation to: ‘Interpreting results from statistical modelling –
a seminar for social scientists’ , Imperial College, 29th April 2008
Dr Paul Lambert and Dr Vernon Gayle
University of Stirling
A seminar for the ESRC National Centre for Research Methods, LancasterWarwick Node on ‘Developing Statistical Modelling in the Social Sciences’
ESRC - NCRM - Apr 2008
1
Key Variables: Social Science Measurement and
Functional Form
1)
Working with variables - ‘Beta’s in
Society’ and ‘Demystifying Coefficients’
2)
Key Variables and social science
measurement
- Harmonisation and standardisation
- Comments and speculation
3)
Functional Form
ESRC - NCRM - Apr 2008
2
‘Beta’s in Society’ and ‘Demystifying Coefficients’
 Dorling, D., & Simpson, S. (Eds.). (1999). Statistics in Society: The Arithmetic of
Politics. London: Arnold.
 Irvine, J., Miles, I., & Evans, J. (Eds.). (1979). Demystifying Social Statistics.
London: Pluto Press.
• Famous works on critical interpretation of social statistics
tend to have a univariate / bivariate focus
– Measuring unemployment; averaging income; bivariate
significance tests; correlation v’s causation
• But social survey analysts usually argue that complex
multivariate analyses are more appropriate..
 Critical interpretation of joint relative effects
 Attention to effects of ‘key variables’ in multivariate analysis
ESRC - NCRM - Apr 2008
3
• “A program like SPSS .. has two main components:
the statistical routines, .. and the data management
facilities. Perhaps surprisingly, it was the latter that
really revolutionised quantitative social research”
[Procter, 2001: 253]
• “Socio-economic processes require comprehensive
approaches as they are very complex (‘everything
depends on everything else’). The data and
computing power needed to disentangle the multiple
mechanisms at work have only just become
available.” [Crouchley and Fligelstone 2004]
ESRC - NCRM - Apr 2008
4
Large scale survey data: 2
technological themes
• We’re data rich (but analysts’ poor)
– Plenty of variables (a thousand is common)
– Plenty of cases
• We work overwhelmingly through individual
analysts’ micro-computing
– impact of mainstream software
– Pressure for simple / accessible / popular analytical techniques
(whatever happened to loglinear models?)
– Propensity for simple ‘data management’
– Specialist development of very complex analytical
packages for very simple sets of variables
ESRC - NCRM - Apr 2008
5
Survey research: Access, manipulate & analyse
patterns in variables (‘variable by case matrix’)
ESRC - NCRM - Apr 2008
6
Working with variables = understanding
‘variable constructions’
processes by which survey measures are defined
and subsequently interpreted by research analysts
• Meaning?
– Coding frames; re-coding decisions; metric
transformations and functional forms; relative
effects in multivariate models
– Data collection and data analysis
– Cf. www.longitudinal.stir.ac.uk/variables/
ESRC - NCRM - Apr 2008
7
β’s - Where’s the action?
•
If we have lots of variables, lots of cases, but
simple techniques and software, the action is in
the variable constructions…
i.
How we chose between alternative measures
ii. How much data management we try
(or bother with)
iii. How we analyse & interpret the coefficients from
the measures we use (..this seminar..)
ESRC - NCRM - Apr 2008
8
i) Choosing measures
See (2) below
• A sensible starting point is with ‘key variables’
• Approaches to standardisation / harmonisation
• {Lack of} awareness of existing resources
See (3) below
• Influence of functional form
ESRC - NCRM - Apr 2008
9
ii) Data management – e.g. recoding data
Count
educ4
1.00
Degree
-9. 00
Highest
-9 Mis sing or wild
educat ional -7 Proxy respondent
qualific ation
1 Higher Degree
2.00
Diploma
Total
323
0
0
0
0
323
982
0
0
0
0
982
0
425
0
0
0
425
2 Firs t Degree
0
1597
0
0
0
1597
3 Teac hing QF
0
0
340
0
0
340
4 Other Higher QF
0
0
3434
0
0
3434
5 Nurs ing QF
0
0
161
0
0
161
6 GCE A Levels
0
0
0
1811
0
1811
7 GCE O Levels or Equiv
0
0
0
0
2518
2518
8 Commercial QF, No O
Levels
0
0
0
331
0
331
9 CSE Grade 2-5,S cot
Grade 4-5
0
0
0
0
421
421
10 Apprentices hip
0
0
0
257
0
257
102
0
0
0
0
102
0
0
0
0
2787
2787
138
0
0
0
0
1545
2022
3935
2399
5726
11 Other QF
12 No QF
13 Still At School No QF
Total
3.00 Higher 4.00 S chool
sc hool or
level or
voc ational
below
ESRC - NCRM - Apr 2008
10
138
15627
ii) Data management – e.g. Missing data / case selection
ESRC - NCRM - Apr 2008
11
ii) Data management – e.g. Linking data
Linking via ‘ojbsoc00’ :
c1-5 =original data / c6 = derived from data / c7 = derived from www.camsis.stir.ac.uk
ESRC - NCRM - Apr 2008
12
Data Management through e-Social
Science (DAMES – www.dames.org.uk)
• Supporting operations on data widely performed by
social science researchers
•
•
•
•
Matching data files together
‘Cleaning’ data
Operationalising variables
Specialist data resources (occupations; education; ethnicity)
• Why is e-Social Science relevant?
• Dealing with distributed, heterogeneous datasets
• Generic data requirements / provisions
• Lack of previous systematic standards (e.g. metadata; security;
citation procedures; resources to review/obtain suitable data)
ESRC - NCRM - Apr 2008
13
• A substantial social science need for improved
standards and resources in data management
UK Data Archive
Qualidata
Flagship social surveys
Office for National Statistics
Administrative data
Specialist academic outputs
DAMES
ONS support
ESDS support
NCRM workshops
Essex summer school
ESRC RDI initiatives
CQeSS
Data Management
Data access / collection
Data Analysis
 In practice, social researchers often spend more time on data
management than any other part of the research process
 A ‘methodology’ of data management is relevant to social science
literatures on ‘harmonisation’, ‘comparability’
ESRC - NCRM - Apr 2008
14
Working with variables – further issues
• Re-inventing the wheel
– …In survey data analysis, somebody else has already
struggled through the variable constructions your are
working on right now…
– Increasing attention to documentation and replicability
[cf Dale 2006; Freese 2007]
• Guidance and support
– In the UK, use www.esds.ac.uk
– Most guidance concerns collecting & harmonising data
– Far less is directed to analytically exploiting measures
ESRC - NCRM - Apr 2008
15
Key Variables: Social Science Measurement and
Functional Form
1)
Working with variables - ‘Beta’s in Society’
and ‘Demystifying Coefficients’
2)
Key Variables and social science
measurement
- Harmonisation and standardisation
- Comments and speculation
3)
Functional Form
ESRC - NCRM - Apr 2008
16
Key variables and social science
measurement
Defining ‘key variables’
- Commonly used concepts with numerous previous
examples
- Methodological research on best practice / best
measurement
[cf. Stacey 1969; Burgess 1986]
ONS harmonisation ‘primary standards’
http://www.statistics.gov.uk/about/data/harmonisation/primary_standards.asp
ESRC - NCRM - Apr 2008
17
Key variables: concepts and measures
Variable
Concept
Something useful
Occupation
Class; stratification;
unemployment
www.geode.stir.ac.uk
Education
Credentials; Ability; Merit
www.equalsoc.org/8
Ethnic group
Ethnicity; race; religion;
national origins
[Bosveld et al 2006]
Age
Age; life course stage;
cohort
[Abbott 2006]
Gender
Gender; household / family www.genet.ac.uk
context
Income
Income; wealth; poverty;
ESRC - NCRM - Apr 2008
www.data-archive.ac.uk
[SN 3909]
18
Key variables –Standardisation
• Much attention to key variables involves proposing
optimum / standard measures
• UK – ONS Harmonisation
• EU – Eurostat standards
• Projects on ‘criterion’ and ‘construct’ validity
– (see later on occupations)
• Standardisation impacts other analyses
– Affects available data
– Affects popular interpretations of data
ESRC - NCRM - Apr 2008
19
Key variables – Harmonisation
(across countries; across time periods)
• “a method for equating conceptually similar but
operationally different variables..” [Harkness et al 2003, p352]
• Input harmonisation
[esp. Harkness et al 2003]
‘harmonising measurement instruments’ [H-Z and Wolf 2003, p394]
– unlikely / impossible in longer-term longitudinal studies
– common in small cross-national and short term lngtl. studies
• Output harmonisation (‘ex-post harmonisation’)
‘harmonising measurement products’ [H-Z and Wolf 2003, p394]
ESRC - NCRM - Apr 2008
20
More on harmonisation
[esp. HZ and Wolf 2003, p393ff]
• Numerous practical resources to help with input
and output harmonisation
– [e.g. ONS www.statistics.gov.uk/about/data/harmonisation ; UN / EU /
NSI’s; LIS project www.lisproject.org; IPUMS www.ipums.org ]
– [Cross-national e.g.: HZ & Wolf 2003; Jowell et al. 2007]
• Room for more work in justifying/ understanding
interpretations after harmonisation
ESRC - NCRM - Apr 2008
21
Equivalence
• “the degree to which
survey measures or
questions are able to
assess identical
phenonema across two
or more cultures”
[Harkness et al 2003, p351]
Measurement
equivalence
involves same instruments
and equality of measures
(e.g. income in pounds)
Functional
equivalence involves
different instruments, but
addresses same concepts
(e.g. inflation adjusted
income)
ESRC - NCRM - Apr 2008
22
“Equivalence is the only meaningful criterion if data is to be
compared from one context to another. However,
equivalence of measures does not necessarily mean that
the measurement instruments used in different countries
are all the same. Instead it is essential that they measure
the same dimension. Thus, functional equivalence is
more precisely what is required”
[HZ and Wolf 2003, p389]
ESRC - NCRM - Apr 2008
23
Harmonisation & equivalence combined
‘Universality’ or ‘specificity’ in variable
constructions
Universality: collect harmonised measures, analyse standardised
schemes
Specificity: collect localised measures, analyse functionally equivalent
schemes
Most prescriptions aim for universality
But specificity is theoretically better
Specificity is more easily obtained than is often realised
Especially for well-known ‘key variables’
ESRC - NCRM - Apr 2008
24
Key variables: comments and
speculation
a) Data manipulation skills and inertia
• I would speculate that around 80% of applications using
key variables don’t consult literature and evaluate
alternative measures, but choose the first convenient
and/or accessible variable in the dataset
 Data supply decisions (‘what is on the archive version’) are critical
• Much of the explanation lies with lack of confidence in data
manipulation / linking data
• Too many under-used resources – cf. www.esds.ac.uk
ESRC - NCRM - Apr 2008
25
b) Software and key variables – a personal view
• Stata is the superior package for secondary
survey data analysis:
• Advanced data management and data analysis functionality
• Supports easy evaluation of alternative measures (e.g. est
store)
• Culture of transparency of programming/data manipulation
• Problems with Stata
• Not available to all users
• {Slow estimation times}
ESRC - NCRM - Apr 2008
26
c) Endogeneity and key variables
• ‘everything depends on everything else’ [Crouchley and
Fligelstone 2004]
• We know a lot about simple properties of key variables
– Key variables often change the main effects of other variables
– Simple decisions about contrast categories can influence
interpretations
– Interaction terms are often significant and influential
• We have only scratched the surface of understanding
key variables in multivariate context and interpretation
– Key variables are often endogenous (because they are ‘key’!)
– Work on standards / techniques for multi-process systems and/or
comparing structural breaks involving key variables is attractive
ESRC - NCRM - Apr 2008
27
Key Variables: Social Science Measurement and
Functional Form
1)
Working with variables - ‘Beta’s in Society’
and ‘Demystifying Coefficients’
2)
Key Variables and social science
measurement
- Harmonisation and standardisation
- Comments and speculation
3)
Functional Form
ESRC - NCRM - Apr 2008
28
‘Functional form’
The way in which measures are
arithmetically incorporated in analysis
a) Level of measurement (nominal, ordinal,
interval, ratio)
b) Alternative models and link functions
c) Other variables and interaction effects
ESRC - NCRM - Apr 2008
29
a) Levels of measurement and the
desire to categorise
• Categories are easier to envisage /
communicate
• Much harmonisation work ≡ locating into categories
• Appearance of measurement equivalence
• But functional equivalence is seldom achieved
• Metrics are better for functional equivalence
• E.g. Standardised income
• How to deal with categorisations?
– The qualitative foundation of quantity [Prandy 2002a]
ESRC - NCRM - Apr 2008
30
In support of scaling…
• Many concepts can be reasonably regarded as metric
– cf. simplified / dichotomisted categorisations
• Comparability / standardisation is easier with scales
• Complex / Multi-process systems are easier with scales
– Structural Equation Models
– Interaction effects
• Growing availability/use of distance score techniques
– Stereotyped ordered logit [‘slogit’ in Stata]
– Correspondence Analysis
– Latent variable models
• …But, scaling seems to be seen as a wicked, positivistic
activity..!
ESRC - NCRM - Apr 2008
31
Practical suggestions on the level of
measurement
• It’s rare not to have a few alternative measures
of the same concepts at different levels of
measurement
Good practice would be to
– try alternative measures and see what difference they
make
– consider treatment of missing values in relation to
measurement instrument choice
– Engage as much as possible with other studies
ESRC - NCRM - Apr 2008
32
b) Alternative models and link
functions
• The functional form of the outcome
variable(s) is of greatest importance
(influences which model is used)
• ‘Link functions’ perform the maths to allow
for alternative functional forms of the
outcome variable
• See [Talk 1] for popular alternative models
ESRC - NCRM - Apr 2008
33
Practical observations on link functions
•
•
Social scientists are unduly conservative
in choosing between alternative models
[We tend to favour binary or metric
outcomes and single process systems]
i. Substantively, this isn’t ideal
ii. Pragmatically, it’s no longer necessary
ESRC - NCRM - Apr 2008
34
i) Substantive risks (of conservative
model choice)
• Attenuated findings
– Concentrate on certain category contrasts
– Ignore or exacerbate extremes of distribution
• Mis-specification
– Ignore / mis-measure relevant β’s
– Ignore / over-emphasise other contextual patterns
• Endogeneity
– ignoring multiprocess system may bias results
(e.g. selection bias)
ESRC - NCRM - Apr 2008
35
ii) Pragmatics of model choice
• General rapid expansion in model
functionality in statistical packages
• Stata stands out for it wide range of data
management and data analysis
functionality
– E.g. ‘statsby’; ‘est table’; ‘outreg2’ for testing and
comparing related models with different
combinations of variables
ESRC - NCRM - Apr 2008
36
c) Other variables and interaction
effects
• A very important influence on one RHS
coefficient is what else is in the RHS and
what it is interacted with
ESRC - NCRM - Apr 2008
37
Advice on Interaction Effects
• Start with main effects – get a good idea how they
work
• Be careful how you fit interaction effects
–
–
–
–
–
–
Often appealing substantively
In practice not always significant (especially higher order)
Hard to interpret higher order interactions
Over-fit - check for replication (e.g. in other datasets)
Always wise to formally test interactions (cf. armchair critics)
Best to construct your own interaction variable(s) and maybe
fit them as a single X (especially complicated categorical
interactions)
ESRC - NCRM - Apr 2008
38
Interpreting other variable effects linear cf. categorical outcomes
GHS Data
OLS: Y = age left education (years)
Logit: Y = Graduate / Non Graduate
X Vars
Female
4-category social Class
(Advantaged; Lower Supervisory; Semi-routine; Routine)
Age (centred at 40)
39
Regression Estimates
A
Female
B
-0.32
Age (40)
-0.06
C
D
E
-0.34
-0.27
-0.06
-0.05
Supervisory
SemiRoutine
-1.83
-1.85
-1.98
-1.88
Routine
-2.40
-2.33
Constant
17.52
17.5 17.75 18.22 18.54
ESRC - NCRM - Apr 2008
40
Linear Regression Models
• 1 unit change in X leading to a b change in Y
• The b is consistent – minor insignificant
random variation (survey data)
• As long as the X vars are uncorrelated
(a classical regression assumption)
ESRC - NCRM - Apr 2008
41
Estimates (logit scale)
Parameterization ??
A
Female
B
C
-0.24
Age (40)
-0.03
D
E
-0.23
-0.20
-0.03
-0.04
Supervisory
-1.46
-1.52
Semi-Routine
-1.82
-1.87
Routine
Constant
-2.65
-0.39
-2.70
-0.04
-0.90
-0.80
ESRC - NCRM - Apr 2008
-0.68
42
Logit Model
• Estimates on a logit scale
• The b estimates a shift from X1=0 to X1=1 leads to a
change in the log odds of y=1
• Even when the X vars are uncorrelated, including
additional variables can lead to changes in b
estimates
• The b estimates the effect given all other X vars in the
model
2/3)
• Fixed variance in theESRC
logit
model
(p
- NCRM - Apr 2008
43
Non-linear outcomes
• Be sensible about how you parameterize coefficients
• Be careful interpreting them…
• Don’t throw variables in like a ‘bull in a china shop’
• Model checking – make sure you understand how
the ‘right hand side’ (rhs) is working
44
Summary – Social science measurement and
functional form
• We argue that the route to better critical
understanding of variable effects combines complex
analysis with many mundane, prosaic tasks in
checking data
– ANALYSIS: Coefficient effects in multivariate models;
multi-process models; understanding interactions; etc
– DATA MANAGEMENT: Re-coding data; linking data;
missing data mechanisms; reviewing literature
• Seldom central to previous methodological reviews
• Cf. www.dames.org.uk
ESRC - NCRM - Apr 2008
45
References
 Abbott, A. (2006). Mobility: What? When? How? In S. L. Morgan, D. B. Grusky & G. S. Fields
(Eds.), Mobility and Inequality. Stanford University Press.
 Bosveld, K., Connolly, H., Rendall, M. S., & (2006). A guide to comparing 1991 and 2001 Census
ethnic group data. London: Office for National Statistics.
 Burgess, R. G. (Ed.). (1986). Key Variables in Social Investigation. London: Routledge.
 Crouchley, R., & Fligelstone, R. (2004). The Potential for High End Computing in the Social
Sciences. Lancaster: Centre for Applied Statistics, Lancaster University, and
http://redress.lancs.ac.uk/document-pool/hecsspotential.pdf.
 Dale, A. (2006). Quality Issues with Survey Research. International Journal of Social Research
Methodology, 9(2), 143-158.
 Dorling, D., & Simpson, S. (Eds.). (1999). Statistics in Society: The Arithmetic of Politics. London:
Arnold.
 Freese, J. (2007). Replication Standards for Quantitative Social Science: Why Not Sociology?
Sociological Methods and Research, 36(2), 2007.
 Harkness, J., van de Vijver, F. J. R., & Mohler, P. P. (Eds.). (2003). Cross-Cultural Survey Methods.
New York: Wiley.
 Hoffmeyer-Zlotnik, J. H. P., & Wolf, C. (Eds.). (2003). Advances in Cross-national Comparison: A
European Working Book for Demographic and Socio-economic Variables. Berlin: Kluwer Academic /
Plenum Publishers.
 Irvine, J., Miles, I., & Evans, J. (Eds.). (1979). Demystifying Social Statistics. London: Pluto Press.
 Jowell, R., Roberts, C., Fitzgerald, R., & Eva, G. (2007). Measuring Attitudes Cross-Nationally.
London: Sage.
 Prandy, K. (2002). Measuring quantities: the qualitative foundation of quantity. Building Research
Capacity, 2, 3-4.
 Procter, M. (2001). Analysing Survey Data. In G. N. Gilbert (Ed.), Researching Social Life, Second
Edition (pp. 252-268). London: Sage.
 Stacey, M. (Ed.). (1969). Comparability in Social Research. London: Heineman.