MEDIATION AND GENERALIZABILITY OF THE BFLPE
1
Big fish in little ponds aspire more: Mediation and cross-cultural generalizability of
school-average ability effects on self-concept and career aspirations in science
-- Supplemental Materials --
Benjamin Nagengast
University of Tübingen, Germany
University of Oxford, United Kingdom
Herbert W. Marsh
University of Oxford, United Kingdom
University of Western Sydney, Australia
King Saud University, Saudi Arabia
MEDIATION AND GENERALIZABILITY OF THE BFLPE
2
Supplemental Materials
To be published on a separate webpage and linked to the published manuscript
Appendix A: Supplemental and Methodological Details
A1. Cultural Dimensions
A2. Standardization
A3. Weights
A4. Missing data
A5. Statistical Models
Appendix B: Additional Tables
Table B1: Intra-class correlation coefficients for the main variables
Table B2: Coefficients of the multi-country BFLPE model for academic self-concept in
Table B3: Coefficients of the multi-country BFLPE model for career aspirations in science
Table B4: Coefficients of the multilevel mediation model for career aspirations
Table B5: Correlations of country-specific parameters of BFLPE and multilevel mediation
models with country-level-predictors as analysed in the canonical correlation analysis
MEDIATION AND GENERALIZABILITY OF THE BFLPE
3
Appendix A: Supplemental and Methodological Details
A1. Cultural Dimensions
Here, we further describe the scores on the original five cultural dimensions (Individualismcollectivism, power distance, uncertainty avoidance, masculinity and long-term orientation)
that were obtained from Hofstede et al. (2010). Item examples are taken from the Values
Survey Module 1994 (Hofstede, 1994).
Individualism-collectivism refers to the degree to which individuals see themselves
integrated into groups within a culture (e.g. “In choosing your ideal job, how important
would it be to you to have sufficient time for your personal or family life?” [higher
importance in individualistic societies]). This was represented by a bipolar index. Higher
values indicate a more individualistic cultural orientation within a country; lower values
indicate a more collectivist cultural orientation within a country. The scores for the countries
in the present sample ranged from 13 to 91 (M = 51.429, SD = 22.680) and were evenly
distributed.
Power distance refers to the acceptance of power differences by less powerful
members of institutions (e.g. “In choosing your ideal job, how important would it be to you to
be consulted by your direct superior in his/ her decisions?” [higher importance in societies
with little power distance]). High values indicate a larger acceptance of inequalities within a
country, low values indicate less acceptance of inequalities within a country. The countryspecific scores in the present sample ranged from 11 to 104 (M = 54.47, SD = 21.09).
Uncertainty avoidance refers to the degree to which members of a society are tolerant
to uncertainty and ambiguity (e.g. “Competition between employees usually does more harm
than good.” [higher agreement in societies high on uncertainty avoidance]). High values
indicate low levels of comfort in unstructured and novel situations, low values indicate high
MEDIATION AND GENERALIZABILITY OF THE BFLPE
4
levels of comfort in unstructured and novel situations. The country-specific values in the
present sample of countries ranged from 23 to 112 (M = 70.35, SD = 21.19).
Masculinity assesses the extent to which emotional roles are endorsed by the genders
within a society on a bipolar scale (e.g. “In choosing your ideal job, how important would it
be to you to work with people who cooperate well with another?” [higher importance in more
feminine societies]). In feminine societies, women strongly endorse traditional feminine
values (e.g., being modest and caring) and men also favour these values somewhat more. In
masculine societies, both women and men more strongly endorse masculine values (e.g.,
being assertive and competitive), although men do so to a greater extent than women. The
country-specific values for masculinity in the present sample ranged from 9 to 110 (M =
48.04, SD = 22.08).
Long-term orientation refers to the degree to which a culture is focused on pragmatic
solutions and adaptations to future challenges, represented by high scale values, as compared
to fostering values related to the past and traditions, represented by low scale values (e.g. “In
your private life, how important is respect for tradition to you?” [higher importance in societies
with more long-term orientation]). The country-specific values for long-term orientation in the
present sample of countries ranged from 13 to 100 (M = 52.60, SD = 21.97).
A2. Standardization
The plausible values for ACH as well as the responses to the six ASC items and four
FUT items were standardized separately for the analyses. First, the average country-values
were subtracted from all variables to control for mean differences in the countries that would
otherwise have confounded variance components and the BFLPE in the total sample analyses
(Moerbeek, 2004). These country-residualized variables were then standardized to have a
mean of zero and a variance of one over the total sample. In order to test the quadratic effect
of L1-ACH, the standardized ACH values were first centered around the corresponding
MEDIATION AND GENERALIZABILITY OF THE BFLPE
5
school-mean and then squared. These squared values were then used as L1-predictors
(defined with the statement WITHIN) in the Mplus models. The standardized ACH values
were used in Mplus to account for the context effects of ACH. In order to keep the variables
in the same metric, the squared and aggregated values were not re-standardized. The same
standardized variables were used in the cross-national comparisons.
A3. Weights
PISA 2006 used a complex sampling design (see OECD, 2009a, for details). As a
consequence, selection probabilities differ between student and schools and all analyses of
the PISA 2006 data have to use the appropriate weights – that additionally correct for school
and student non-response – to obtain representative results (see OECD, 2009a). Following the
current recommendations for weighting in multilevel models (Asparouhov, 2006, 2008;
Carle, 2009; Stapleton, 2006), we used the final school weight as weight at L2 and the
within-school student weight as weight at L1. The within-school weight was obtained by
taking the ratio of the final student weight and the final school weight provided in the
international PISA database. The weights at both levels were scaled to sum up to the
corresponding sample sizes (the default option in Mplus, see Muthén & Muthén, 1998-2010).
All models used the MLR-estimator in Mplus that yields standard errors and model fit
statistics that are robust to non-normality and non-independence of responses. The analysis
for the total international sample used the design-based correction of standard errors and fit
statistics treating country as a stratification variable implemented to control for nesting of
students and schools within countries (see Sterba, 2009, for an accessible introduction).
A4. Missing data
There were no missing data for the plausible values of ACH. The amount of missing
data for questionnaire items was small (ranging from 2.3 to 2.4% per item for the four
indicators of FUT and from 7.9 to 8.5% per item for the six indicators of ASC). Multiple
MEDIATION AND GENERALIZABILITY OF THE BFLPE
6
imputation (Schafer, 1997; Graham, 2009) was used to deal with missing data in the items.
Ten imputations were conducted in SPSS 17 using MCMC imputation. Each imputation used
different subsets of plausible values for science, reading and maths ACH (see earlier
discussion of the appropriate use of plausible values in PISA data) and a variety of
background variables at the student- and the school-level. The contextual variables at the
school-level accounted for the between-school differences. All analyses were run separately
for the ten imputed datasets. The individual results were combined to obtain the final
parameter estimates and standard errors using the imputation facility in Mplus 6.
A5. Statistical Models
Assessment of model fit. Assessment of fit of multilevel structural equation models
is an evolving area of research. Ryu and West (2009) suggested evaluating the model fit
separately for the L1 and the L2-models and provided strategies for estimating level-specific
-test statistics and fit indices. We followed their strategy in an initial investigation of fit of
the multilevel CFA-model. However, their strategy does not apply to models that constrain
parameters to be equal across the levels that we tested next. In addition, no clear guidelines
for the interpretation and cut-off values for model fit indices in multilevel SEM exist. Hence,
we relied on criteria of model fit for conventional singlel-evel SEMs. Following Marsh,
Balla, and Hau (1996; also see Marsh, Balla, & McDonald, 1988; Marsh, Hau, & Wen, 2004)
we considered the Tucker-Lewis index (TLI), the comparative fit index (CFI), and the root
mean square error of approximation (RMSEA) to evaluate goodness of fit, as well as the test statistic and an evaluation of parameter estimates. The TLI and CFI vary along a 0-to-1
continuum in which values greater than .90 and .95 are typically taken to reflect acceptable
and excellent fits to the data, respectively. RMSEA values of less than .06 are taken to reflect
a reasonable fit. Whereas, RMSEA values greater than .10 are unacceptable, although no
golden rule exists (Chen, Curran, Bollen, Kirby, & Paxton, 2008; Hu & Bentler, 1999;
MEDIATION AND GENERALIZABILITY OF THE BFLPE
7
Marsh, et al., 2004). The CFI contains no penalty for a lack of parsimony, so that improved
fit due to the introduction of additional parameters may reflect capitalization on chance,
whereas the TLI and RMSEA contain penalties for a lack of parsimony (for further
discussion see Cheung & Rensvold, 2002; Hu & Bentler, 1999; Marsh et al., 2004). For
comparison of nested models by differences in fit indices, we followed the recommendations
given by Chen (2007) and Cheung and Rensvold (2002) who suggested that a decrease in fit
for the more parsimonious model of less than .01 for incremental fit indices like the CFI,
should be treated as support for this model. Chen (2007) suggested that when the RMSEA
increases by less than .015 there is support for the more constrained model.
Calculation of contextual effects. The contextual effects models and the multilevel
mediation model were estimated using the reflective aggregation procedure in Mplus
(Muthén & Muthén, 1998-2010) that uses implicit group-mean centering of all L1-variables.
This implies that the partial regression weights associated with L1-variables reflect L1effects, while the partial regression weights associated with L2-variables reflect L2-effects
that are not controlled for L1-differences (Enders & Tofighi, 2007; Kreft et al., 1995).
Estimates of contextual effects, that represent the effect of L2-variables after controlling for
L1-differences, can be obtained by subtracting the L1-effect from the L2-effect (Enders &
Tofighi, 2007; Kreft et al., 1995)
context = L2 - L1
(1)
where L2 is the L2-effect, L1 is the L1-effect and context is the contextual effect. The
standard error for the contextual effect was obtained with the multivariate delta method (see
Raykov & Marcoulides, 2004).
A similar logic applied to the effect estimates in the multilevel mediation model.
Estimates for contextual direct effects of ACH on ASC and on FUT and of ASC on FUT
could simply be obtained by applying Equation (1) to the corresponding parameter estimates.
MEDIATION AND GENERALIZABILITY OF THE BFLPE
8
An estimate for the indirect contextual effect was obtained by subtracting the indirect L1effect from the indirect L2-effect
INDcontext = INDL2 – INDL1= (1 L2 * L2) – (1 L1* L1)
(2)
where 1 L1 is the linear effect of L1-ACH on ASC, L1 is the effect of L1-ASC on FUT, 1 L2
is the linear effect of L2-ACH on ASC and L2 is the linear effect of L2-ASC on FUT. Again,
the standard error for the indirect contextual effect was obtained with the multivariate delta
method.
A reviewer suggested an alternative equivalent decomposition of the indirect
contextual effect that emphasizes the 2-1/2-1 mediational chain at its heart. Here, the indirect
contextual effect is decomposed into one component that is transmitted by the individuallevel mediator, i.e. total ASC and into a second component that assesses the additional
contribution of the class-average of the mediator, i.e. of L2-ASC. More formally, the indirect
contextual effect can be decomposed as follows:
INDcontext = IND2-1-1 + IND2-2-1
= 1context*L1 + 1L2*context
(3)
where IND2-1-1 , defined as the product of 1context, the contextual effect of ACH on ASC, and
L1 the L1-effect of ASC on FUT, is the component of the indirect contextual effect that is
transmitted by individual ASC and IND2-2-1, defined as the product of 1L2, the L2-effect of
ACH on ASC and context , the contextual effect of ASC on FUT, is the component transmitted
by L2-ASC in addition to individual ACH. The equivalence of Equations (2) and (3) can be
easily proofed by substituting the definitions of 1context = 1 L2 - 1 L1 and context = L2 - L1
into Equation (3).
Effect size for indirect effects. The effect size of the indirect effects was evaluated as
the proportion of the total effect and with Preacher and Kelley’s (2011) 2-statistic that
relates the observed indirect effect to the maximum possible indirect effect given the
MEDIATION AND GENERALIZABILITY OF THE BFLPE
9
constraints of the variance-covariance matrix of the involved variables. For evaluating the
indirect effect at L1, the maximum indirect effect could be calculated from the variancecovariance matrix of L1-ACH, L1-ASC and L1-FUT following the rules set out by Preacher
and Kelley (2011). For the indirect contextual effect, we computed the maximum possible
indirect effect by subtracting the maximum possible indirect effect at L1 from the maximum
possible indirect effect at L2 and compared the contextual effect obtained from Equation (2)
to this quantity following Preacher and Kelley’s (2011) formulae.
Calculating the BFLPE in the presence of quadratic L1-effects. Most BFLPE
research has found a quadratic relation between L1-ACH and L1-ASC (e.g. Marsh & Hau,
2003; Marsh & Rowe, 1996; Seaton et al., 2009). The presence of a quadratic relation at L1
complicates the interpretation of the contextual effect. It implies that the conventional
contextual effect defined in Equation (1) can only be interpreted as the BFLPE for a student
with an achievement equal to the average of the school he is attending. If the L1-relation is a
non-linear function that can be represented as linear in its parameters, the contextual effect
can be generally defined as the difference between the L2 effect and the first partial
derivative of the L1-function (see Hayes & Preacher, 2010; Stolzenberg, 1980).
context_f = L2 - ASC / L1-ACH).
(4)
In case of a linear relation at L1, Equation (4) simplifies to Equation (1). For a quadratic
relation at L1, the contextual effect context_f varies with L1-ACH according to the following
function
context_f = L2 - L1 – 2 L12 L1-ACH
(5)
where L12 is the regression weight for the quadratic component L1-ACH2. The definition in
Equation (5) will be equal to the definition of a contextual effect in Equation (1) when L1ACH is equal to zero. In other words, Equation (1) represents the BFLPE for an average
student in a class, for whom group-mean-centered L1-ACH will be equal to zero. The change
MEDIATION AND GENERALIZABILITY OF THE BFLPE
10
in the BFLPE depends on the coefficient of L12: If the quadratic effect is positive, the
resulting BFLPE will be larger for relatively high-achieving students in a school compared to
relatively low-achieving students. If the quadratic effect is negative, the resulting BFLPE will
be larger for relatively low-achieving students in a school.
Instantaneous indirect effects. The quadratic relation between L1-ACH and L1-ASC
also complicate the definition of indirect effects (Hayes & Preacher, 2010; Imai, Keele &
Tingley, 2010; Pearl, 2010). Recently, Hayes and Preacher (2010) derived an approach that
addresses non-linearities when the relation between predictor and mediator can be described
with a differentiable function of the predictor. They showed that whenever there are nonlinear relations between predictor and mediator, the conventional linear indirect effect
estimator is only a conditional estimator at the point at which the predictor variable has the
value of zero. A non-linear predictor-mediator relation implies that the indirect effects vary
depending on the predictor variable. Furthermore, they devised a general method for
accounting of non-linearities in the relation between predictor and mediator and showed how
the instantaneous indirect effect and its dependency on the predictor variable could be
modelled.
In the present investigation, we extended Hayes and Preacher (2010) approach to a
multilevel mediation model with contextual effects. As a quadratic effect of L1-ACH on ASC
was included in the model, the indirect effect of L1-ACH on FUT and the indirect contextual
effect varied as a function of L1-ACH (Hayes & Preacher, 2010). The unconditional indirect
effects as obtained above reflect the indirect effects when L1-ACH is zero corresponding to
the grand mean in our analysis. In line with the rules outlined by Hayes and Preacher (2010),
instantaneous indirect effects were obtained for both L1-ACH and the contextual effect of
L2-ACH that showed changes in the indirect effects as functions of L1-ACH. The indirect
effects of L1-ACH on FUT could be expressed as
MEDIATION AND GENERALIZABILITY OF THE BFLPE
11
INDL1 = (1 L1 + 22 L1 * L1-ACH) * L1
= 1 L1* L1 + 22 L1* L1 * L1-ACH
(6)
where 1 L1 is the linear effect of L1-ACH on ASC, 2 L1 is the quadratic effect of L1-ACH
on ASC and L1 is the effect of L1-ASC on FUT. The indirect contextual effect of L2-ACH
on FUT as a function of L1-ACH could be expressed as
INDcontext = (1 L2 * L2) – INDL1
= (1 L2 * L2) - 1 L1* L1 - 22 L1* L1 * L1-ACH
(7)
where 1 L2 is the linear effect of L2-ACH on ASC and L2 is the linear effect of L2-ASC on
FUT. This derivation shows that the indirect contextual effect of L2-ACH on FUT varies
only as a function of L1-ACH when there are quadratic effects of L1-ACH.
MEDIATION AND GENERALIZABILITY OF THE BFLPE
12
Appendix B: Additional Tables
To be published on a separate webpage and linked to the published manuscript
Table B1: Intra-class correlation coefficients for the main variables
,.
Country
Azerbaijan
Argentina
Australia
Austria
Belgium
Brazil
Bulgaria
Canada
Chile
Chinese Taipei
Colombia
Croatia
Czech Republic
Denmark
Estonia
Finland
France
Germany
Greece
Hong Kong
Hungary
Iceland
Indonesia
Ireland
Israel
Italy
Japan
Jordan
Korea
Kyrgyzstan
Latvia
Lithuania
Luxembourg
Macao-China
Mexico
Montenegro
Netherlands
New Zealand
Norway
Poland
Portugal
Qatar
Science
Achievement
0.511
0.550
0.181
0.578
0.573
0.502
0.478
0.259
0.592
0.431
0.311
0.438
0.487
0.162
0.202
0.068
0.443
0.595
0.488
0.379
0.586
0.091
0.445
0.203
0.327
0.560
0.501
0.249
0.309
0.392
0.171
0.234
0.296
0.272
0.459
0.304
0.664
0.206
0.135
0.189
0.336
0.538
Academic SelfConcept in
Science
0.142
0.060
0.034
0.064
0.092
0.049
0.059
0.021
0.037
0.030
0.053
0.033
0.049
0.046
0.025
0.014
0.028
0.035
0.027
0.024
0.053
0.027
0.146
0.043
0.095
0.071
0.065
0.045
0.101
0.098
0.037
0.032
0.018
0.026
0.035
0.030
0.041
0.028
0.027
0.039
0.018
0.032
Science Career
Aspirations
0.114
0.067
0.032
0.096
0.072
0.093
0.098
0.044
0.049
0.048
0.095
0.058
0.073
0.023
0.072
0.011
0.069
0.043
0.029
0.005
0.115
0.013
0.155
0.024
0.099
0.120
0.098
0.046
0.128
0.184
0.064
0.037
0.017
0.010
0.134
0.094
0.061
0.042
0.027
0.022
0.057
0.064
MEDIATION AND GENERALIZABILITY OF THE BFLPE
13
Uruguay
0.383
0.278
0.474
0.377
0.602
0.200
0.161
0.403
0.257
0.443
0.574
0.269
0.323
0.422
0.037
0.094
0.042
0.065
0.028
0.038
0.028
0.054
0.126
0.031
0.068
0.022
0.051
0.022
0.024
0.149
0.070
0.104
0.110
0.032
0.037
0.104
0.069
0.032
0.069
0.029
0.000
0.047
Total Sample
0.413
0.061
0.078
Romania
Russian Federation
Serbia
Slovak Republic
Slovenia
Spain
Sweden
Switzerland
Thailand
Tunisia
Turkey
United Kingdom
United States
MEDIATION AND GENERALIZABILITY OF THE BFLPE
14
Table B2: Coefficients of the multi-country BFLPE model for academic self-concept in
science
Country
Predictors
L1Achievement
SE
Squared L1Achievement
SE
Contextual
effect of L2Achievement
SE
Azerbaijan
Argentina
Australia
Austria
Belgium
Brazil
Bulgaria
Canada
Chile
Chinese Taipei
Colombia
Croatia
Czech Republic
Denmark
Estonia
Finland
France
Germany
Greece
Hong Kong
Hungary
Iceland
Indonesia
Ireland
Israel
Italy
Japan
Jordan
Korea
Kyrgyzstan
Latvia
Lithuania
Luxembourg
Macao-China
Mexico
Montenegro
Netherlands
New Zealand
Norway
Poland
Portugal
Qatar
Romania
Russian Federation
Serbia
Slovak Republic
Slovenia
Spain
Sweden
Switzerland
Thailand
Tunisia
Turkey
United Kingdom
United States
Uruguay
0.120
0.107
0.324
0.301
0.282
0.066
0.094
0.347
0.168
0.107
0.056
0.161
0.165
0.300
0.223
0.303
0.301
0.315
0.175
0.267
0.166
0.391
-0.082
0.331
0.269
0.193
0.220
0.176
0.251
-0.032
0.098
0.148
0.232
0.162
0.045
0.076
0.300
0.275
0.290
0.168
0.225
0.226
0.066
0.122
0.119
0.192
0.171
0.288
0.353
0.306
-0.015
0.099
0.187
0.275
0.282
0.142
0.033
0.021
0.011
0.024
0.015
0.019
0.018
0.010
0.023
0.026
0.018
0.019
0.015
0.013
0.016
0.015
0.024
0.019
0.018
0.017
0.024
0.015
0.026
0.015
0.025
0.013
0.019
0.016
0.045
0.018
0.018
0.015
0.019
0.018
0.021
0.022
0.022
0.015
0.015
0.013
0.021
0.022
0.034
0.012
0.026
0.015
0.022
0.012
0.016
0.014
0.030
0.023
0.034
0.016
0.017
0.019
0.021
0.027
0.044
0.073
0.078
0.021
0.024
0.039
0.062
0.074
0.045
0.046
0.059
0.072
0.042
0.041
0.071
0.054
0.058
0.055
0.046
0.063
0.009
0.053
0.037
0.057
0.042
0.045
0.099
0.003
0.036
0.069
0.068
0.065
0.031
0.047
0.091
0.061
0.063
0.074
0.060
0.014
0.042
0.039
0.067
0.043
0.035
0.082
0.048
0.077
-0.025
0.050
0.022
0.042
0.054
0.021
0.047
0.023
0.006
0.022
0.015
0.025
0.018
0.007
0.024
0.018
0.016
0.017
0.017
0.012
0.016
0.014
0.020
0.018
0.018
0.017
0.023
0.011
0.034
0.014
0.013
0.012
0.016
0.012
0.040
0.014
0.019
0.013
0.014
0.015
0.025
0.014
0.022
0.009
0.012
0.011
0.019
0.019
0.033
0.012
0.020
0.014
0.022
0.012
0.012
0.011
0.034
0.025
0.060
0.008
0.013
0.018
-0.154
-0.177
-0.168
-0.231
-0.183
-0.118
-0.073
-0.234
-0.118
-0.080
-0.129
-0.123
-0.221
-0.190
-0.182
-0.254
-0.226
-0.301
-0.148
-0.209
-0.209
-0.173
-0.195
-0.191
-0.222
-0.212
-0.097
-0.105
0.050
-0.187
-0.118
-0.135
-0.076
-0.160
-0.061
-0.136
-0.287
-0.235
-0.198
-0.126
-0.274
-0.269
-0.087
-0.222
-0.141
-0.189
-0.188
-0.080
-0.177
-0.198
-0.176
-0.117
-0.109
-0.225
-0.352
-0.158
0.093
0.051
0.034
0.038
0.036
0.033
0.043
0.031
0.043
0.039
0.038
0.043
0.024
0.057
0.035
0.126
0.033
0.031
0.031
0.042
0.038
0.075
0.044
0.063
0.062
0.034
0.031
0.044
0.078
0.034
0.043
0.039
0.032
0.055
0.036
0.053
0.041
0.043
0.050
0.049
0.036
0.035
0.056
0.045
0.042
0.036
0.033
0.031
0.087
0.042
0.077
0.039
0.056
0.034
0.090
0.031
MEDIATION AND GENERALIZABILITY OF THE BFLPE
15
Note: Effects that are significant at p = 0.05 are printed in bold. L1 = student-level, L2 =
school-level, SE = standard error.
MEDIATION AND GENERALIZABILITY OF THE BFLPE
16
Table B3: Coefficients of the multi-country BFLPE model for career aspirations in science
Country
Predictors
L1Achievement
SE
Squared L1Achievement
SE
Contextual
effect of L2Achievement
SE
Azerbaijan
Argentina
Australia
Austria
Belgium
Brazil
Bulgaria
Canada
Chile
Chinese Taipei
Colombia
Croatia
Czech Republic
Denmark
Estonia
Finland
France
Germany
Greece
Hong Kong
Hungary
Iceland
Indonesia
Ireland
Israel
Italy
Japan
Jordan
Korea
Kyrgyzstan
Latvia
Lithuania
Luxembourg
Macao-China
Mexico
Montenegro
Netherlands
New Zealand
Norway
Poland
Portugal
Qatar
Romania
Russian Federation
Serbia
Slovak Republic
Slovenia
Spain
Sweden
Switzerland
Thailand
Tunisia
Turkey
United Kingdom
United States
Uruguay
-0.037
0.083
0.292
0.160
0.252
-0.030
-0.046
0.318
0.118
0.123
-0.056
0.074
0.061
0.216
0.092
0.307
0.231
0.176
0.153
0.308
0.080
0.331
-0.039
0.278
0.216
0.118
0.173
0.193
0.211
-0.082
0.033
0.047
0.145
0.109
-0.009
-0.114
0.259
0.253
0.175
0.049
0.276
0.213
-0.086
-0.031
0.016
0.056
0.126
0.322
0.226
0.158
-0.011
0.116
0.142
0.271
0.222
0.015
0.043
0.026
0.012
0.026
0.020
0.031
0.025
0.015
0.033
0.024
0.026
0.024
0.019
0.019
0.021
0.014
0.024
0.027
0.028
0.026
0.027
0.019
0.022
0.023
0.027
0.017
0.021
0.019
0.053
0.020
0.024
0.022
0.026
0.020
0.034
0.029
0.032
0.018
0.021
0.016
0.025
0.030
0.038
0.017
0.029
0.024
0.021
0.017
0.019
0.018
0.027
0.025
0.055
0.016
0.024
0.028
-0.019
0.054
0.067
0.046
0.103
0.049
0.032
0.046
0.073
0.078
0.027
0.059
0.046
0.113
0.070
0.082
0.105
0.059
0.071
0.087
0.060
0.087
0.035
0.058
0.069
0.068
0.052
0.071
0.134
-0.024
0.054
0.070
0.097
0.099
0.054
0.089
0.100
0.077
0.084
0.082
0.126
0.068
0.058
0.035
0.081
0.032
0.004
0.113
0.070
0.077
0.039
0.050
-0.014
0.071
0.058
0.053
0.055
0.025
0.009
0.027
0.018
0.030
0.019
0.009
0.026
0.019
0.035
0.021
0.021
0.013
0.017
0.012
0.022
0.024
0.021
0.018
0.027
0.014
0.040
0.016
0.018
0.016
0.018
0.017
0.026
0.019
0.017
0.017
0.016
0.016
0.031
0.021
0.031
0.012
0.016
0.015
0.025
0.025
0.039
0.015
0.024
0.019
0.029
0.016
0.014
0.016
0.031
0.029
0.070
0.011
0.021
0.021
-0.100
-0.266
-0.106
-0.059
-0.117
-0.208
-0.118
-0.183
-0.113
-0.004
-0.233
-0.065
-0.012
-0.104
-0.218
-0.306
-0.039
-0.113
-0.178
-0.214
-0.153
-0.250
-0.165
-0.087
-0.269
-0.154
0.046
-0.255
0.088
-0.291
0.049
-0.026
0.024
-0.080
-0.356
-0.173
-0.222
-0.219
-0.055
0.002
-0.135
-0.304
-0.033
-0.220
-0.196
-0.035
-0.010
-0.091
-0.102
-0.002
-0.124
-0.065
-0.030
-0.191
-0.340
-0.121
0.096
0.055
0.040
0.051
0.035
0.051
0.047
0.033
0.049
0.056
0.066
0.063
0.043
0.062
0.054
0.096
0.039
0.045
0.051
0.050
0.045
0.095
0.058
0.044
0.080
0.044
0.036
0.055
0.142
0.061
0.080
0.046
0.043
0.049
0.060
0.099
0.054
0.061
0.068
0.055
0.055
0.052
0.047
0.067
0.055
0.053
0.040
0.049
0.070
0.086
0.070
0.043
0.105
0.039
0.042
0.043
MEDIATION AND GENERALIZABILITY OF THE BFLPE
17
Note: Effects that are significant at p = 0.05 are printed in bold. L1 = student-level, L2 =
school-level, SE = standard error.
MEDIATION AND GENERALIZABILITY OF THE BFLPE
18
Table B4: Coefficients of the multilevel mediation model for career aspirations
Country
Predictors
L1-ACH
(direct)
Azerbaijan
Argentina
Australia
Austria
Belgium
Brazil
Bulgaria
Canada
Chile
Chinese Taipei
Colombia
Croatia
Czech Republic
Denmark
Estonia
Finland
France
Germany
Greece
Hong Kong
Hungary
Iceland
Indonesia
Ireland
Israel
Italy
Japan
Jordan
Korea
-0.111
0.020
0.066
0.033
0.071
-0.071
-0.099
0.114
-0.004
0.060
-0.090
-0.003
-0.024
0.058
-0.031
0.162
0.036
-0.012
0.021
0.140
0.002
0.101
0.008
0.054
0.050
0.004
0.062
0.064
0.048
Squared
L1-ACH
SE
0.043
0.025
0.011
0.023
0.016
0.028
0.022
0.014
0.031
0.016
0.024
0.023
0.018
0.018
0.019
0.014
0.022
0.024
0.021
0.024
0.026
0.018
0.020
0.022
0.022
0.015
0.020
0.015
0.038
-0.032
0.038
0.037
0.015
0.051
0.036
0.018
0.024
0.029
0.032
0.000
0.038
0.018
0.075
0.047
0.062
0.059
0.026
0.027
0.053
0.038
0.050
0.030
0.022
0.048
0.035
0.030
0.038
0.069
SE
L1-ASC
0.062
0.020
0.008
0.029
0.016
0.025
0.021
0.009
0.025
0.016
0.038
0.019
0.020
0.012
0.017
0.010
0.018
0.022
0.018
0.015
0.026
0.012
0.033
0.013
0.015
0.014
0.015
0.015
0.025
0.609
0.585
0.689
0.420
0.639
0.618
0.558
0.583
0.718
0.586
0.610
0.475
0.511
0.522
0.543
0.473
0.643
0.592
0.748
0.620
0.467
0.582
0.574
0.670
0.610
0.583
0.503
0.731
0.645
L1-ACH
(indirect)
SE
0.029
0.041
0.015
0.025
0.021
0.032
0.032
0.015
0.035
0.020
0.041
0.026
0.029
0.021
0.032
0.024
0.025
0.026
0.023
0.020
0.025
0.020
0.027
0.024
0.028
0.020
0.025
0.028
0.028
0.073
0.064
0.226
0.128
0.182
0.041
0.053
0.204
0.121
0.063
0.034
0.077
0.085
0.158
0.123
0.145
0.195
0.189
0.132
0.167
0.078
0.230
-0.047
0.224
0.166
0.114
0.111
0.130
0.163
L2-ACH
(context)
SE
0.020
0.013
0.009
0.013
0.012
0.012
0.011
0.008
0.018
0.016
0.011
0.011
0.009
0.009
0.012
0.010
0.017
0.014
0.014
0.013
0.012
0.011
0.015
0.013
0.017
0.008
0.010
0.013
0.029
0.000
-0.159
-0.008
0.018
0.000
-0.113
-0.078
-0.104
-0.050
0.027
-0.152
-0.014
0.088
-0.015
-0.126
-0.222
0.076
0.063
-0.071
-0.076
-0.022
-0.138
0.030
0.062
-0.144
0.004
0.083
-0.186
-0.180
L2-ASC
(context)
SE
0.076
0.042
0.035
0.047
0.032
0.045
0.049
0.039
0.051
0.035
0.067
0.058
0.038
0.047
0.059
0.092
0.045
0.037
0.040
0.041
0.046
0.088
0.077
0.047
0.042
0.031
0.036
0.045
0.084
0.104
0.078
0.124
0.273
0.025
0.393
0.128
0.532
0.507
0.539
0.014
0.176
-0.249
0.067
0.290
0.555
0.431
0.310
0.205
-0.116
0.595
-0.032
0.288
-0.153
0.524
0.698
0.119
0.120
0.755
L2-ACH
(indirect,
context)
SE
0.187
0.348
0.142
0.186
0.128
0.282
0.483
0.229
0.317
0.181
0.365
0.380
0.243
0.133
0.341
0.350
0.392
0.226
0.262
0.178
0.201
0.202
0.174
0.162
0.110
0.127
0.194
0.238
0.353
-0.097
-0.112
-0.098
-0.080
-0.116
-0.095
-0.041
-0.076
-0.063
-0.031
-0.081
-0.053
-0.100
-0.092
-0.087
-0.093
-0.116
-0.178
-0.108
-0.138
-0.129
-0.110
-0.194
-0.151
-0.117
-0.155
-0.035
-0.069
0.268
SE
0.068
0.042
0.031
0.025
0.024
0.033
0.027
0.042
0.054
0.039
0.033
0.029
0.019
0.040
0.034
0.133
0.038
0.026
0.029
0.024
0.037
0.063
0.062
0.042
0.069
0.043
0.028
0.034
0.173
MEDIATION AND GENERALIZABILITY OF THE BFLPE
Kyrgyzstan
Latvia
Lithuania
Luxembourg
Macao-China
Mexico
Montenegro
Netherlands
New Zealand
Norway
Poland
Portugal
Qatar
Romania
Russian
Federation
Serbia
Slovak Republic
Slovenia
Spain
Sweden
Switzerland
Thailand
Tunisia
Turkey
United Kingdom
United States
Uruguay
19
-0.063
-0.019
-0.021
0.014
0.034
-0.032
-0.164
0.066
0.065
0.020
-0.043
0.140
0.045
-0.134
0.018
0.023
0.020
0.020
0.018
0.031
0.023
0.031
0.015
0.021
0.014
0.025
0.023
0.033
-0.026
0.035
0.039
0.059
0.069
0.037
0.057
0.041
0.034
0.050
0.041
0.090
0.058
0.027
0.018
0.015
0.016
0.013
0.013
0.029
0.020
0.027
0.009
0.015
0.015
0.019
0.019
0.034
0.577
0.534
0.456
0.559
0.460
0.514
0.648
0.639
0.681
0.531
0.541
0.601
0.736
0.741
0.024
0.036
0.032
0.023
0.020
0.033
0.024
0.030
0.023
0.024
0.027
0.025
0.018
0.039
-0.019
0.053
0.068
0.131
0.075
0.023
0.050
0.193
0.189
0.155
0.092
0.137
0.168
0.049
0.010
0.011
0.008
0.013
0.009
0.011
0.015
0.017
0.012
0.010
0.008
0.015
0.017
0.024
0.001
0.118
0.035
0.103
-0.010
-0.320
0.033
-0.045
-0.077
0.007
0.065
0.066
-0.076
0.031
0.059
0.070
0.042
0.570
0.044
0.062
0.068
0.046
0.040
0.062
0.047
0.052
0.041
0.042
0.911
0.589
0.332
-0.221
-0.089
0.389
1.486
0.704
0.658
0.391
0.041
0.657
0.648
0.063
0.172
0.356
0.357
3.521
0.142
0.282
0.281
0.221
0.248
0.221
0.170
0.527
0.227
0.163
-0.295
-0.072
-0.059
-0.078
-0.074
-0.037
-0.207
-0.178
-0.142
-0.064
-0.066
-0.203
-0.227
-0.067
0.063
0.048
0.029
0.561
0.021
0.029
0.115
0.051
0.058
0.049
0.030
0.053
0.040
0.045
-0.097
-0.055
-0.034
0.035
0.126
0.035
-0.001
-0.003
0.047
0.011
0.082
0.047
-0.086
0.015
0.029
0.023
0.020
0.014
0.017
0.016
0.022
0.021
0.043
0.014
0.023
0.024
0.014
0.041
0.012
-0.016
0.056
0.044
0.036
0.052
0.016
-0.025
0.043
0.026
0.039
0.013
0.022
0.018
0.024
0.011
0.013
0.015
0.022
0.030
0.055
0.010
0.018
0.018
0.542
0.596
0.465
0.535
0.677
0.538
0.521
0.517
0.698
0.695
0.683
0.612
0.701
0.027
0.030
0.029
0.030
0.017
0.024
0.020
0.033
0.034
0.030
0.019
0.028
0.030
0.067
0.072
0.090
0.092
0.197
0.192
0.161
-0.008
0.070
0.132
0.189
0.174
0.101
0.007
0.018
0.009
0.013
0.009
0.013
0.010
0.016
0.017
0.025
0.014
0.014
0.014
-0.049
-0.088
0.049
0.096
-0.046
-0.085
0.030
-0.078
0.021
0.038
-0.050
-0.160
0.014
0.045
0.043
0.048
0.032
0.058
0.078
0.063
0.084
0.035
0.070
0.034
0.039
0.043
0.600
0.587
0.248
0.524
0.050
0.440
0.594
-0.234
0.278
0.214
0.299
-0.526
1.075
0.152
0.223
0.248
0.317
0.174
0.270
0.256
0.309
0.264
0.209
0.247
0.128
0.506
-0.183
-0.102
-0.087
-0.110
-0.044
-0.022
-0.037
-0.046
-0.088
-0.063
-0.141
-0.179
-0.131
0.054
0.044
0.025
0.028
0.043
0.079
0.054
0.050
0.036
0.053
0.033
0.019
0.052
Note: Effects that are significant at p = 0.05 are printed in bold. ASC = academic self-concept in science; ACH = science achievement; FUT =
career aspirations in science; L1 = student-level; L2: school-level; context = contextual effect; direct = direct effect; indirect = indirect effect,
mediated by ASC; instantaneous = coefficient of instantaneous indirect effect; SE = standard error.
MEDIATION AND GENERALIZABILITY OF THE BFLPE
20
Table B5: Correlations of country-specific parameters of BFLPE and multilevel mediation models with country-level-predictors as analysed in
the canonical correlation analysis
1
2
3
4
5
6
7
8
9
10
11
12
Cultural factors
1 Human Development Index
2 Power Distance
3 Individualism
4 Masculinity
5 Uncertainty Avoidance
6 Long-Term Orientation
1
-0.63
0.66
0.05
-0.32
0.06
1
-0.64
0.15
0.48
0.17
1
0.09
-0.49
0.02
1
0.08
0.06
1
0.07
1
Country-specific effects
7 ASC on L1-ACH
8 FUT on L1-ACH (direct)
9 FUT on L1-ACH (total)
10 ASC on L2-ACH
11 FUT on L2-ACH (direct)
12 FUT on L2-ACH (total)
0.91
0.66
0.84
-0.31
0.11
0.11
-0.63
-0.48
-0.58
0.37
0.04
0.09
0.68
0.35
0.54
-0.51
0.15
-0.04
0.01
-0.05
-0.03
-0.06
0.03
-0.04
-0.40
-0.35
-0.38
0.36
0.06
0.13
0.06
-0.11
-0.07
0.15
0.32
0.47
1
0.64
0.88
-0.38
0.08
0.07
1
0.91
-0.35
-0.08
-0.12
1
-0.38
-0.01
-0.05
1
-0.09
0.45
1
0.75
1
Standard Deviation
0.12
0.07
0.09
0.10
0.06
21.20
22.49
22.54
21.64
21.25
0.10
0.12
Note: The correlations and standard deviations are based on the countries with complete data for the cultural factors. ASC = science selfconcept; L1-ACH = student-level achievement in science; FUT = career aspirations in science; L2-ACH = school-level achievement in science;
direct = direct effect; total = total effect. All effects of L2-ACH are contextual effects.
MEDIATION AND GENERALIZABILITY OF THE BFLPE
21
References
Asparouhov, T. (2006) General multi-level modelling with sampling weights.
Communications in Statistics – Theory and Methods, 35, 439-460.
Asparouhov T. (2008): Scaling of sampling weights for two level models in Mplus 4.2. Los
Angeles, CA: Muthén and Muthén.
Carle, A. (2009). Fitting multilevel models in complex survey data with design weights:
Recommendations. BMC Medical Research Methodology, 9,49.
Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance.
Structural Equation Modeling, 14, 464-504.
Chen, F., Curran, P. J., Bollen, K. A., Kirby, J., & Paxton, P. (2008). An empirical evaluation
of the use of fixed cutoff points in RMSEA test statistic in structural equation models.
Sociological Methods & Research, 36, 462-494. doi: 10.1177/0049124108314720.
Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing
measurement invariance. Structural Equation Modeling, 9, 233-255.
Enders, C. K. & Tofighi, D. (2007). Centering predictor variables in cross-sectional
multilevel models: A new look at an old issue. Psychological Methods, 12,121-138.
Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annual
Review of Psychology, 60, 549-576.
Hayes, A. F. & Preacher, K. J. (2010). Quantifying and testing indirect effects in simple
mediation models when the constituent paths are nonlinear. Multivariate Behavioral
Research, 45, 627-660. doi: 10.1080/00273171.2010.498290.
Hofstede, G. (1994). Values Survey Module 1994. Manual. Retrieved from
http://www.geerthofstede.nl/research--vsm/vsm-94.aspx on August 15th, 2011.
MEDIATION AND GENERALIZABILITY OF THE BFLPE
22
Hofstede, G., Hofstede, G. J., & Minkov, M. (2010). Cultures and organizations: Software of
the mind (3rd edition). New York: McGraw-Hill.
Hu, L. & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis:
Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.
doi: 10.1080/10705519909540118.
Imai, K., Keele, L., & Tingley, D. (2010). A general approach to causal mediation analysis.
Psychological Methods, 15, 309-334. doi: 10.1037/a0020761.
Kreft, I. G. G., de Leeuw, J., & Aiken, L. S. (1995). The effect of different forms of centering
in hierarchical linear models. Multivariate Behavioral Research, 30, 1–21.
Marsh, H. W., Balla, J. R., & Hau, K. T. (1996). An evaluation of incremental fit indexes: A
clarification of mathematical and empirical properties. In G. A. Marcoulides & R. E.
Schumacker (Eds.), Advanced structural equation modeling techniques (pp. 315-353 .
Mahwah , NJ : Lawrence Erlbaum.
Marsh, H. W., Balla, J. R., & McDonald, R. P. (1988). Goodness-of-fit indexes in
confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103,
391– 410.
Marsh, H. W., & Hau, K. T. (2003). Big-fish-little-pond effect on academic self-concept. A
cross-cultural (26-country) test of the negative effects of academically selective
schools. American Psychologist, 58(5), 364–376.
Marsh, H. W., Hau, K. T., & Wen, Z. (2004). In search of golden rules: Comment on
hypothesis-testing approaches to setting cutoff values for fit indices and dangers in
overgeneralizing Hu and Bentler’s (1999) findings. Structural Equation Modeling, 11,
320-341.
MEDIATION AND GENERALIZABILITY OF THE BFLPE
23
Marsh, H. W., & Rowe, K. J. (1996). The negative effects of school-average ability on
academic self-concept: An application of multilevel modelling. Australian Journal of
Education, 40, 65–87.
Moerbeek, M. (2004). The consequence of ignoring a level of nesting in multilevel analysis.
Multivariate Behavioral Research, 39, 129-149.
Muthén, L.K. and Muthén, B.O. (1998-2010). Mplus User’s Guide (6th edition). Los
Angeles, CA: Muthén & Muthén.
Organisation for Economic Co-operation and Development. (2009a). PISA 2006 Technical
Report. Paris: Author.
Pearl, J. (2010). The mediation formula: A guide to the assessment of causal pathways in
non-linear models. Retrieved from ftp://ftp.cs.ucla.edu/pub/stat_ser/r363.pdf.
Preacher, K. J., & Kelley, K. (2011). Effect size measures for mediation models: Quantitative
strategies for communicating indirect effects. Psychological Methods, 16, 93-115.
doi: 10.1037/a0022658
Raykov, T., & Marcoulides, G. A. (2004). Using the delta method for approximate interval
estimation of parameter functions in SEM. Structural Equation Modeling, 11, 621637.
Ryu, E., & West, S. G. (2009). Level-specific evaluation of model fit in multilevel structural
equation modelling. Strutural Equation Modeling, 16, 583-601. doi:
10.1080/10705510903203466.
Schafer, J. L. (1997). Analysis of incomplete multivariate data. Boca Raton, FL: Chapman &
Hall.
Seaton, M., Marsh, H. W., & Craven, R. G. (2009). Earning its place as a pan-human theory:
Universality of the big-fish-little-pond effect across 41 culturally diverse countries.
Journal of Educational Psychology, 101,403-419.
MEDIATION AND GENERALIZABILITY OF THE BFLPE
24
Stapleton, L. (2006). An assessment of practical solutions for structural equation modelling
with complex sample data. Structural Equation Modeling, 13, 28-58.
Sterba, S. K. (2009) Alternative model-based and design-based frameworks for inference
from samples to populations: From polarization to integration. Multivariate
Behavioral Research, 44, 711-740. doi: 10.1080/00273170903333574.
Stolzenberg, R. M. (1980). The measurement and decomposition of causal effects in
nonlinear and nonadditive models. Sociological Methodology, 11, 459–488.