High School Experience:
Comparing Self-Report and Transcript Data
from the NLSY97
A. Rupa Datta
Parvati Krishnamurty
NORC at the University of Chicago
NLSY97
May 2008
Anniversary Conference
(draft 5/24/08)
10th
1
Introduction
The National Longitudinal Survey of Youth, 1997 Cohort (NLSY97) dataset
include two sources of information about respondents’ high school experiences: selfreports from annual interviews with individuals throughout their high school years, and
abstracted information from their high school transcripts. Although the transcripts and
interview data were designed to complement one another, their co-existence offers the
opportunity to compare interview and transcript data as alternative sources for some key
pieces of data about educational experience. In this paper, we describe the two types of
data collected from these sources and assess the concordance of some measures. We
conclude with some comments about the relative merits and weaknesses of each type of
data for measuring different aspects of high school experience.
The NLSY97 high school transcript data differ from other high school transcript
data sets in a number of ways. Perhaps most relevant is that the NLSY97 (together with
the NLSY79) may be the only American high school transcript data set that does not stem
from a school-based sample. Although there are other, larger transcript data sets available
(for example, NELS:88, ELS, Add Health, NEAP), all of these stem from an originally
school-based sample design. In contrast, the NLSY97 respondents derive from an areaprobability sample. As a result, there are on average fewer than three respondents per
high school in the NLSY97 data.
Another distinctive feature of the NLSY97 transcript data is that all youths who
attended high school were included in the study. Many other transcript data sets include
only high school graduates. Including non-graduates complicates the structure of the
2
dataset and analyses of the data, but obviously lends richness and supports a host of
research questions that are not relevant to a dataset of graduates only.
Also, the NLSY97 schooling experience data are collected in an intervieweradministered format, whereas most major educational studies have collected data in selfadministered form, especially in the early school-based waves. The literature on mode
effects generally finds that self-administration matters more in sensitive items. In the
mode effects literature, sensitive items are often identified as delinquency, income, or
sexual behavior. It’s unclear whether or not some dimensions of high-school experience
may also behave as sensitive items. If so, we would trade off the ability of interviewers
to establish rapport and assist respondents in recall and response construction (thus
improving the quality of self-report data) against social desirability bias which would
diminish the accuracy of self-reports.
Finally, the NLSY97 is the only large high school transcript data set that has such
a wealth of interview data in conjunction with the transcript data. The various
Department of Education data sets (NEAP, ELS, NELS:88, etc.) as well as AddHealth,
all involve infrequent longitudinal interviews at best (the NEAP is cross-sectional). In
contrast, the NLSY97 respondents were interviewed annually throughout high school and
so provide a wealth of interview data that can be considered in tandem with the transcript
data.
Although there are some published studies comparing self-reports with transcript
data on educational experience, these distinctive features of the NLSY97 transcript
provide some motivation for looking to these data to extend the previous body of work
on the relative merits of transcripts and self-reports.
3
Kane, Rouse and Staiger (1999) did compare transcript and self-report data on
post-secondary enrollment in order to improve estimates of the returns to schooling.
Using data from the National Longitudinal Survey of the Class of 1972 as well as the
National Educational Longitudinal Study: High School Class of 1988, they estimated
that in binary categorizations (such as graduation from 4-year college), self-report data
could suffer less reporting error than transcript data. They estimated also that reporting
of years of schooling among non-graduates from college and among high school dropouts
could be quite inaccurate in both transcripts and self-reports.
Naifeh and Shakrani (1996) compared self-reports and transcript data among high
school graduates who had participated in the 12th grade mathematics NEAP in 1990.
They found that concordance rates were low (fewer than 50% exact matches) in reports
of individual math courses taken, numbers of courses taken, highest math course taken,
and combinations of courses taken. Among discordant cases, however, the vast majority
differed by very small magnitudes, typically just one course difference out of eleven
courses tallied. Although there was some tendency for interview data to over-report
relative to transcript data, there were instances of the opposite, including times when an
individual student under-reported some course enrollment and over-reported others
relative to the transcript data.1 The authors found that transcript data exhibited somewhat
better behavior in association with NEAP math achievement test scores, for example,
transcript data but not self-reports were monotonic in percent proficient against number
of math courses. In general, both self-reports and transcript data were associated in
similar (and expected) ways with the NEAP math scores.
1
This might indicate mis-classification of students into the categories of math courses asked about.
4
Kuncel, Crede and Thomas (2005) provide a review of the previously published
literature comparing transcript and self-report data, including a meta-analysis of results
on the accuracy of grades, class ranks, and test scores. They find consistent evidence that
students with more success in school (i.e., higher grades, lower class rank, etc.) report
more accurately. They also find evidence that recall bias results in deterioration of
accuracy as the elapsed time from school enrollment increases. Many of these studies
use data that were collected administratively rather than in a research context. Students
self-reporting grades or class-rank as part of standardized test administration (like the
ACT) or as part of an admissions application may be acutely aware of negative
consequences for mis-reporting, or the likelihood that consumers of their data may also
have access to their high school transcripts for comparison purpose. This threat of
corroboration may induce more accurate reporting than in the no-consequences context of
a survey interview. The reviewed studies clearly take the position that transcript data
represent ‘truth’ and deviation from them in self-reports is indication of inaccurate selfreport data.
Zimmerman, Caldwell, and Bernat (2002) take a different approach to the
transcript/self-report comparison, suggesting that the size and direction of the
discrepancy between the two measures of grade-point average (GPA) may itself be
indicative of a youth’s psychological state, academic beliefs, and problem behaviors. In
their case, they find that self-reports and discrepancies are more predictive of problem
behaviors than are transcript measures of GPA.
Background:
NLSY97 and NLSY97 Transcript Data
5
The NLSY97 sample consists of 8,984 American youths who were born 1980 to
1984 and living in a U.S. household in 1997. The nationally representative sample
includes oversamples of African-American and Hispanic youths. The approximately
9,000 NLSY97 individuals have been sought for interviews annually since then.
Interviews are approximately one hour in duration and most often conducted in person
with a substantial subset of items administered using Audio Self-Administered
technology to provide greater privacy for collection of sensitive items. Data are collected
on a wide range of topical domains, including schooling, employment, family formation,
delinquency, income and assets, and health. Data in key substantive areas are collected
in event history format. Respondents are solicited for each interview regardless of their
participation status in prior interviews. In the ninth round of data collection in 2005, 81.7
percent of the original sample completed an interview. Because many respondents
complete interviews after spells of attrition, 91 percent of the original sample have now
provided information about their lives through 2005.
High school transcript data were collected in 2000 and 2004 concurrent with main
interview data collection in those years. In 2000, high school transcripts were sought for
1622 youths who had 1) previously reported graduating from high school or who had
dropped out of high school and were at least 18 years old, 2) an identifiable high school
reported in interview data, and 3) provided signed permission for NORC to seek access to
their school records. Through unrelated circumstances, several hundred signed waivers
were destroyed, and transcripts could be sought only for those respondents who provided
permission on a newly signed waiver. Transcripts were obtained for 1417 respondents.
6
In 2004, transcripts were sought for 5701 youths whose transcripts were not
received in the initial wave of transcript collection. Wave 2 data was obtained for 4815
respondents. A total of 6,232 transcripts were received and coded for youths across the
two waves of transcript collection. These 6,232 largely comprise the sample for this
analysis. Most of the NLSY97 respondents had completed their high school career by
2004, so we have complete high school transcripts for them.
The data collection was done by mailing a transcript request packet to each school
from which a respondent received his or her high school diploma or the last school the
respondent reported attending during a NLSY97 interview. The packet included a cover
letter addressed to the principal, a one page cover-sheet questionnaire collecting school
specific grading and transcript policies, a student request list listing the sampled students
in the school and the signed permission forms for these students. Transcript, course
catalog and information on special programs were provided by the school.
Transcript data could not be obtained for 2752 youths. Most of the nonrespondents did not sign a permission form or did not have a school identified for mailing
out the transcript requests. In some cases, the school record was not found or the school
or school district refused to release the information. Table 1 shows the number of
NLSY97 respondents for whom transcripts were obtained, those who were not
interviewed and the reasons for non-response.
Table 1: Transcript survey status
Category
Collected
transcripts
Sub-category
Total
6232
% of Sample
69.4%
7
Wave 1
Wave 2
Wave 2 fielded, not
collected
Refusal at district
level
Refusal at School
level
Student record not
found
Student record not
available
1417
4815
886
111
231
427
117
1866
Not fielded for
transcript survey
Permission not
secured
School not
identified
Not mailed, final
blocked
Total
9.9%
20.8%
1572
231
63
8984
100%
Structure of the NLSY97 schooling data
In the schooling section of the NLSY97 Youth Interview, respondents were asked
about their current or most recent school in round 1. From round 2 onwards, data was
collected in a retrospective format so respondents were asked about all the schools they
attended since the last interview. The data is organized so that enrollment dates and
institutions are identified, enrollment spells are rostered and then information about each
enrollment spell is captured.
Self reported data on the respondent’s high school experience include:

Schooling attainment (highest grade completed and highest degree received)

Course of study (college prep, vocational, technical)
8

Types of math, science and other courses taken

Overall grades they received (8th grade, end of high school)

Highest SAT math, verbal or ACT composite score

Number of schools attended

Type of school attended
Additionally there is data on school based learning programs, grade progression,
enrollment status and school characteristics.
Structure of the NLSY97 transcript data
The NLSY97 high school transcript data consists of a set of student-level
variables, information from the school which sent the transcripts, term-specific and
course-specific information. The school from which the high school transcript was
collected is referred to as the primary school in the data. Transcripts often include
information on courses taken at transfer schools and summer programs.
The transcript data include the following:

Information on the primary school: school sector, number of students in the
district, percentage of districts students in grades 9-12, coursework offered (e.g.
Calculus, AP, IB, vocational education)

Term-specific information for up to 18 terms including dates, type of school year,
academic year, grade level, number of credits

Course specific variables including course codes (SST-R), grade and credit value
of the course. Course grades are converted into standard grades.

Carnegie credits for each academic year
9

Cumulative percentage of New Basics curriculum requirements completed by
academic year

Grade point average, school’s report as well as Carnegie credit weighted GPA
overall and by academic year

Test scores for SAT, ACT, SAT-II, PSAT and AP tests.

Number of days absent from school or tardy

School completion status

Enrollment dates

Participation in special, gifted or bilingual programs.

Specialization variables (academic/vocational specialist or concentrator)

Pipeline variables to measure progression in math, foreign language and science
Comparing NLSY97 Self-report and Transcript Data
The NLSY97 youth interview and high school transcript data were designed for
different purposes and with different constraints. Thus, although they both describe high
school experience, they are not intended to provide identical measures of the same
constructs. Rather, each provides some information about the respondent’s high school
experience.
As with other sections of the NLSY97 youth interview, the section on high school
experience was designed with several principles in mind. The data are collected in
month-level event-history format, and great emphasis is placed on asking questions that
respondents can reasonably answer, formulated in ways that are cognitively reasonable.
(An alternative would be to ask what researchers might most want to know – cumulative
10
GPA to the third decimal place, for example – whether or not respondents can know and
report such information accurately.)
Some of the design of the NLSY97 schooling data was informed by experiences
with the NLSY79 and other earlier data sets. In these, students were often observed with
inexplicable regression or jumps in grade of enrollment. It was also difficult to
differentiate among non-graduates, for example, those completing all but one term and
almost all requirements versus those who had never completed a full year’s worth of
courses. With these issues in mind, great emphasis was placed on collecting data to
support very nuanced data on students’ progression throughout high school. Enrollment
status and grades of enrollment are collected for every month so that researchers can
identify and understand regression and jumps in grade level. Progression is addressed in
other ways also, including expected graduation date and point-in-time status reports
supplementing the event history reports. All gaps in enrollment are also collected. The
emphasis on grade progression and month-by-month enrollment status comes instead of
greater detail on high school activities – for example, courses taken, disciplinary actions,
counseling or other services received, extra curricular activities.
Because accurate enrollment status was a principal objective of the NLSY97 in
the early years of data collection, the fielding period was restricted to overlap with the
traditional 10-month American school year. This timing has the advantage of ensuring
that a maximum number of respondents are indeed enrolled at the time of interview. On
the other hand, it implies that the reference period for the interview – from the prior
interview to the current one – will typically span two school years, neither in its entirety.
This framework does not suit asking questions about completed grades, or about course
11
enrollments, since recall periods can be long and salience may be low (much to the
dismay of high school teachers).
The abstraction of data from high school transcripts is also constrained in many
ways. The transcripts tend to have more information on the grades from the primary
school. Information on grades, courses and grading scales from transfer schools or
summer schools may be sketchy. The school may lack complete information on
respondent’s test scores and school attendance as well.
The sample for the main survey and the Transcript survey are compared in Table 2 to see
if there were any major differences in demographic variables, background variables or
aptitude test scores between the two samples.
Table 2: Comparison of the NLSY97 sample and the Transcript sample
Percentage of respondents in each category (Weighted)
Demographic
variable
Sex
Birth year
Race-ethnicity
Region
Mother’s
Value
Male
Female
1980
1981
1982
1983
1984
Black
Hispanic
Mixed
NHNB
Northeast
North Central
South
West
Less than high
NLSY97 sample
(8984 respondents)
51.32
48.68
20.09
19.94
20.27
19.44
20.26
15.42
12.86
1.22
70.51
18.6
26.22
34.29
20.89
23.91
Transcript survey
(6232 respondents)
50.62
49.38
22.52
21.22
19.42
17.72
19.12
15.1
11.91
1.06
71.93
18.34
27.77
32.95
20.93
21.62
12
education
PIAT Math score
(quartiles)
ASVAB scores
(quartiles)
school grad
High school grad
More than high
school
Q1
Q2
Q3
Q4
Q1
Q2
Q3
Q4
34.32
41.77
34.59
43.79
59.16
16.42
14.29
10.13
39.26
20.03
20.18
20.53
59.61
15.77
14.21
10.41
34.26
20.68
22.15
22.91
Transcript survey respondents have slightly higher ASVAB scores, larger percentage of
mothers who are educated beyond high school (and fewer mothers who are high school
dropouts), There are also slight differences in the transcript and main sample by region.
For instance, the transcript survey respondents have a slightly higher percentage of
NHNB (white), older and female respondents compared to the main survey. The
differences are minor and most are in the 1-2% range.
Comparing self-report and transcript measures:
There are a wide variety of measures on which data are available from the
transcripts as well as interviews. Reasons for discrepancies between the two sources can
be categorized as differences in the designs of the data, response errors in respondents’
reports, administrative errors or other differences in interpretation/procedure in school
records, or differences in time periods for which the measures apply. Differences in
design of the data include differences in concepts, wording and time frame of the
questions in the two surveys. Response errors made by respondents include social
desirability bias, cognitive difficulties with answering questions, recall and recency
effects. Administrative errors due to lack of information or differences in procedures
13
could have occurred in the transcript survey. One indication of this is that there was high
item non-response for certain questions in the survey. The two surveys were timed very
differently and therefore could be measuring slightly different things. Respondents were
asked about schooling at different points while the high school transcript was collected at
the end of high school.
In this paper, we compare some important schooling measures like high school
graduation, grade point average, number of schools attended, math courses taken and
standardized test scores.
1. High school graduation
One item that both data sources can capture is whether or not the youth graduated from
high school. Table 3 below shows that in approximately 93 percent of cases where both
self-report and transcript data are available, the two sources are in concordance on
graduation status:
Table 3: A comparison of high school graduation status from self reports and transcripts
Self
Report
Transcript
No HS
HS Diploma
Diploma
1049
117
No HS
Diploma
HS Diploma 314
4574
Total cases with both self-report and transcript graduation status: 6054
The transcript variable shown here is coded from an item that asked school administrators
to provide the reason that a student ceased enrollment at a school (other options included
dropped out, expelled, transferred to another school, unknown, etc.) The self-report
variable asks youths directly whether or not they have received a high school diploma.
14
Of the 117 youths whose transcripts indicate graduation, but self-report indicates
nongraduate, 52 youths had not been interviewed since their transcript-reported
graduation dates, so there is not necessarily a contradiction between the two sources. Of
those who had been interviewed since their transcript-reported graduation date, none of
the remaining youths report a highest grade completed higher than 11, although the
majority had some 12th grade coursework on their transcripts.
There are 314 youths who self-report graduation but whose primary transcript
survey school did not indicate graduation. Of these, 156 youths report a graduation date
that comes after the departure date given by the school. There is no logical contradiction
in these cases. Fifty youths self-report a graduation date that is the same as the departure
date from school given in the transcript records. Their transcripts showed the following
reasons for departure: transferred 9 youths), dropped out (20), GED (14), withdrew (5),
discharged (age greater than 18) (2). These all suggest contradiction between transcripts
and self-reports. Thirty-eight youths self-report a departure date that precedes the schoolprovided departure date. These could be contradictions or they could be cases in which
the youth had completed in one school, but transcript school was not aware of the
transfer. Seventy youths had missing date information that leaves ambiguous whether or
not a true contradiction exists.
It is worth noting that the transcript variable in this case was recorded in a
questionnaire that school administrators were asked to complete as a supplement to the
transcript itself. As such, the transcript errors could include simple transcription errors,
school staff who relied on their own recollections rather than referencing documentation,
or other non-systematic reasons for poor data quality.
15
Overall, it appears that approximately half of the discordant cases have genuinely
contradictory information across the two sources. The evidence does not favor either the
self-reports or the transcript data as the superior data source in determining high school
graduation status.
2. GPA
Respondents reported their grades in high school as mostly A’s, half As half Bs and so
on. We excluded those who reported mixed As through Cs, mixed grades and those who
report that their high school was ungraded, since it was difficult to classify these into a
four point scale to make it comparable to the transcript grades. These were reclassified
into a scale of 0 to 4.
There were various transcript variables that provided the GPA. We used the
transcript survey created variable that calculates Carnegie-credit weighted GPA on a four
point scale. These were recoded to scale of 0 to 4. Table 4 shows a comparison of the
high school GPA variable from the self reports and transcripts.
Table 4: Comparison of high school GPA from self reports and transcripts
Transcript GPA
Self
reported
GPA
0: Mostly
below Ds
1: Half Cs and
Ds or mostly
Ds
2: Half Bs and
Cs or mostly
Cs
3: Half As and
Bs or mostly
Bs
0
1
2
3
4
Below 1.0
1.0 to 1.9
2.0 to 2.9
3.0 to 3.8
3.8 and above
3
22
20
3
0
9
153
261
26
1
15
3
1617
260
1
4
58
897
1361
11
16
4:Mostly As
Total: 5753
0
5
40
500
227
High school GPA categories for respondents match in about 58.4% of cases. In
many cases, the transcript GPA seems to be higher than self reported for lower grades (C
and D’s and below) and lower than self reports for higher GPAs (As and B’s).
There could be several reasons why the self reports do not match the transcript reports for
the GPA variable. There were some differences in design. In the transcript survey, the
GPA is weighted using Carnegie credits while the self report is from a question that asks
for the respondent’s impression of their overall grades in high school. Thus the two
reports are measuring different concepts. There is also the possibility of response errors.
Since the question about high school grades is not very specific, and respondents may
give their overall impression of their grades in high school. This makes their response
subject to various errors including recall errors and recency effects. For instance,
respondents may not be able to correctly recall their grades in the early years of high
school and may base their response on the most recent year of high school. Respondents
are also likely to report better grades than they actually received because of social
desirability effects.
There may be various administrative and classification issues that affect the
transcript reports. Calculations in the transcript which were made based on grades from
transfer/summer schools may be subject to errors, since the information may not be
complete and all the grades may not be known by the primary school. This could be a
particular problem for cases where the schools reported different types of grades and did
not provide grading specifications. Differences in how grades are classified may also
17
contribute to the differences we observe between the self and transcript data. The
respondent’s reports are based on their specific school’s grades while the transcript
variable is standardized and weighted.
3.Number of schools attended
Comparing the number of schools attended can provide a crude measure of the
concordance between the two data sources. In addition, the number of schools can be of
substantive interest in understanding disruptions to schooling, and the level of transiency
experienced. Table 5 below shows the number of schools reported by youths in their
interviews, compared with the number of schools identified on the youth’s high school
transcript.
During each NLSY97 interview, youths are asked to report every ‘regular’ school
they have attended since the prior interview period. An over-time roster permits
identification of schools the youth has previously attended, so that transitions can be
accurately measured. For this variable, we have identified unique schools that a youth
reported attending and that the youth classified as a high school (as opposed to a 2 year
college, a middle school, or other schooling levels). On the transcript side, each distinct
school appearing in the record is counted.
Table 5 : Comparison of number of schools reported in self reports and transcripts
0
Self
Report
0
1
2
3
4 or more
1
69
3735
580
131
29
Transcript
2
3
730
396
90
34
3
0
131
98
57
19
4 or
more
0
42
54
20
14
18
Total number of cases with self-report and transcript numbers of schools attended: 6232
Sixty-seven percent of youths had exact matches on the number of schools attended
during grades 9-12. An additional 25.4 percent were discrepant by one school, with more
self-reported school numbers exceeding transcript numbers than vice versa (897 vs. 670).
Since only youths with a received transcript are included in this analysis,
transcript data must show at least one school. On the other hand, self-report data may
inaccurately report a school as a middle school only when in fact the youth began
enrollment at a middle school, but then continued in its high school program without
reclassifying the school. In addition, students may have attended 9th grade in a middle
school. Since transcript data are recorded for the full ninth through twelfth grades, some
middle schools will appear in the transcript counts but not in the self-report counts.
Conversely, transcript data may differ on their treatment of dual enrollment. In
the event that a student was primarily enrolled in a comprehensive high school but also
took courses at a local vocational school that had linkages to the first school, the
interview data would likely count only the comprehensive school. There may be some
differences in how school transcripts handled such a situation.
Also, the definition of ‘regular’ school in the questionnaire reads,
“REGULAR SCHOOL IS ONE THAT OFFERS AN ACADEMIC DIPLOMA OR DEGREE; E.G., ELEMENTARY
SCHOOL, HIGH SCHOOL, COLLEGE, GRADUATE SCHOOL, LAW SCHOOL, OR NURSING PROGRAM
LEADING TO AN RN DEGREE. NOT INCLUDED AS REGULAR SCHOOL ARE: TRAINING AT A
TECHNICAL INSTITUTE, LICENSE TRADE PROGRAMS, ETC. UNLESS THE CREDITS OBTAINED ARE
TRANSFERRABLE TO A REGULAR SCHOOL AND COULD COUNT TOWARD AN ACADEMIC DIPLOMA
OR DEGREE.”
It appears that a large number of youths had enrollments during the summer that appear
on their transcripts but that the youths did not mention during their interviews.
Although credits, course codes, and quality grades typically appear in the
transcripts for all prior high schools where the youth earned credit, there is often no
19
additional information about the prior schools. For that reason, it’s relatively difficult to
examine the transcript schools to ascertain whether or not they are valid institutions.
We have two indications of whether or not all reported schools are in fact valid
places of enrollment for the students. Out of 8,984 students, there were 427 cases in
which the student record was not found. These are instances in which the schools were
unable to identify a student in their records who had attended their school. In large part,
these were matching problems – variations in spelling, birthdates not matching closely
enough, etc. In some cases, schools had misplaced some subset of their previous student
records, especially when a school had closed within a district. Some of the remaining
cases in this disposition would be instances in which the youth reported attending a
school that he or she had indeed never attended. An additional problem indicator is the
231 youths who had self-reported enrollment, but whose last school could not be
identified. These are cases in which the school name or address could not be linked to a
currently or previously existing institution. The 7.3 percent of the sample falling into one
of these two categories probably offers a ceiling on the rate of students falsely reporting
enrollment in schools.
For this measure, it would appear that neither self-reports nor transcripts
dominate, and in fact some combination may provide the most accurate information.
4.Math courses taken
We compared the math courses reported by the respondent to those reported in the
transcript to see if the math course progression was similar in the self reports and
transcript reports.
20
Respondents reported the math courses/categories they took in every round and we used
this information to calculate the highest level of math courses they took between 1997
and 2003. These were coded as shown below for 1998-2003, so the 1997 reports were
reclassified to conform to the same code frame which is shown below.
0
1
2
3
4
5
6
7
8
9
None
General/basic/vocational math
Algebra.I
Geometry
Algebra II
Trigonometry
Precalculus/Adv. Algebra
Calculus
Other advanced math
Other math class
Note that the category 9 “Other math class” is problematic since it is unclear what math
courses are included. Therefore, we did the analysis both including and excluding that
category.
In the transcript survey, pipeline variables were created to indicate the highest
level of math courses taken by the respondent based on course information in their
transcripts. These are based on NCES standards with some minor modifications to
account for the R-SST course coding scheme that was used. The pipeline variables for
progress in Math are coded as follows:
100 No math
200 Non academic: highest course is General/consumer or occupational math
300 Low academic: highest course is Pre-algebra2*
400 Middle academic 1: highest course is Algebra I ,Geometry or Unified math (2 or
fewer yrs)
500 Middle academic 2: one or fewer Carnegie credits in Algebra II to pre-calculus or 3
years of unified math
600 Advanced academic 1: more than 1 credit in Algebra II through Pre-calculus,
Trigonometry, Advanced Math
2
There is no corresponding category for this in self reports.
21
700 Advanced academic 2: highest course is Other Advanced Math
800 Advanced academic 3: highest course is Advanced math—Calculus, IB or AP math
Table 6 compares the math courses reported by the respondent to the transcript
variable. When no math courses were reported, the math pipeline variable does not
correspond well with the respondent reports. There are respondents who reported taking
math courses even though the transcript reported that they didn’t and vice versa. In other
categories there isn’t always a clear correspondence between categories in the transcript
variable and the self reports. For instance, low academic (300) does not have an
equivalent category in the self reports. There are fairly large numbers in all cells for a self
report of 9 (other math courses), probably because this category is a little ambiguous.
We find that the non-academic category from the pipeline variable corresponds to
the self report of basic/ vocational or algebra I and low academic to algebra I. Middle
academic I corresponds to algebra I, geometry and algebra II and Middle academic II to
algebra II, though some report their highest math course as geometry, trigonometry or
pre-calculus. Advanced math I corresponds to pre-calculus with some respondents
reporting algebra II and trigonometry as the highest math course. For advanced math II
we see many respondents reporting algebra II, pre-calculus or other advanced math as the
highest course they took. For advanced math III, most of the respondents report their
highest course as calculus while some report taking other advanced math classes. Some
of the spread we see in middle academic categories could be due to other criteria being
22
used in the construction of the pipeline variables like including the number of years of
unified math courses and the number of course credits.
Transcript
report
100
200
300
400
500
600
700
800
0
Table 6: Comparison of math courses taken
Self-reports
1
2
3
4
5
6
7
8
9
2
27
12
9
2
0
0
0
7
103
21
34
6
1
2
0
6
126
56
413
305
158
75
71
8
88
55
385
51
7
4
3
6
27
17
363
130
10
5
0
8
63
25
371
877
114
52
2
1
11
3
54
150
177
30
9
2
13
3
52
156
479
57
47
2
17
2
22
23
41
17
348
2
16
8
49
72
108
53
97
Total: 6227
In Table 7, we ignored other math courses while calculating the highest course taken.
While there are some differences, the patterns are similar to those in Table 6.
Table 7: Comparison of math courses taken,
(excluding the “any other math” category while calculating the highest level of math
course taken)
Transcript
report
100
200
300
400
500
600
700
800
0
1
Self-reports
2
3
3
33
15
21
6
1
0
0
9
142
31
49
10
1
3
0
8
122
80
509
80
10
7
3
6
42
20
482
163
12
11
0
4
5
6
7
8
10
78
34
456
1037
154
79
6
1
17
3
64
176
195
42
13
2
16
4
65
181
538
68
56
2
18
4
27
29
48
19
380
3
21
11
78
91
136
66
119
Total: 6225
There could be several reasons for the differences we observe between the
transcript and self reports of math coursework. There are some key differences in the
23
design of the questions, especially in the categories of math courses. The categories in
the self reports and transcript reports do not correspond well and in many cases the
mapping between the categories is not clear. In some cases there is no comparable
category (e.g. low academic/pre-algebra), some categories of courses are not well-defined
(e.g. other math courses) and the pipeline variables in the transcript are classified based
on additional criteria that do not apply to respondent reports (e.g. number of credits,
number of years of courses). Respondents may also have been reporting courses they did
not complete or did not pass.
Various response errors could have occurred in the self reports. Respondents may
have problems recalling the math courses they took (especially if they missed rounds) or
not be able to correctly identify or classify their math courses into categories. Differences
in timing could also be a factor. Respondents reported the level of math courses they took
between 1997 and 2003 while the transcript reports are from 1999 or 2003-04
4. Standardized test scores
We compared self reported highest scores for SAT Math, SAT Verbal and ACT
composite scores with transcript reports of the most recent score. These were the only test
scores that were available from both sources. Respondents reported their scores in ranges.
We used the created variable for the highest score (created in 2005) but did not include
scores reported after 2003 when the second wave of transcripts were collected. From the
transcript survey data, we obtained the exact scores from the last administration of the
test. We converted these into categories to correspond with the self-reported scores.
24
When we compare the self reports and transcript reports we find that while a large
proportion of reports do match, there are a sizeable number of cases with transcript scores
lower than self reported scores. These comparisons are shown below in Tables 8 through
10. For SAT math, the score categories match in 63.6% of cases as shown in Table 8.
About 62% of the score categories match for SAT verbal scores (Table 9). The self and
transcript reports of the ACT composite scores match in about 81% of the cases (see
Table 10). For SAT scores over 400, there are large numbers of respondents who reported
higher scores than the scores from their transcripts. This is also true for higher scores for
the ACT composite. Self reported scores could be higher because respondents were asked
to report their highest score while the transcripts reported the respondent’s most recent
score.
Table 8: Comparison of SAT Math scores from self reports and transcripts
Self
reported
scores
200-300
301-400
401-500
501-600
601-700
701-800
200-300
Transcript scores
301-400
401-500
501-600
6
6
5
3
9
55
18
1
6
78
228
21
4
20
129
267
3
4
11
60
3
7
15
14
Total number of scores available from both sources: 1228
601-700
701-800
0
0
0
9
170
18
0
0
0
0
3
55
Table 9: Comparison of SAT Verbal scores from self reports and transcripts
Self
reported
scores
200-300
301-400
401-500
501-600
601-700
701-800
200-300
6
8
7
7
1
3
Transcript scores
301-400
401-500
501-600
601-700
6
1
1
0
55
21
1
0
76
214
23
2
16
135
296
9
7
23
80
142
0
10
8
19
Total number of scores available from both sources: 1224
701-800
0
0
0
0
3
44
25
Table 10: Comparison of ACT composite scores from self reports and transcripts
Self
reported
scores
0-6
0-6
7-12
Transcript scores
13-18
19-24
25-30
31-36
0
0
0
0
0
0
7-12
0
3
3
1
0
0
13-18
0
11
217
11
1
0
19-24
0
3
98
429
6
1
25-30
0
0
7
59
220
1
31-36
0
0
0
6
10
50
Total number of scores available from both sources: 1137
Some of the differences we see between self reports and transcript reports of test scores
could be due to differences in design and various response errors. There were conceptual
and wording differences in question wording and concepts between the interview
question and the transcript. In particular, respondents reported their highest score, while
transcript reports contain the most recent score as of the transcript survey date. Response
errors that may have occurred include social desirability effects, difficulties with
classification of scores and recall errors. Respondents may have overstated their test
scores to the interviewer due to social desirability effects. Both the conceptual difference
in the question and the social desirability effect could make the self reported test score
higher than the transcript report. Respondents reported their scores in ranges while the
transcripts recorded the actual scores. Respondents may have made errors while
classifying their scores into ranges. They could also have had problems with recall of
their actual test score, particularly if they missed rounds of the survey.
26
5. Characteristics of Respondents with Concordant/Discordant Reports
In Table 11 below, we examine four of our selected outcomes to see the extent to
which concordance of self-report and transcript measures may be related to other
characteristics. The table includes such basic demographic characteristics as gender,
race, ethnicity, and region of residence at age 21. Previous literature has found fairly
consistently that students of greater academic achievement are more likely to have selfreports that match transcript reports than are students of lower academic achievement.
Toward this end, we consider three measures that are likely to be related to a youth’s
academic achievement: the educational attainment of the youth’s mother, the youth’s
Armed Services Vocational Aptitude Battery (ASVAB) percentile score (if available),
and the youth’s self-reported highest grade completed as of age 21.
Across the four measures, high school graduation, grade-point average, number of
high schools attended, and standardized test scores, the last looks most different from the
others in terms of its demographic characteristics. For example, many fewer students
with self-report and transcript test scores available are male, Hispanic, have mothers with
less than a high school diploma, or ASVAB scores in the lowest quartile. Similarly,
students with college-educated mothers or ASVAB scores in the highest quartile are
over-represented in this measure. This is presumably simply due to the fact that the ACT
and SAT tests are generally only taken by students who are considering 4-year colleges,
not a representative group of American high school students.
Curiously, the GPA measure seems to exhibit the smallest differences between
concordant and discordant cases, although it may be the most subject to social desirability
bias. These results may simply be reinforcing the generally poor quality of GPA
27
Table 11: Weighted Demographic Characteristics by Match Status of Key Variables (shown only for
youths with both self-report and transcript measures available)
HS Graduation
GPA
School Number
Test Scores
All
Match
No
Match
No
Match
No
All
At
(n=6232) (n=5617) Match
(n=3374) Match
(n=4194) Match
Match
Least
(n=431)
(n=2392)
(n=2038) (n=1229) One
NonMatch
(n=865)
Male
Hispanic
Black
White
Mother’s
Education:
> 12 years
Mother’s
Education:
=12 years
Mother’s
Education:
< 12 years
NEast (at
age 21)
NCentral
(at age 21)
South (at
age 21)
West (at
age 21)
ASVAB
missing
ASVAB
<= 25%
ASVAB
25-75%
ASVAB
> 75%
Highest
Grade
Completed
at age 21
0.51
0.12
0.15
0.73
0.44
0.51
0.11
0.14
0.74
0.45
0.50
0.16
0.22
0.62
0.35
0.51
0.12
0.14
0.74
0.46
0.50
0.12
0.15
0.73
0.42
0.51
0.10
0.14
0.75
0.45
0.50
0.15
0.17
0.68
0.41
0.43
0.04
0.08
0.87
0.62
0.48
0.09
0.12
0.79
0.57
0.35
0.34
0.35
0.35
0.34
0.35
0.33
0.28
0.29
0.22
0.21
0.29
0.19
0.24
0.20
0.26
0.10
0.13
0.14
0.15
0.08
0.13
0.16
0.16
0.11
0.14
0.23
0.23
0.23
0.18
0.23
0.24
0.24
0.20
0.33
0.17
0.29
0.29
0.23
0.30
0.28
0.26
0.33
0.26
0.32
0.17
0.18
0.16
0.19
0.16
0.16
0.20
0.14
0.14
0.17
0.16
0.21
0.16
0.17
0.16
0.18
0.11
0.14
0.34
0.33
0.48
0.32
0.35
0.32
0.39
0.14
0.20
0.49
0.43
0.41
0.44
0.42
0.44
0.41
0.39
0.47
0.23
0.24
0.11
0.24
0.23
0.24
0.20
0.46
0.34
12.9
12.96
12.47
13.1
12.77
13.06
12.54
14.16
13.96
28
measures, especially given significant variability across schools, and the vague nature of
the self-report item.
Hispanic youths are generally over-represented among discordant cases, as are
Black youths, although less so. The patterns for the four Census regions are inconsistent
across the four measures.
Among the measures related to academic achievement, in all four measures,
youths with concordant reports have higher average value of highest grade completed at
age 21 than do youths with discordant reports. Youths with college-educated mothers are
over-represented among concordant reports, while youths with mothers who did not
complete high school are over-represented among discordant reports. These results are
generally consistent with previously documented patterns from other data sets. Youths
with ASVAB scores in the lowest quartile are over-represented among discordant reports,
and those with scores in the highest quartile are over-represented. This is most true for
high school graduation status, and least true for grade-point average. Alexander,
Entwistle, and Bedinger (1994) have suggested previously that among fourth graders and
their parents, cognitive ability to understand and recall grades received in school may
contribute to discrepancies in accuracy. The cross-tabulations of ASVAB score with
concordance status are consistent with this hypothesis.
Sample Regressions Using Self-Report and Transcript Measures
A key issue in evaluating the relative merits of the NLSY97 self-report and
transcript data on high school experience is the extent to which the two generate
substantively disparate results when used in analyses. If the discordances are essentially
29
random, then they may be of little worry to researchers. If the two different sources of
data generate substantially different analytic results, then it may be crucially important to
try to reconcile or to determine which is the ‘true’ value. In Tables 12 and 13, we
provide example regressions of three outcomes: no post secondary schooling by age 21,
highest grade completed by age 21, and annual weeks worked in year of 21st birthday.
The purpose is not so much to estimate the contribution of the self-report or transcript
measures in determining these outcomes. Rather, our intent is to see the extent to which
results differ by our choice of measure.
Table 11 shows each post-secondary schooling outcome modeled using selfreport or transcript measures of high school graduation status and number of high schools
attended. The patterns are similar for all four pairs of regressions. The coefficients on
the self-report/transcript measures are quite different from one another, just as the
measures themselves can be. Yet, the remainder of the estimated model is quite similar.
Aside from race and ethnicity, all other variables yield similar results in both the selfreport and transcript regression, including whether or not the variable is significant, and
the approximate magnitude and direction of the coefficient. There may be a slight
tendency for the self-report measure to actually make a greater contribution to each
outcome than does the corresponding transcript measure.
Table 12 shows a pair of regressions using the self-report and transcript measures
on high-school graduation status to predict annual weeks worked at age 21. Here again,
the basic patterns of Table 11 obtain.
30
Table 11: Predicting Post-secondary educational attainment using self-report and transcript measures
No Post-secondary schooling by age 21
Highest Grade Completed at age 21
Self-Report
HS
Graduation
Transcript
HS
Graduation
Self-Report
Number of
Schools
Transcript
Number of
Schools
Male
Hispanic
Black
Mother’s Ed.
12 years
Mother’s Ed.
> 12 years
NCentral (at
age 21)
South (at age
21)
West (at age
21)
ASVAB
missing
ASVAB
25-75%
ASVAB
> 75%
Constant
-2.620**
(0.11)
2.691**
(34.2)
-2.273**
(0.09)
2.348**
(30.26)
0.629**
(0.05)
-0.479 (10.29)**
0.154**
(0.04)
0.524**
0.531**
0.595**
0.587**
(0.07)
(0.07)
(0.06)
(0.06)
-0.016
-0.139
-0.175*
-0.159
(0.10)
(0.09)
(0.09)
(0.09)
-0.213*
-0.146
-0.187*
-0.157*
(0.09)
(0.09)
(0.08)
(0.08)
-0.193*
-0.253** -0.400** -0.416**
(0.09)
(0.08)
(0.08)
(0.08)
-1.169** -1.221** -1.363** -1.363**
(0.09)
(0.09)
(0.08)
(0.08)
0.266**
0.265**
0.201*
0.204*
(0.10)
(0.10)
(0.09)
(0.09)
0.125
0.088
0.069
0.052
(0.09)
(0.09)
(0.08)
(0.08)
-0.067
-0.042
-0.238*
-0.172
(0.11)
(0.10)
(0.10)
(0.10)
-0.708** -0.710** -0.720** -0.734**
(0.10)
(0.10)
(0.09)
(0.091)
-1.037** -1.091 ** -1.287** -1.308**
(0.09)
(0.09)
(0.08)
(0.08)
-2.194** -2.303** -2.559** -2.626**
(0.13)
(0.13)
(0.12)
(0.12)
2.503**
3.013**
0.248
0.877**
(0.14)
(0.16)
(0.13)
(0.12)
N=5578
N=5748
N=5748
N=5748
-2LogL
-2LogL
-2LogL
-2LogL
= 5374
= 7778
=6177
Logistic Regression
Logistic Regression
Standard errors given Standard errors given in
in parentheses
parentheses
-0.125
(-2.93)**
-0.391
(-6.19)**
0.034
(0.37)
0.188
(2.24)*
0.381
(4.46)**
0.889
(10.31)**
-0.123
(-1.22)
-0.088
(-0.94)
0.027
(0.25)
0.512
(5.05)**
1.126
(12.80)**
2.079
(19.32)**
11.709
(86.94)**
N=5135
Rsq=0.14
OLS
-0.384 (6.14)**
0.051
(0.56)
0.210
(2.52)*
0.357
(4.23)**
0.860
(10.1)**
-0.115
(-1.15)
-0.104
(-1.11)
0.066
(0.64)
0.481
(4.79)**
1.075
(12.32)**
1.974
(18.42)**
12.215
(87.88)**
N=5135
Rsq=0.16
T-values given in
parentheses
-0.287**
(-4.84)
-0.018
(-0.21)
0.215**
(2.72)
0.070
(0.86)
0.467**
(5.70)
-0.171
(-1.81)
-0.146
(-1.66)
-0.085
(-0.86)
0.311**
(3.24)
0.604**
(7.13)
1.349**
(13.05)
10.390**
(83.7)
N=4989
Rsq=0.27
OLS
-0.263**
(-4.61)
-0.041
(-0.50)
0.147
(1.94)
0.098
(1.27)
0.500**
(6.36)
-0.153
(-1.68)
-0.100
(-1.18)
-0.101
(-1.07)
0.334**
(3.64)
0.627**
(7.77)
1.387**
(13.97)
9.950**
(81.54)
N=5135
Rsq=0.30
T-values given in
parentheses
31
These two tables (11 and 12) suggest that although there may be considerable
variability across the two sources in the coefficients of the disputed measures themselves,
the choice of source may not induce great variability in other results.
Table 12: Predicting post-high school employment using self-report and
transcript measures
Weeks Worked in year of 21st birthday
Self-Report
4.632
HS Graduation
(4.96)**
Transcript
4.425
HS Graduation
(4.92)**
Male
1.095
1.219
(1.82)
(2.01)*
Hispanic
5.120
5.106
(4.28)**
(4.22)**
Mixed Race
0.082
1.132
(0.03)
(0.35)
Non-Hispanic, Non-Black
5.700
5.754
(6.20)**
(6.19)**
Mother’s Education: =12
2.286
2.158
years
(2.61)**
(2.42)*
Mother’s Education:
-0.576
-0.653
> 12 years
(0.66)
(0.74)
NCentral (at age 21)
0.808
0.716
(0.92)
(0.81)
South (at age 21)
-1.906
-2.143
(2.13)*
(2.38)*
West (at age 21)
-1.919
-2.150
(1.91)
(2.12)*
ASVAB Percentile
0.002
-0.003
(0.13)
(0.26)
Constant
26.189
26.867
(19.11)**
(19.79)**
N=4225
N=4115
R-sq=0.03
R-sq=0.04
OLS
Conclusions
In this paper, we have examined selected constructs of high school experience for which
measures can be constructed from self-report and transcript data in the NLSY97. We
find that although the concordance rates vary and can be quite low for some measures,
32
neither source emerges strongly as the preferred ‘gold standard.’ Rather, each source is
better at providing information about certain aspects about high school experience. We
then consider the characteristics of youths with concordant vs. discordant data on selected
measures. Mirroring previous work, we find evidence that students with better academic
results are more likely to have matching self-report and transcript data. This systematic
response error suggests that analytic results would likely be biased across the two sources
(although it’s unclear which gives the more accurate estimate). With that in mind, we
estimate a handful of regressions in order to illustrate the sensitivity of analytic results to
the choice of data source.
In summary, we find that although data can be quite different between self-report
and transcript sources, neither is the clearly better source. For many purposes, it may be
adequate to simply have one or other data.
33
References
Alexander, K, D. Entwistle, and S. Bedinger (1994) “When Expectations Work: Race and
Socioeconomic Differences in School Performance” Social Psychology Quarterly, Vol. 57, No.
4, pp. 283-299.
Kane, T, C. Rouse, and D Staiger (1999) “Estimating Returns to Schooling when
Schooling is Misreported.” Working Paper #419, Princeton University Industrial
Relations Section, June 1999.
Kuncel, N, M. Crede, and L Thomas (2005) “The Validity of Self-reported Grade Point
Averages, Class Ranks, and Test Scores: A Meta-Analysis and Review of the Literature”
Review of Educational Research Spring, 2005. 75(1):63-82.
Naifeh, M and S Shakrani (1996). “Mathematics Course-Taking and NAEP Math
Proficiency: Comparing Students’ Reports with their Transcripts” Paper presented at the
Annual Meeting of the National Council on Measurement in Education (New York:
April).
Talento-Miller, E and J Peyton (2006). “Moderators of the Accuracy of Self-report Grade
Point Average” Graduate Management Admission Council Research Reports RR-06-IO.
McLean, Virginia.
Zimmerman, M, C Caldwell, D Bernat (2002). “Discrepancy between self-report and
school-record grade point average: Correlates with psychosocial outcomes among
African-American adolescents.” Journal of Applied Social Psychology, 32, 86-109.
34