MEMORANDUM
TO:
INSTITUTIONAL DATA COMMITTEE
FROM:
RANA GLASGAL, BILL HAYWARD, DENNIS HENGSTLER
SUBJECT:
UNDERGRADUATE STUDENT DATA NEEDS
DATE:
4/19/2005
The AAU Executive Committee has charged the Institutional Data Committee (IDC) with evaluating
university data collection efforts in the areas of graduate education, undergraduate education, and
faculty activities. Specifically, the IDC has been asked to answer the following questions:
1)
What data are currently collected, are there systematic flaws or redundancies in the data
collected, and how can any such problems could be rectified?
2)
What data are not being collected that would be useful for both institutional and national
planning and policy development, and what data collection procedures should be implemented to produce
relevant and comparable data across AAU universities?
This paper will address the above questions related to undergraduate education, and companion papers will
address the other areas of graduate education and faculty activities. In reviewing the various data on
undergraduate education, the IDC will need to consider why it is important to collect such information and
what questions could be answered with the data collected. It is assumed that the primary focus of the data
collection efforts is to provide data to AAU member institutions for internal policy decisions and that the data
could greatly facilitate the government relations and administration (e.g., member selection) of the AAU
organization.
A comprehensive look at undergraduate data shows that the data can be organized into five principal
categories:
1.
2.
3.
4.
5.
6.
Enrollment Demographics
Incoming Student Quality
Financial Aid and Cost of Attendance
Engagement
Resources
Outcomes
In this paper, we will define the data in each category, discuss current data collection efforts and their
strengths and weaknesses, and make suggestions for future data gathering efforts.
Other important questions, issues, and principles that apply to all of the types of data examined by the IDC
are:
 Should the data be collected by discipline or by organizational unit? If so, what level of detail should
be collected (e.g., broad discipline, sub-discipline), and what taxonomy of disciplines (discipline
classifications) should be reported (CIP, NRC, NSF, ISI, etc)? If discipline level data is desired the
DRAFT




AAU Data exchange recommends using CIP codes and that appropriate crosswalks be developed
between discipline taxonomies.
How should data be treated when there is a small number (N) to ensure the privacy of individuals
and as well as the reliability and generality of the data? Also, what constitutes a small N (e.g., 1, 3, 5,
10)?
Should historical data be included if it is comparable to current data?
How should data be collected (electronic vs. hardcopy; unit-record data vs. aggregate or summary
data), how often should it be collected, who should collect and check the data, how should the data
be maintained (e.g., should the data reside the AAU Data Exchange data warehouse), what standard
reports should be generated from the data, and what principles and guidelines should be developed
for other reporting of the data (e.g., who has access to what data; under what conditions can data be
released and to whom)?
What infrastructure and funding will be necessary to support the desired data collection and
reporting efforts?
1. Enrollment Demographics
Definition
This category includes the data that describes the student body, including headcounts and FTE
counts by ethnicity and gender. This is commonly collected data that is frequently used for
normalizing and as the basis for ratios such as student to faculty, endowment per undergraduate
student, state funding per FTE student, or the percentage of the school population that is
undergraduate.
Current Data Collection
Undergraduate enrollment data is collected by many government agencies and data sharing consortia.
Current sources, which usually count graduate students as well, include:
1. IPEDS fall enrollment – undergraduate students separated out by first-time, first-year vs. all
others, gender, full-time vs. part-time, ethnicity, student level, and age. IPEDS also collects
the residence of first-time undergraduates, and an unduplicated 12-month headcount of all
undergraduates by gender and ethnicity. Data are collected for the total institution and for
nine fields: education, engineering, law, biological sciences/life sciences, mathematics,
physical sciences, medicine, dentistry, and business management/administrative sciences.
The nine fields are coded at the 2-digit CIP level. The annual survey is completed by the
institution, usually by an institutional research or registrar’s office. Data are public, and are
available through a variety of retrieval mechanisms including the AAUDE warehouse.
Expansion of this survey to collect unit-record data is currently under study at the National
Center for Education Statistics (NCES) of the US Department of Education, the survey
sponsor. Submission of IPEDS data is required for institutions receiving any federal
funding.
2
DRAFT
2. Common Data Set (CDS) – In an effort to standardize and reduce the amount of paperwork
required, publishers and the higher education community have collaborated to produce a
common data set and format which includes general information about the campus,
undergraduate fall enrollment (by gender, ethnicity, full- vs. part-time status) and persistence,
freshmen and transfer admission, academic offerings, student life, annual expenses, financial
aid, faculty and class size, and degrees. Like IPEDS, it separates out first-time
undergraduates as well, but goes a step further, separating first-time freshman from firsttimers who are not freshmen.
3. AAUDE downloads IPEDS enrollment data for AAU institutions into the AAUDE data
warehouse to facilitate linking enrollment with other institutional data.
4. COFHE11 Institutional Profiles fall enrollment – Fall undergraduate enrollment by gender,
ethnicity, and school for six discipline group categories (Arts and Sciences, Engineering,
Business, Law, Medicine, Other).
Data Issues
Although this data is collected frequently, not all of the definitions are consistent. For example, the
IPEDS definition does not allow for inclusion of students studying abroad at a branch campus, but
the COFHE definition does. Also, there is no consistent, national definition of how to calculate
student FTE. Many schools use a proxy based on full-time headcount plus one-third of part-time
headcount, but this is just an approximation. Other methods include collecting student credit hours
taken for each year, from which an annual FTE can be computed.
One thing lacking in current enrollment data collection is enrollment by discipline, and by extension,
enrollment by ethnicity and gender by discipline. IPEDS collects enrollment by 2-digit CIP code for
selected discipline areas. The data are of limited comparative use because of different practices on
when students are admitted into a major and whether the CIP coding is consistently applied across
institutions.
Recommendations
Given the widespread use of the IPEDS enrollment data (and its inclusion already as an AAUDE
data warehouse item), the AAU should continue to use this data as the defacto standard for
enrollment demographics. A common, agreed-upon definition of student FTE should be adopted.
For the public AAU institutions, an issue on some campuses is the mix in residency and citizenship
status (in-state, out-of-state, and foreign), so more access to this type of data would be helpful.
Other potentially useful data that are not generally available right now are the number of students
who: are first generation (parental education level), low-income (family income), have a single parent,
COFHE, the Consortium on Financing Higher Education, is an unincorporated association of thirty-one
private colleges and universities. COFHE’s main focus is to collect and report on members’ historical data
relating to admissions, the financing of high education, and institutional data for the primary purpose of
institutional self-study and planning. Most of the AAU private institutions are COFHE members, but the
publics are not. COFHE surveys are presented here as examples of data collection efforts.
1
3
DRAFT
have dependent children., come from private high schools vs. public high schools, have English as a
second language, or participated in outreach programs in high school.
2. Incoming Student Quality
Definition
This category tries to do a very difficult thing – quantify the quality of a school’s incoming freshman
class. While much of what makes up a freshman class is intangible, for many years college guide
publications have been trying to collect quantifiable proxy measures of quality. Items in this category
can include SAT/ACT scores, high school class rank, AP test scores, and high school GPA. Another
proxy for the quality of the incoming freshman class is how competitive it was for a student to get in,
i.e., the admit rate. The yield rate can also fit in this category as it can be interpreted to mean that the
higher the yield rate, the more likely the highest-quality students are attending.
Current Data Collection
1. Common Data Set - used by many college guide publications – Collects apply/admit/enroll
data; 25th/75th percentile SAT and ACT, percentage of students by ranges for SAT and ACT
scores; percent in top tenth, top quarter, top half, or bottom half of their high schools, high
school GPA ranges and average high school GPA.
2. US News collects most of the C DS data plus the averages for SAT and ACT scores. (This
is not made public by US News but could be shared among AAUDE members.)
3. AAUDE Freshman Profile – collects apply/admit/enroll statistics for the incoming
freshman class, and can be used to compute admissions and yield rates for all AAUDE
institutions that respond to it.
4. COFHE Admissions Statistics (Redbook) – collects admissions statistics for private schools
at a greater level of detail, such as apply/admit/enroll data by gender and ethnicity, 25th/75th
percentile SAT scores by gender and ethnicity and by applied, admitted and enrolling
students.
5. CIRP entering freshman survey – this national, widely used survey administered by HERI at
UCLA may be useful for skills self-assessment, college expectations and other profile
dimensions. A few AAU institutions participate annual in the CIRP survey.
Data Issues
Some of this data is becoming less useful as more high schools decline to compute class rank and
fewer institutions require it from their applicants. Also, high schools may vary significantly in quality,
and GPAs are calculated in so many different ways as to be difficult to compare. For example, some
high schools assign a score of 5 for an A in an honors or AP class, while others assign a 4.
Also, given the recent changes in the SAT, anyone who uses the combined SAT scores as a single
metric will need to adjust to the new 3-score test, and some historical continuity will be lost.
4
DRAFT
However, the SAT math and verbal scores can still be separately compared with historical data and
the new 3rd score can readily be added as a separate item. Also, most schools are not collecting
comprehensive data on SAT II test scores.
The use of apply/admit/enroll statistics has been called into question due to concerns over the
impact of binding early decision programs in inflating yield rates for some institutions, or institutions’
attempts to encourage applications from many students who they are not likely to admit. What is
required to use these data is a clear understanding of which schools have binding early decision
programs, which have non-binding early action and what percentage of the freshman class entered
through these early decision programs. This kind of data is collected by the CDS for college guides
and by COFHE as well.
There is little data currently collected and analyzed regarding transfer students.
Recommendations
Some of the widely used and reliably comparable data for this category can easily be (or is already)
collected by AAU universities based on information already widely compiled for the Common Data
Set, such as apply/admit/enroll statistics, and 25th and 75th percentile SAT and ACT scores.
It may be worthwhile to collect data on AP tests – how many students took them and how they did.
Because these are standardized, national examinations, they may help to even out some of the
regional differences in grading schemes and high school GPAs.
3. Financial Aid and Cost of Attendance
Definition
This category of data asks how undergraduates pay for their education. Some simple categories of
data such as tuition and fees are already being collected. In addition, the AAU may want more
complex data on financial aid and family income. From this data, AAU schools can gauge their
availability to students from varied socio-economic backgrounds, and calculate the net cost of getting
their undergraduate degree.
Current Data Collection
1. IPEDS financial aid – gathers types of aid in summarized fashion on the percent of students
receiving aid by type of aid and the average amount of aid received, such as institutional
grants, state and federal grants, and loans.
2. Common Data Set (CDS) - aid to enrolled undergraduates (need-based and non-need-based
scholarship/grant aid, self help and loans by source). Includes both the total financial aid
dollars awarded and the number of aid recipients.
3. AAUDE Tuition and Fees – gathers resident and non-resident tuition and fees.
5
DRAFT
4. AAUDE Current Developments of Interest - annually collects tuition and fees for
undergraduate and graduate students and selected professional schools (e.g., law, medicine,
dentistry, pharmacy, veterinary medicine, and pharmacy) by residency status; also reports
percentage increase over the prior year.
5. COFHE Freshman Financial Aid/Admissions Survey (Blue Book) – for the private AAU
schools, shows average financial aid award by type of aid, and the income profiles of the
current freshman class. Much of this information is also shown by ethnicity.
6. COFHE Sources of Undergraduate Grant Aid at the COFHE Institutions (Brown Book) –
for the private AAU schools, shows how total financial aid grant money given to students is
obtained by the university – general funds, outside agencies, etc.
7. Harvard Tuition and Fees survey – gathers tuition and fee data from selected private
universities, including all types of graduate and professional school tuition.
Data Issues
Tuition and fees are well covered by AAUDE surveys. Although all AAU institutions complete the
IPEDS financial aid survey, it is not a very extensive survey.
Recommendations
Comparisons of finances across public and private universities should be handled carefully. A
common language and method of comparison is important so that one type of school or the other is
not unfairly considered. While the COFHE blue and brown books cover the topic well for the
private schools, there is no such comprehensive data collection by all AAU schools. The CDS
section on financial aid could be used as a starting point for discussion on how to collect comparable
data for the AAU. The debt level of students upon graduation is a good indicator that some schools
may calculate already, but which is not generally shared.
Higher education is often taken to task for being expensive (even the public universities are not
immune to this criticism), so data showing the variety of financial circumstances that AAU students
come from is informative. A profile of the family incomes of students would be a helpful item to
collect. Some information is currently collected on student and parent surveys about family income
ranges, but income data reported by students can be unreliable. If this data is tracked more broadly
and over time, it will help the AAU determine if the door continues to remain open to students of all
income backgrounds. Family income data is hard to collect, except for students who apply for aid
and complete the FAFSA form. Perhaps a simple common measure could be adopted, such as the
percentage of entering freshmen whose family income falls below a certain threshold, e.g., $30,000.
But given widespread differences in the cost of living across the country, we also would need to
acknowledge that this measure may mean different things in different states, or use Runzheimer’s
cost of living indices to account for regional differences.
4. Engagement
Definition
6
DRAFT
This is perhaps the most difficult category to collect consistent and meaningful data. Engagement
measures how much a student is involved in his or her education, uses the resources available, and
achieves his or her academic goals.
Current Data Collection
1. National Survey of Student Engagement (NSSE) - This is an extensive survey of
undergraduate experiences designed to obtain information about student participation in
programs and activities that institutions provide for their learning and personal development.
The results provide an estimate of how undergraduates spend their time and what they gain
from attending college. Several AAU institutions currently participate in the NSSE survey
and some institutions make their data available to AAUDE.
2. COFHE Enrolled Student Survey - All of the AAU privates have used this relatively new
survey. Like NSSE, it is designed to find out how students spend their time, and how often
they take advantage of the resources available to them. Some of the questions on the
Enrolled Student Survey are very similar to questions on NSSE.
3. AQUER Value-Added Project – This ambitious AAU-sponsored project attempts to
measure the amount of learning that takes place between the freshman and senior year. The
primary objective is to measure the extent to which an undergraduate education at a research
university influences the ability of students to think critically and reason analytically. The
project is currently housed at the University of Maryland.
Data Issues
Surveys of student engagement such as NSSE, COFHE’s enrolled student survey and senior survey
provide important information on student satisfaction, but it is difficult to get all schools to conduct
the same survey. While some “core” questions are present in almost all student satisfaction surveys,
they may be worded slightly differently, or use different scales of measurement. So the opportunities
for comparative data or for aggregate data about AAU universities are restricted. Also, some schools
and university systems have their own internally-designed surveys that provide them with
longitudinal data, so they do not wish to either change to a different survey or administer an
additional one. Another problem is that most institutions consider their student satisfaction surveys
proprietary information and are not willing to release the data publicly, though they may be more
open to sharing it with other AAU schools for internal use.
Systematic data on other areas of undergraduate engagement, such as the availability, quality, and
participation in undergraduate research, internationalization and global issues and service learning
activities, are not currently collected. Student surveys currently have questions about participation in
these areas, but with varying specificity. In the area of undergraduate research, for example, many
questions need to be answered, such as what constitutes undergraduate research and how can one
recognize it when it is not labeled as such? While many undergraduate research experiences occur in
the classroom, others occur outside the classroom. How can the AAU capture and document the
diversity of these experiences, which occur in different ways in different disciplines, and across a
wide range of venues – labs, libraries, classrooms, studios, internships, off campus field sites, etc.?
The task of counting credited undergraduate research experiences is quite complex and varies
considerably across disciplines. A few years ago, the University of California system developed a set
of guidelines for categorizing the undergraduate curriculum (i.e., courses for which students received
7
DRAFT
academic credit) in terms of whether it involved advanced or apprentice undergraduate research, or
none at all. But while the initial classifications were internally consistent within a campus, the results
were not consistent across campuses, and expansion beyond the UC system would surely be even
more inconsistent. In short, the process of defining and counting undergraduate research courses is
fraught with ambiguity.
Another area of engagement missing from current undergraduate data collections is service learning,
either for credit or not for credit. This is another complex area that is relatively new and not easy to
define. It will take a concerted effort to determine how to define and collect such data, but this area
is often mentioned as a differentiator between university experiences.
The AAU’s AQUER project attempted to measure the amount of learning that takes place between
the freshman and senior year (value-added) by testing students at the freshman and senior levels in
critical and analytical thinking. Two pilots were conducted (at Columbia University and the
University of Maryland). While this is an intriguing way to measure the extent to which an
undergraduate education at a research university influences the ability of students to think critically
and reason analytically, it is costly and time consuming, and it is too early to see if this will be a useful
and accepted tool that leads to institutional improvement. Further research is needed before its
efficacy is judged and a recommendation can be made.
Recommendations
Measuring student engagement will require much effort and coordination.
tasks that could be considered:
Here are some of the
1. Choose a set of core questions for student engagement, and ask all the AAU schools to use
those questions and a common scale on whatever survey instrument they currently use, so
that comparing data for the core set of questions will be possible. Data sharing could be
facilitated through AAUDE.
2. Continue to develop ways to measure the types of undergraduate research; this will require a
greater understanding of how undergraduate research is conducted on AAU campuses.
3. Consider the AQUER value-added project so that other campuses may learn from these
innovative pilots, and consider the feasibility of this approach.
5. Resources
Definition
Measures of resources illustrate what programs, tools, or opportunities are available to
undergraduates, and how accessible they are.
Current Data Collection
1. Common Data Set (CDS) – Number of instructional faculty, percent of faculty that are fulltime, percent with terminal degree, gender and minority status of faculty, number of total
faculty in stand-alone graduate/professional programs, student to faculty ratio, class size of
undergraduate classes and use of graduate teaching assistants.
8
DRAFT
2. AAUP Faculty Salaries – This extensively-used survey is the standard for measuring faculty
salaries and benefits. This is sometimes used as a proxy for faculty quality, a student
resource. The data is collected at a highly aggregated level, by rank and gender. (The topic
of faculty salary surveys is covered in greater depth by the Faculty Activity Data white
paper.)
3. AAUDE Faculty Salaries By Discipline Survey – Again, faculty salaries can be used as a
proxy for faculty quality. Unlike the AAUP survey, the AAUDE faculty salary data allows
analysis at the departmental level and collects high, low, and average salaries by rank and by
discipline using the 6-digit Classification of Instructional Programs (CIP) code. It also
collects the average faculty salary for new hires. (Again, the topic of faculty salary surveys is
covered in greater depth by the Faculty Activity Data white paper.)
4. Libraries – The Association of Research Libraries (ARL) conducts an annual survey on the
number of volumes available, and the number of journals kept by libraries on campus.
5. Delaware Teaching Load and Cost Study - This data collection is voluntary and is presently
in its thirteenth year of operation. Originally funded by a FIPSE (Fund for the Improvement
of Post-Secondary Education) grant, the purpose of the data collection is to collect data
from institutions on matters related to teaching load, faculty productivity, and instructional
cost. Institutions are asked to report the following information by discipline (academic
department): faculty FTE by type (regular, other regular, supplemental, teaching assistances),
student credit hours by level, organized class sections by level, and direct expenditures for
salaries, benefits, other personnel expenditures, as well as the separately budgeted research
activity and public service activity. Although several AAU campuses currently participate in
the Delaware study, they do not report the expenditure data. The Delaware Study does not
attempt to capture data on standard teaching loads, credit hour equivalents and course
reductions.
6. IPEDS Finance Survey – This annual survey reports the total revenues by source of funds
and expenditures by function (instruction, research, public service, institutional support, etc).
No revenue or expenditure data are reported by discipline.
Data Issues
Some of the conventional ways for measuring the resources available to undergraduates are open for
improvement. For example, is the percent of faculty that are full-time a good measure of a
committed and engaged professoriate? And do faculty salaries really indicate the quality of the
undergraduate instruction at an institution? Also, more might be done by looking at resources using
qualitative measures, such as a list of programs available in undergraduate research, study abroad,
service learning, internships, cohort-specific experiences, and the like, along with some indicators of
the percent of undergraduates who take advantage of these opportunities.
In the area of library resources, the very high and rising cost of academic journals is causing many
university library systems to consolidate and rely more heavily on intercampus transfers of books and
journals. Therefore, are library collections in general a valid measure of a student resource? This is
a relatively uncharted measure, and will need much more consideration before it is included as a data
collection item.
9
DRAFT
The Delaware Study has been widely recognized as a primary source of comparative data related to
faculty teaching workload. The study often requires extensive campus resources to develop the
reports needed to answer the survey. For this reason, several AAU institutions have declined further
participation having once participated in the study. Also, many objections have been raised regarding
the instruction cost portion of the Delaware study (e.g., failure to include the indirect instructional
costs).
There are numerous issues related to the identifying the cost of instruction and education per student
FTE or by credit hour. Obtaining consensus among the AAU group on this issue will be
problematic.
Recommendations
1. Consider more extensive use of qualitative lists of resources and programs available to
students to supplement quantitative measures.
2. Decide what additional measures of faculty resources and profiles should be commonly
collected and shared
including diversity measures, awards and national academy
memberships.
3. Consider whether any library information is useful and relevant.
6. Outcomes
Definition
Undergraduate student outcomes measure the success of students as they graduate and enter the
workforce or graduate school. Two IPEDS reports – the Graduation Rate Survey and the
Completions Survey - provide a basic set of data on graduation and retention rates and degrees
earned, but there are more complex aspects of undergraduate outcomes such as job placement,
graduate school placement, or graduate scholarship recipients, that may also measure the success of
research universities in educating their undergraduate students.
Current Data Collection
1. IPEDS Completions – Number of graduates each year by degree type, ethnicity, gender, and
discipline.
2. IPEDS and NCAA retention and graduation rates surveys – 4-, 5-, and 6-year graduation
rates.
3. AAUDE/CSRDE Graduation and Retention Survey – graduation (4-, 5-, and 6-year) and
retention data, by gender, ethnicity, residency status, and gender/ethnicity combinations.
Includes historical files for many institutions.
4. Senior and alumni surveys – Collect data on planned pursuit of advanced degrees plus a selfassessment of educational experience, outcomes, and learning.
5. NSF Survey of Earned Doctorates – This survey provides information on the undergraduate
degree institution for graduating PhD students each year. This widely-used and long-
10
DRAFT
running survey is covered more extensively in the white paper on graduate data. The SED
counts of baccalaureate origin can be normalized with average undergraduate enrollments
from IPEDS for the corresponding period 6 to 8 years earlier and the trends tracked over
time. All AAU schools participate in this survey, and AAUDE is trying to work with the
NSF to get a download of current and historical survey data to put into the AAUDE data
warehouse.
6. National Student Clearinghouse – Tracks students as they transfer from school to school, so
it can aid in getting a true graduation rate. Can also be used to help find out where an
institution’s baccalaureate recipients go on to graduate school.
7. PhDs 10 Years Out – A very interesting and extensive follow up study for a few disciplines
based upon the PhD graduates from a few selected schools. Could be used in a modified
form to look at undergraduate “feeder” schools.
8. Honors and Awards: Rhodes scholars, Marshall scholars, Truman scholars, Mellon
fellowships, Fullbright fellows, NSF graduate research fellowships, etc.
Data Issues
It would be difficult to collect all prestigious scholarships won by undergraduates, but if the AAU
decides on a core set of awards to pursue, it can be coordinated with the agencies that give out the
awards. It is also quite difficult to track students’ post-graduation activity. At many schools, senior
surveys do not elicit high response rates.
A limitation of the IPEDS Graduation Rates Survey is that it only collects information on full-time,
first time freshmen. Graduation rates on students who transfer between institutions or nontraditional students who only enroll part-time are not available nationally. This limitation is part of
what is behind Congressional interest in NCES’s proposal to collect IPEDS data at the unit-record
level, i.e., the student level.
Recommendations
The AAU should consider developing a core set of questions for use in exit survey and/or alumni
surveys, to track students’ post-graduation plans and outcomes. The Survey of Earned Doctorates
could be used more effectively once the AAUDE data warehouse gathers all the historical data, but
this will take quite a bit of work. Also, many public institutions do not have substantial alumni
organizations and don’t survey their alumni as easily and frequently as private institutions, so
coordination of an alumni survey may be problematic.
The PhD “10 years out” study required a lot of time and funding, so it is not realistic to expect AAU
campuses to replicate it, but it could provide inspiration for a more practical methodology.
Other, less traditional, ideas for measuring student outcomes include some way to measures of civic
engagement, such as questions on an alumni survey, or counting graduates in leadership roles such as
the Forbes top CEO’s, government leadership positions, etc.
11