Summary of Methodology http://www.nap.edu/catalog.php?record_id=12676
Michigan State University

• National Academies of Science “rate” research doctorate programs approx. every 13 years
( 1982, 1995, 2009 ).
http://sites.nationalacademies.org/pga/Resdoc/index.htm
1995 was largely a reputational (only) survey.
• In 2003, NRC published a study on the methods used in 1995 with recommendations on changes for 2005-06 data collection (for
2009 ratings).
http://books.nap.edu/catalog.php?record_id=10859
2

• 222 institutions
• 61 fields (MSU ranked in 54 fields)
• 5006 programs across the 61 fields
• Each field of study had produced at least 500
Ph.D.s in the 5 years prior to 2004-05.
• To be included, each University program must have granted 5 Ph.D. degrees in the 5 years prior to 2005-06.
3

• The taxonomy of fields was hotly debated.
Used CIP codes (US Dept of Ed taxonomy),
NSF fields, & info from scholarly societies.
• It is what it is! Not very useful for interdisciplinary programs as faculty effort was sub-divided. Universities use different names for similar programs.
• Learn more: http://sites.nationalacademies.org/PGA/Resdoc/PGA_044478 4

Agricultural Economics*
American Studies*
Animal Science
Anthropology
Astrophysics & Astronomy*
Biochem & Molecular Biology
Biosystems Engineering
Cell & Molecular Biology
Chemical Engineering
Chemistry
Civil & Environmental Eng
Communications
Comm Arts Media & Info Studies
Computer Science
Criminal Justice
Crop & Soil Sciences
Ecology, Evol Bio, & Behavior
Economics
Electrical Engineering
English
Mechanical Engineering
Microbiol & Molec Genetic
Entomology* Music Education
Environmental Tox icology Neuroscience
Fisheries & Wildlife
Food Science*
Forestry*
Pathobiol & Diagnostic Inv
Pharmacology/Toxicology
Philosophy
Genetics
Geography
Geological Sciences
History
Physics
Physiology
Plant Biology
Plant Breeding & Genetics
Horticulture
Human Nutrition
Plant Pathology
Political Science
Kinesiology Psychology
Large Animal Clinical Sci Sociology
Linguistics Spanish Lang, Lit, Culture
Materials Science & Eng Statistics
Mathematics Zoology
5

Faculty effort assignment to program(s)
(not always based on tenure/salary home unit):
– CORE
(supervise dissertations and/or on admissions/curriculum committees for PhD)
– NEW
(hired in previous 3 years & expected to become Core)
– ASSOCIATED
(not Core in program, but regular faculty at institution)
Note:
A faculty member’s program assignment may be divided between various programs, but will always total 100% Dept chairs made final decisions.
6

• Data collection was in 2005-06
• Data points collected over various spans of time (Question G)
• Approx. 25%+ of MSU faculty changed between 05-06 and present
• Approx. 94% questionnaire participation rate by MSU faculty
7

1. Institutional Questionnaire: U practices and a list of doctoral programs
2. Program Questionnaire: Characteristics of students, faculty, program
3. Faculty Questionnaire: Faculty work history, grants, pubs, and important characteristics of a quality program (Question G)
4. Student Questionnaire: Students ( post comps in English,
Chem Eng, Economics, Physics, Neuroscience
) background, faculty interactions, & post-graduation plans
8

5. Rating Questionnaire: Asked faculty to rate programs in their field
• Raters classified by rank, geographic region, faculty size in program. Each faculty member rated 15 programs with data provided. Not asked about basis for rating.
9

• In addition to the questionnaires, publications and citations data were collected thru ISI database http://isiwebofknowledge.com/.
• Humanities CVs submitted with faculty questionnaire were used to count books and publications.
• Honors and awards data came from 224 scholarly societies.
10

The 20 Variables (Question G on Faculty Questionnaire)
•
Publications per Allocated Faculty*, 2001-2006 (going back to 1986 for faculty in humanities fields)
• Average Citations per Publication (citations in 2001-2006 to articles dating back to
1981—for all fields except the humanities)
•
Number of Grants per Allocated Faculty*
•
Percent Interdisciplinary (% Associated Faculty)
• Percent Non-Asian Minority Faculty for Core or New Faculty, 2006**
•
Percent Female Faculty for Core or New Faculty, 2006**
• Awards per Allocated Faculty*
•
Average GRE, 2004-2006 (Verbal measure for the humanities, Quantitative measure for all other fields)
•
Percent students receiving full support in the first year, (fall 2006)
• Percent first year students with external funding, 2006
* Faculty members who served in more than one program were allocated to those programs based on whether they were core in the program and the share of that program of total dissertations supervised.
11

The 20 Variables (continued)
•
Percent Non-Asian Minority Students, 2006
• Percent Female Students, 2006
•
Percent International Students, 2006
•
Average annual PhDs graduated 2002 to 2006
• Average completions (8 year completion percentage for humanities fields, 6 years for other fields)
•
Time-to-Degree (for Full and Part Time graduates)
• Percent PhDs with definite plans for an academic position, 2001-2005 (including postdoctoral fellowships)
•
Student Work Space [1 = 100% of students with workspace, -1 if <100% of students with workspace]
• Health Insurance [1= provides health insurance, -1 = does not provide health insurance]
•
Student Activities (number offered from a list of 18)
** “Core” Faculty are those whose primary appointment is in the doctoral program.
“New” faculty are faculty with tenure track appointments who were appointed in the past 3 years.
12

MSU task force of statistical/survey experts to study the report and provide input:
Brian Silver
--
Director, Center for Statistical Consulting, Political Science
Neal Schmitt -- Chair, Department of Psychology
Les Manderscheid -- Ag, Food and Resource Economics and the Graduate School
Mary Black -- Assistant Director, Office of Planning and Budgets
Kyle Sweitzer -- Data Analyst, Office of Planning and Budgets
CONVENER: Karen Klomparens -- Dean, Graduate School
Methodology was complex and valid, with sources of uncertainty dealt with appropriately.
PROGRAMS WILL PROVIDE FURTHER INTERPRETATION OF THE DATA
13

14

1. Directly: Faculty chose the most important quality characteristics from Question G
2. Reputational rating: statistically related faculty ratings to the Question G variables.
Combined the direct and reputational weights, rank ordered. Used first and third quartiles of these ratings to yield a RANGE
OF RATINGS for each program
15

• No student outcomes are reflected in the overall rating, so 3 supplemental measures added
• NRC preparing a way for us to weight variables differently and run own ratings
• Not reporting data with less than 5 per cell
• Combined small progs with others to calculate weights
• Preparing a way for students to use data when considering programs
16

Research Impact
–
Publications/faculty member
– Citations/publication
–
Percent of faculty holding grants
– Honors and awards per faculty member
Student Support and Outcomes
–
Percent of students having full support in first year
– Percent of students with portable fellowships in first year
–
Percentage of students with RAships
– Percentage of students with TAships
–
Time-to-degree
–
Percent who complete in 6 years (sciences) or 8 years (humanities)
– Placement in an academic position or postdoc after graduation
Source: C. Kuh, NAS
17

Diversity of the Academic Environment
–
Percent of students who are female
–
Percent of faculty who are female
– Percent of students from underrepresented minority group
– Percent of faculty from underrepresented minority group
–
Percent of students who are international
Source: C. Kuh, NAS
18

• Programs will be arranged alphabetically and the range of ratings will be given for each.
• Ranges overlap other ranges for most programs. This means that there may be a number of programs of roughly the same quality.
• You should identify those similar programs in discussing the quality of your programs.
Source: C. Kuh, NAS
19

1) A list of the values of variables that your program supplied to the NRC or that was calculated from those variables
2) The normalized values for those variables
3) The median combined coefficient (statistical + direct) for each variable and its standard deviation
4) The range of the normalized variable values
5) The range of the combined effects of the coefficients in the random halves calculation
Source: C. Kuh, NAS
20

TABLE 5-1 Data and Coefficient Table for a Program in Economics
Standardized Program Values and Range of Combined Coefficients Institution Name: xxx Program Name: yyy
Column 1 Column 2 Column 3 Column 4 Column 5 Column 6
Combined Coefficients**
Program Program Value
Description Variable Value Standardized Minus 1 SD TO Plus 1 SD
Publications per Allocated Faculty
Cites per Publication
Percent of Faculty with Grants
Percent of Faculty Interdisciplinary
Percent of Non-Asian Minority Faculty
V1
V2
V3
V4
V5
1.074
1.171
25.50%
5.90%
7.70%
2.180
-0.234
-0.583
-0.641
0.547
0.118
0.276
0.084
n.s.# n.s.#
0.132
0.307
0.091
n.s.# n.s.# n.s.#
0.060
0.096
0.056
0.033
Percent Female Faculty
Awards per allocated faculty
Average GRE-Q
Percent 1 st yr. Students w/full support
Percent 1rst yr. Students w/portable fellowships
Percent non-Asian Minority Students
Percent Female Students
Percent International Students
Average PhDs 2002 to 2006
Percent Completing within 6 years
Time to Degree full and Part time
Percent students in Academic Positions
V6
V7
V8
V9
V10
V11
V12
V13
V14
V15
V16
V17
12.50%
0
746
100.00%
0.00%
10.00%
44.40%
53.30%
5.4
27.60%
5.67
11.10%
-0/440
-0.546
-0.165
0.980
-.544
0.069
0.678
-0.509
-.0355
-0.638
0.232
-1.405
n.s.#
0.043
0.092
0.036
0.021
Student Work Space
Health Insurance
Number of Student Activities Offered
V18
V19
V20
1
1
17
1
1
0.439
* Col 3 is based on data submitted by the program or calculated from these data.
+
Col 4 is standardized across all program values in the field, with mean of 0 and variance of 1.
** Col 5 is Minus 1 Standard Deviation from the Mean for the combined coefficients for the field as a whole
** Col 6 is Plus 1 Standard Deviation from the Mean for the combined coefficients for the field as a whole
# n.s. in a cell means the coefficient was not significantly different from 0 at the p=.05 level.
n.s.# n.s.#
0.026
n.s.#
-0.038
n.s.#
0.120
n.s.#
-0.028
0.049
21 n.s.#
-0.030
n.s.#
0.144
n.s.#
-0.017
0.065
n.s.# n.s.#
0.037

22

23

24

25

1. Identify variables with the largest contribution. These had the greatest effect on the range of ratings by program.
2. Compare your variable values with programs in other institutions. These will be available in an online database.
3. Consider any additional relevant comments about your program and the
NRC methods.
26

• The data will be made available 24-72 hours before they are publically released, and programs may review their data during this time. COMMENTS TO KK!
• Additional analyses will be conducted upon release of the data (by CIC, AAU, and by you!)
• This study improved MSU’s ability to have data readily available in a consistent fashion about students and committee memberships. Data collection has expanded to include ALL graduate programs (Master’s and Doctoral) via the GradInfo database.
www.gradinfo.msu.edu
See next slide…
27

28

29

Broad Field
Biological Sciences
Health Sciences
Engineering
Physical Sciences
Agricultural Sciences
Social Sciences
Humanities
Pubs per
Faculty
0.30
0.35
0.29
0.28
0.34
0.36
0.53
Cites per
Pub
0.21
0.16
0.25
0.26
0.18
0.26
------
Pct Fac w grants
0.36
0.37
0.29
0.29
0.34
0.22
0.15
Awards per fac
0.13
0.11
0.17
0.17
0.13
0.16
0.32
Source: C. Kuh, NAS
30

Broad Field
Biological Sciences
Health Sciences
Engineering
Physical Sciences
Agricultural Sciences
Social Sciences
Humanities
Pct full Finish Median Student
Supp 6 Yrs TTD Placmt
0.26
0.24
0.26
0.29
0.14
0.14
0.17
0.16
0.34
0.29
0.28
0.27
0.29
0.20
0.23
0.23
0.24
0.25
0.10
0.12
0.12
0.12
0.11
0.18
0.18
0.18
0.18
0.17
Collect
Outcome
0.17
0.16
0.18
0.18
0.18
0.18
0.17
Source: C. Kuh, NAS
31

Broad Field
Biological Sciences
Health Sciences
Engineering
Physical Sciences
Agricultural Sciences
Social Sciences
Humanities
Pct fac Pct fac Pct stud Pct stud Pct stud
Minority Female Minority Female Int’l
0.14
0.24
0.23
0.14
0.30
0.38
0.25
0.18
0.09
0.07
0.13
0.10
0.15
0.22
0.21
0.16
0.20
0.17
0.20
0.24
0.26
0.20
0.30
0.26
0.20
0.27
0.29
0.22
0.17
0.18
0.18
0.21
0.15
0.16
0.16
Source: C. Kuh, NAS
32