Efficiency as a Measure of
Knowledge Production of
Research Universities
Amy W. Apon* Linh B. Ngo*
Michael E. Payne*
Paul W. Wilson+
School of Computing* and Department of Economics +
Clemson University
1
Content
•
•
•
•
•
2
Motivation
Methodology
Data Description
Case Studies
Conclusion
Motivation
• Recent economic and social events motivate
universities and federal agencies to seek more
measures from which to gain insights on return on
investment
3
Motivation
• Traditional measures of productivity:
– Expenditures, counts of publications, citations, student
enrollment, retention, graduation …
• These may not be adequate for strategic decision
making
• Traditional Measures of Institutions’ Research
Productivity:
– Are primarily parametric-based
– Often ignore the scale of operation
4
Research Question
• What makes this institution more efficient in
producing research?
• What makes this group of institutions more efficient
in producing research?
• How do we show statistically that one group of
institutions is more efficient than the other group
5
Efficiency as a Measure
• Using efficiency as a measure of knowledge
production of universities
– Extends traditional metrics
– Utilizes non-parametric statistical methods
• Non-parametric estimations of relative efficiency of production
units
• No endogeneity: we are not estimating conditional mean function
because we are not working in a regression framework
• Scale of operations is taken into consideration
– Rigorous hypothesis testing
6
Background
• We define 𝑃 as the set of feasible combinations of p inputs
and q outputs, also called the production set.
Output
Feasible set
Input
7
• There exists a maximum
level of output on a given
input (the concept of
efficiency)
• The efficiency score is an
estimation with regard to
the true efficiency frontier
• Range: [0,1]
Hypothesis Testing Procedure
8
Convexity
• Test for Convexity
– Null hypothesis: The production set is convex
– Alternative: The production set is not convex
Output
Output
Feasible set
Input
9
Output
Feasible set
Input
Feasible set
Input
Constant Returns to Scale
• Test for Constant Returns to Scale
– Null hypothesis: The production set has constant returns to
scale
– Alternative: The production set has variable returns to scale
Output
Output
Feasible set
Input
10
Feasible set
Input
Group Distribution Comparison
• Test for Equivalent Means (EM)
– Null hypothesis: 𝜇1 = 𝜇2
– Alternative: 𝜇1 > 𝜇2
• Test for First Order Stochastic Dominance (SD)
between the two efficiency distributions:
– Null hypothesis: distribution 1 does not dominate
distribution 2
– Alternative: distribution 1 dominates distribution 2
11
Case Studies
• University Level
• Departmental Level
• Grouping Categories
–
–
–
–
12
EPSCoR vs. NonEPSCoR
Public vs. Private
Very High Research vs. High Research
“Has HPC” versus “Does not have HPC”
Hypotheses
• Institutions from states with more federal funding
(NonEPSCoR) will be more efficient than institutions
from states with less federal funding (EPSCoR)
• Private institutions will be more efficient than public
institutions
• Institutions with very high research activities will be
more efficient than institutions with high research
activities
13
University: Data Description
•
•
•
•
•
•
14
NCSES Academic Institution Profiles
NSF WebCASPAR
Web of Science
Aggregated data from 2003-2009
Input: Faculty Count, Federal Expenditures
Output: PhD Graduates, Publication Counts
University
• Test of Convexity:
– p = 0.4951: Fail to reject the null hypothesis of convexity
• Test of Constant Returns to Scale:
– p = 0.9244: Fail to reject the null hypothesis of constant
return to scale
15
University:
EPSCoR vs NonEPSCoR
EPSCoR
NonEPSCoR
p-values for EM and SD tests
Group 1: EPSCoR
Group 2: NonEPSCoR
p-values for EM and SD tests
Group 1: NonEPSCoR
Group 2: EPSCoR
Count
45
118
EM
SD
EM
SD
Mean
Efficiency
0.325
0.385
4.3 × 10−26
0.993
0.999
--
• While the first set of EM/SD tests indicates that the distribution of
efficiency scores for EPSCoR institutions does not dominate the
distribution of efficiency scores for NonEPSCoR institutions,
• The second set of EM/SD tests also rejects the notion that the distribution
of efficiency scores for NonEPSCoR institutions is greater than the
distribution of efficiency scores for EPSCoR institutions.
• This implies that NonEPSCoR institutions are at least as efficient as EPSCoR
institutions
16
University:
Public vs. Private
Public
Private
p-values for EM and SD tests
Group 1: Public
Group 2: Private
p-values for EM and SD tests
Group 1: Private
Group 2: Public
Count
110
53
EM
SD
EM
SD
Mean
Efficiency
0.396
0.311
3.1 × 10−86
0.011
0.999
--
• The first set of EM/SD tests indicates that the distribution of efficiency
scores for public institutions dominates the distribution of efficiency
scores for private institutions,
• The second set of EM/SD tests also supports this result by rejects the
notion that the distribution of efficiency scores for public institutions is
greater than the distribution of efficiency scores for private institutions.
• This result shows strong evidence that public institutions are more
efficient than private institutions
17
University:
VHR vs. HR
VHR
HR
p-values for EM and SD tests
Group 1: VHR
Group 2: HR
p-values for EM and SD tests
Group 1: HR
Group 2: VHR
Count
80
83
EM
SD
EM
SD
Mean
Efficiency
0.398
0.338
9.1 × 10−90
0.021
0.999
--
• This result shows strong evidence that institutions with very high research
activities are more efficient than institutions with only high research
activities
18
Department: Data Description
• National Research Council: Data-Based Assessment of
Research-Doctorate Programs in the U.S. for 2005-2006
• Input: Faculty Count, Average GRE Scores
• Output: PhD Graduates, Publication Counts
• 8 academic fields have sufficient data:
–
–
–
–
–
–
–
–
19
Biology
Chemistry
Computer Science
Electrical and Computer Engineering
English
History
Math
Physics
Department
Department
20
p-values
Test for Convexity
Test for Constant Returns to Scale
Biology
0.032
--
Chemistry
0.466
0.060
Computer Science
0.368
0.999
Electrical and Computer
Engineering
0.078
--
English
0.003
--
History
8.4 × 10−5
--
Mathematics
0.626
0.894
Physics
0.214
0.999
Department:
EPSCoR vs NonEPSCoR
EPSCoR
NonEPSCoR
Count/Mean Efficiency
21
p-value for EM/SD tests:
Group 1: EPSCoR
Group 2: NonEPSCoR
p-values for EM/SD tests:
Group 1: NonEPSCoR
Group 2: EPSCoR
EM
SD
EM
SD
Biology
35/0.81
86/0.88
2.9 × 10−26
0.997
0.999
--
Chemistry
54/0.39
126/0.51
3.3 × 10−31
0.858
0.999
--
Computer
Science
30/0.3
97/0.49
3.1 × 10−17
0.999
0.999
--
Electrical and
Computer
Engineering
34/0.66
102/0.87
1.7 × 10−6
0.999
0.999
--
English
27/0.91
92/0.89
9.5 × 10−272
0.648
0.999
--
History
30/0.92
107/0.92
0.0000
0.802
0.999
--
Mathematics
32/0.48
95/0.59
7.9 × 10−6
0.953
0.999
--
Physics
41/0.44
120/0.59
4.4 × 10−24
0.999
0.999
--
Department:
Public vs. Private
Public
Private
Count/Mean Efficiency
22
p-value for EM/SD tests:
Group 1: Public
Group 2: Private
p-values for EM/SD tests:
Group 1: Private
Group 2: Public
EM
SD
EM
SD
Biology
82/0.85
39/0.89
0.999
--
2.8 × 10−28
0.230
Chemistry
130/0.45
50/0.53
0.999
--
5.3 × 10−48
0.096
Computer
Science
92/0.42
35/0.5
4.5 × 10−217
0.984
0.999
--
Electrical and
Computer
Engineering
97/0.79
39/0.86
0.999
--
1.7 × 10−96
0.127
English
81/0.89
38/0.92
0.9999
--
9.6 × 10−233
0.3626
History
87/0.92
50/0.91
0.9999
--
4.4 × 10−228
0.9318
Mathematics
90/0.55
37/0.59
0.1734
--
0.8265
--
Physics
11/0.54
50/0.59
0.9138
--
0.0861
0.1917
Department:
VHR vs. HR
VHR
HR
Count/Efficiency
23
p-value for EM/SD tests:
Group 1: VHR
Group 2: HR
p-values for EM/SD tests:
Group 1: HR
Group 2: VHR
EM
SD
EM
SD
Biology
67/0.89
40/0.79
0.999
--
2.5 × 10−24
0.999
Chemistry
115/0.56
57/0.35
0.010
6.1 × 10−10
0.989
--
Computer
Science
95/0.5
29/0.28
0.999
--
2.5 × 10−12
0.999
Electrical and
Computer
Engineering
94/0.83
37/0.77
0.999
--
English
85/0.89
32/0.91
0.999
--
5.5 × 10−76
0.246
History
101/0.92
33/0.91
0.999
--
0.0000
0.935
Mathematics
94/0.61
32/0.42
0.0001
1.3 × 10−5
0.999
--
Physics
117/0.63
42/0.35
0.968
--
0.999
--
1.2 × 10−24
0.950
Implication
• Efficiency estimations, together with hypothesis
testing, provide insights for strategic decision
making, particularly at departmental level.
• Lower efficiency estimate does not mean a program
is not doing well.
• Issues:
– Lack of data and integration/curation of data
24
Questions
25