Industry/University
Cooperative Research
Centers
Highlights of Membership and Process/Outcome
Data
FY 2009-2010
IUCRC Evaluator’s Meeting
June 10, 2011
Denis Gray, Lindsey McGowen,
Sarah DeYoung & Landon LaPorte
North Carolina State University
Industry/University
Cooperative Research
Centers
MEMBERSHIP REPORT DATA
Take Home Message
• Tracking “membership” is important
because it may be the most important
measure of success for centers and the
IUCRC program
• But…
– We need to be very careful in interpreting
changes in membership overtime
Industry/University
Cooperative Research
Centers
Interpreting Membership Changes Over
Time
• Changes in membership numbers over time are
influenced by changes at different levels:
Industry/University
Cooperative Research
Centers
– Members: Individual members will leave a center and new
members will be added.
– Centers: mature centers will graduate (and their members will
be dropped) and new centers will be created (and their
members added).
• Recently, graduated centers have re-emerged as Phase 3 centers
– Sites: Generally speaking, new sites are added to existing
centers but not always
• Changes in program total and average can be due to any
combination of these factors.
– Totals are particularly unstable
– DIMS database may allow this to be completely disaggregated
INDUSTRIAL MEMBERSHIPS BY YEAR
Industry/University
Cooperative Research
Centers
AVERAGE NUMBER
OF MEMBERS PER CENTER
TOTAL NUMBER OF
MEMBERS
800
25
700
20
600
TOT. MEMBERS
UP; NUMBER OF
CENTERS UP
500
400
15
DECLINE IN
AVERAGE
MEMBERS:
GRADUATION OF
LARGE ADDITION
OF SMALL NEW?
10
300
200
5
100
'10
'07
'04
'01
98
95
92
89
'10
'08
'06
'04
'02
'00
98
96
94
92
90
88
85
85
0
0
AVERAGE MEMBERSHIP TURNOVER
Members Added this FY
8
7
5
Members Left this FY
COUNTERINTUITIVE
BUT THIS ONLY
INCLUDES
CONTINUING
CENTERS
4
3
2
1
Fiscal Year
'10
'09
'08
'07
'06
'05
'04
'03
'02
'01
'00
99
98
97
96
95
94
93
92
91
90
0
89
Members
6
Industry/University
Cooperative Research
Centers
Industry/University
Cooperative Research
Centers
What about the composition of IUCRC
membership?
Center Director
Membership Summary
Industry/University
Cooperative Research
Centers
Count
Total memberships
Percent
759
100%
Industry: large firms (500 + employees)
359
47.4%
Industry: small firms (< 500 employees)
234
30.8%
US Government: Federal
102
13.5%
Non-US Government
4
0.2%
US Government: State or local
28
3.8%
Non-Profit
22
2.9%
Other
10
1.4%
Number of organizations with
memberships
548
--
Member Composition 2006-2010
Industry/University
Cooperative Research
Centers
70%
60%
50%
40%
30%
20%
10%
0%
2006
2007
Large
2008
Small
2009
Fed
State
2010
Others
^ Categories comprising Others include: non-profit, non-US government, and other organization
Member Composition 2006-2010
(without outlier)
Industry/University
Cooperative Research
Centers
70%
60%
50%
40%
30%
20%
10%
0%
2006
2007
2008*
Large
Small
2009*
Fed
2010*
State
Others
* Years with Advanced Forestry excluded as a small business outlier: ‘08 Small = 36, ‘09 Small = 49, ‘10 Small = 57
^ Categories comprising Others include: non-profit, non-US gov’t, and other org
Membership: Organizations with the
Most Memberships
# of Memberships Organizations
26
U.S. Army
18
Dept of Energy
15
Lockheed Martin
13
Boeing, NASA
11
Air Force
10
GM, IBM
7
Intel Corporation, Raytheon
5
DoD, Navy
Dropped
GE
Industry/University
Cooperative Research
Centers
Membership: Organizations with the
Most Memberships Over Time
1995
2000
Industry/University
Cooperative Research
Centers
2005
2010
N
Name
N
Name
N
Name
N
Name
10
Dow Chem.
10
Motorola
17
DoE
26
Army
9
DuPont
9
Army
13
Army
18
DoE
8
3M, Motorola,
Ford
8
Lucent Tech., DoE
10
Boeing
15
Lockheed Martin
7
General Motors
7
Boeing, IBM,
Honeywell
8
Intel, Air Force
13
Boeing, NASA
6
Amoco, Boeing,
Texas Inst.
6
Daimler Chrysler
7
Raytheon, HRL Labs,
DoD
11
Air Force
--
--
5
Ford
6
Honeywell, Navy
10
GM, IBM
--
--
--
--
5
Dow Chem., Siemens
7
Intel, Raytheon
--
--
defense/contractor
--Increased
----
5
DoD, Navy
Dropped:
Dow Chem., DuPont, 3M,
General Motors, Amoco,
Texas Inst., EPA, Mavy
Motorola, Lucent Tech., IBM,
Daimler Chrysler, Ford
HRL Labs, Honeywell,
Dow Chem, Siemens
New:
Lucent Tech., IBM,
Honeywell, Daimler
Chrysler, DoD
Intel, Air Force, Raytheon,
HRL Labs, Dow Chem.,
Siemens, Navy
General Motors,
Lockheed Martin, NASA,
IBM, Raytheon
Government Members
134 memberships, 54 organizations (17.5% of all memberships; see slide 8)
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Air Force Research Laboratory
Arkansas Public Service Commission
Army
Bureau of Land Management, Eugene District Office
California Association of Sanitation Agencies (CASA)
Center for Advance Technology in Telecom
CERDEC
City of Columbus
City of Glendale
City of Peoria
City of Peoria
City of Phoenix
City of San Pedro (Coahuila, Mexico)
City of Scottsdale
City of Tucson
County Sanitation Districts of Los Angeles
DHS
DoD
DoE
Ector County Medical
Center Federal Aviation Administration
Federal Bureau of Investigation
Florida Space Grant Consortium
ICTAS
Idaho Department of Lands
Industrial Technology Research Institute
King County - Northwest Biosolids Management Assoc
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Industry/University
Cooperative Research
Centers
Knolls Atomic Power Laboratory
Maine Bureau of Parks and Lands
Montana Department of Natural Resources and Conservation
NASA
National Highway Traffic Safety Administration
National Institute for Occupational Safety and Health (NIOSH)
National Institute of Standards & Technology (NIST)
National Oceanographic & Atmospheric Administration (NOAA)
National Reconnaissance Office
National Security Agency
Navy
Ohio Department of Job and Family Service (ODJFS)
Oklahoma City Air Logistics Center
Oklahoma Department of Transportation
Orange County Sanitation District
Oregon Department of Forestry
PA Department of Community and Economic Development
Pima County Regional Wastewater Reclamation Department
Port of Guaymas/State of Sonora
SENAI-CETA
Space Engineering Institute
Space Florida
State of Hawaii Strategic Industries Division
Town of Brookhaven
United States Bureau of Reclamation
USDA Forest Service Research
Single vs. Multi-Site Members
Industry/University
Cooperative Research
Centers
Single
Multiple
Count of Centers
9
33
Count of Sites
9
107 (~3.24 per center)
Members
87
672
Avg per Center
9.7
20.4
Avg per Site
--
6.3
Lost members 2009
19
117
Avg per Center
2.1
3.5
Avg per Site
--
1.1
14
131
Avg per Center
1.6
4
Avg per Site
--
1.2
Recruited members 2009
Conclusion and a caution …
Industry/University
Cooperative Research
Centers
• Membership looks improved, losses have stopped, big
increase in new centers, but…
• All our analyses of membership changes over time need
to be interpreted cautiously because
– There is a six-month lag in data, so we are not sure about
current snapshot
– Because membership is affected by factors operating at multiple
levels of analysis
• NCSU group attempted to change the data collection method
starting this year in order to accurately capture all these change
mechanisms.
– We asked centers to report members by site, but this option was not widely
used…
• When operational, DIMS system will provide a more fine-grained
view of sources of member ebb and flow
– Stay tuned
Industry/University
Cooperative Research
Centers
PROCESS – OUTCOME DATA
Process/Outcome
2009-2010 Response Rates
Industry/University
Cooperative Research
Centers
Center level
Individual level
Industry
Faculty
Industry
Faculty
39
39
743
604
+5
+5
+35
+28
-3
-3
-26
-24
+2
+1
+6
+8
47
46
788
651
Centers Excused from Evaluation
-4
-4
-22
-18
Centers that did not return data
0
0
0
0
Available Population
43
42
766
633
Data Received
43
42
325
266
Response frequencies
Population from CD report
1st Year Reporting Population from CD
report
Retired/Defunct Centers
Retired/Defunct Centers Reporting
Population
Response Rates
Received / Population
91.5%
91.3%
41.2%
40.9%
Received / Available Population
100%
100%
42.4%
42.0%
Industry Response Rate
100.00%
Industry/University
Cooperative Research
Centers
80.00%
Center Level
Received/Available
Population
60.00%
Center Level
Received/Population
Individual Level
Received/Available
Population
40.00%
20.00%
Individual Level
Received/Population
0.00%
Note: 2000-2001 & 2002-2003 data not included
Industry/University
Cooperative Research
Centers
Industry Questionnaire
Select Results
What percentage of Center research projects do
you take an active interest in?
90%
Industry/University
Cooperative Research
Centers
5
70%
4
50%
3
30%
2
10%
1
2.88
3.01
2.91
2.63
05-06
06-07
07-08
Mean
2.76
08-09
09-10
*percentage values on y-axis represent midpoint of response range
What percentage of Center research projects do
you take an active interest in?
Industry/University
Cooperative Research
Centers
30
Percent
20
10
0
0-19%
20-39%
2005-2006: M=2.88
2008-2009: M=2.76
40-59%
60-79%
M=2.91
Percentage2006-2007:
of
research
projects
2009-2010: M=3.01
80-100%
2007-2008: M=2.63
During the past year, how satisfied were you with the
capabilities of the researchers and quality of the research
program?*
Industry/University
Cooperative Research
Centers
Mean
5
4.3
4.33
4.13
4.3
4.22
4
3
2
1
2005-2006
2006-2007
2007-2008
2008-2009
2009-2010
* Prior to 2006-2007, this question asked about capabilities of faculty and graduate students
During the past year, how satisfied were you with
the Center’s breadth of research topics covered?
Mean
5
4
Industry/University
Cooperative Research
Centers
3.86
4.02
3.87
3.93
3.96
3
2
1
2005-2006
2006-2007
2007-2008
2008-2009
2009-2010
During the past year, how satisfied were you with
the Center’s focus of the research?
Industry/University
Cooperative Research
Centers
5
4
3.81
3.93
3.8
3.85
3.89
2008-2009
2009-2010
3
2
1
2005-2006
2006-2007
2007-2008
During the past year, how satisfied were you with
relevance of the research to my organizations needs?*
short
long
needs
5
4
3.88
3.74
3.68
2006-2007
2007-2008
3.81
3.78
2008-2009
2009-2010
3.43
3
2
1
2005-2006
* Prior to 2006-2007, respondents reported on both long and short term needs.
Industry/University
Cooperative Research
Centers
During the past year, to what extent has participation in the
Center contributed to the following benefits for your
organization?
Industry/University
Cooperative Research
Centers
Research and Development
4
3.42
3.18
2.94
3.12
3.25
3
2
1
2005-2006
2006-2007
2007-2008
2008-2009
2009-2010
During the past year, to what extent has participation in the
Center contributed to the following benefits for your
organization?
Industry/University
Cooperative Research
Centers
Commercialization
5
4
3
2.32
2.27
2.03
2.14
2.16
2008-2009
2009-2010
2
1
2005-2006
2006-2007
2007-2008
During the past year, to what extent has participation in the
Center contributed to the following benefits for your
organization?
Industry/University
Cooperative Research
Centers
Professional Networking
5
4
3.64
3.53
3.37
3.39
3.33
2007-2008
2008-2009
2009-2010
3
2
1
2005-2006
2006-2007
During the past year, how satisfied were you with center
administrative operations?
Industry/University
Cooperative Research
Centers
5
4.1
4.13
4.04
3.93
3.97
2008-2009
2009-2010
4
3
2
1
2005-2006
2006-2007
2007-2008
Industry/University
Cooperative Research
Centers
Outcomes: Value of Center-Stimulated
Projects
15 Year Trend – New Research Projects
(reported median)
2.5
2009-2010
Total Sample, M = 2.07
1+ Projects, M = 2.64
Total Sample
Industry/University
Cooperative Research
Centers
1+ Projects
1.5
1
0.5
0
19941995
19951996
19961997
19971998
19981999
19992000
20002001
20012002
20032004
20042005
20052006
20062007
20072008
20082009
20092010
N of projects
2
Note: 2002-2003 data not included
15 Year Trend - Total Dollar Value of Projects
(reported median)
Industry/University
Cooperative Research
Centers
300,000
$$ per Member
$$ per Member if 1+ Projects
$$ per Project
250,000
Dollars
200,000
150,000
100,000
50,000
0
1994- 1995- 1996- 1997- 1998- 1999- 2000- 2001- 2003- 2004- 2005- 2006- 2007- 2008- 20091995 1996 1997 1998 1999 2000 2001 2002 2004 2005 2006 2007 2008 2009 2010
Note: 2002-2003 data not included
5 Year Trend - Total Dollar Value of Projects
(reported mean)
Industry/University
Cooperative Research
Centers
1000
Dollars (in thousands)
$$ per Member
$$ per Member if 1+ Projects
$$ per Project
800
600
400
200
0
2005-2006
2006-2007
2007-2008
2008-2009
* Data collected by forced choice format beginning 2006-7
2009-2010
Value of Center-Stimulated Projects:
FY 2009-2010
Industry/University
Cooperative Research
Centers
50
40
30
20
10
0
0
10
25
50
60
80
100
150
200
250
300
400
500
600
700
800
900
1,000
1,500
2,000
2,500
3,000
4,000
5,000
Number of members
60
Dollars (in thousands)
Estimating IUCRC-Wide CenterStimulated Funding
Industry/University
Cooperative Research
Centers
• Program-wide center-stimulated funding for 09-10 is
$86.8 million
• There is a fundamental problem estimating both
center and program-wide value of center stimulated
projects
– Response rate is running at ~41% of total population
• Our reported value is a VERY conservative estimate
– Assumes none of the 59% non-responders invested in
center stimulated projects
• Need to find a defensible approach to estimating
– Non-responders
• mean
• median
• 50% of median
Estimating IUCRC-Wide Follow-on
Funding
300
Total Reported
(N=283)
241.6
250
200
150
100
86.8
Industry/University
Cooperative Research
Centers
112.1 99.5
50
0
2009-10 (in Millions of Dollars)
Non-respondents
Estimated (N=
788) Based on
Mean
Non-respondents
Estimated
(N=788) Based on
Median
Non-respondents
Estimated
(N=788) Based on
50% Median
Industry/University
Cooperative Research
Centers
Students Hired
During the past year, how many students trained in the Center
projects were hired by your organization?
0.6
0.5
mean
0.4
0.3
0.2
0.1
Note: 2002-2003 data based on subset of Centers
Industry/University
Cooperative Research
Centers
Will your organization renew its membership?
Industry/University
Cooperative Research
Centers
5
4.09
3.91
4.11
4.01
4.0
4.13
4.17
4
3
2
1
2003-2004 2004-2005 2005-2006 2006-2007 2007-2008 2008-2009 2009-2010
Industry Summary
Industry/University
Cooperative Research
Centers
• Satisfaction and benefits appear relatively stable with some
signs of improvement
– Percent of projects interested in; impact on R&D; commercialization;
student hires; renewal
• Follow-on projects and funding continues to be very volatile
– Investments still being made
– Median # of projects reported & Median investments are up
– Mean declines are likely based on outlier from 2008-2009 ($50M)
• Student hiring showing strong rebound
• Will see increased emphasis on questions that get at economic
impact: cost avoidance, cost savings, etc.
Industry/University
Cooperative Research
Centers
Faculty Questionnaire
Select Results
Faculty Long and Short Forms
Long Form
Industry/University
Cooperative Research
Centers
Short Form
# of items
13
6
# of questions in common
6
6
# of unique questions
7
0
# of centers using form
26
13
Sample size
145
121
Compared to the research projects that you typically
conduct outside the Center, would you describe your
Center-funded research as:
Scope
5
Much more basic------Much more applied
Industry/University
Cooperative Research
Centers
4
3.3
3.5
3.5
3.57
3.68
2008-2009
2009-2010
3
2
1
2005-2006
2006-2007
2007-2008
During the past year, how satisfied were you with the
quality of center-supported research program?
Industry/University
Cooperative Research
Centers
5
4.13
4
4.29
4.29
4.27
2007-2008
2008-2009
2009-2010
3.76
3
2
1
2005-2006
2006-2007
During the past year, how satisfied were you with the
relevance of the Center’s research program to my
professional goals?
Industry/University
Cooperative Research
Centers
5
4.01
4
4.26
4.25
2007-2008
2008-2009
4.2
3.7
3
2
1
2005-2006
2006-2007
2009-2010
During the past year, what impact has participation in the
Center had for YOU in the following areas?
Industry/University
Cooperative Research
Centers
The feeling of accomplishment I get from the
research I do
5
4.0
4
3.46
3.58
3.67
2006-2007
2007-2008
3.84
3
2
1
2005-2006
2008-2009
2009-2010
During the past year, what impact has participation in the
Center had for YOU in the following areas?
Industry/University
Cooperative Research
Centers
Opportunities for research contracts/grants
5
3.75
4
3.65
3.74
3.72
2007-2008
2008-2009
2009-2010
3.32
3
2
1
2005-2006
2006-2007
During the past year, what impact has participation in the
Center had for YOU in the following areas?
Industry/University
Cooperative Research
Centers
Recognition I receive for the work I do
5
4
3.25
3.13
2005-2006
2006-2007
3.49
3.63
3.64
2007-2008
2008-2009
2009-2010
3
2
1
During the past year, what impact has participation in the
Center had for YOU in the following areas?
Industry/University
Cooperative Research
Centers
Access to useful equipment
5
4
3
2.79
2.95
3.03
2006-2007
2007-2008
3.28
3.07
2
1
2005-2006
2008-2009
2009-2010
During the past year, what impact has participation in the
Center had for YOU in the following areas?
Industry/University
Cooperative Research
Centers
Ability to support graduate students
5
3.62
4
3.65
3.91
3.32
3.24
3
2
1
2005-2006
2006-2007
2007-2008
2008-2009
2009-2010
During the past year, what impact has participation in the
Center had for YOU in the following areas?
Industry/University
Cooperative Research
Centers
Ability to publish my work in quality proceedings
and journals
5
4
3.22
3.17
3.49
3.42
3
2
1
2006-2007
2007-2008
2008-2009
2009-2010
Prior to 2006-2007 faculty were asked to report the number of publications
rather than a rating of ability to publish.
During the past year, how satisfied were you with center
administrative operations?
Industry/University
Cooperative Research
Centers
5
3.90
3.9
4.11
4.24
4.23
4
3
2
1
2004-2005
2005-2006
2006-2007
2007-2008
2008-2009
Faculty Summary
Industry/University
Cooperative Research
Centers
• Most indicators are relatively stable; some exceptions …
– Steady creep toward more applied research over past few years;
decline in sense of accomplishment
– Noticeable increase in ability to support graduate students
– Steady increase in satisfaction with administrative operations
• Interpretation (with caution):
– Increasing pressure to deliver more immediately applicable
results
– Temporary ARRA effect of satisfaction and student support; will
need to monitor
– Response bias (questionnaires returned by more positive faculty)
and cohort effect (more younger centers) make interpretation
tentative
How Should These Process/Outcome
Results be Used?
Industry/University
Cooperative Research
Centers
• Trends are probably much more interpretable at local center level
– Director leaves; research direction changes; move from one-on-one to
consortial center
• Benchmark center against previous year and national norms
– By comparing means, medians, and standard deviations, evaluators can
see how their centers compare to national “norms”
– Insert national data into industry software tools for this year/last year
comparisons with your center
– Move current center means to previous year in e-mail software package
• Caution:
– Deteriorating response rate undermines the validity and usefulness of
feedback
• Need to reinvent our methods
• Cautiously considered interpretation