Smith Student Outcomes and Baccalaureate Origins Overview of Outcomes Research
Overview of Outcomes Research
• Need for outcomes data and analysis
y
–
–
–
–
Grant applications
Program design
Program assessment
Program assessment
Continuous improvement
• What do we know? –
–
–
–
National context
Baccalaureate origins of PhDs data set
Smith’ss alumnae survey
Smith
alumnae survey
SURF research
• Discussion: What do we want to know?
What do we Know?
•
Sources for National Context
•
National Science Board. 2010. Science and Engineering Indicators 2010. Arlington, VA: National Science Foundation (NSB 10‐01).
http://www.nsf.gov/statistics/seind10/start.htm
•
AAUW. 2010. Why so few? Women in science, technology, engineering, and mathematics. Washington: hill, c., corbett, c., st. rose, a.
p //
g/
/ p
/ y
p
http://www.aauw.org/research/upload/whysofew.pdf
•
National Science Foundation, Division of Science Resources Studies. 2006. Baccalaureate Origins of S&E Doctorate Recipients (NSF 08‐311).
http://www.nsf.gov/statistics/infbrief/nsf08311/
NSB: Science and Engineering Indicators 2010
• The number of S&E (includes social and behavioral sciences) bachelor’s degrees has risen steadily over the past 15 years. S&E bachelor’s degrees have consistently accounted for 1/3 of all bachelor’s degrees in the past 15 years.
g
p
y
• The share of Bachelor’s degrees awarded to women increased in many major S&E fields from 1993‐2007.
• W
Women have earned 58% of all bachelors degrees since h
d 58% f ll b h l
d
i
2002, and have earned about half of all S&E bachelor’s degrees since 2000, but major variations exist.
NSB: Science and Engineering Indicators 2010
• In
In 2007, men earned a majority of bachelor
2007, men earned a majority of bachelor’ss degrees awarded degrees awarded
in engineering, computer sciences, and physics (81%, 81%, and 79%, respectively). • Women earned half or more of bachelor’s degrees in psychology (77%), biological sciences (60%), social sciences (54%), agricultural sciences (50%), and chemistry (50%) (A recent l
l
(5 ) d h
(5 ) (
increase from 41% since 1996).
• Women’s share of bachelor’s degrees in computer sciences, mathematics, and engineering has declined in recent years.
AAUW: Why So Few? 2010
AAUW: Why So Few? 2010
AAUW: Why So Few? 2010
What do we Know About Smith?
What do we Know About Smith?
Majors by Division from 1982 to 2008
Division
vso I
Linear (Division I)
Division
v s o II
Linear (Division II)
Division
v s o III
Linear (Division III)
Interdivisional/other
te d v s o a /ot e
Linear (Interdivisional/other)
50
% Declared M
Majors
40
30
20
10
0
1982
1985
1988
1992
1996
2000
Pl notet that
Please
th t scalel on the
th Y axisi has
h a maximum
i
value
l off 50%.
50% Thi
This chart
h t shows
h allll ddeclared
l d majors;
j ddouble
bl majors
j are counted
t d twice.
ti
Source: Office of the Registrar. Majors Official Counts, 1982‐2008.
2004
2008
Doctorates Awarded in Science and Engineering and non‐S&E Fields: 1998–2008
35,000
30,000
25,000
20,000
Science and engineering
Non‐science and engineering
15,000
10,000
5 000
5,000
0
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
SOURCE: NSF/NIH/USED/NEH/USDA/NASA, 2008 Survey of Earned Doctorates.
2008
AAUW: Why So Few? 2010
What do we Know About Outcomes?
•
Sources for Outcomes Data
–
Higher Education Data Sharing (HEDS) Consortium. M h 2010 HEDS W i h d B
March 2010. HEDS Weighted Baccalaureate Origins l
O i i
Study.
•
•
•
–
–
Carnegie Foundation for the Advancement of Teaching (CFAT). F b
February 2010. Web. 17 February 2010. 2010 W b 17 F b
2010
http://classifications.carnegiefoundation.org/resources/cc2005_pu
blic_file_021110.xls
National Center for Education Statistics (NCES). February 2010. Integrated Postsecondary Education Data Survey (IPEDS). Web. 12 February 2010. http://nces.ed.gov/ipeds
National Science Foundation (NSF). February 2010. WebCASPAR. Web. 12 February 2010. http://caspar.nsf.gov
y
p // p
g
2009 Alumnae Survey
Banner Outcomes Data
PhD Production
PhD Production
•Baccalaureate Origins Data (SED) is very reliable, but is limited to examining one desirable outcome (ignores any other terminal degrees, including professional degrees like MDs and JDs).
like MDs and JDs).
•Recent NSF InfoBrief listed top 50 institutions by weighted baccalaureate origin in S&E; Smith did not eighted baccala reate origin in S&E Smith did not
appear. Further inquiries identified Smith’s rank as 64 and S&E PhDs per 100 origins as 4.9 (50th ranked institution had 5.3 PhDs per 100 origins). PhD Production
PhD Production
•There are many ways to slice data given the pieces of information compiled and cleaned HEDS (Final universe of 1533 BO colleges and
compiled and cleaned HEDS (Final universe of 1533 BO colleges and universities)
• Carnegie Classification
• Baccalaureate Origins (Overall and Women‐Only)
• Bachelor’s Degrees Awarded (Overall and Women‐Only)
•For this presentation, universe of study narrowed to “women‐only “ (for both Baccalaureate Origins and Bachelor’ss Degrees Awarded) and (for both Baccalaureate Origins and Bachelor
Degrees Awarded) and
by Carnegie classified Baccalaureate Colleges.
•Data
Data presented in 10 Year Sums to smooth out year to year presented in 10 Year Sums to smooth out year to year
variability. Non‐overlapping samples.
PhD Production
PhD Production
•Different
Different ways of summarizing data (and for ways of summarizing data (and for
different purposes)
•Baccalaureate Origins: Count of Doctorate recipients who earned their Bachelor’s degree at the listed institutions
•Proportion: Compare institutions’ origins divided by p
p
g
y
the numbers of Bachelor’s Degrees awarded (weight or yield)
•Rank institutions by unweighted counts of origins
•Rank institutions by unweighted
counts of origins
•Rank institutions’ origins weighted by their bachelor’s degree production
Doctorates Awarded to Smith Alumnae
700
600
5500
400
Overall
Life Science
PPsychology
hl
300
200
100
0
1977-1986
1987-1996
Overall
1977-1986
1987-1996
1997-2006
1997-2006
Life Science Psychology
96
604
95
83
560
99
97
615
96
Source: Weighted Baccalaureate Origins Study. March 2010. The Higher Education Data Sharing (HEDS) Consortium
PhDs per 100 Smith Graduates
12 00
12.00
10.00
8 00
8.00
Overall
Life Science
Psychology
6.00
4.00
2.00
0.00
1977-1986
1987-1996
Overall
1997-2006
Life Science
Psychology
1977 1986
1977-1986
10 79
10.79
1 70
1.70
1 72
1.72
1987-1996
8.25
1.46
1.22
1997-2006
8.63
1.35
1.36
Proportion calculated is number of PhDs awarded to alumnae over number of all baccalaureates awarded 10 years prior
(example: PhDs awarded to alum 1997‐2006 divided by Smith Baccalaureate Completions 1987‐1996)
Source: Weighted Baccalaureate Origins Study. March 2010. The Higher Education Data Sharing (HEDS) Consortium
Unweighted Rank for Baccalaureate Origins (Women Only)
among Baccalaureate Colleges
1977-1986
1987-1996
1997-2006
0
5
10
15
R
Rank
20
25
30
35
40
45
50
Source: Weighted Baccalaureate Origins Study. March 2010. The Higher Education Data Sharing (HEDS) Consortium
Overall
Life Science
Psychology
Weighted Rank for Baccalaureate Origins (Women Only) among Bacclaureate Colleges
1977‐1986
1987‐1996
1997‐2006
0
5
10
15
Ra
ank
20
11
18
25
21
25
30
35
40
22
28
36
41
45
50
Source: Weighted Baccalaureate Origins Study. March 2010. The Higher Education Data Sharing (HEDS) Consortium
48
Overall
Life Science
P h l
Psychology
What are They Doing?
What are They Doing?
• Alumnae Survey 2009
– Administered annually
Ad i i
d
ll
– For 2009:2, 5, 10, 15, and 20 years out
– Response Rates
•
•
•
•
•
2 yr out ‐ 53.6%
5 yr out ‐ 55.8%
5 yr out 55 8%
10 yr out ‐ 50.7%
15 yr out ‐ 48.8%
20
20 yr out ‐
53 1%
53.1%
– Online questionnaire with mail and phone outreach
Advanced Education
Advanced Education
100%
89.3%
90%
80%
75.7%
70%
60%
50%
40%
30%
20%
10%
0%
Grad/Professional Post‐Grad Enrollment
Prepared 'more than adequately' or 'very well'
Source: 2009 Smith College Alumnae Survey data for those 10,15, and 20 years post‐graduation.
Education and Work: Relation of Career to Major
l
f
100%
90%
80%
70%
60%
52 7%
52.7%
50%
40%
32.1%
30%
20%
15.2%
10%
0%
Same field as major
Related to major
Is your current position related to your undergraduate field(s) of study?
Source: 2009 Smith College Alumnae Survey data for those 10,15, and 20 years post‐graduation.
Not related
Education and Work: Career Preparation & Satisfaction
100%
90%
80%
79.7%
83.5%
70%
60%
50%
40%
30%
20%
10%
0%
Prepared "more than adequately" or "very well" Satisfied with the course of career
Source: 2009 Smith College Alumnae Survey data for those 10,15, and 20 years post‐graduation.
Satisfaction with Smith Education & Current Life
f
100%
95.6%
90.5%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Undergraduate education
Undergraduate education
Your life right now
Your life right now
Very or Generally Satisfied
Source: 2009 Smith College Alumnae Survey data for those 10,15, and 20 years post‐graduation.
Perspective on Smith Education: Skills
How well do you think your undergraduate experience prepared you?
100%
90%
80%
87.5%
84 9%
84.9%
87.4%
81.9%
73.0%
70%
60%
51 5%
51.5%
50%
40%
30%
20%
10%
0%
Write effectively Communicate well orally Think analytically Formulate original ideas and solutions
Acquire new skills and knowledge "More Than Adequately" or "Very Well"
Source: 2009 Smith College Alumnae Survey data for those 10,15, and 20 years post‐graduation.
Use quantitative tools
Perspective on Smith Education: Skills
How well do you think your undergraduate experience How
well do you think your undergraduate experience
prepared you?
100%
90%
80%
81.7%
77.1%
73.9%
68.1%
70%
72.4%
60%
50%
40%
39.9%
30%
20%
10%
0%
Be an effective Work effectively Be self‐confident
Gain in‐depth Read or speak a Understand leader
as a member of a knowledge of a foreign language social problems field
team
team "More Than Adequately" or "Very Well"
Source: 2009 Smith College Alumnae Survey data for those 10,15, and 20 years post‐graduation.
Perspective on Smith Education: Skills
How well do you think your undergraduate experience How
well do you think your undergraduate experience
prepared you?
100%
90%
80%
70%
82.9%
78.8%
70.4%
60%
50%
42.8%
49.3%
40%
30%
20%
10%
0%
Be an active member of your community
Maintain a healthy Relate well to different Understand role of Understand the lifestyle
races/nations/religions science and technology significance of art, in society
music, literature, & drama
"More
More Than Adequately
Than Adequately" or or "Very
Very Well
Well"
Source: 2009 Smith College Alumnae Survey data for those 10,15, and 20 years post‐graduation.
Impact of Science Programs at Smith
• Alumnae survey
• Science majors j
• Classes 1974‐2007
• 67% response rate (n=786)
• Comparison of graduate degree attainment • SURF participants
• non‐SURF
non SURF • Results based on bivariate analysis
•73%
73% of SURF participants with graduate degree outcome of SURF participants with graduate degree outcome
•57% of non‐SURF participants with graduate degree outcome
Source: Lopatto, David and Carol Trosset. “Report on the Smith College Alumnae Survey.” Clark Science Center Director’s Office, Smith College. (2008)
Banner Office of the Registrar, Smith College. Reasons for Re‐Analyzing Results from Prior Study
•Systematic
Systematic differences between SURF participants and non
differences between SURF participants and non‐SURF
SURF •Control for confounding variables •Total SAT score
•Admissions rating for the student
g
•Race/ethnicity •Graduation year •The outcome variable is dichotomous indicator of completion of a graduate degree
•Fit a multiple logistic regression model, a multiple logistic regression model controlling for propensity score, and a matched analysis using the propensity scores
it
Source: Smith College Science Alumnae Survey. Office of Instituional Research and Educational Assessment. (2008)
Banner. Office of the Registrar, Smith College. SURF Re‐Analysis Results
y
Multiple logistic regression model
• Controlling for total SAT score and the other possible confounders, SURF participants had 1 9 times the odds (95% confidence interval 1 2 3 0) of completing an advanced degree
1.9 times the odds (95% confidence interval 1.2‐3.0) of completing an advanced degree relative to non‐SURF participants
Propensity score model
• Controlling for propensity score when including only total SAT score and the other possible Controlling for propensity score when including only total SAT score and the other possible
confounders, SURF participants had 1.9 times the odds (95% confidence interval 1.2‐2.9) of completing an advance degree relative to non‐SURF participants Banner Data
Banner Data
•Outcomes data fields for
•Contact
Contact information
information
•Spousal relationship
•Most recent degree •Most recent institution •Next most recent degree •Next most recent institution •3rd most recent degree •3rd most recent institution •Information mostly self‐reported, but manually updated internally
•Used primarily by Development Office and Alumnae Association for outreach and fund‐raising
Next Steps
Next Steps
• Annual alumnae surveys (2, 5, 10 years out)
• Development of alumnae outcomes data set
– Early design phase, Institutional Intelligence project
– Purpose: assessment, grant application support, outcomes evaluation
– Ideas
•
•
•
•
Tracking exam scores
Senior Survey data
y
Alumnae Survey data
Self‐reported alumnae outcomes
• What else? What do we want to know? • What assessment questions can we answer with outcomes data?
Wh t
t
ti
ith t
d t ?
• How can Smith better support faculty with outcomes data?