Susan Carter, J.D.
Office of Research Development Services
University of California Merced
May 29, 2014
University of California at Merced
1
Overview of Today’s Workshop

Introductions

How can RDS help?

Quick Overview of Early Career Opportunities

Discussion/planning activity

Focus on NSF CAREER

Discussion/next steps/close
University of California at Merced
2
How we can help











Identification of funding opportunities;
Support for Program Officer contacts;
Connections to educational activities, evaluators,
collaborators;
Assistance drafting Dean/Dept. letters;
Strategies to integrate education/research activities
(NSF CAREER);
Feedback on objectives, approach, activities;
Editing suggestions geared to solicitation;
Post-doc mentoring and data management plans;
Resource section;
Budget/Budget justification;
Sample proposals, help with internal review
University of California at Merced
3
Overview: Early Career Opportunities



Geared to tenure track, not yet tenured
faculty.
Most from private foundations and
professional societies (COS/Pivot search
‘young’ and ‘investigator’ for federal
opportunities: n= 118; for all opportunities
n=2339)
The opportunities are a good way to build a
relationship with an agency; some agencies
encourage ‘first time’ applicants/recipients.
University of California at Merced
4
Your proposal



Should advance you toward your academic
goals.
Should be compatible with your institution’s
goals.
Should represent a contribution to society at
large.
University of California at Merced
5
Early Career Opportunities: A Very Partial List




NIH Career Development (K series) Awards:
◦ http://grants.nih.gov/training/careerdevelopmentawards.ht
m
DOE Early Career Research Program:
◦ http://science.energy.gov/early-career/
◦ 2014 Recipient: Florin Rusu, UC Merced, ‘Scalable and
Energy-efficient Methods for Interactive Exploration of
Scientific Data’ (Office of Advanced Scientific Computing
Research)
Air Force Young Investigator Research Program:
◦ https://www.fbo.gov/index?s=opportunity&mode=form&id
=c9b1754afe23dd6231f6ba85299a5218&tab=core&_cview
=0
DARPA Young Faculty Award:
◦ http://www.darpa.mil/Opportunities/Universities/Young_Fa
culty.aspx
University of California at Merced
6
Early Career Resources


Most agencies have extended webpages; e.g.
NIH K Kiosk linked above, and other internet
resources and publications.
NIH has a YouTube Channel:
◦ https://www.youtube.com/user/nihgrants

Professional societies.

Program Officers.

Colleagues.
University of California at Merced
7
Presentations from NSF hosted workshops:
http://aries.imse.ksu.edu/nsf/nsfcareer2013/resource.htm
http://cs.gmu.edu/events/nsfcisecareer2014/wpcontent/uploads/2014/03/slides.pdf
http://aries.imse.ksu.edu/nsf/NSF2014/Workshopdoc.htm
You
Tubehttp://www.youtube.com/watch?v=taYMgx_U3YY&featur
e=share&list=PLS7JwklCxwEYiXvzKJD3IjKj7a2gF0lfx
AND sample NSF Briefings for review panels:
http://crewman.uta.edu/NSF-CAREERWorkshop2013/Files/CAREER-Panel-Briefing.pdf
http://cs.gmu.edu/events/nsfcisecareer2014/?page_id=15
University of California at Merced
8
NSF CAREER Resources, contd.




Presentation on what NOT to do:
◦ http://www.umass.edu/research/system/files/Rutgers_Pazzani_n
ear-misses.pdf
Sample outline:
◦ http://www.ohio.edu/engineering/grants/upload/CAREERSample-Outline-2.pdf
Information from NSF on its review process:
◦ http://www.nsf.gov/bfa/dias/policy/outreach/grantsconf/meritre
view_march13.pdf
◦ See also:
http://www.nsf.gov/nsb/publications/2012/nsb1228.pdf
 (appendices include information on funding rates, often by
Directorate)
Some successful CAREER proposals in Geosciences:
◦ http://serc.carleton.edu/NAGTWorkshops/earlycareer/research/N
SFgrants.html
University of California at Merced
9
Campus and UC Resources
RDS
http://rds.ucmerced.edu/


UC Educational Evaluation Center:
http://education.ucsb.edu/ucec/faq.html

Institutional Research and Decision Support:
(institutional data repository; can do survey design and
administration): http://ipa.ucmerced.edu/

Assessment at UC Merced (includes links to programs,
people, resources at school and institutional level).
◦ http://assessment.ucmerced.edu/
University of California at Merced
10
Campus resources, contd.

UC Merced Center for Research on Teaching Excellence
(CRTE): http://crte.ucmerced.edu/faculty-services
◦ Program Learning Outcomes Resources:
 Assessment Guide: http://crte.ucmerced.edu/program-learningoutcomes-resources
◦ The MERLOT ELIXR Initiative: offers a digital case story
repository that hosts more than 70 discipline-specific multimedia
stories.
◦ Students Assessing Teaching and Learning (SATAL):
 http://crte.ucmerced.edu/node/64
 Upon faculty request, SATAL collects and analyzes evidence of student
learning for formative assessment purposes

CRTE also offers individual consultations.
University of California at Merced
11
Discussion/Questions this far?
University of California at Merced
12
CAREER Proposal Planning Activity

Your five, ten and twenty year plans: where
will you be, what will you be doing?
University of California at Merced
13
Your CAREER Planning

Where are you today?
◦ List three areas of research expertise and interest
◦ List three areas of education expertise and interest
◦ List three resources that you/ your institution bring
to the table

List three career/life goals: where do you
want to be:
◦ In five years?
◦ In ten years?
◦ In twenty years?

How do you get from here to there?
University of California at Merced
14
Early Career Proposal Planning: Initial Steps (an
iterative process)

Sketch out your five year career plan.
◦ Where do you want to be in five years:
 Your research agenda, goals, projects, collaborators,
publications?
 Your teaching: goals, courses, collaborators, curriculum
development?
 Outreach: goals, activities, collaborators, integration with your
research and teaching?
 Where do you want to be in ten years? Twenty years?
(Remember: Support for a lifetime of achievement).


Where do your early career submissions fit into this
plan?
Discuss your goals and plans with your Dean, Dept.
head, mentors, peers.
University of California at Merced
15
Before you do much writing…

Know your field:
◦
◦
◦
◦
◦
◦
What is the current state-of-the-art?
Who are the top ten researchers?
What they are doing right now?
Where they get their funding?
What they consider to be the key research issues?
Who would likely review your proposal?
 Source, Hazelrigg, NSF
Before you do much writing….Ask yourself these
basic questions (and then answer them in your
project description!):
◦ What is it about (objectives)?
◦ How will you do it (methodology, technical approach,
activities, timeline, integration of research and
educational activities)?
◦ Can you do it (you and your facilities)?
◦ Is it innovative, exciting and worth doing? Will it
contribute to national needs and priorities? (intellectual
merit and broader impact)?
◦ How does it fit with your 5/10/20 year plans and goals?
University of California at Merced
17
Initial Steps

If you haven’t already done so, identify the most appropriate
Division/Program and email and then call the Program Officers
◦ Do this SOON (a month before the deadline would be considered
‘too late’.

Be prepared to answer in 25 words or less:
◦ What are your research objectives for this funding?

Questions to ask the Program Officer:
◦ Do my research objectives fit well with your program?
◦ If my project is not a good fit, are there other programs I should
consider?
◦ What is the maximum size of your Division/Program’s early career
awards?
◦ How are early career proposals submitted to your program
reviewed?
University of California at Merced
18
Questions this far?
University of California at Merced
19
Focus on the NSF CAREER
University of California at Merced
20
Why CAREER?
From the Solicitation:
“CAREER: The Faculty Early Career Development
(CAREER) Program is a Foundation-wide activity that
offers the National Science Foundation's most
prestigious awards in support of junior faculty who
exemplify the role of teacher-scholars through
outstanding research, excellent education and the
integration of education and research within the
context of the mission of their organizations. Such
activities should build a firm foundation for a lifetime
of leadership in integrating education and research.”
University of California at Merced
21
A quick overview….






Minimum award: $400K over five years; Bio or PLR
$500K (includes IDC.)
Funding rates vary by Directorate, but about 20%
overall (per year).
Eligibility: Untenured, in tenure track; can only
apply three times.
No Co-PIs or Senior Persons named or in budget.
Requires letter from Dean/Dept. head to
demonstrate institutional support and verify
eligibility. (At UC Merced this is By-Law 55 unit
Chair other than in ENG).
In any area normally supported by NSF.
University of California at Merced
22
Quick Overview continued.

Mark your Calendars for the Due Dates:
◦ Monday July 21: BIO, CISE, EHR
◦ Tuesday, July 22: ENG
◦ Wednesday, July 23: GEO, MPS, SBE


Division and Directorate Contacts:
http://www.nsf.gov/crssprgm/career/cont
acts.jsp
More including abstracts of recent awards:
◦ https://www.nsf.gov/funding/pgm_summ.jsp?pims
_id=503214
University of California at Merced
23
Quick Overview, contd.



Funds the academic career development of
new faculty; it is not just a research award;
All CAREER proposals must have an
integrated research and education plan at
their core.
“NSF encourages all applicants to think
creatively about how their research will
impact their education goals and,
conversely, how their education activities
will feed back into their research. These
plans should reflect both the proposer's
own disciplinary and educational interests
and goals, as well as the needs and context
of his or her organization.”
An Overview of the CAREER Proposal Components











Cover Page (generated in FastLane).
Project Summary (1 page equivalent)
Description (15 page limit):
PI Biosketch: Standard NSF format, but include both research
and educational activities and accomplishments.
References.
Budget and budget justification.
Facilities and resources.
Current and pending support.
Post-doc mentoring plan (if applicable; 1 page limit).
Data Management plan (2 page limit);
Supplemental Documents:
 Departmental/Dean Letter: (2 pages)
 Letters of Collaboration (1 page each).
University of California at Merced
25
Don’t forget….
From the Solicitation:
“Proposers are encouraged to communicate
with the CAREER contact or cognizant
Program Officer in the Division closest to
their area of research to discuss the
expectations and approaches that are most
appropriate for that area.”

http://www.nsf.gov/crssprgm/career/contacts.
jsp
University of California at Merced
26
Don’t forget The Funder’s Goals!





Stable support for junior PIs.
Career development of outstanding ‘teacherscholars’ within the context of their
institution.
Build a foundation for a lifetime of integrated
contributions to research and education.
Give universities incentives to value
integration of research and education.
Increase participation of traditionally
underrepresented groups in STEM.
◦ Source: Fisher, NSF CISE; wman.uta.edu/NSF-CAREERWorkshop2013/Files/NSF-Presentation-CAREERProgram.pdf
University of California at Merced
27
CAREER Awards at UC Merced

Past UC Merced CAREER Recipients include:
◦
◦
◦
◦
◦
◦
◦
◦
◦
◦
◦
Asmeret Asefaw Berhe, LES (EAR)
Elliott Campbell, Environmental Engineering (CBET)
Miguel Carreira-Perpinan, EECS (IIS)
Alberto Cerpa, EECS (CNS)
Sayantani Ghosh, Physics (DMR)
Linda Hirst, Physics (DMR)
Christopher Kello, Cognitive Science (BCS)
Kevin Mitchell, Physics (PHY)
Shawn Newsam, EECS (IIS) (PECASE Recipient)
Lin Tian, Physics (DMR)
Ming-Hsuan Yang, EECS (IIS)
University of California at Merced
28
Elements of the Project Description
FROM THE SOLICITATION:
“The Project Description should be developed
in consultation with the department head or
equivalent…”
Therefore:
◦
◦
◦
◦
Consult with RDS;
Develop a written outline;
Email and make an apt. to call the Program Contact;
Make an apt and discuss with your Department
head or Dean.
University of California at Merced
29
Elements of the Project Description, contd.



Description of the proposed research, including:
◦ expected significance;
◦ objectives;
◦ Preliminary data;
◦ methods and approach.
Description of the proposed educational activities;
◦ including evaluation and assessment.
Description of how the research and educational activities are
integrated;

Must address intellectual merit and broader impacts;

Results of prior NSF support, if applicable.
University of California at Merced
30
Elements of departmental letter



Statement that PI is eligible for the CAREER;
Indication that proposed research and
education activities are supported by and
integrated into educational and research
goals of the department and UC Merced;
Indication that department/school is
committed to the support and professional
development of the PI.
◦ See handout
University of California at Merced
31
Elements of departmental letter contd.


A description of the relationship between the
CAREER project, the PI's career goals and job
responsibilities, and the goals of his/her
department/UC Merced;
A description of ways in which the department
head (or equivalent) will ensure mentoring and
professional development of the PI;
◦ From the RFA: “A letter that fails to acknowledge
institutional commitment to the professional
development and mentoring of the PI in both
research and education may disadvantage an
otherwise outstanding proposal.”
University of California at Merced
32
Questions this far?
University of California at Merced
33
University of California at Merced
34
Background and a bit of history

Basic two criteria required by NSB since 1997

Requirement that both be explicitly stated in Project
Summary and Description since 2002; if not to be
returned without review (see
http://www.nsf.gov/pubs/2002/iin127/imptnot.pdf)

Emphasis on transformative work in both intellectual
merit and broader impacts since 2007;

Criteria revised effective January 2013; even more
explicit about the need to address both intellectual merit
and broader impacts.
University of California at Merced
35
Background, NSB Merit Review TF Report,
contd.

NSF wanted to retain researchers’ freedom and
creativity to posit their own broader impacts, rather than
relying on a predetermined list of ‘National Needs’,’
which was considered but abandoned in the revised,
final report.

Report included a call for developing better metrics for
broader impacts activities and noted that broader
impacts may be better measured at levels beyond the
individual project — such as at the institutional level.
◦ Source: Holbrook, J.B.; “Re-assessing the science – society relation: The case of
the US National Science Foundation’s broader impacts merit review criterion
(1997 – 2011)”; Center for the Study of Interdisciplinarity;
http://www.peerev.org/data/7cce53cf90577442771720a370c3c723/PE_doc_28403
.pdf
University of California at Merced
36
Background, contd.

Revisions based on recommendations from the NSB
appointed Task Force on Merit Review:
http://www.nsf.gov/nsb/publications/2011/meritreviewcriteria.pdf

TF Report conclusion: The Intellectual Merit and Broader
Impacts review criteria together capture the important
elements that should guide evaluation of NSF proposals.

TF Report recommended revisions to the descriptions of
the Broader Impacts criterion and how it is implemented.

Use of the two basic review criteria should be informed
by a guiding set of three core principles and five review
elements.
University of California at Merced
37
Three Guiding Core Principles from the Merit Review TF
Report

All NSF projects should be of the highest quality and
have the potential to advance, if not transform, the
frontiers of knowledge.

NSF projects, in the aggregate, should contribute more
broadly to achieving societal goals.

Meaningful assessment and evaluation of NSF funded
projects should be based on appropriate metrics,
keeping in mind the likely correlation between the effect
of broader impacts and the resources provided to
implement projects.
University of California at Merced
38
Reminder, Two Basic NSF Review Criteria
•
•
•
•
Applies to all NSF Proposals, including CAREER:
Intellectual Merit: potential to advance knowledge
and understanding;
Broader Impacts: potential to benefit society and
contribute to the achievement of specific, desired
societal outcomes.
NSF will sometimes use additional criteria as
required to highlight the specific objectives of
certain programs and activities.
University of California at Merced
39
Five Elements of the Two Criteria: Some Questions
the reviewers will ask:



Do the proposed activities suggest and explore
creative, original, or potentially transformative
concepts?
Is the plan for carrying out the activities wellreasoned, well-organized, and based on a sound
rationale?
How well qualified is the individual, team, or
organization to conduct the proposed activities?
University of California at Merced
40
Five elements, contd.


Are there adequate resources available to
the PI (either at the home organization or
through collaborations) to carry out the
proposed activities?
Does the plan incorporate a mechanism to
assess success?
University of California at Merced
41
A bit more history: Why does NSF care about Broader
Impacts?


Because Congress cares!
It is a question of policy AND politics.
◦ See e.g.:
◦ Holbrook, 2012 (citation above);
◦ Schienke, E. W., Tuana, N., Brown, D. A., Davis, K. J., Keller, K.,
Shortle, J. S., ... & Baum, S. D. (2009). The Role of the National
Science Foundation Broader Impacts Criterion in Enhancing
Research Ethics Pedagogy. Social Epistemology, 23(3-4), 317336.
◦ Rothenberg, M. (2010). Making Judgments about Grant
Proposals: A Brief History of the Merit Review Criteria at the
National Science Foundation. Technology & Innovation, 12(3),
189-195.
University of California at Merced
42
So, where do I start when
drafting the Broader Impacts
portion of my proposal?
University of California at Merced
43
Start by reading the Solicitation…


“Both criteria are to be given full consideration
during the review and decision-making processes;
each criterion is necessary but neither, by itself, is
sufficient.”
“Reviewers will be asked to consider what the
proposers want to do, why they want to do it, how
they plan to do it, how they will know if they
succeed, and what benefits could accrue if the
project is successful. These issues apply both to
the technical aspects of the proposal and the way
in which the project may make broader
contributions.”
 NSF 14 - 532, emphasis added
University of California at Merced
44
An analogy?

“I shall not today attempt further to define
the kinds of material I understand to be
embraced within that shorthand description;
and perhaps I could never succeed in
intelligibly doing so. But I know it when I see
it.” (emphasis added)
◦ Justice Potter Stewart,concurring opinion in
Jacobellis v. Ohio 378 U.S. 184 (1964)
University of California at Merced
45
More from the Solicitation:

“Broader impacts may be accomplished through
the research itself, through the activities that are
directly related to specific research projects, or
through activities that are supported by, but are
complementary to, the project.”

However, a note of caution: Don’t confuse
educational activities and/or outreach with
‘broader impacts.’

You must be specific about how the project
activities support and encompass both IM and BI!
University of California at Merced
46
Then look at the Grants Proposal Guide
(GPG):

GPG Chapter III.C outlines the changes to the
merit review process effective January 14, 2013.
◦ ALL applicants should read this, several times:
http://www.nsf.gov/pubs/policydocs/pappguide/nsf1300
1/gpg_3.jsp
◦ Refers to the three merit review principles, two criteria,
five elements.

GPG Chapter II.C.2.d.i. contains information for
use by proposers in development of the Project
Description section of the proposal.

“Reviewers are strongly encouraged to review
the criteria, including GPG Chapter II.C.2.d.i.,
prior to the review of a proposal.” (NSF 11-690)
From the GPG:

NSF values the advancement of scientific knowledge and
activities that contribute to achievement of societally
relevant outcomes, including, but not limited to:
 full participation of women, persons with disabilities, and
underrepresented minorities in STEM;
 improved STEM education and educator development at any level;
 increased public scientific literacy and public engagement with
science and technology;
 improved well-being of individuals in society;
 development of a diverse, globally competitive STEM workforce;
 increased partnerships between academia, industry, and others;
 improved national security;
 increased economic competitiveness of the United States;
 enhanced infrastructure for research and education.
University of California at Merced
48
A Few Resources and References on Broader
Impacts


NSF Presentation on changes to Merit Review:
http://www.nsf.gov/bfa/dias/policy/merit_re
view/overview.pdf
“Revised Merit Review Resources for the
External Community” (1/14/2013):
◦ http://www.nsf.gov/bfa/dias/policy/merit_review/r
esources.jsp
◦ Includes FAQs:
 http://www.nsf.gov/bfa/dias/policy/merit_review/mrf
aqs.jsp
University of California at Merced
49
Resources, contd.


Frodeman, R., Holbrook, J. B., Bourexis, P. S.,
Cook, S. B., Diederick, L., & Tankersley, R. A.
(2013). Broader Impacts 2.0: Seeing—and
Seizing—the Opportunity. BioScience, 63(3),
153-154.
Center for the Study of Interdisciplinarity,
University of North Texas:
◦ http://www.csid.unt.edu/topics/NSF-broaderimpacts-criterion.html
University of California at Merced
50
Questions this far?
University of California at Merced
51
The Educational Component:
What works, what doesn’t
University of California at Merced
52
Developing the Educational Component; the basics.

Activities “should be consistent with research and
best practices in curriculum, pedagogy, and
evaluation” (from the Solicitation).
◦ Read (and cite, where appropriate!) the resources listed
in the Solicitation!
◦ Cite educational research and research on human
behavior; remember that you are expected to include
both research and educational citations in your proposal.

Proposed activities can be in a broad range of
areas and can by directed at any level;
◦ K-12; u/g; grad students and/or the general public.
◦ See examples in Solicitation, but don’t limit your activities
to those!
University of California at Merced
53
Developing the Educational Component, contd.

Education and research activities must be integrated.
◦ Ask yourself and answer: how are they related and how do they
support each other?
◦ Ask yourself: what are creative ways to reach out to underserved
groups and to disseminate research findings?

Activities should be developed and described within the
context of your institution.
◦ Hint: build on UC Merced’s strengths and existing resources
◦ UC Merced’s HSI status in this context.

Funds should be included in budget to support outreach
and education activities; (can include funds for
evaluation.)
University of California at Merced
54
Developing the Educational Component, contd.

Education and research activities must be integrated.
◦ Ask yourself and answer: how are they related and how do they
support each other?
◦ Ask yourself: what are creative ways to reach out to underserved
groups and to disseminate research findings?

Activities should be developed and described within the
context of your institution.
◦ Hint: build on UC Merced’s strengths and existing resources
◦ UC Merced’s HSI status in this context.

Funds should be included in budget to support outreach
and education activities; (can include funds for
evaluation.)
University of California at Merced
55
Examples of integrated educational activities



Research, field work and lab involvement (UG, G, K12, adult
learners, the public);
Implement existing instructional innovations in your own
courses; work with future teachers (pre-service) – research
experiences, course content;
Work with in-service teachers – workshops, curriculum units,
research experiences;

Work with high school/middle school students;

Provide content expertise to existing education projects;

Workshops or tutorials
University of California at Merced
56
More examples:

Curriculum development (UG, G, K12, 6-12
teachers, inter- or cross-disciplinary course
development, online course development);

Assessment of curriculum;

Mentoring (UG, G, 9-12);

Linking activities to industrial, international,
or cross-disciplinary work.
University of California at Merced
57
Educational Activities: what works





Describe how the activities will improve your teaching
(develop your own pedagogical content knowledge as it
relates to your field);
Describe how new learning materials or strategies are guided
by research on teaching and learning, as well as advances
within your discipline;
Describe how the specific activities support your educational
goals;
Show you can do it! (preliminary results count here too!);
Highlight what you’ve already done: include the obvious but
extend your reach to include innovative change and outreach.
◦ Important, however, that the activities be doable.
University of California at Merced
58
Educational Activities: what works, contd.



Partner with communities traditionally underrepresented in
STEM
Think systemically
◦ i.e. how to involve teachers and students if looking at K-12
activities.
◦ How will your activities have the broadest impact?
◦ Who is the intended audience?
◦ Nice to refer to alignment of K-12 activities with State
curriculum standards
 See http://www.cde.ca.gov/be/st/ss/
 Consult with the teachers and administrators you will be
involving in your work.
 But don’t forget to think outside the box in delivery!
Assess: How will you know that your work was successful
(how will you know that you achieved the goals)?
University of California at Merced
59
Educational Component: what doesn’t work

Don’t over-commit!

Don’t only outline activities that are part of your current
position (such as current courses taught).

Has it been done a million (or even a hundred) times?
◦ If you can catch the reviewers’ attention they are more
likely to believe that you can catch the intended
audience’s attention.

Don’t try to do it alone: you probably aren’t an expert, but
even if you are, your reach will be limited if you don’t
involve others.
University of California at Merced
60
Educational/outreach resources at UC
Merced
◦ RDS
◦ Undergraduate Research Opportunities Center
(UROC): http://uroc.ucmerced.edu/
◦ School based instructional assessment
◦ Institutional Planning and Analysis:
 http://ipa.ucmerced.edu/
◦ UC Educational Evaluation Center
 http://education.ucsb.edu/ucec/
 At UCSB, UCM is a member
◦ Center for Educational Partnerships
 http://cep.ucmerced.edu/
◦ Center for Research on Teaching Excellence:
 http://crte.ucmerced.edu/
University of California at Merced
61
Educational/outreach resources at UC Merced
◦ RDS (to make connections)
◦ Undergraduate Research Opportunities Center (UROC):
http://uroc.ucmerced.edu/
◦ STEM Resource Center:
 http://stemresourcecenter.campuscms.ucmerced.edu/
◦ School based instructional assessment
◦ IRDS:
 http://ipa.ucmerced.edu/
◦ UC Educational Evaluation Center
 http://ucec.gseis.ucla.edu/
 At UCSB, UCM is a member
◦ Center for Educational Partnerships(K-12 outreach)
 http://cep.ucmerced.edu/
◦ Center for Research on Teaching Excellence:
 http://crte.ucmerced.edu/
University of California at Merced
62
Questions this far?
University of California at Merced
63
University of California at Merced
64
Start here:

The 2002 User Friendly Handbook for Project Evaluation
(NSF 02-057)

http://www.nsf.gov/pubs/2002/nsf02057/start.htm

Includes a glossary and references to the scientific
literature in education and in evaluation.
University of California at Merced
65
A few more resources:

Online Evaluation Resource Library: http://oerl.sri.com/

Practical Assessment, Research and Evaluation (an online peer reviewed
journal): http://pareonline.net/

National Institute for Science Education: http://archive.wceruw.org/nise/

National Research Council. (2001). Knowing what students know: The
science and design of educational assessment. Committee on the
Foundations of Assessment. Pellegrino, J., Chudowsky, N., and Glaser, R.,
Editors

UC Merced Office of Institutional Assessment:
◦ http://assessment.ucmerced.edu/assessment-resources
University of California at Merced
66
A few tips:

Be specific about how you will know whether your research and
education plans were successful.
◦ For example, if you state that your goals include improved skill
development and retention in STEM, reviewers will want to know how
you will show that your project succeeded. Will you track students?

Your assessment plan should restate your project’s major objectives
and ideally should describe the metric for each objective.

Limit the objectives to be measured to the most important for the
overall success of your project.

Try to use multiple measures for the same objective so you can
assess different aspects to “triangulate” your data.
University of California at Merced
67
Why should you evaluate your educational activities?
Encouraged by the CAREER Solicitation! While your research goals
can be measured in traditional ways (findings, publications, new
directions), NSF expects you to develop a plan for assessment of your
educational goals. Note: you may want to include funding for this,
even if you don’t include an outside evaluator.
Provides information to help improve the project; you can re-assess
how the project is going and whether goals are being met as you
develop your activities.
Often provides new insights or new insights. “Unanticipated
consequences” of a program may be the most useful outcomes of the
assessment enterprise.
Source: NSF 02-057; User Friendly Handbook.
University of California at Merced
68
Ask Yourself:

What are the goals of your educational activities and
component?

Why are they important?

How will you know that your work was successful
and you achieved your goals?
University of California at Merced
69
From the NSF CAREER Solicitation….

“A CAREER proposal must indicate the goals and
objectives of the proposed education activities, how it
will be integrated with the research component, and the
criteria for assessing how these goals will be met.
Principal investigators are strongly encouraged to
describe how the impact of the educational activities will
be assessed or evaluated. A helpful document for
information on evaluating educational activities is the
NSF publication The 2002 User-Friendly Handbook for
Project Evaluation (NSF 02-057).”
University of California at Merced
70
From the NSF CAREER FAQs…

Q: Do I need to involve an outside evaluator for the
education component?

A: You do not need to utilize an outside evaluator.
However, you should have some sort of evaluation plan
to provide feedback that could lead to future
improvements. See if you can find someone within your
organization to help you if you have never done an
evaluation. Most academic institutions will either have an
education department where you might find
collaborators, or have some sort of an evaluation system
in place for coursework that could be modified to suit
your needs.
University of California at Merced
71
Discussion…
University of California at Merced
72