Ning Fang
Don Millard
Division of Undergraduate Education
National Science Foundation
November 10, 2010







Introduction
Overview of the review process
Instructions on producing a panel review
Organization of participants into teams and designation of panel chairs
Local teams discuss strengths and weaknesses
Teams locally report/discuss results
Reporting to virtual group with ratings – note: facilitators will be asked to select an individual to report
BREAK (15min)








Individuals consider ways to improve the proposal
Local teams discuss suggestions for improvement
Teams locally report/discuss results (5min)
Report back to virtual group - note: facilitator picks an individual to report
PD commentary on responses
Think – What have I learned today that I will use in preparing my next proposal?
Share your thoughts with local participants
Facilitator reports results back to virtual group
Wrap-up Q&A
Mock Review Webinar - 2

Help participants to:
 Become more familiar with the proposal review
process
 Better understand the TUES-specific criteria
 Better understand the use of intellectual merit/broader
impact criteria in reviewing proposals
 Develop more competitive proposals that effectively
meet the expectations of the TUES program
Mock Review Webinar - 3
Mock Review Webinar - 4

Title changed to emphasize the special interest in projects that have the
potential to transform undergraduate STEM education

Review criteria was modified to emphasize the desire for projects that:
 Propose materials, processes, or models that have the potential to
 Enhance student learning
 Be adapted easily by other sites
 Involve a significant effort to facilitate adaptation at other sites
 Institutionalize the approach at the investigator's college or university as appropriate
(e.g., for the Type)
 Have the potential to contribute to a paradigm shift in undergraduate STEM
education
Mock Review Webinar - 5

Vision: Excellent STEM education for all undergraduate
students

Reflects national concerns about producing:
 Skilled STEM professionals (including K-12 teachers)
 Citizens knowledgeable about STEM and how it relates to
their lives

Seeks to build a community of faculty committed to improving
undergraduate STEM education

Encourages projects with potential to transform undergraduate
STEM education
Mock Review Webinar - 6

Creating Learning Materials and Strategies
 Guided by research on teaching and learning
 Incorporate and be inspired by advances within the discipline

Implementing New Instructional Strategies
 Contribute to understanding on how existing strategies:
 Can be widely adopted
 Are transferred to diverse settings
 Impact student learning in diverse settings

Developing Faculty Expertise
 Enable faculty to acquire new knowledge and skills in order to revise their
curricula and teaching practices
 Involve a diverse group of faculty
Mock Review Webinar - 7

Assessing and Evaluating Student Achievement:
 Develop and disseminate valid and reliable tests of STEM knowledge
 Collect, synthesize, and interpret information about student understanding,
reasoning, practical skills, interests, attitudes or other valued outcomes

Conducting Research on Undergraduate STEM Education:
 Explore how:
 Effective teaching strategies and curricula enhance learning and attitudes
 Widespread practices have diffused through the community
 Faculty and programs implement changes in their curriculum
Mock Review Webinar - 8

Projects developing instructional materials and methods
should:
 Be based on how students learn
 Consider transferability and dissemination throughout the project's
lifetime
 Involve efforts to facilitate adaptation at other sites in more advanced
projects
Mock Review Webinar - 9


Expect to award approximately 10%
Total budget: up to $200,000 for 2 to 3 years
 $250,000 when 4-year and 2-year schools collaborate





Typically involve a single institution & one program
component – but there are exceptions
Contribute to the understanding of undergraduate STEM
education
Informative evaluation effort based on the project's specific
expected outcomes
Institutionalized at the participating colleges and universities
Deadlines:
 May 26, 2011 (A-M)
 May 27, 2011 (N-W)
Mock Review Webinar - 10

Type 2 Projects
 20 to 25 awards expected
 Total budget: up to $600K for 2 to 4 years

Type 3 Projects
 3 to 5 awards expected
 Budget negotiable, but not to exceed $5M over 5 years

Tues Central Resource Projects
 1 to 3 awards expected
 Budget negotiable, depending on the scope and scale of the activity
 Small focused workshop projects -- 1 to 2 years & up to $100K
 Large scale projects -- 3 to 5 years & $300K to $3M
Deadline:
January 14, 2011
Mock Review Webinar - 11

Reviewers are solicited by program directors
 2010 TUES Type 1 Proposals Review:




4 panels/program director (60-70 proposals/PD)
6-8 reviewers/panel
17 proposals/panel – not all read by every panelist
Approximately 130 engineering reviewers

Reviewers assign individual ratings and develop a
review 2-3 weeks prior to coming to the panel
meeting

Panel meeting is held in DC area – over a 2 day
period
Mock Review Webinar - 13

Panel Chair (picked by program director ahead of
time) establishes order of proposal review process

Proposals are discussed individually

A “scribe” is designated to capture all of the points
brought up in discussion and produce a summary
review – called the “Panel Summary”

Panel returns on day 2 to review all the proposals’
panel summaries
Mock Review Webinar - 14

NSF program directors
 Informs recommendations relative to funding
 Guides pre-award negotiations

Applicants
 If proposal is funded:
 Provides suggestions for improving project
 If proposal is not funded:
 Provides information to guide a revision of the proposal
Mock Review Webinar - 15

All proposals are evaluated using the NSB-approved
review criterion
 Intellectual merit
 Broader impacts

The TUES Solicitation provides two sets of suggested
questions to help define these criteria
 Standard NSF set
 TUES specific set
Mock Review Webinar - 16

Suggested questions are only a guide for considering
intellectual merit and broader impacts

Suggested questions are NOT:
 A complete list of “requirements”
 Applicable to every proposal
 An official checklist
Mock Review Webinar - 17

Will the project:
 Include activities important in advancing knowledge?
 Involve qualified proposer(s)?
 Contain creative and original concepts?
 Have a well conceived and organized plan?
 Include sufficient access to resources?
Mock Review Webinar - 18

Will the project:
 Advance discovery - promote teaching & learning?
 Broaden participation of underrepresented groups?
 Enhance the infrastructure?
 Include broad dissemination?
 Benefit society?
Mock Review Webinar - 19
Will the project:

Produce one or more of the following:
 Exemplary materials, processes, or models that enhance student learning
and can be adopted by other sites




 Important findings related to student learning?
Build on existing knowledge about STEM education?
Have explicit and appropriate expected measurable outcomes integrated into an
evaluation plan?
Include an evaluation effort that is likely to produce useful information?
Institutionalize the approach at the investigator's college or
university (as appropriate for the Type)
Mock Review Webinar - 20
Will the project:
 Involve a significant effort to facilitate adaptation at other
sites?
 Contribute to the understanding of STEM education?
 Help build and diversify the STEM education community?
 Have a broad impact on STEM education in an area of
recognized need or opportunity?
 Have the potential to contribute to a paradigm shift in
undergraduate STEM education?
Mock Review Webinar - 21
Mock Review Webinar - 22

The Entire Proposal is Used to Inform Reviewers





Project Summary
Project Description
Biographical Sketches
Budget
Supplementary Documentation
23
Mock Review Webinar - 23

A review should indicate an opinion on the merit of the project

The rating should indicate an overall evaluation of the
proposal’s merit
 Excellent, Very Good, Good, Fair, and Poor

Text boxes are provided to:
 Describe positive aspects
 Not just list them -- Provide details
 Identify concerns (or weaknesses)
 Not just list them -- Provide details
 Offer suggestions for improvement
Rating and text should be consistent
Mock Review Webinar - 24

Describe the strengths and weaknesses (or concerns) relative
to the review criteria
 Include all of the positive aspects, concerns, and issues

Use a comfortable style and format

Use good grammar, style and complete sentences

A very brief review is not very helpful
Mock Review Webinar - 25

Provide any additional information that you want to transmit

One-sentence summary of the proposal

Summary of the critique

Statement indicating why the proposed project is important or
not

Suggestions for improvements

Other comments
Mock Review Webinar - 26






Uses appropriate style
Contains adequate details
Contains understandable, specific, and complete statements
Relates strengths and weakness to review criteria
Indicates why an item is a strength or weakness
Justifies the proposal rating in the written critique
A reader should be able to guess the rating from the written text
Mock Review Webinar - 27

Identify a strength or weaknesses
 “The evaluation plan is a strength.”

Identify a strength or weaknesses and indicate
why it is one

“The evaluation plan includes a competent, independent evaluator...”
 “The background discussion is well referenced, shows a good
understanding of the prior work, supports the proposed work...”
Mock Review Webinar - 28

Identify a strength or weaknesses and indicate why it is one and
why it is important
 “The evaluation plan includes a competent, independent evaluator, … and it
will guide the investigators as the project evolves and provide a measure of
its effectiveness at the end.”

Identify a strength, indicate why it is one, why it is important, and
how it could be improved
 “The evaluation plan is a strength because it includes a competent,
independent evaluator, … and this will guide the investigators as the project
evolves and provide a measure of its effectiveness at the end. It could be
improved by adding …”
Mock Review Webinar - 29

Be constructive in your written comments
 Provide suggestions to help applicants improve their proposals

Do not be overly critical in your ratings
 Most meritorious proposals (fundable proposals) have some
weaknesses
 Some are correctable through negotiations
Mock Review Webinar - 30





How many strengths did you note?
How many weaknesses did you note?
Are these numbers consistent with your rating?
Do your statements indicate why an item is a strength or
weakness?
Do they indicate why a strength or weakness is
important?
Mock Review Webinar - 31
Mock Review Webinar - 32

Local teams discuss the strengths and weaknesses identified
in the individual reviews (20min)
 Team “scribe” takes notes during discussion

Teams locally report/discuss results with facilitator’s guidance
(10min)
 Facilitators guide discussion and selection of an individual to report to
the full virtual group


Participating organizations will be called upon by webinar
moderator – PLEASE BE READY TO REPORT BACK
Program Directors will offer comments on reports (15min)
Mock Review Webinar - 33


Redundancy is OK – indicative of common perception
Responses will be solicited as follows:
 Intellectual Merit
 Strengths
 Weaknesses
 Broader Impacts
 Strengths
 Weaknesses
 Overall Perception
 Is this a proposal worthy of funding?
Mock Review Webinar - 34
Mock Review Webinar - 35


Redundancy is OK – indicative of common perception
Responses will be solicited as follows:
 Intellectual Merit
 Strengths
 Weaknesses
 Broader Impacts
 Strengths
 Weaknesses
 Overall Perception
 Is this a proposal worthy of funding?
Mock Review Webinar - 36
Mock Review Webinar - 37
Mock Review Webinar - 38
Mock Review Webinar - 39
Mock Review Webinar - 40


Individuals consider ways that the proposal could be improved
– create a list (5min)
Local teams discuss the suggestions for improvement (10min)
 Team “scribe” takes notes during discussion

Teams locally report/discuss results with facilitator’s guidance
(5min)
 Facilitators guide discussion and selection of an individual to report to
the full virtual group


Participating organizations will be called upon by webinar
moderator – PLEASE BE READY TO REPORT BACK
Program Directors will offer comments on reports (15min)
Mock Review Webinar - 41


Try to minimize redundant responses
Identify approaches for:
 Building on strengths
 Overcoming weaknesses

Responses should include potential improvements to:






Idea
Project infrastructure
Project implementation plan
Evaluation plan
Dissemination plan
Proposal participation/involvement
Mock Review Webinar - 42
Mock Review Webinar - 43


Try to minimize redundant responses
Identify approaches for:
 Building on strengths
 Overcoming weaknesses

Responses should include potential improvements to:






Idea
Project infrastructure
Project implementation plan
Evaluation plan
Dissemination plan
Proposal participation/involvement
Mock Review Webinar - 44
Panel Summary and Reviewer Comments




The proposal is the evolution of the successful studio
pedagogy developed at RPI to a mobile studio pedagogy
In this way the proposal builds on a instructional methodology
accepted in the STEM literature.
The primary innovation is the development of a low cost virtual
instrument board that, when coupled with a computer/laptop,
provides students with a portable experimental platform.
The proposal is student focused and grounded in the STEM
knowledge base.
Mock Review Webinar - 46



This experimental platform serves as a low cost replacement
for the laboratory equipment found in the original RPI studio
classrooms and provides the students with an "anywhere,
anytime" experimental platform. The PIs are well qualified to
carry out the work.
In addition, the proposal was responsive to the cyclic model of
knowledge creation contained in the CCLI request for
proposals.
The panel felt the proposal would be strengthened by a more
detailed evaluation plan with clearly stated, measurable
outcomes.
Mock Review Webinar - 47



The broader impacts of the proposed work include the
potential to significantly impact a large number of under served
students by providing a low cost, portable experimental
platform.
In addition, the inclusion of an historically black college or
university and a community college should magnify the impact
of the proposed pedagogy.
This partnership was made concrete through an instructor
exchange program between the institutions.
Mock Review Webinar - 48




While the evaluation team appears to have worked together for
several years, some basic information about the qualifications of
those individuals should have been included in the proposal.
It is not clear that it will positively impact student learning -- just
because a student has the means to perform an experiment 24/7
does not mean that they will.
Proposal could benefit from clarifying or identifying the role of
laboratory experiences where actual industrial-quality laboratory
equipment is utilized, as too much reliance on the proposed
pedagogy might also leave the student with limited experience in
the use of the real industrial grade and scale laboratory equipment.
The proposal could be strengthened with more definition of the role
of each partner.
Mock Review Webinar - 49





Moving from the original studio to the mobile studio essentially
means that the new work is now based on an untested model.
The proposal could be strengthened by addressing this issue.
It has not been demonstrated that the mobile studio is as
effective as the original studio with its significant faculty
involvement and structure.
This proposal would have been much stronger as a Phase I or
if some preliminary data on efficacy were given.
The proposal mentions a test that was done at Howard
University with this work but no details are given.
It fails to explicitly address how it will focus on components of
the cyclic model for innovation in STEM education.
Mock Review Webinar - 50
Mock Review Webinar - 51
Reviewers have:
 Many proposals
 Ten or more from several areas

Limited time for your proposal
 20 minutes for first read

Different experiences in review process
 Veterans to novices

Different levels of knowledge in proposal area
 Experts to outsiders

Discussions of proposals’ merits at panel meeting
 Share expertise and experience
Mock Review Webinar - 52
5
 Write down a list of suggestions (guidelines) that you
would suggest that a colleague should follow - to deal
with practical aspects of the Review Process
 2 minutes
Mock Review Webinar - 53
5

Use good style (clarity, organization, etc.)






Be concise, but complete
Write simply but professionally
Avoid jargon and acronyms
Check grammar and spelling
Use sections, headings, short paragraphs & bullets (Avoid dense,
compact text)
Reinforce your ideas
 Summarize; Highlight (bolding, italics)

Give examples
Mock Review Webinar - 54
5

Use tables, figures – where it makes sense

Reinforce your ideas
 Summarize; Highlight (bolding, italics)

Give examples
Mock Review Webinar - 55
5

Provide appropriate level of detail

Pay special attention to Project Summary
 Summarize goals, rationale, methods, and evaluation and dissemination plans
 Address intellectual merit and broader impacts
 Explicitly and independently
 Three paragraphs with headings:
 “Summary”
 “Intellectual Merit”
 “Broader Impacts”
Mock Review Webinar - 56
5

Follow the solicitation and GPG
 Adhere to page, font size, and margin limitations
 Use allotted space but don’t pad the proposal
 Follow suggested (or implied) organization
 Use appendices sparingly (check solicitation to see if allowed)
 Include letters showing commitments from others
 “Support letters” are not allowed
 Avoid form letters
Mock Review Webinar - 57
5

Prepare credible budget
 Consistent with the scope of project
 Clearly explain and justify each item

Address prior funding when appropriate
 Emphasize results

Sell your ideas but don’t over promote

Proofread the proposal

“Tell a story” and turn a good idea into a competitive proposal
Mock Review Webinar - 58
5
Good idea + need
 Right people + infrastructure
 Assessment of outcomes that measure effect on
student learning (with goals/objectives linked to
evaluation)
 Active dissemination plan
 Efforts to broaden participation of underrepresented
groups

Mock Review Webinar - 59
Mock Review Webinar - 60

Individuals consider:
What have I learned today that I will be able to used in
preparing my next proposal?

Create a list (5min)
Share your list with local participants (5min)
Facilitators report back to virtual group (5min)



Participating organizations will be called upon by webinar
moderators – PLEASE BE READY TO REPORT BACK
Mock Review Webinar - 61
Mock Review Webinar - 62

For an impact on undergraduate education to be
transformative, it needs to provoke a shift in learning
Sky and Water I
Mock Review Webinar - 63
by: M. C. Escher. Woodcut, 1938





Model good practices that increase/improve learning
Facilitate direct experience with the methods and processes of
inquiry/experimentation
Empower the student
Foster a sense of wonder, the excitement of discovery and
“out-of-the-box” thinking
Enhance ability to produce innovative results
 Products
 Solutions to problems
Mock Review Webinar - 64
Good idea + need
 Right people + infrastructure
 Can be readily adopted at other sites
 Assessment of outcomes that measure effect on
student learning (with goals/objectives linked to
evaluation)
 Active dissemination plan
 Shows promise for institutionalization
 Efforts to broaden participation of underrepresented
groups

Mock Review Webinar - 65
Mock Review Webinar - 66
Ning Fang – nfang@nsf.gov
Don Millard – dmillard@nsf.gov