National Science Foundation
Division of Civil, Mechanical and Manufacturing Innovation
Cooperative Agreement CMMI-0927178
NEES Operations FY 2010 – FY 2014
Purdue University: Dr. Julio Ramirez, PI
Annual Site Visit Review (Year Four/Year Five)
November 7-8, 2013
Site Visit Team
Mr. William Allcock
Director of Operations
Argonne Leadership Computing Facility
Argonne National Laboratory
Argonne, IL
Dr. Janos Gergely
Associate Professor
Department of Civil and Environmental
Engineering
University of North Carolina at Charlotte
Charlotte, NC
Dr. Vijaya Gopu
Professor and Edward G. Schlieder
Endowed Chair in Civil Engineering
Louisiana Transportation Research Center
University of New Orleans
New Orleans, LA
Dr. Gerald "Stinger" Guala
Branch Chief, Eco-Science Synthesis
Director of Biodiversity Information Serving Our
Nation (BISON)
Director of the Integrated Taxonomic Information
System (ITIS)
Core Science Analytics and Synthesis
Core Science Systems
U.S. Geological Survey
Reston, VA
Ms. Ardoth Hassler
Associate Vice President
University Information Services
Georgetown University
Washington, DC
Dr. Miriam Heller
Director of Council Affairs
American Society of Engineering Education
Washington, DC
Dr. Marty Humphrey
Associate Professor
Department of Computer Science
University of Virginia
Charlottesville, VA
Dr. Joann Jacullo-Noto (virtual)
Director, M.A.T. Program
Drew University
Madison, NJ
Dr. Kaye Shedlock
Consultant
Golden, CO
National Science Foundation Staff
Dr. Joy Pauschke (Site Visit Coordinator)
NEES Program Director
Division of Civil, Mechanical and Manufacturing
Innovation
Matthew Carnavos (November 7-8)
Program Analyst
Division of Civil, Mechanical and Manufacturing
Innovation
Dr. Richard Fragaszy (November 7-8)
Program Director, Geotechnical Engineering
Division of Civil, Mechanical and Manufacturing
Innovation
Dr. George Hazelrigg (November 7)
Deputy Division Director
Division of Civil, Mechanical and Manufacturing
Innovation
Dr. Scott Horner (November 7)
Deputy Director for Large Facilities
Office of Budget, Finance and Award Administration
Dr. Kishor Mehta (November 7)
Program Director, Hazard Mitigation and Structural
Engineering
Division of Civil, Mechanical and Manufacturing
Innovation
Dr. Carol Wilkinson (November 7)
Program Director, Large Facilities Office
Office of Budget, Finance and Award Administration
2
A. Summary of Site Visit
The combined year four/year five annual review for NEES Operations was held on November 7-8, 2013.
The Purdue/NEEScomm team participated via WebEx and the site visit team (SVT) participated as a group
from the National Science Foundation (NSF), with NSF staff. One SVT member participated as a virtual
SVT member. The first day consisted of presentations by NEEScomm. At the end of the first day, the SVT
gave NEEScomm a list of questions to address overnight. NEEScomm presented the response to the
questions at the start of the second day. The SVT then met in closed session to write the site visit report.
At the end of the second day, the NSF and the SVT briefed NEEScomm on the recommendations and
SWOT analysis.
B. Recommendations and Areas for Improvement
1. NEEScomm must provide vision and leadership, above and beyond award management, during this
final year of the award. To ensure the future of NEES, NEEScomm needs to determine the “right
things to do” and not just “do things right.”
2. NEEScomm must start planning immediately for an orderly closeout of this award. A timeline, with
required resources, should be developed for the various closeout steps.
3. The SVT recommends to NSF that the NEEScomm final spending plan, with all closeout activities fully
enumerated and accounted for, be fully reviewed by the SVT or another appropriate panel of peers.
4. As the stewards of critical national assets and this being one of NSF’s largest projects and long-term
investments, NEEScomm must provide a plan that ensures the option to continue support and
availability of the cyberinfrastructure and the equipment sites. NEEScomm is optimally positioned to
offer a visionary plan for its continued involvement to support and make available the NEEShub.
5. The NEEScomm Governance Board must assume a stronger role in the oversight of the closure of the
award.
6. NEEScomm needs to deepen the assessment of the education, outreach, and training (EOT) activities
in order to identify their impact.
7. It is important to identify the "lessons learned" from this project, specifically addressing the
successes and challenges associated with providing centralized support and management to
distributed, large-scale equipment sites and the effectiveness and efficiency of the
cyberinfrastructure.
C. SWOT: Strengths, Weaknesses, Opportunities, and Threats
Strengths
•
•
•
•
The NEES network continues to provide access to unique, world-class research facilities that are
highly available and well run, enabling national and international research and collaboration.
The progress NEES has made in advancing hybrid simulation (HS) is laudable.
NEEScomm is commended for the significant progress they have made in the curation of data and
information. The growth in the number of curated projects is impressive.
NEES continues to expand the use and availability of high performance computing (HPC) by its
researchers and collaborators. This is excellent leveraging of the NSF investment in
cyberinfrastructure.
3
•
•
•
•
•
The integration of undergraduates into the research, both REU and local students, continues to
improve. The SVT specifically recognizes NEES’ efforts in its service to underserved minorities.
NEEScomm responded to the past EOT recommendations and is demonstrating some quantifiable
outcomes.
The Balanced Scorecard has been significantly improved to provide useful management information
and meaningful indications of activities to the NEES Governance Board, NSF, and the SVT.
NEEScomm has successfully increased the publicity about NEES, earthquake engineering research,
and research facilities.
NEEScomm continues to perform system-wide risk assessments, and looks at its cyberinfrastructure
as a “whole” and not just the individual “parts.”
Weaknesses
•
•
•
•
•
NEEScomm is managing the award, rather than providing community leadership and vision.
NEEScomm has inadequately planned for the future. It should have already outlined three parallel
planning scenarios for NEES, especially addressing intellectual property concerns:
 NEEScomm continuing at Purdue.
 NEES operations transferring to another organization.
 NEES program discontinuance.
The involvement and participation of the Governance Board is not readily evident. They must
become more involved.
The timeline for archiving data after the last completed experiment is aggressive. Moving materials to
“cold storage” is a non-trivial operation.
The SVT notes that NEEScomm responded to the request to shorten the Annual Report and provide
links. However, the report is still hard to navigate and ascertain impacts. The topic sentences should
contain the most important point of the paragraph. The narrative should include a “summary
sentence” of outcomes in addition to providing links to the products. Outcomes/successes should be
obvious without the reader having to analyze each table fully. An “index” to important attachments,
such as the Balanced Scorecard, Cybersecurity Plan, etc., is recommended.
Opportunities
•
•
•
•
NEEScomm needs to be planning for award closeout now and stop waiting for direction from NSF.
NEEScomm has the opportunity to develop a unique plan for continuing and enhancing partnerships,
preservation of research data and results, and the reuse of the repository.
NEEScomm has the opportunity to use its fifth year to lead the development of a vision for NEES for
years 11-15.
NEEScomm should continue to increase racial and ethnic diversity and participation of women across
all aspects of the network.
Threats
•
•
Incomplete transition plans are a significant threat. Not having plans to ensure that the data, results,
training, and outreach programs will be continued into the future is a threat to the broad community
and researchers.
Failure to retain necessary staff through project closure is a serious threat.
4
•
Waiting for direction from NSF is a serious threat. What happens if NEEScomm does not get a
response from NSF prior to January 15, 2014, and there is another lapse in Federal appropriations?
Lost time cannot be recovered.
D. NSF Merit Review Criterion: Intellectual Merit
The NSF Intellectual Merit criterion encompasses the potential to advance knowledge:
a. What is the potential for the proposed activity to advance knowledge and understanding within its own
field or across different fields?
b. To what extent do the proposed activities suggest and explore creative, original, or potentially
transformative concepts?
c. Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound
rationale? Does the plan incorporate a mechanism to assess success?
d. How well qualified is the individual, team, or organization to conduct the proposed activities?
e. Are there adequate resources available to the PI (either at the home organization or through
collaborations) to carry out the proposed activities?







NEEScomm has been very successful in significantly increasing the number of projects curated in year
four.
NEEScomm's efforts to focus its EOT activities and be responsive to SVT recommendations made
during an earlier site visit are very encouraging.
The continued enhancement of the NEEShub in providing more tools, educational material, etc., is an
important contribution to the professional community.
The scheduling of experiments at the various equipment sites has been streamlined and the system is
working effectively. Plans to ensure completion of all experiments before September 30, 2014, are
sound.
The various activities undertaken to promote the use of HS have helped enhance the research
capabilities of the nation's earthquake engineering research community. This is a major contribution
of NEEScomm.
NEEScomm has facilitated easy access to HPC resources for earthquake engineering researchers.
NEEScomm has established numerous international collaborations over the past four years.
E. NSF Merit Review Criterion: Broader Impacts
The Broader Impacts criterion encompasses the potential to benefit society and contribute to the achievement of
specific, desired societal outcomes:
a. What is the potential for the proposed activity to benefit society or advance desired societal outcomes?
b. To what extent do the proposed activities suggest and explore creative, original, or potentially
transformative concepts?
c. Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound
rationale? Does the plan incorporate a mechanism to assess success?
d. How well qualified is the individual, team, or organization to conduct the proposed activities?
e. Are there adequate resources available to the PI (either at the home organization or through
collaborations) to carry out the proposed activities?
The potential broader impacts of the network's activities are clearly articulated in the vision of NEES: "A
global infrastructure network that improves the resilience of new and existing construction, and supports
the education of the next generation of engineers and scientists." Close to 200 NEES research (NEESR)
projects have been performed at the equipment sites. The EOT program has been conducted reaching a
5
wide range of audiences, from K-12 students to science teachers, from undergraduate to graduate
students, and from design professionals to the general public.
NEEScomm did not articulate how its partnerships and the related activities advance community
understanding of the seismic vulnerability of our built infrastructure. The outreach activities, including
museum exhibits, workshops, webinars, TV documentaries, and especially those engaging
underrepresented and minority groups at all level, are notable and indicate the potential for broad
outreach.
The Annual Report did not include enough examples describing the international collaborations between
NEES and their European, Canadian, Japanese and Chinese counterparts. The multi-story wood frame
experiment performed at the E-Defense site was highlighted; however, more examples of additional (and
more specific) research partnerships would enable better assessment of the direct impact of the
international effort.
NEESR findings are published in conventional channels, in data papers (in collaboration with EERI’s
Earthquake Spectra), and through various media in order to reach a broader audience. Furthermore,
NEEShub improved the existing database and user interface, although the actual data usage has not been
tracked.
A primary goal of NEES has been to improve the resilience of the built infrastructure. NEEScomm listed
the proposed changes and additions within the ASCE and ACI, as well as AISC, building code requirements
as evidence of its influence on the state-of-the-practice. Had there been more leadership from
NEEScomm in encouraging the development of safer and more robust code provisions, the broader
impact of the network's efforts could have been greater, especially given the scope and funding level for
this award.
F. How well has NEEScomm and the facilities responded to the issues raised in the August 2012 annual
site visit report and the facility site visit reports (August 2012 – February 2013)?
NEEScomm provided documentation describing steps taken in response to eight areas for improvement
identified by the 2012 annual review SVT. The area in need of greatest attention was EOT. The 2012 SVT
recommended that NEEScomm refocus and downsize the existing EOT effort, since the progress to date
seemed insufficient given the number of staff positions. NEEScomm did downsize the EOT staff,
redirecting personnel to the IT effort (as suggested by the SVT). NEEScomm stressed that all EOT
responsibilities specified in the Cooperative Agreement would be met, the NEESacademy would be
maintained and refined, but no further advanced features would be developed. Improvements in public
outreach via media were developed, including cooperation with NSF and AAAS to include NEESR
successes via LiveScience.com, EurekAlert, the Discovery Channel’s Daily Planet, PBS’ NOVA, and Purdue
Marketing and Media. NEEScomm is also working on several fronts to increase awareness via museums
and appropriate print and electronic media, including the creation of the "newsroom" on nees.org.
The data curation and preservation pipeline was cited as a threat last year, and the SVT provided specific,
detailed recommendations. NEEScomm responded with specific plans addressing all of the
recommendations, including new curation guides and data, metadata, and documentation upload
mandates. NEEScomm increased network staff to provide more upload capabilities, and significant
6
progress is evident. Additionally, NEEScomm and some sites have been working together to build
automatic data capture software that should be a great help. NEEScomm has also been strengthening the
preservation infrastructure, as recommended.
NEEScomm also:
 Established a required acknowledgement for all papers that use NEES data and provides usage
information via nees.org,
 Continued and expanded outreach to underrepresented students to increase diversity,
 Strengthened safety awareness throughout the network, and
 Simplified quarterly and annual reporting, utilizing the NEEShub to provide links to supporting
documentation.
NSF made four facility site visits since the last annual review. Perhaps fortuitously, all four are heavily
involved in HS. A great outcome of these site visits is the increased attention directed to HS across the
NEES community, ongoing development of a NEEShub-based “guide to hybrid simulation,” and associated
activities. Workshops have been held, task groups are now active, NEEShub hosts a HS Wiki, and a
Simulation Steering Committee guides overall collaboration. Advances in HS are likely to be among the
most important legacies of NEES.
Each facility site visit report involved mostly site-specific recommendations that NEEScomm appears to
have worked diligently with each site to address. NEEScomm has provided supplemental funding for
safety improvements, increased productivity, and other recommendations as needed. Follow-through by
the sites and NEEScomm appears to be thorough and appropriate, since no major issues or weaknesses
have caused disruption to the sites or network to date.
Lack of leadership on the part of NEEScomm appears repeatedly in various forms in each site review, best
illustrated by these verbatim sentences from the EOT section of one of the facility site visit response
documents:
SVT statement: There is a lack of leadership on the part of NEEScomm in EOT across the network.
NEEScomm response: NEEScomm has provided, and will continue to provide, ongoing guidance and
leadership to the equipment sites through the practices it has cultivated.
A famous quote from very successful management consultant, professor, and author Peter Drucker is:
"management is doing things right; leadership is doing the right things." He, and many others, describes
great leaders as possessing impressive social intelligence, no fear of change, and most importantly,
vision.
NEEScomm has provided management and guidance through the practices it has cultivated, moving
earthquake engineering towards collaborative research. However, NEEScomm has done so reactively,
rather than proactively. NEEScomm has “done things right” when the community determined what they
would like to have done. But NEEScomm has rarely demonstrated the leadership to push boundaries. For
example, NEEScomm's increased attention on HS is a reactive result of multiple instances of feedback
over time from the sites and various SVTs rather than NEEScomm recognizing early on that HS is an
important new direction and encouraging/facilitating advances proactively. A current example is the lack
7
of closeout scenarios. NEES Operations is a large NSF award, and waiting for a decision is reactive;
examining all likely scenarios and mitigating the effects of each for the future of NEES is proactive.
G. How well has NEEScomm met its performance metrics and how have the metrics been used for
project assessment and improvement?
NEEScomm performance metrics development and use have followed a trajectory that might be
expected for a large, multi-faceted program, i.e., performance metrics were developed, utilized, and
modified as their effectiveness became evident. NEEScomm met most of its performance objectives and
where they did not, made changes necessary to meet their objectives by the next review. In the first few
years of this award, NEEScomm attempted to include all input from the sites and NSF, and all levels of
granularity. The Balanced Scorecard was modified and grew rapidly, until it was too large and, in some
respects, of questionable value (lots of counting, very little impact assessment). The 2012 SVT
recommended that NEEScomm “Evaluate all performance metrics with the purpose of streamlining the
operational metrics and include other metrics that provide real information on the scientific and broader
impacts of NEES.” NEEScomm did a very good job of simplifying the Balanced Scorecard and formulating
some impact assessment metrics. Some of the metrics and the rationale for some of the targets still need
to be fully defined in the supporting documentation. Although NEEScomm did not meet all of its
performance metrics for year four, the simplified performance metrics themselves and NEEScomm's
management performance overall are improvements.
H. Award Closure (overall management – preliminary plans)
Quality and appropriateness of:
a. Priorities
b. Plans and timeline
c. Personnel
d. Budget
NEEScomm appropriately states priorities for award closure are ensuring that all experiments are
completed by September 30, 2014, and all data are uploaded, curated, and archived. However,
NEEScomm does not address the transfer or decommissioning of the cyberinfrastructure, and long-term
preservation of the NEES assets (the data, and also the EOT material, tools, etc.). This is a serious
concern.
With the exception of the UC Berkeley equipment site, there is no plan or information about the future in
the Annual Report (Volume 2) for the sites in the absence of NEES funding. The SVT accepts that it could
be argued that this is outside of the scope of NEES. However, as the stewards of critical national assets
and this being one of the largest NSF projects and long-term investments, a visionary plan for ongoing
support and availability of these facilities to the community would have been welcomed.
NEEScomm must begin contingency planning immediately even though the path forward is uncertain. If
plans from NSF are too long in coming, NEEScomm will be forced into action because, and, based on its
statements, it will not have funding to continue supporting the cyberinfrastructure. There is substantial
risk of catastrophic loss of vital community assets if NEEScomm does not develop a plan.
The SVT strongly recommends that NEEScomm begin developing contingencies that include at least the
following scenarios:
8
•
•
•
Preparing for handover to another awardee, if one has been identified prior to September 30, 2014.
Requesting funding required to maintain cyberinfrastructure to bridge a gap between this award and
the next. This may include: no development, no ticket response, and basic cybersecurity resources
(e.g., scans, IDS, vulnerability patches, etc.) only. The SVT is fully aware that this is non-trivial, but is
also aware that the costs, considering that Purdue is running the same infrastructure for other hubs
(Oracle instances, etc.), are incremental and should be reasonable.
Developing a “worst case scenario” to ensure the long-term survival of the assets. For example, for
software developed outside of NEEScomm, either identify a home that already exists (for instance
opensees.berkeley.edu) or create one, for instance on Github. For EOT assets, find someone (EERI,
etc.) that can host them. For the Project Warehouse, databases, data papers, etc., determine how
the data and required metadata and documentation can be stored in a way that makes them
accessible, and has a very high probability of maintaining its integrity.
The SVT found no explicit description of the personnel needed to be involved in closure activities. Are all
staff expected to do their current job up until the last day or will some tasks be finished sooner than
others, resulting in staffing levels declining over time? Though the SVT saw no clear discussion of
contingencies, NEEScomm has obviously given it some thought, as the budget changes based on whether
or not Purdue is the awardee. Given the lack of detail about plans, it is impossible to ascertain if the
budget is reasonable. The assumptions listed seem reasonable.
I. Facility Operations
a.
b.
c.
d.
What is the quality of the progress made during year four?
What is the quality of the plans and budget for year five?
What is the quality of the plan to close-out the Major Equipment Repair Fund?
What is the quality of the plans for each facility to complete its work by the end of year five and close-out
site operations (e.g., staffing retention, project throughput/completion, safety, decommissioning, etc., as
needed).
The equipment sites are on track to finish their scheduled experiments by the end of the NEEScomm
award, September 30, 2014, with the exception of two sites. NEEScomm continually monitors the
activities at each site on a monthly or, when warranted, on a bi-weekly schedule; contingency plans are in
place to deal with schedule delays and unexpected equipment failures. NEEScomm allocated close to
$0.5 million in productivity supplemental funding to eight sites, providing additional resources at these
facilities. In addition, a risk assessment identified no high-risk facilities.
The Annual Report does not address the proper closeout of site operations. Even though this was a
constant question or request during past facility site visits, only two sites addressed this important
question. The report does mention the issue of staff retention through the award end. However, a
proper site closeout involves more than personnel plans for retention and severance. A proper closeout
plan should also include site decommissioning, robust post-NEES research plans, industry partnerships,
and evidence of plans to seek external funding. About a third of the facilities are already engaged in nonNEES research activities, and this is expected to continue beyond September 30, 2014. Since the facilities
and equipment will be permanently housed at the home institutions, clear closeout and transition plans
should have been communicated in multiple sections of the Annual Report.
9
To improve an already good safety record, considerable safety-related activities have been performed.
This includes the distribution of posters, development of online safety trainings, and the identification of
site-specific hazards.
No major equipment updates are planned for FY 2014. However, a significant amount has been carried
over in the Major Equipment Repair Fund (in addition to the funds to be allocated for the last fiscal year).
If not used for equipment repair, these funds could be spent to cover other needs, subject to NSF
approval.
J. Cyberinfrastructure
a.
b.
c.
What is the quality of the progress made during year four?
What is the quality of the plans and budget for year five?
What is the quality of the plans for closing the cyberinfrastructure, including data repository, curation and
archiving; simulation tools; and documentation, at the end of year five (e.g., staffing retention, final
documentation)?
The NEEScomm information technology (IT) area continues to do a good job serving the community. The
hardware resources appear to be ample to meet user needs. The user requirements gathering
mechanism incorporates input from the community on what is needed and then prioritizes those needs.
The use of virtual machines is a cost-effective means of operating the infrastructure. As in the past,
NEEScomm IT staff continues to use software development best practices, such as version control,
regular unit testing, and testing of “servers” before production deployment.
Per recommendations in last year's site visit report, there was a strong emphasis on HS and visualization.
New visualization tools were added (N3DV and 3DDV). There was an increase in hybrid testing projects,
as well as some new real time hybrid testing capabilities added. Providing easier access to HPC resources
has clearly been a great success and the increase in SU consumption is impressive.
There have been new features added to make reuse of data easier (datastore, data papers). Ways must
be found to identify and document data reuse as this is potentially one of the greatest legacies of NEES.
The preservation pipeline continues to make improvements. The use of students knowledgeable in
earthquake engineering allowed the data curator to focus on curation activities. This is an excellent idea
and has resulted in a significant increase in curated projects. Other improvements include the curation
guide, better tracking of experiments via both the site activities page and personal interaction, storage
audits with checksums, audit trails based on PREMIS events, and data format preservation action plans.
An area that is still lacking is automated capture of data from the instruments. NEEScomm is aware of
this and has assigned resources to two equipment sites (UIUC and University of Minnesota) to improve
this over the next year.
NEEScomm proactively addresses cybersecurity at headquarters and with the other sites in its detailed
security plan. The “Site Operations Risk Assessments” appropriately include IT. The NEES cybersecurity
plan v2.1 found online (nees.org/topics/CyberSecurity) produces a more recent version than the link
included in the Annual Report. The version originally provided appeared to have last been updated in
10
2011. Version 2.1 shows a 2013 revision. (The SVT found this new version by accident, thus averting a
criticism that the plan was not updated annually as described.)
NEEScomm made a conscious decision not to pursue becoming an InCommon Service Provider (SP)
[(http://www.incommonfederation.org/)]. The SVT understands that Purdue would need to lead the way
on this with HUBzero. However, NEEScomm is urged to keep the possibilities of accepting Federated
Identity credentials and utilizing EDUROAM (see http://www.eduroamus.org/) on its radar. As of this
writing, all of the institutions with NEES sites are InCommon members and presumably somewhere on a
continuum of being or becoming Identity Providers for their employees and students. The number of
research-focused organizations who are InCommon members has increased to 29, including NIH, NSF and
numerous NSF Large Facilities and DOE facilities. The SVT does not expect NEES in its final year to
implement identity federation in response to this paragraph.
K. Education, Outreach, and Training (EOT)
a.
b.
c.
d.
What is the quality of the progress made during year four?
What is the quality of the plans and budget for year five? How have the plans for year five been informed
by the assessments conducted in year four?
How well has the project conveyed the broader impacts of the EOT activities beyond quantitative metrics?
What is the quality of the plans for closing out EOT on this award by the end of year five?
The SVT commends NEEScomm on its responsiveness to the recommendations from the prior year
annual review. NEEScomm has clearly refocused and realigned EOT activities and personnel; prioritized
and expanded public outreach; enhanced dissemination mechanisms; and evaluated, improved and
added performance metrics to more meaningfully evaluate and communicate the value of EOT efforts.
Focus was placed on a few select EOT activities through NEES headquarters. The EOT activities from year
four, some highlighted below, reflect well NEEScomm’s overall goals for this component and has the
potential to report impacts achieved.
The SVT supports the action by NEEScomm to:
 Assume a management role in the assessment, selection, and coordination of all future educational
and outreach activities by and across sites,
 Use supplemental funds to extend the education and outreach goals, and
 Redesign the NEESacademy to reflect the useful lesson plans identified in NEES education efforts to
be shared with K-12 school partners.
The assistance NEEScomm has provided to equipment sites’ EOT staff in developing assessment tools for
the site tours is important and shows a response that simultaneously addresses the needs of NEEScomm
and the individual sites. Preliminary results point to fewer, higher priority EOT activities that will be more
effective, and implemented and completed in a timely fashion. Based on initial project selection, the
process also achieves the intended goal of enabling network management of EOT site activities to
achieve greater collaboration of multiple sites and more uniform presentation of outreach activities. The
reporting of impacts collected from the use of this assessment tool should be part of the final report to
NSF. In general, assessment tools should include knowledge gained by those impacted by education
activities as well as impacts on attitudes.
The refocused efforts on the NEESacademy allowed development of more coordinated online
educational material. The NEES safety training course, two new NEEShub Boot Camp short courses, Wood
11
Education Institute, and REU enrichment courses are available through Moodle. New metrics were also
put in place to track access and use. For instance, the flipbook, a relatively new content delivery
medium, doubled the number of accesses. New assessment instruments have been developed and put in
place for some of these courses.
A significant development for EOT is the development of the affordable, controlled motion, shake table
for wider dissemination and engagement in the EOT NEES community. The cost has gone down to $500
as opposed to the $23,000 price tag for the commercial version.
Moving the media specialist to NEEScomm HQ has enhanced the potential exposure of NEES to the
public. The addition of performance metrics on public awareness and knowledge transfer should
incentivize such activities. The addition of the Newsroom to the NEEShub offers users information on
current media coverage and events pertaining to NEES. The use of the Purdue media office for weekly
distribution and tracking of NEES announcements is well presented in the Annual Report and is evidence
of the commitment to dissemination. The total number of participants in NEES activities has increased by
20%, which NEEScomm notes as a trend. This may indicate the increased effectiveness of outreach since
the number of outreach activities has remained the same as in the previous year.
NEEScomm contributions to the recent 2013 PBS NOVA television program on “Making Stuff Safer,” in
the press, in archival journals, and through paper publications all indicate higher exposure of NEES and
the value of earthquake engineering. Such efforts should be continued.
NEEScomm added specific performance metrics for the number of publications. Two special issues of the
Journal of Structural Engineering focused on NEES research. NEEScomm reported 471 publications in
2012, including 97 refereed journal articles and 41 general papers, indicating that NEES research findings
are significant and are being disseminated widely.
The use of supplemental funds to the equipment sites for public outreach through museum exhibits
shows promise and offers long-term opportunities for ongoing public outreach and engagement. The
careful tracking of the impacts of these exhibits on increased public knowledge of earthquake research is
needed.
The education and outreach project activities reported from the Ambassador Program to Family
Engineering Day Programs demonstrate impact on students at all levels, and the public. In particular, the
increased leadership and communication skills acquired by participants indicate potential to positively
affect the future engineering workforce. The communication of these results through journal articles is
lauded. Opportunities should be seized to exploit assessment instruments to reinforce content learning.
NEEScomm is urged to focus on assessment of impacts of all such activities in its final report to NSF.
EOT efforts focused on education to serve diverse groups are noteworthy. NEEScomm has made
important contributions through the NEES Undergraduate Research Pathway program at Morgan State
University and at the University of Texas at El Paso. This program will impact underrepresented students
for whom research experiences as undergraduates are minimal. Also, the success of the Virtual Peer
Teams offers cost-effective methods for exposing these students to the use of large-scale equipment. The
distributed collaboration skills gained by these students are desirable in the 21st century global
workforce. Both of these unique benefits should be analyzed and reported.
12
Training in OpenSees, provided in collaboration with the Pacific Earthquake Engineering Research (PEER)
Center, with an online audience of 67 researchers in 2012 was reported. This was followed by a series of
additional training sessions provided in 2013 for a total or 325 faculty and graduate student researchers.
This NEEScomm activity and training sessions should be continued and assessment of its usefulness to
researchers should be reported. The data collected can serve to improve the training provided.
The significant attendance reported at the Research to Practice and other webinars indicates a high level
of utility to the professional community; the attendance at general workshops and seminars shows NEES’
utility in training and outreach. A selected sample of attendees at webinars should be assessed for
further improvement.
While the SVT is heartened by the reporting of quantitative EOT impact data, there are several areas of
concern. The failure of NEEScomm to lead efforts to track building code changes resulting from NEES
research limits the opportunity to report broader impacts. Such tracking and reporting should be a key
part of the final project report to NSF. This is a key mission of NEES work.
The request for carryover funds for EOT should be presented in terms of outcomes to be achieved. While
the work described is worthy, the absence of quantitative outcomes weakens the plan.
The closeout plan for NEES in EOT should include a plan for other organizations to take over the
NEESacademy to enable teachers and others to use the resources in that site. This has not been
addressed.
Finally, the SVT encourages NEEScomm to better define and showcase all quantitative impacts achieved
by NEES in all public announcements related to the project in the last year of project work. This will serve
to inform the public of the usefulness of this project.
L. Budget
a.
b.
Is the overall budget formulation for year five operations reasonable?
Does the budget enable activities for a successful closure of this award?
Given the current uncertainty in NSF’s plans, and the amount of final year and carryover funding yet to be
dispersed, there is considerable flexibility remaining in the budget. As the award process becomes
clearer, NSF will request an explicit closeout plan for the current award. The SVT recommends to NSF that
the NEEScomm final spending plan, with all closeout activities fully enumerated and accounted for, be
fully reviewed by the SVT or another appropriate panel of peers.
Plans within the current Annual Report for use of carryover funds are not sufficiently focused on
deliverables. The final spending plan for the currently expiring agreement should clearly couple
deliverables to timelines and costs.
Within the general budget data provided, the SVT saw no major deviations from community norms in
terms of general costs associated with personnel and equipment.
13