National Science Foundation Division of Civil, Mechanical and Manufacturing Innovation Cooperative Agreement CMMI-0927178 NEES Operations FY 2010 – FY 2014 Purdue University: Dr. Julio Ramirez, PI Annual Site Visit Review (Year Four/Year Five) November 7-8, 2013 Site Visit Team Mr. William Allcock Director of Operations Argonne Leadership Computing Facility Argonne National Laboratory Argonne, IL Dr. Janos Gergely Associate Professor Department of Civil and Environmental Engineering University of North Carolina at Charlotte Charlotte, NC Dr. Vijaya Gopu Professor and Edward G. Schlieder Endowed Chair in Civil Engineering Louisiana Transportation Research Center University of New Orleans New Orleans, LA Dr. Gerald "Stinger" Guala Branch Chief, Eco-Science Synthesis Director of Biodiversity Information Serving Our Nation (BISON) Director of the Integrated Taxonomic Information System (ITIS) Core Science Analytics and Synthesis Core Science Systems U.S. Geological Survey Reston, VA Ms. Ardoth Hassler Associate Vice President University Information Services Georgetown University Washington, DC Dr. Miriam Heller Director of Council Affairs American Society of Engineering Education Washington, DC Dr. Marty Humphrey Associate Professor Department of Computer Science University of Virginia Charlottesville, VA Dr. Joann Jacullo-Noto (virtual) Director, M.A.T. Program Drew University Madison, NJ Dr. Kaye Shedlock Consultant Golden, CO National Science Foundation Staff Dr. Joy Pauschke (Site Visit Coordinator) NEES Program Director Division of Civil, Mechanical and Manufacturing Innovation Matthew Carnavos (November 7-8) Program Analyst Division of Civil, Mechanical and Manufacturing Innovation Dr. Richard Fragaszy (November 7-8) Program Director, Geotechnical Engineering Division of Civil, Mechanical and Manufacturing Innovation Dr. George Hazelrigg (November 7) Deputy Division Director Division of Civil, Mechanical and Manufacturing Innovation Dr. Scott Horner (November 7) Deputy Director for Large Facilities Office of Budget, Finance and Award Administration Dr. Kishor Mehta (November 7) Program Director, Hazard Mitigation and Structural Engineering Division of Civil, Mechanical and Manufacturing Innovation Dr. Carol Wilkinson (November 7) Program Director, Large Facilities Office Office of Budget, Finance and Award Administration 2 A. Summary of Site Visit The combined year four/year five annual review for NEES Operations was held on November 7-8, 2013. The Purdue/NEEScomm team participated via WebEx and the site visit team (SVT) participated as a group from the National Science Foundation (NSF), with NSF staff. One SVT member participated as a virtual SVT member. The first day consisted of presentations by NEEScomm. At the end of the first day, the SVT gave NEEScomm a list of questions to address overnight. NEEScomm presented the response to the questions at the start of the second day. The SVT then met in closed session to write the site visit report. At the end of the second day, the NSF and the SVT briefed NEEScomm on the recommendations and SWOT analysis. B. Recommendations and Areas for Improvement 1. NEEScomm must provide vision and leadership, above and beyond award management, during this final year of the award. To ensure the future of NEES, NEEScomm needs to determine the “right things to do” and not just “do things right.” 2. NEEScomm must start planning immediately for an orderly closeout of this award. A timeline, with required resources, should be developed for the various closeout steps. 3. The SVT recommends to NSF that the NEEScomm final spending plan, with all closeout activities fully enumerated and accounted for, be fully reviewed by the SVT or another appropriate panel of peers. 4. As the stewards of critical national assets and this being one of NSF’s largest projects and long-term investments, NEEScomm must provide a plan that ensures the option to continue support and availability of the cyberinfrastructure and the equipment sites. NEEScomm is optimally positioned to offer a visionary plan for its continued involvement to support and make available the NEEShub. 5. The NEEScomm Governance Board must assume a stronger role in the oversight of the closure of the award. 6. NEEScomm needs to deepen the assessment of the education, outreach, and training (EOT) activities in order to identify their impact. 7. It is important to identify the "lessons learned" from this project, specifically addressing the successes and challenges associated with providing centralized support and management to distributed, large-scale equipment sites and the effectiveness and efficiency of the cyberinfrastructure. C. SWOT: Strengths, Weaknesses, Opportunities, and Threats Strengths • • • • The NEES network continues to provide access to unique, world-class research facilities that are highly available and well run, enabling national and international research and collaboration. The progress NEES has made in advancing hybrid simulation (HS) is laudable. NEEScomm is commended for the significant progress they have made in the curation of data and information. The growth in the number of curated projects is impressive. NEES continues to expand the use and availability of high performance computing (HPC) by its researchers and collaborators. This is excellent leveraging of the NSF investment in cyberinfrastructure. 3 • • • • • The integration of undergraduates into the research, both REU and local students, continues to improve. The SVT specifically recognizes NEES’ efforts in its service to underserved minorities. NEEScomm responded to the past EOT recommendations and is demonstrating some quantifiable outcomes. The Balanced Scorecard has been significantly improved to provide useful management information and meaningful indications of activities to the NEES Governance Board, NSF, and the SVT. NEEScomm has successfully increased the publicity about NEES, earthquake engineering research, and research facilities. NEEScomm continues to perform system-wide risk assessments, and looks at its cyberinfrastructure as a “whole” and not just the individual “parts.” Weaknesses • • • • • NEEScomm is managing the award, rather than providing community leadership and vision. NEEScomm has inadequately planned for the future. It should have already outlined three parallel planning scenarios for NEES, especially addressing intellectual property concerns: NEEScomm continuing at Purdue. NEES operations transferring to another organization. NEES program discontinuance. The involvement and participation of the Governance Board is not readily evident. They must become more involved. The timeline for archiving data after the last completed experiment is aggressive. Moving materials to “cold storage” is a non-trivial operation. The SVT notes that NEEScomm responded to the request to shorten the Annual Report and provide links. However, the report is still hard to navigate and ascertain impacts. The topic sentences should contain the most important point of the paragraph. The narrative should include a “summary sentence” of outcomes in addition to providing links to the products. Outcomes/successes should be obvious without the reader having to analyze each table fully. An “index” to important attachments, such as the Balanced Scorecard, Cybersecurity Plan, etc., is recommended. Opportunities • • • • NEEScomm needs to be planning for award closeout now and stop waiting for direction from NSF. NEEScomm has the opportunity to develop a unique plan for continuing and enhancing partnerships, preservation of research data and results, and the reuse of the repository. NEEScomm has the opportunity to use its fifth year to lead the development of a vision for NEES for years 11-15. NEEScomm should continue to increase racial and ethnic diversity and participation of women across all aspects of the network. Threats • • Incomplete transition plans are a significant threat. Not having plans to ensure that the data, results, training, and outreach programs will be continued into the future is a threat to the broad community and researchers. Failure to retain necessary staff through project closure is a serious threat. 4 • Waiting for direction from NSF is a serious threat. What happens if NEEScomm does not get a response from NSF prior to January 15, 2014, and there is another lapse in Federal appropriations? Lost time cannot be recovered. D. NSF Merit Review Criterion: Intellectual Merit The NSF Intellectual Merit criterion encompasses the potential to advance knowledge: a. What is the potential for the proposed activity to advance knowledge and understanding within its own field or across different fields? b. To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts? c. Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success? d. How well qualified is the individual, team, or organization to conduct the proposed activities? e. Are there adequate resources available to the PI (either at the home organization or through collaborations) to carry out the proposed activities? NEEScomm has been very successful in significantly increasing the number of projects curated in year four. NEEScomm's efforts to focus its EOT activities and be responsive to SVT recommendations made during an earlier site visit are very encouraging. The continued enhancement of the NEEShub in providing more tools, educational material, etc., is an important contribution to the professional community. The scheduling of experiments at the various equipment sites has been streamlined and the system is working effectively. Plans to ensure completion of all experiments before September 30, 2014, are sound. The various activities undertaken to promote the use of HS have helped enhance the research capabilities of the nation's earthquake engineering research community. This is a major contribution of NEEScomm. NEEScomm has facilitated easy access to HPC resources for earthquake engineering researchers. NEEScomm has established numerous international collaborations over the past four years. E. NSF Merit Review Criterion: Broader Impacts The Broader Impacts criterion encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes: a. What is the potential for the proposed activity to benefit society or advance desired societal outcomes? b. To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts? c. Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success? d. How well qualified is the individual, team, or organization to conduct the proposed activities? e. Are there adequate resources available to the PI (either at the home organization or through collaborations) to carry out the proposed activities? The potential broader impacts of the network's activities are clearly articulated in the vision of NEES: "A global infrastructure network that improves the resilience of new and existing construction, and supports the education of the next generation of engineers and scientists." Close to 200 NEES research (NEESR) projects have been performed at the equipment sites. The EOT program has been conducted reaching a 5 wide range of audiences, from K-12 students to science teachers, from undergraduate to graduate students, and from design professionals to the general public. NEEScomm did not articulate how its partnerships and the related activities advance community understanding of the seismic vulnerability of our built infrastructure. The outreach activities, including museum exhibits, workshops, webinars, TV documentaries, and especially those engaging underrepresented and minority groups at all level, are notable and indicate the potential for broad outreach. The Annual Report did not include enough examples describing the international collaborations between NEES and their European, Canadian, Japanese and Chinese counterparts. The multi-story wood frame experiment performed at the E-Defense site was highlighted; however, more examples of additional (and more specific) research partnerships would enable better assessment of the direct impact of the international effort. NEESR findings are published in conventional channels, in data papers (in collaboration with EERI’s Earthquake Spectra), and through various media in order to reach a broader audience. Furthermore, NEEShub improved the existing database and user interface, although the actual data usage has not been tracked. A primary goal of NEES has been to improve the resilience of the built infrastructure. NEEScomm listed the proposed changes and additions within the ASCE and ACI, as well as AISC, building code requirements as evidence of its influence on the state-of-the-practice. Had there been more leadership from NEEScomm in encouraging the development of safer and more robust code provisions, the broader impact of the network's efforts could have been greater, especially given the scope and funding level for this award. F. How well has NEEScomm and the facilities responded to the issues raised in the August 2012 annual site visit report and the facility site visit reports (August 2012 – February 2013)? NEEScomm provided documentation describing steps taken in response to eight areas for improvement identified by the 2012 annual review SVT. The area in need of greatest attention was EOT. The 2012 SVT recommended that NEEScomm refocus and downsize the existing EOT effort, since the progress to date seemed insufficient given the number of staff positions. NEEScomm did downsize the EOT staff, redirecting personnel to the IT effort (as suggested by the SVT). NEEScomm stressed that all EOT responsibilities specified in the Cooperative Agreement would be met, the NEESacademy would be maintained and refined, but no further advanced features would be developed. Improvements in public outreach via media were developed, including cooperation with NSF and AAAS to include NEESR successes via LiveScience.com, EurekAlert, the Discovery Channel’s Daily Planet, PBS’ NOVA, and Purdue Marketing and Media. NEEScomm is also working on several fronts to increase awareness via museums and appropriate print and electronic media, including the creation of the "newsroom" on nees.org. The data curation and preservation pipeline was cited as a threat last year, and the SVT provided specific, detailed recommendations. NEEScomm responded with specific plans addressing all of the recommendations, including new curation guides and data, metadata, and documentation upload mandates. NEEScomm increased network staff to provide more upload capabilities, and significant 6 progress is evident. Additionally, NEEScomm and some sites have been working together to build automatic data capture software that should be a great help. NEEScomm has also been strengthening the preservation infrastructure, as recommended. NEEScomm also: Established a required acknowledgement for all papers that use NEES data and provides usage information via nees.org, Continued and expanded outreach to underrepresented students to increase diversity, Strengthened safety awareness throughout the network, and Simplified quarterly and annual reporting, utilizing the NEEShub to provide links to supporting documentation. NSF made four facility site visits since the last annual review. Perhaps fortuitously, all four are heavily involved in HS. A great outcome of these site visits is the increased attention directed to HS across the NEES community, ongoing development of a NEEShub-based “guide to hybrid simulation,” and associated activities. Workshops have been held, task groups are now active, NEEShub hosts a HS Wiki, and a Simulation Steering Committee guides overall collaboration. Advances in HS are likely to be among the most important legacies of NEES. Each facility site visit report involved mostly site-specific recommendations that NEEScomm appears to have worked diligently with each site to address. NEEScomm has provided supplemental funding for safety improvements, increased productivity, and other recommendations as needed. Follow-through by the sites and NEEScomm appears to be thorough and appropriate, since no major issues or weaknesses have caused disruption to the sites or network to date. Lack of leadership on the part of NEEScomm appears repeatedly in various forms in each site review, best illustrated by these verbatim sentences from the EOT section of one of the facility site visit response documents: SVT statement: There is a lack of leadership on the part of NEEScomm in EOT across the network. NEEScomm response: NEEScomm has provided, and will continue to provide, ongoing guidance and leadership to the equipment sites through the practices it has cultivated. A famous quote from very successful management consultant, professor, and author Peter Drucker is: "management is doing things right; leadership is doing the right things." He, and many others, describes great leaders as possessing impressive social intelligence, no fear of change, and most importantly, vision. NEEScomm has provided management and guidance through the practices it has cultivated, moving earthquake engineering towards collaborative research. However, NEEScomm has done so reactively, rather than proactively. NEEScomm has “done things right” when the community determined what they would like to have done. But NEEScomm has rarely demonstrated the leadership to push boundaries. For example, NEEScomm's increased attention on HS is a reactive result of multiple instances of feedback over time from the sites and various SVTs rather than NEEScomm recognizing early on that HS is an important new direction and encouraging/facilitating advances proactively. A current example is the lack 7 of closeout scenarios. NEES Operations is a large NSF award, and waiting for a decision is reactive; examining all likely scenarios and mitigating the effects of each for the future of NEES is proactive. G. How well has NEEScomm met its performance metrics and how have the metrics been used for project assessment and improvement? NEEScomm performance metrics development and use have followed a trajectory that might be expected for a large, multi-faceted program, i.e., performance metrics were developed, utilized, and modified as their effectiveness became evident. NEEScomm met most of its performance objectives and where they did not, made changes necessary to meet their objectives by the next review. In the first few years of this award, NEEScomm attempted to include all input from the sites and NSF, and all levels of granularity. The Balanced Scorecard was modified and grew rapidly, until it was too large and, in some respects, of questionable value (lots of counting, very little impact assessment). The 2012 SVT recommended that NEEScomm “Evaluate all performance metrics with the purpose of streamlining the operational metrics and include other metrics that provide real information on the scientific and broader impacts of NEES.” NEEScomm did a very good job of simplifying the Balanced Scorecard and formulating some impact assessment metrics. Some of the metrics and the rationale for some of the targets still need to be fully defined in the supporting documentation. Although NEEScomm did not meet all of its performance metrics for year four, the simplified performance metrics themselves and NEEScomm's management performance overall are improvements. H. Award Closure (overall management – preliminary plans) Quality and appropriateness of: a. Priorities b. Plans and timeline c. Personnel d. Budget NEEScomm appropriately states priorities for award closure are ensuring that all experiments are completed by September 30, 2014, and all data are uploaded, curated, and archived. However, NEEScomm does not address the transfer or decommissioning of the cyberinfrastructure, and long-term preservation of the NEES assets (the data, and also the EOT material, tools, etc.). This is a serious concern. With the exception of the UC Berkeley equipment site, there is no plan or information about the future in the Annual Report (Volume 2) for the sites in the absence of NEES funding. The SVT accepts that it could be argued that this is outside of the scope of NEES. However, as the stewards of critical national assets and this being one of the largest NSF projects and long-term investments, a visionary plan for ongoing support and availability of these facilities to the community would have been welcomed. NEEScomm must begin contingency planning immediately even though the path forward is uncertain. If plans from NSF are too long in coming, NEEScomm will be forced into action because, and, based on its statements, it will not have funding to continue supporting the cyberinfrastructure. There is substantial risk of catastrophic loss of vital community assets if NEEScomm does not develop a plan. The SVT strongly recommends that NEEScomm begin developing contingencies that include at least the following scenarios: 8 • • • Preparing for handover to another awardee, if one has been identified prior to September 30, 2014. Requesting funding required to maintain cyberinfrastructure to bridge a gap between this award and the next. This may include: no development, no ticket response, and basic cybersecurity resources (e.g., scans, IDS, vulnerability patches, etc.) only. The SVT is fully aware that this is non-trivial, but is also aware that the costs, considering that Purdue is running the same infrastructure for other hubs (Oracle instances, etc.), are incremental and should be reasonable. Developing a “worst case scenario” to ensure the long-term survival of the assets. For example, for software developed outside of NEEScomm, either identify a home that already exists (for instance opensees.berkeley.edu) or create one, for instance on Github. For EOT assets, find someone (EERI, etc.) that can host them. For the Project Warehouse, databases, data papers, etc., determine how the data and required metadata and documentation can be stored in a way that makes them accessible, and has a very high probability of maintaining its integrity. The SVT found no explicit description of the personnel needed to be involved in closure activities. Are all staff expected to do their current job up until the last day or will some tasks be finished sooner than others, resulting in staffing levels declining over time? Though the SVT saw no clear discussion of contingencies, NEEScomm has obviously given it some thought, as the budget changes based on whether or not Purdue is the awardee. Given the lack of detail about plans, it is impossible to ascertain if the budget is reasonable. The assumptions listed seem reasonable. I. Facility Operations a. b. c. d. What is the quality of the progress made during year four? What is the quality of the plans and budget for year five? What is the quality of the plan to close-out the Major Equipment Repair Fund? What is the quality of the plans for each facility to complete its work by the end of year five and close-out site operations (e.g., staffing retention, project throughput/completion, safety, decommissioning, etc., as needed). The equipment sites are on track to finish their scheduled experiments by the end of the NEEScomm award, September 30, 2014, with the exception of two sites. NEEScomm continually monitors the activities at each site on a monthly or, when warranted, on a bi-weekly schedule; contingency plans are in place to deal with schedule delays and unexpected equipment failures. NEEScomm allocated close to $0.5 million in productivity supplemental funding to eight sites, providing additional resources at these facilities. In addition, a risk assessment identified no high-risk facilities. The Annual Report does not address the proper closeout of site operations. Even though this was a constant question or request during past facility site visits, only two sites addressed this important question. The report does mention the issue of staff retention through the award end. However, a proper site closeout involves more than personnel plans for retention and severance. A proper closeout plan should also include site decommissioning, robust post-NEES research plans, industry partnerships, and evidence of plans to seek external funding. About a third of the facilities are already engaged in nonNEES research activities, and this is expected to continue beyond September 30, 2014. Since the facilities and equipment will be permanently housed at the home institutions, clear closeout and transition plans should have been communicated in multiple sections of the Annual Report. 9 To improve an already good safety record, considerable safety-related activities have been performed. This includes the distribution of posters, development of online safety trainings, and the identification of site-specific hazards. No major equipment updates are planned for FY 2014. However, a significant amount has been carried over in the Major Equipment Repair Fund (in addition to the funds to be allocated for the last fiscal year). If not used for equipment repair, these funds could be spent to cover other needs, subject to NSF approval. J. Cyberinfrastructure a. b. c. What is the quality of the progress made during year four? What is the quality of the plans and budget for year five? What is the quality of the plans for closing the cyberinfrastructure, including data repository, curation and archiving; simulation tools; and documentation, at the end of year five (e.g., staffing retention, final documentation)? The NEEScomm information technology (IT) area continues to do a good job serving the community. The hardware resources appear to be ample to meet user needs. The user requirements gathering mechanism incorporates input from the community on what is needed and then prioritizes those needs. The use of virtual machines is a cost-effective means of operating the infrastructure. As in the past, NEEScomm IT staff continues to use software development best practices, such as version control, regular unit testing, and testing of “servers” before production deployment. Per recommendations in last year's site visit report, there was a strong emphasis on HS and visualization. New visualization tools were added (N3DV and 3DDV). There was an increase in hybrid testing projects, as well as some new real time hybrid testing capabilities added. Providing easier access to HPC resources has clearly been a great success and the increase in SU consumption is impressive. There have been new features added to make reuse of data easier (datastore, data papers). Ways must be found to identify and document data reuse as this is potentially one of the greatest legacies of NEES. The preservation pipeline continues to make improvements. The use of students knowledgeable in earthquake engineering allowed the data curator to focus on curation activities. This is an excellent idea and has resulted in a significant increase in curated projects. Other improvements include the curation guide, better tracking of experiments via both the site activities page and personal interaction, storage audits with checksums, audit trails based on PREMIS events, and data format preservation action plans. An area that is still lacking is automated capture of data from the instruments. NEEScomm is aware of this and has assigned resources to two equipment sites (UIUC and University of Minnesota) to improve this over the next year. NEEScomm proactively addresses cybersecurity at headquarters and with the other sites in its detailed security plan. The “Site Operations Risk Assessments” appropriately include IT. The NEES cybersecurity plan v2.1 found online (nees.org/topics/CyberSecurity) produces a more recent version than the link included in the Annual Report. The version originally provided appeared to have last been updated in 10 2011. Version 2.1 shows a 2013 revision. (The SVT found this new version by accident, thus averting a criticism that the plan was not updated annually as described.) NEEScomm made a conscious decision not to pursue becoming an InCommon Service Provider (SP) [(http://www.incommonfederation.org/)]. The SVT understands that Purdue would need to lead the way on this with HUBzero. However, NEEScomm is urged to keep the possibilities of accepting Federated Identity credentials and utilizing EDUROAM (see http://www.eduroamus.org/) on its radar. As of this writing, all of the institutions with NEES sites are InCommon members and presumably somewhere on a continuum of being or becoming Identity Providers for their employees and students. The number of research-focused organizations who are InCommon members has increased to 29, including NIH, NSF and numerous NSF Large Facilities and DOE facilities. The SVT does not expect NEES in its final year to implement identity federation in response to this paragraph. K. Education, Outreach, and Training (EOT) a. b. c. d. What is the quality of the progress made during year four? What is the quality of the plans and budget for year five? How have the plans for year five been informed by the assessments conducted in year four? How well has the project conveyed the broader impacts of the EOT activities beyond quantitative metrics? What is the quality of the plans for closing out EOT on this award by the end of year five? The SVT commends NEEScomm on its responsiveness to the recommendations from the prior year annual review. NEEScomm has clearly refocused and realigned EOT activities and personnel; prioritized and expanded public outreach; enhanced dissemination mechanisms; and evaluated, improved and added performance metrics to more meaningfully evaluate and communicate the value of EOT efforts. Focus was placed on a few select EOT activities through NEES headquarters. The EOT activities from year four, some highlighted below, reflect well NEEScomm’s overall goals for this component and has the potential to report impacts achieved. The SVT supports the action by NEEScomm to: Assume a management role in the assessment, selection, and coordination of all future educational and outreach activities by and across sites, Use supplemental funds to extend the education and outreach goals, and Redesign the NEESacademy to reflect the useful lesson plans identified in NEES education efforts to be shared with K-12 school partners. The assistance NEEScomm has provided to equipment sites’ EOT staff in developing assessment tools for the site tours is important and shows a response that simultaneously addresses the needs of NEEScomm and the individual sites. Preliminary results point to fewer, higher priority EOT activities that will be more effective, and implemented and completed in a timely fashion. Based on initial project selection, the process also achieves the intended goal of enabling network management of EOT site activities to achieve greater collaboration of multiple sites and more uniform presentation of outreach activities. The reporting of impacts collected from the use of this assessment tool should be part of the final report to NSF. In general, assessment tools should include knowledge gained by those impacted by education activities as well as impacts on attitudes. The refocused efforts on the NEESacademy allowed development of more coordinated online educational material. The NEES safety training course, two new NEEShub Boot Camp short courses, Wood 11 Education Institute, and REU enrichment courses are available through Moodle. New metrics were also put in place to track access and use. For instance, the flipbook, a relatively new content delivery medium, doubled the number of accesses. New assessment instruments have been developed and put in place for some of these courses. A significant development for EOT is the development of the affordable, controlled motion, shake table for wider dissemination and engagement in the EOT NEES community. The cost has gone down to $500 as opposed to the $23,000 price tag for the commercial version. Moving the media specialist to NEEScomm HQ has enhanced the potential exposure of NEES to the public. The addition of performance metrics on public awareness and knowledge transfer should incentivize such activities. The addition of the Newsroom to the NEEShub offers users information on current media coverage and events pertaining to NEES. The use of the Purdue media office for weekly distribution and tracking of NEES announcements is well presented in the Annual Report and is evidence of the commitment to dissemination. The total number of participants in NEES activities has increased by 20%, which NEEScomm notes as a trend. This may indicate the increased effectiveness of outreach since the number of outreach activities has remained the same as in the previous year. NEEScomm contributions to the recent 2013 PBS NOVA television program on “Making Stuff Safer,” in the press, in archival journals, and through paper publications all indicate higher exposure of NEES and the value of earthquake engineering. Such efforts should be continued. NEEScomm added specific performance metrics for the number of publications. Two special issues of the Journal of Structural Engineering focused on NEES research. NEEScomm reported 471 publications in 2012, including 97 refereed journal articles and 41 general papers, indicating that NEES research findings are significant and are being disseminated widely. The use of supplemental funds to the equipment sites for public outreach through museum exhibits shows promise and offers long-term opportunities for ongoing public outreach and engagement. The careful tracking of the impacts of these exhibits on increased public knowledge of earthquake research is needed. The education and outreach project activities reported from the Ambassador Program to Family Engineering Day Programs demonstrate impact on students at all levels, and the public. In particular, the increased leadership and communication skills acquired by participants indicate potential to positively affect the future engineering workforce. The communication of these results through journal articles is lauded. Opportunities should be seized to exploit assessment instruments to reinforce content learning. NEEScomm is urged to focus on assessment of impacts of all such activities in its final report to NSF. EOT efforts focused on education to serve diverse groups are noteworthy. NEEScomm has made important contributions through the NEES Undergraduate Research Pathway program at Morgan State University and at the University of Texas at El Paso. This program will impact underrepresented students for whom research experiences as undergraduates are minimal. Also, the success of the Virtual Peer Teams offers cost-effective methods for exposing these students to the use of large-scale equipment. The distributed collaboration skills gained by these students are desirable in the 21st century global workforce. Both of these unique benefits should be analyzed and reported. 12 Training in OpenSees, provided in collaboration with the Pacific Earthquake Engineering Research (PEER) Center, with an online audience of 67 researchers in 2012 was reported. This was followed by a series of additional training sessions provided in 2013 for a total or 325 faculty and graduate student researchers. This NEEScomm activity and training sessions should be continued and assessment of its usefulness to researchers should be reported. The data collected can serve to improve the training provided. The significant attendance reported at the Research to Practice and other webinars indicates a high level of utility to the professional community; the attendance at general workshops and seminars shows NEES’ utility in training and outreach. A selected sample of attendees at webinars should be assessed for further improvement. While the SVT is heartened by the reporting of quantitative EOT impact data, there are several areas of concern. The failure of NEEScomm to lead efforts to track building code changes resulting from NEES research limits the opportunity to report broader impacts. Such tracking and reporting should be a key part of the final project report to NSF. This is a key mission of NEES work. The request for carryover funds for EOT should be presented in terms of outcomes to be achieved. While the work described is worthy, the absence of quantitative outcomes weakens the plan. The closeout plan for NEES in EOT should include a plan for other organizations to take over the NEESacademy to enable teachers and others to use the resources in that site. This has not been addressed. Finally, the SVT encourages NEEScomm to better define and showcase all quantitative impacts achieved by NEES in all public announcements related to the project in the last year of project work. This will serve to inform the public of the usefulness of this project. L. Budget a. b. Is the overall budget formulation for year five operations reasonable? Does the budget enable activities for a successful closure of this award? Given the current uncertainty in NSF’s plans, and the amount of final year and carryover funding yet to be dispersed, there is considerable flexibility remaining in the budget. As the award process becomes clearer, NSF will request an explicit closeout plan for the current award. The SVT recommends to NSF that the NEEScomm final spending plan, with all closeout activities fully enumerated and accounted for, be fully reviewed by the SVT or another appropriate panel of peers. Plans within the current Annual Report for use of carryover funds are not sufficiently focused on deliverables. The final spending plan for the currently expiring agreement should clearly couple deliverables to timelines and costs. Within the general budget data provided, the SVT saw no major deviations from community norms in terms of general costs associated with personnel and equipment. 13