Valuing Software and Other
Research Outputs
Daniel S. Katz
dkatz@nsf.gov & d.katz@ieee.org
@danielskatz
Program Director, Division of
Advanced Cyberinfrastructure
1am:London – The Altmetrics Conference
26 September 2014
National Science Foundation
• Federal agency created in 1950 "to promote the
progress of science; to advance the national
health, prosperity, and welfare; to secure the
national defense…”
• Annual budget of $7.2 billion (FY 2014)
• Funds 24 percent of all federally supported
basic research at US colleges and universities
• In many fields such as mathematics, computer
science and the social sciences, NSF is the
major source of federal funds
NSF
OFFICE OF DIVERSITY &
INCLUSION (ODI)
NATIONAL SCIENCE FOUNDATION
Claudia J. Postell, Head
703.292.8020
OFFICE OF THE DIRECTOR
NATIONAL SCIENCE BOARD (NSB)
703.292.8000
Dan E. Arvizu
Chair
Kelvin K. Droegemeier
Vice Chair
OFFICE OF THE GENERAL
COUNSEL (OGC)
Lawrence Rudolph,
General Counsel
Peggy Hoyle, Deputy GC
France A. Córdova
703.292.7000
703.292.8060
Director
OFFICE OF INTERNATIONAL &
INTEGRATIVE ACTIVITIES (OIIA)
NATIONAL SCIENCE BOARD
OFFICE
Michael Van Woert
Wanda Ward, Head
Executive Officr
703.292.7000
Vacant
703.292.8040
Deputy Director
OFFICE OF INSPECTOR
GENERAL (OIG)
OFFICE OF LEGISLATIVE & PUBLIC
AFFAIRS (OLPA)
Allison C. Lerner, Inspector General
e
703.292.7100
DIRECTORATE FOR
BIOLOGICAL
SCIENCES
(BIO)
John C.l Wingfied,
Assistant Director
Jane Silverthorne,
Deputy AD
703.292.8400
DIVISION OF BIOLOGICAL
INFRASTRUCTURE (DBI)
Scott Edwards,
Division Director
703.292.8470
DIVISION OF ENVIRONMENT AL
BIOLOGY (DEB)
Penelope Firth,
Division Director
703.292.8480
DIVISION OF INTEGRA TIVE
ORGANISMAL SYSTEMS (IOS)
Michelle Elekonich,
Acting Division Director
703.292.8420
DIVISION OF MOLECULAR &
CELLULAR BIOSCIENCES (MCB)
Gregory Warr,
Acting Division Director
703.292.8440
Judy Gan, Head
703.292.8070
DIRECTORATE FOR
COMPUTER &
INFORMATION SCIENCE &
ENGINEERING (CISE)
DIRECTORATE FOR
EDUCATION & HUMAN
RESOURCES
(EHR)
Suzane Iacono,
Acting Assistant Director
Joan Ferrini-Mundy,
Assistant Director
Vacant,
Deputy AD
Jermelina L. Tupas,
Acting Deputy AD
DIRECTORATE FOR
ENGINEERING
(ENG)
Pramod P.
Khargonekar,
Assistant Director
Grace Wang,
Deputy AD
703.292.8600
703.292.8900
DIVISION OF COMPUTER & e
NETWORK SYSTEMS (CNS)
Keith Marzullo,
Division Director
703.292.8950
DIVISION OF COMPUTING &
COMMUNICATION
FOUNDATIONS (CCF)
Rao Kosaraju,
Division Director
703.292.8910
DIVISION OF ADVANCED
CYBERINFRASTRUCTURE (ACI)
Irene Qualters,
Division Director
703.292.8970
DIVISION OF INFORMATION &
INTELLIGENT SYSTEMS (IIS)
Deborah Lockhart,
Acting Division Director
703.292.8930
DIRECTORATE FOR
GEOSCIENCES
(GEO)
DIVISION OF HUMAN RESOURCE
DEVELOPMENT (HRD)
Sylvia James,
Division Director
703.292.8640
DIVISION OF RESEARCH ON
LEARNING IN FORMAL &
INFORMAL SETTINGS (DRL)
Sarah McDonald,
Acting Division Director
703.292.8620
DIVISION OF UNDERGRADUATE
EDUCATION (DUE)
Susan Singer,
Division Director
703.292.8670
OFFICE OF EMERGING
FRONTIERS (EF)
Charles Liarakos,
Acting Division Director
703.292.8508
National Science Foundation
4201 Wilson Boulevard
Arlington, Virginia 22230
TEL: 703.292.5111 | FIRS: 800.877.8339 | TDD: 800.281.8749
e
DIVISION OF CHEMICAL,
BIOENGINEERING, ENVIRONMENTAL &
TRANSPORT SYSTEMS (CBET)
JoAnn Lighty,
Division Director
703.292.8320
DIVISION OF CIVIL,
MECHANICAL & MANUFACTURING
INNOVATION (CMMI)
George Hazelrigg,
Acting Division Director
703.292.8360
DIVISION OF ELECTRICAL,
COMMUNICATIONS & CYBER
SYSTEMS (ECCS)
Samir El-Ghazaly,
Division Director
703.292.8339
DIVISION OF ENGINEERING
EDUCATION & CENTERS (EEC)
Theresa Maldonado,
Division Director
703.292.8380
DIVISION OF INDUSTRIAL
INNOVATION & PARTNERSHIPS (IIP)
Vacant,
Division Director
703.292.8050
OFFICE OF EMERGING
FRONTIERS IN RESEARCH &
INNOVATION (EFRI)
Sohi Rastegar,
Senior Advisor
703.292.8301
DIRECTORATE FOR
SOCIAL, BEHAVIORAL, &
ECONOMIC SCIENCES
(SBE)
Roger Wakimoto,
Assistant Director
Fleming Crim,
Assistant Director g
Fay Cook,
Assistant Director
Margaret Cavanaugh,
Deputy AD
Celeste M. Rohlfin,
Deputy AD
Deborah Olster,
Acting Deputy AD
703.292.8500
703.292.8800
703.292.8700
703.292.8300
DIVISION OF GRADUATE
EDUCATION (DGE)
Valerie Wilson,
Acting Division Director
703.292.8630
DIRECTORATE FOR
MATHEMATICAL &
PHYSICAL SCIENCES
(MPS)
OFFICE OF BUDGET,
FINANCE, & AWARD
MANAGEMENT
(BFA)
Martha A. Rubenstein,
Head e
/ Chief Financial
Officr
Joanna E. Rom,
Deputy Head
OFFICE OF INFORMATION
& RESOURCE
MANAGEMENT
(OIRM)
Clifford Gabriel,
Acting Head
Amy Northcutt,
Chief Information Officr
703.292.8100
703.292.8200
DIVISION OF ATMOSPHERIC &
GEOSPACE SCIENCES (AGS)
Scott Borg,
Division Director
703.292.8520
DIVISION OF ASTRONOMICAL
SCIENCES (AST)
James Ulvestad,
Division Director
703.292.8820
DIVISION OF BEHAVIORAL &
COGNITIVE SCIENCES (BCS)
Mark Weiss,
Division Director
703.292.8740
BUDGET DIVISION (BUD)
Michael Sieverts,
Division Director
703.292.8260
DIVISION OF ADMINISTRATIVE
SERVICES (DAS)
Mercedes Eugenia,
Division Director
703.292.8190
DIVISION OF EARTH
SCIENCES (EAR)
Sonia Esperança
Acting Division Director
703.292.8550
DIVISION OF CHEMISTRY (CHE)
Jacquelyn Gervay-Hague,
Division Director
703.292.8840
DIVISION OF SOCIAL &
ECONOMIC SCIENCES (SES)
Jeryl Mumpower,
Division Director
703.292.8760
DIVISION OF ACQUISITION AND
COOPERATIVE SUPPORT (DACS)
Jeffery Lupis,
Division Director
703.292.8240
DIVISION OF INFORMATION
SYSTEMS (DIS)
Dorothy Aronson,
Division Director
703.292.8150
DIVISION OF OCEAN
SCIENCES (OCE)
Deborah Bronk,
Acting Division Director
703.292.8580
DIVISION OF MATERIALS
RESEARCH (DMR)
Mary Galvin-Donoghue,
Division Director
703.292.8810
NATIONAL CENTER FOR
SCIENCE AND ENGINEERING
STATISTICS (NCSES)
John Gawalt,
Division Director
703.292.8780
DIVISION OF FINANCIAL
MANAGEMENT (DFM)
,
Shirl Ruffin
Division Director / Deputy CFO
703.292.8280
DIVISION OF HUMAN RESOURCE
MANAGEMENT (HRM)
Judy Sunley,
Division Director /
Acting Chief Human Capital Officr
703.292.8180
DIVISION OF
POLAR PROGRAMS (PLR)
Kelly Falkner,
Division Director
703.292.8030
DIVISION OF MATHEMATICAL
SCIENCES (DMS)
Michael Vogelius,
Division Director
703.292.8870
DIVISION OF GRANTS &
AGREEMENTS (DGA)
Karen Tiplady,
Division Director
703.292.8210
DIVISION OF PHYSICS (PHY)
Denise Caldwell,
Division Director
703.292.8890
DIVISION OF INSTITUTION &
AWARD SUPPORT (DIAS)
Mary Santonastasso,
Division Director
703.292.8230
OFFICE OF MULTIDISCIPLINARY
ACTIVITIES (OMA)
ClarkeCooper,
Offic He ad
703.292.8800
LARGE FACILITIES OFFICE
Matthew Hawkins,
Acting Deputy Director
703.292.4416
September 2014
Advanced Cyberinfrastructure
(ACI) Division
• Supports and coordinates the development,
acquisition, and provision of state-of-the-art
cyberinfrastructure resources, tools, and
services
• Supports forward-looking research and
education to expand the future capabilities of
cyberinfrastructure
• Serves the growing community of scientists and
engineers, across all disciplines, whose work
relies on the power of advanced computation,
data-handling, and networking
Cyberinfrastructure
“Cyberinfrastructure consists of
computing systems,
data storage systems,
advanced instruments and
data repositories,
visualization environments, and
people,
all linked together by
software and
high performance networks,
to improve research productivity and
enable breakthroughs not otherwise possible.”
-- Craig Stewart
Software as Infrastructure
Science
• Software (including services) essential for
the bulk of science
-
•
About half the papers in recent issues of
Software
Science were software-intensive projects
Research becoming dependent upon
advances in software
Computing
Significant software development being
Infrastructure
conducted across NSF: NEON, OOI,
NEES, NCN, iPlant, etc
Wide range of software types: system, applications, modeling,
gateways, analysis, algorithms, middleware, libraries
• Software is not a one-time effort, it must be sustained
•
•
•
Development, production, and maintenance are people intensive
Software life-times are long vs hardware
Software has under-appreciated value
For software to be sustainable,
it must become infrastructure
NSF Software Infrastructure Projects
5 rounds of funding,
65 SSEs
4 rounds of funding,
35 SSIs
2 rounds of funding,
14 S2I2
conceptualizations
See http://bit.ly/sw-ci for current projects
SSE & SSI – NSF 14-520: Cross-NSF, all Directorates participating
Next SSEs due Feb 2015; Next SSIs due June 2015
SI2 Solicitation and Decision Process
• Proposal reviews well -> my role becomes
matchmaking
– I want to find program officers with funds, and convince them
that they should spend their funds on the proposal
• Unidisciplinary project (e.g. bioinformatics app)
– Work with single program officer, either likes the proposal or
not
• Multidisciplinary project (e.g., molecular
dynamics)
– Work with multiple program officers, ...
• Omnidisciplinary project (e.g. http, math library)
– Try to work with all program officers, often am told “it’s your
responsibility”
To judge software, need to
understand/forecast impact
Measuring Impact – Scenarios
1. Developer of open source physics simulation
– Possible metrics
•
•
•
•
•
How many downloads? (easiest to measure, least value)
How many contributors?
How many uses?
How many papers cite it?
How many papers that cite it are cited? (hardest to measure,
most value)
2. Developer of open source math library
– Possible metrics are similar, but citations are less
likely
– What if users don’t download it?
•
•
•
•
It’s part of a distro
It’s pre-installed (and optimized) on an HPC system
It’s part of a cloud image
It’s a service
• Future impacts – let proposers suggest
ACI Software Cluster Programs
• In these programs, ACI works with other NSF
units to support projects that lead to software
as an element of infrastructure
• Issue: amount of software that is
infrastructure grows over time, and grows
faster than NSF funding
Q: How can NSF ensure that software as
infrastructure continues to appear, without
funding all of it?
A: Incentives
• The devil is in the details
Other Software Discussions
• Working Towards Sustainable Software for
Science: Practice and Experience (WSSSPE)
– Google: WSSSPE
– 2 previous workshops, one upcoming
– WSSSPE2 @ SC14 (Nov 16, New Orleans)
• Lessons:
Many of the issues in developing
sustainable software are social, not
technical
Software work is inadequately visible in
ways that “count” within the reputation
system underlying science
Where We Are
• To judge software, need to understand/forecast impact
• Q: How can NSF ensure that software as infrastructure
continues to appear, without funding all of it?
• A: Incentives
• Many of the issues in developing sustainable software are
social, not technical
• Software work is inadequately visible in ways that “count”
within the reputation system underlying science
Hypothesis: better measurement of
contributions can lead to rewards
(incentives), leading to career paths,
willingness to join communities, leading to
more sustainable software
Moving Forward - NSF
• Recent CISE/ACI & SBE/SES Dear Colleague
Letter: Supporting Scientific Discovery through
Norms and Practices for Software and Data
Citation and Attribution (NSF 14-059,
http://www.nsf.gov/pubs/2014/nsf14059/nsf14059
.jsp)
– Need well-developed metrics to assess the
impact and quality of scientific software and
data
– Explore new norms and practices for software
and data citation and attribution, so that data
producers, software and tool developers, and
data curators are credited
• 6 projects and 3 collaborative workshops funded
Moving Forward - Dan
• Products (software, paper, data set) are
registered
– Credit map (weighted list of contributors—
people, products, etc.) is an input
– DOI is an output
– Leads to transitive credit1
• E.g., paper 1 provides 25% credit to software A, and
software A provides 10% credit to library X -> library
X gets 2.5% credit for paper 1
• Helps developer show: “my tools are important”
– Issues:
• Social: Trust in person who registers a product
• Technological: How2, Registration system
1D.
S. Katz, "Transitive Credit as a Means to Address Social and Technological Concerns Stemming from Citation
and Attribution of Digital Products," Journal of Open Research Software, v.2(1): e20, 2014. DOI: 10.5334/jors.be
2D. S. Katz, A. M. Smith, "Implementing Transitive Credit with JSON-LD," 2nd Workshop on Sustainable Software
for Science: Practice and Experiences (WSSSPE2), 2014. URL: http://arxiv.org/abs/1407.5117
Moving Forward - Community
• Career paths – Is there a role for non-tenuretrack researchers who produce software, data,
etc. in universities?
– Assuming yes, do universities recognize and support
this? If not, how to get them to?
• What is needed to support reproducibility of
science, in terms of data and software?
• Lots of entities with similar interests in both
software and data, e.g. JISC, RCUK, NIH, DOE,
Sloan & Moore, Mozilla, Apache, etc.
• Participate in WSSSPE
• Other ideas and questions are welcome, now or
later
– dkatz@nsf.gov or d.katz@ieee.org
Resources
•
•
•
NSF Software as Infrastructure Vision:
http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf12113
Implementation of NSF Software Vision:
http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=504817
Software Infrastructure for Sustained Innovation (SI2) Program
–
–
–
–
•
Working towards Sustainable Software for Science: Practice and Experiences
(WSSSPE)
–
–
–
•
Home: http://wssspe.researchcomputing.org.uk (includes links to all slides & papers)
1st workshop paper: http://arxiv.org/abs/1404.7414
2nd workshop site: http://wssspe.researchcomputing.org.uk/wssspe2/
NSF 14-059: “Dear Colleague Letter - Supporting Scientific Discovery through
Norms and Practices for Software and Data Citation and Attribution”
–
•
Scientific Software Elements (SSE) & Scientific Software Integration (SSI) solicitation:
http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf14520
2013 PI meeting: https://sites.google.com/site/si2pimeeting/
2014 PI meeting: https://sites.google.com/site/si2pimeeting2014/
Awards: http://bit.ly/sw-ci
http://www.nsf.gov/pubs/2014/nsf14059/nsf14059.jsp
Transitive Credit Papers
–
–
http://dx.doi.org/10.5334/jors.be
http://arxiv.org/abs/1407.5117
Credits:
•
SI2 Program:
– Current program officers: Daniel S. Katz, Rudolf Eigenmann, William
Y. B. Chang, John C. Cherniavsky, Almadena Y. Chtchelkanova,
Cheryl L. Eavey, Evelyn Goldfield, Sol Greenspan, Daryl W. Hess,
Peter H. McCartney, Bogdan Mihaila, Dimitrios V. Papavassiliou,
Andrew D. Pollington, Barbara Ransom, Thomas Russell, Massimo
Ruzzene, Nigel A. Sharp, Paul Werbos, Eva Zanzerkia
– Formerly-involved program officers: Manish Parashar, Gabrielle
Allen, Sumanta Acharya, Eduardo Misawa, Jean Cottam-Allen,
Thomas Siegmund
•
WSSSPE:
– Organizers: Daniel S. Katz, Gabrielle Allen, Neil Chue Hong, Karen
Cranston, Manish Parashar, David Proctor, Matthew Turk, Colin C.
Venters, Nancy Wilkins-Diehr
– WSSSPE1 Summary paper authors: Daniel S. Katz, Sou-Cheng T.
Choi, Hilmar Lapp, Ketan Maheshwari, Frank Löffler, Matthew Turk,
Marcus D. Hanwell, Nancy Wilkins-Diehr, James Hetherington,
James Howison, Shel Swenson, Gabrielle D. Allen, Anne C. Elster,
Bruce Berriman, Colin Venters
– WSSSPE1 Keynote speakers: Phil Bourne, Arfon Smith