Prospective Evaluation of NCI’s Physical
Sciences – Oncology Centers Program:
Evaluating trans-disciplinary collaborations
and field convergence
Nicole Moore, ScD, Program Officer, OPSO
Office of Physical Sciences - Oncology, Center for Strategic Scientific Initiatives
(CSSI), National Cancer Institute (NCI), National Institutes of Health (NIH)
March 19, 2013
Physical Sciences-Oncology Centers
(PS-OC) Program 2009-2014
FY09
FY14
Phase I
Pre-Award
Phase I – Physical Sciences - Oncology
Centers: Bringing a new, ‘physical
sciences perspective’ to cancer
research.
 12 U54 PS-OCs
 100+ institutions and 600+
investigators worldwide
 8 of 12 PIs new to NCI
Future
PS-OC Scientific Themes:
 Physics (Physical Laws and Principles)
of Cancer
 Evolution and Evolutionary Theory of
Cancer
 Information Coding, Decoding,
Transfer, and Translation in Cancer
 De-convoluting Cancer’s Complexity
Physical Sciences Perspective
Basic Biology
Research
Basic Cancer
Research
Cancer Technology
Research
Cancer Research w/
Clinical Implications
Pre-Clinical
Clinical
PS-OC Program Goals
The Specific Goals for the PS-OC Program:
Build trans-disciplinary teams and infrastructure to better
understand and control cancer through the convergence of physical
sciences and cancer biology.
To generate new knowledge and catalyze new fields of study in
cancer research by utilizing physical sciences/engineering principles to
enable a better understanding of cancer and its behavior at all scales.
Not looking for new tools to do “better” science, but new perspectives
and approaches to do paradigm-shifting science that will lead to
exponential progress against cancer.
New – “Schools of Thought”
PS-OC Program Structure:
Interactions inside and outside of network
PS-OC Network
External
Research
Communities
PS-OC
Organizing Framework
Project
1
Project
2
Project
3
Project
4
Core B
Core A
Education and Training Unit
Pilot
Projects
Trans-Network
Admin Unit
Core B
Core A
Project
1
Project
2
Project
3
Project
4
Trans-Network
Admin Unit
Pilot
Projects
Organizing Framework
PS-OC
Outreach
Projects
TransNetwork
Projects
Trainee Exchange
Education and Training Unit
Outreach and
Dissem Unit
Outreach and
Dissem Unit
Outreach
Projects
PS-OC Program Evaluation Plan
Recommended by STPI in 2009
“Given the groundbreaking nature of PS-OCs, it is expected that
ongoing evaluation….will be needed to assess progress of developing
physical science-based frameworks.” -PS-OC RFA Concept (2008)
Logic Model
Recommended by STPI in 2010
PS-OC Program Evaluation
2009 - 2013
Prospective Evaluation
Structured Evaluation
Analysis for “in-the-moment“
program improvement
Preliminary outcomes after 3 years of operation
•
Prospective evaluation plan
developed in collaboration with
GAO, NIAID, DPCSPI, OSPA, and
STPI
•
Logic model and progress report
structure developed with STPI
•
•
Data collected via
semi-annual progress
reports
Development of new metrics,
indicators, and tools for monitoring
progress toward program goals
with Discovery Logic
• Partnered with NSF and NIBIB to evaluate the
state of the physical sciences in oncology
portfolio
• Survey of ~400 individuals from physical
sciences in oncology and broader scientific
community
• Independent panel evaluation by non-PS-OC
affiliated NCI and NIH experts
PS-OC Prospective Evaluation
Data collection
1. Semi Annual Progress Reports (PHS 2590)






Progress of research towards aims
All personnel list
Publications, Presentations, Patents
List of trainees (w/ discipline)
Student exchanges and transitions
New collaborations within Center, within Network, or outside
Network
2. Web of Science (Bibliometric Information)
3. Additional information provide by Program Officials
84 Progress Reports Collected to Date (7 per Center)
PS-OC Prospective Evaluation
Data Model and Processing
Challenge: Organize and structure data for monitoring and
analysis in real-time
• Data model developed for visual representation of the structure
of the progress report data
• Used to formally describe data relationships
• Used to design SQL database for data storage
Example relationships:
• Which people contributed
to a project?
• Which projects does a progress
report identify?
• Which publications has a person
written?
PS-OC Prospective Evaluation
iTRAQR
Interdisciplinary Team Reporting, Analysis, and
Query Resource (iTRAQR)
Data Entry
User Interface
Analysis
User Interface
• Stores data entered through the Data Entry User Interface
• Entered data can be queried to support charting and data
export in the Analysis User Interface
SQL
Database
PS-OC Prospective Evaluation
Example metrics and indicators
•
•
•
•
•
Investigators and Trainees
Network and Center Collaborations
Bibliometrics (Impact Factor, Citations)
Trans-Disciplinary Authorship Collaborations
Field Convergence (Physical Sciences/Oncology)
PS-OC Prospective Evaluation
Data outputs
Trainees
Growth of the PS-OC Network
400
350
300
250
200
150
100
50
0
Landscape and Number of Collaborations
Post-Doc
Grad
Undergrad
Other
Year 03
Year 01
Jan Jun Dec Jun Dec Jun Dec
2010 2010 2010 2011 2011 2012 2012
Average Publication Count
Per Collaboration
Effectiveness of PS-OC Program
Elements on Collaboration Outputs
4
3.5
3
2.5
2
1.5
1
0.5
0
Range of Training Opportunities
20
3.38
15
10
0.69
0.79
0.23
5
17
0
15
8
4
0
Research
Projects
Cores
TransPilot
Outreach
Network Projects
Pilot
Projects
Projects
Courses
Hands-On Symposia
Training
Journal
Clubs
Trans-Disciplinary Collaborations
Analysis of authorship collaborations
PS-OC Investigator Collaborations
Pre-Award (2006-2008) vs. Post-award (2009 - present)
25
per year
Node: Network PI (Physical Scientist, Oncologist)
Line: Network Co-authorship PI pair (PS-PS, OC-OC, PS-OC)
Trans-Disciplinary Collaborations
Analysis of authorship collaborations
PS-OC Investigator Collaborations
Pre-Award (2006-2008) vs. Post-award (2009 - present)
25
per year
62
per year
Analysis of Collaborations
Pre-award
29%
Post-award
24%
32%
46%
47%
22%
Physical Sciences PIs
Physical Sciences & Cancer PIs
Cancer PIs
Node: Network PI (Physical Scientist, Oncologist)
Line: Network Co-authorship PI pair (PS-PS, OC-OC, PS-OC)
17% increase in percentage of
trans-disciplinary authorship
collaborations since the start of
the PS-OC program
Field Convergence
Physical sciences & oncology fields
PWS FEA
SMR Hypoxia
PS-OC Network
“Ingredients”
Death
Monte Protein
Galaxy
MS
Carlo
Chip-Seq
RESIC AFM
model
SILAC PET
Microrray nanofabrication
Motility
Karyotyping
TIRF MRI
BIN
ab initio
IHC
Game
Theory ChIP-on-chip
Stochastic
FISH DIC
Behavior
EM
TFM
Automata FCM
JUMNA
IVM
Theoretical /
Computational Physics
Trans-Disciplinary
Research Output
CT
model
Evolution Single cell
Dynamics epigentics
Bayesian
Cell CT Network
Field Convergence
Field Convergence
Development of indicator using publication content
Journal Category
Physical Sciences
Terms
Physical Sciences
Journals
Cancer
Journals
Publications
Publications
1,643
571
Content Analysis on Titles/Abstracts
Terms
Terms
Term Filters
Physical
Sciences
Terms
Cancer
Terms
Cancer Terms
Field Convergence
Analysis of Trans-disciplinary publications by topic
100%
Trans-Disciplinary
Publications (%)
90%
Start of PS-OC program
(September 2009)
80%
70%
60%
50%
40%
30%
20%
10%
0%
2004
2005
2006
2007
2008
2009
2010
2011
2012
Year
30% increase in percentage of trans-disciplinary
publications during PS-OC grant years
Data analysis through June 2012
Field Convergence
Breakdown by investigator discipline
Physical Scientists
Cancer Researchers
80%
100%
Baseline (2004-2008)
Grant Years (2010 - 2012)
Baseline (2004-2008)
Percent Publications
Percent Publications
100%
60%
40%
20%
80%
Grant Years (2010 - 2012)
60%
40%
20%
0%
0%
PS
OC
PS-OC
Publication Category
OTHER
PS
OC
PS-OC
Publication Category
OTHER
Both physical scientists and cancer researchers have increased the
percentage of trans-disciplinary publication output during the grant years
Prospective Evaluation
Impact on Program
• Direct feedback for investigators
• 17% of projects have undergone PI
initiated changes
• Increased reporting quality
• Increased Center connectivity
• Direct feedback for Program Officials
• Ongoing comparison of Centers
• Change program elements (ex. Outreach pilot projects)
• More emphasis on cancer researchers applying
physical sciences principles
Physical Sciences in Oncology
(physics.cancer.gov)
More Information
Larry A. Nagahara, Ph.D.
Email: larry.nagahara@nih.gov
Nicole M. Moore, Sc.D.
Email: nicole.moore@nih.gov