RMS Phase 1B Six-month Evaluation
Summary of Results
OE Pre-award Implementation Team
October 2012
Contents
•
•
•
•
•
•
•
•
Introduction
Executive Summary of Phase 1B results
Client satisfaction measures
Quality of work measures
Internal processes measures
Implementation Challenges
Implications and Future Direction
Appendix: Methods
2
Introduction
• The six-month progress report measured and
evaluated the implementation, operations and
performance of 2 RMS teams and 6 departments that
started receiving services in Phase 1B of RMS'
implementation schedule.
• The purposes of the six-month evaluation include:
• Guiding continued successful implementation of RMS
• Establishing an accurate record of organizational
development as it occurs
• Providing data for decision-making in on-going operations
• Benchmarking progress toward achievement of goals

Comparing results to the Phase 1A six-month evaluation
3
Introduction (cont'd)
• The Phase 1B evaluation period is 1/17/12- 7/16/12.
• The Phase 1B RMS Teams included two Teams (D and
E), each comprised of:
 1 Team Manager
 7 Research Services Coordinators (RSC)
 3 Associate RSCs
• The 6 Phase 1B department clients:
–
–
–
–
–
–
Dermatology
Diabetes Center
Laboratory Medicine
Medicine (SFGH divisions and CAPS division)
Microbiology & Immunology
Pathology
4
MEASURES (results
at the six-month mark for both Phases)
Client Satisfaction
Department Managers & Post-award Staff satisfaction
Quality of Work
# of proposals rejected by sponsor due to critical errors
# of proposals submitted by sponsor's deadline
Internal Processes
RMS staff satisfaction
Proposals per RSC
Accountability- Monthly Activity Reports
Ph
as
e
Ph 1A
as
e
1B
*
Summary of Phase 1B Six-month Evaluation Results
Good/ Positive
Neutral/ No change
Needs Improvement
*The results varied between the two Teams; however, results are
reported together in this report.
5
Executive Summary
The overall results for Phase 1B teams are more varied than Phase IA. Although the Faculty survey
response rate was very low and therefore statistically invalid, we listened carefully to the key concerns
that were voiced through focus groups and the surveys. This important feedback is guiding our 'course
adjustments.'
On slides 6- 8 is a brief overview of the Phase IB evaluation results divided into the following categories:



Satisfaction levels
Other performance measures
Key implications and future direction
 Satisfaction Levels
–
–
–
Department Managers, Division Managers and Post-award Staff expressed a range of satisfaction levels, from
satisfied to generally dissatisfied with RMS services.
As in Phase IA, RSCs are perceived by Department Managers and Post-award Staff as responsive and
knowledgeable, and managers like the cross coverage provided by the team.
RMS staff satisfaction levels ranged from satisfied to dissatisfied.
•
–
Some of the lower satisfaction levels may be attributed, in part, to the stress of a peak proposal volume period
compounded by an office relocation within the first few months of operation.
RMS staff appreciate the team environment and increasing their knowledge about research administration.
However, RMS staff are concerned about how to ensure standardization and appropriate workloads and how
to mitigate late proposal notification and last minute proposal submissions by PIs. In addition, the
delineations between pre- and post-award can create confusion.
6
Executive Summary (cont’d)
 Other Performance Measures
– Number of proposals per RSC team average has met and exceeded the
performance target of 83 proposals per RSC team average.
– 100% of proposals were submitted by the sponsor deadline and no
proposals were rejected by the sponsor for critical errors.
– Producing the monthly reports continues to be challenging and
reports were not provided to clients on time.
7
Executive Summary (cont’d)
 Key Implications and Future Direction
•
RMS is committed to improving its services and stabilizing the new organization with
effective processes and systems to provide pre-award services. Two key action items to
address process issues are currently underway:
–
–
•
RMS is leveraging the positive aspects of the teams and using these to ensure consistent high
quality performance across all of the teams.
–
•
RMS is working diligently to foster better relations between pre- and post-award through the
Pre/Post Think Tank, a group established to provide recommendations on improving pre /post clarity
and communications.
RMS is implementing a new process to improve the timeliness of the monthly reports.
The early success of the RMS organizational model is the team environment and the knowledge
sharing that results. As RMS continues to increase the effectiveness of it's internal communications,
this knowledge sharing potentially could increase and affect staff satisfaction levels as well as quality
of work outcomes.
Continuous tracking of key performance metrics is essential for identifying areas for
improvement.
–
–
Volume and peak periods of proposal and other pre-award matters (such as out-going subcontracts)
between Teams may change as Departments change, and re-balancing of a Team's workload volume
may be required.
Preliminarily, staff workload appears to be manageable at the 83 proposal per RSC team average.
However, close monitoring of RSC workload and satisfaction will continue.
8
CLIENT SATISFACTION RESULTS
9
Client Satisfaction
Results- Satisfaction Levels and Positive Feedback
Satisfaction Levels
Department Managers, Division Managers and Post-award Staff expressed a range of
satisfaction levels, from satisfied or neutral to generally dissatisfied with RMS services and
feel there are areas that need improvement.
Faculty were invited to provide feedback via emailed surveys after they submitted a
proposal through RMS; however, responses rates were too low to provide valid results.
Positive Feedback
Department Managers, Division Managers, and Post-award Staff expressed appreciation of:
• responsiveness of the RSCs
• improvements they saw in RMS service delivery over the six-month period
• RMS' staffing cross-coverage because they know the proposals will always get submitted
• document sharing between RMS and clients on the department/RMS shared drive
• proposals being submitted on time
"The first few months were a bit difficult but after meeting with [the RSC] and [the RMS Manager] to
discuss the faculty’s issues, we were able to work through a lot of the problems."
10
Client Satisfaction
Department Managers & Post-award Staff
Results- Areas that Need Improvement
The following issues were identified as top priorities by the respondents:
Areas for Improvement
Action Items
Some Post-award Staff feel that they are now responsible for
work that they consider to be pre-award.
Developed the Pre/Post Think Tank to
provide recommendations for resolving
"The RSC is pushing pre-award work on me. When things come up there roles and responsibilities issues. (See
is push and pull on who does what. For example who responds to the
Appendix for List of Members on Pre/Post
NoA and forwards it to C&G?"
Think Tank)
Department Managers are not consistently receiving monthly
reports, despite requesting them multiple times.
"I've requested a report of proposal activity from the [RMS] Manager
multiple times and still no response."
In some cases, RMS is not consistently uploading the final
proposal documents and internal budgets to Proposal Express.
"I'd like RMS to keep proposal and budget documents in RAS."
Some Department Managers and Post-award Staff want more
proactive communication, especially about status of proposals
and other pre-award matters.
"Our PIs would like [the RSC] to be able to propose solutions and solve
problems without having to consult the PI for every last detail."
Currently revising the monthly reports
process to make it more efficient and
effective, and plan to implement this Fall.
RMS is currently developing exception
reports to help monitor staff's
compliance with the new system's
business processes and identify where
training is needed.
RMS Team Managers are encouraging
staff to be more proactive, and
proactivity is part of the annual staff
performance evaluations.
11
Client Satisfaction
Department Managers & Post-award Staff
Results- Areas that Need Improvement, continued
The following issues were identified as top priorities by the respondents:
Areas for Improvement
Action Items
Some Department Staff expressed a lack of trust in the
RSC, which may result in the Department Staff being
more involved in the proposal development process
than necessary.
RMS Team Managers are encouraging
more in-person meetings between
RSCs, Post-award Staff and Campus
Managers to foster improved
relationships.
"The PIs don't trust the RSC so they send everything through
me [the Post-Award Staff]."
Some Department Staff believe RMS' business systems, Provide ongoing communications to
processes, and tools are implemented too slowly and
campus community via the web,
inconsistently.
Chatter and email updates about
"It is difficult to understand what RMS processes are in place- evolution of RMS tools, systems and
not sure how much of this is related to model vs person's skill best practices.
set."
Some Department Staff would like better
communication about how RMS is managing staff
performance issues.
"While the manager is receptive to feedback, we don't know
how she is addressing performance issues."
Communicate via Schools' Manager
Forums regarding RMS staff
performance management strategies
and practices.
12
QUALITY OF WORK RESULTS
13
Quality of workCritical Errors
During the Phase 1B six-month period for Teams
D and E:
 0% proposals were rejected* by the sponsor
due to critical errors
 100% proposals were submitted by the
sponsor's deadline
* This metric measures any proposal that was rejected by the sponsor,
resulting in loss of potential funding. This does not include sponsor
requests for revisions to a proposal before acceptance as a submission.
14
INTERNAL PROCESSES RESULTS
15
Internal ProcessesRMS staff satisfaction- Level of satisfaction and Positive
aspects of the job
• There was a range of satisfaction levels across the two Teams,
from satisfied to dissatisfied.
• RMS staff greatly appreciate the collaborative and supportive
team environment. It fosters knowledge-sharing and learning.
• Some ARSCs are especially appreciative of the help and training
that the RSCs give them.
• Some RSCs find their workload to be manageable.
• Other positive aspects across the Teams include:
– shared drive tool
– technology tools that enable mobility to work at client departments
location
– flexible peer-review process
16
Internal ProcessesRMS staff satisfaction- Challenges of the job
Challenges
Action Items
Late proposal notification and/or last minute
proposals submissions by PIs is stressful and
affects the level of service/quality that RMS
can provide.
All staff will begin tracking the PI
Notification Date for proposals to identify
areas where this issue is found.
Lack of standardized processes across all
Teams.
Implementing standardized processes is an
on-going effort.
Un-defined ARSC roles and responsibilities
and career path in the RMS organization.
Will form an RMS Workgroup to address this
issue.
Equitable workload distribution
Develop tools for measuring workload
distribution.
Too much workload tracking in multiple
systems and Excel spreadsheets.
Revising the workload tracking process, and
plan to implement in the Fall.
Uncertainty about pre/post roles and
responsibilities.
Developed the Pre/Post Think Tank to
provide recommendations for resolving
roles and responsibilities issues.
17
Internal processesProposal volume- Phase 1B
Average
number of
proposals
per Team
completed
during
Phase 1B
Teams D
and E
average
272
Average
number of
Other Preaward Matters
completed per
Team (e.g. internal
Annualized
average annual
proposals per
Team
proposals, sponsor
correspondence)
235
630*
Phase 1B
Departments'
Pre-RMS
Annual Average
Proposals per
Team
(2009,2010)
580
* Estimated average annual proposals based on less than six-months of fully implemented Phase 1B
Teams. One of the Team's proposal volume was very high in March- May ,and historically 35-40% of
proposal volume occurs during those months for those departments; therefore, the above annualized
proposal volume may be over-estimated.
18
Internal processesProposals per RSC
RMS Average
annualized
proposals per
RSC
Phase 1B
Teams average
Phase 1A
Team
Baseline
Average
annualized
proposals per
RSA
End state
performance target
annual proposals per
RSC
90*
81**
83
66
58
N/A
* Estimated average annual proposals based on less than six-months of fully implemented Phase 1B
Teams. One of the Team's proposal volume was very high in March- May, and historically 35-40% of
proposal volume occurs during those months for those departments; therefore, the above annualized
proposal volume may be overestimated.
** Metric derived from 6 Phase 1B Departments' self-report in Fall 2011.
19
Internal processesAccountability
Measure:
Monthly Activity
Reports
Teams D and E
Number of Monthly Reports
Submitted to Each Client
Department during Phase 1B
• 0 - 2 reports submitted per
Department over six months.
• Most Departments received 0
monthly reports during the 6month evaluation period.
20
Internal Processes- Staff Attrition
Measure:
Staff Attrition
Teams D and E
Number of staff
changes during Phase
1B
0
21
Implementation Monitoring
Implementation Environment- Phase 1B
Implementation challenges impacted the RMS Phase 1B teams
including:
• Team E had a significantly longer transition period because
their assigned space at SFGH was not ready in January, so
they were temporarily located at Laurel Heights until March.
• Transition Services for pre-RMS departments impacted the
Phase 1B Teams because they helped with proposal workload
overflow as needed.
• Implementation of Phase 2 started almost immediately after
Phase 1B go-live date which took project team attention
away from Phase 1B Teams.
22
Implications and Future Direction
• RMS is committed to improving its services and stabilizing the new
organization with effective processes and systems to provide pre-award
services. Two key action items to address process issues are currently
underway:
– RMS is working diligently to foster better relations between pre- and postaward through the Pre/Post Think Tank, a group established to provide
recommendations on improving pre/post clarity and communications.
– RMS is implementing a new process to improve the timeliness of the monthly
reports.
• RMS is leveraging the positive aspects of the teams and using these to
ensure consistent high quality performance across all of the teams.
– The early success of the RMS organizational model is the team environment
and the knowledge sharing that results. As RMS continues to increase the
effectiveness of its internal communications, this knowledge sharing
potentially could increase and affect staff satisfaction levels as well as quality
of work outcomes.
23
Implications and Future Direction (cont'd)
• Many of the measures used in the Phase 1A and 1B evaluations will be used
for on-going performance measurement. With all Teams implemented, RMS
will begin developing its long-term Performance Metrics and Evaluation
(PME) plan that will be linked to its strategic goals and objectives.
– The results of the Phase 1A & 1B six-month evaluations will serve as the
baseline from which to compare future performance of RMS.
– Future data collection efforts will be more closely coordinated with the
Department Chair and MSO to improve response rates.
• Continuous tracking of key performance metrics is essential for identifying
areas for improvement.
– Volume and peak periods of proposal and other pre-award matters (such as outgoing subcontracts) between Teams may change as Departments change, and
re-balancing of a Team's workload volume may be required.
– Preliminarily, staff workload appears to be manageable at the 83 proposal per
RSC team average. However, close monitoring of RSC workload and
satisfaction will continue.
24
APPENDIX: METHODS
25
Client Satisfaction‘How's My Driving?’ Phase 1B Faculty Quick
Survey Methods
• Brief survey delivered to faculty clients to determine what went
well and what needs improvement
• Survey distributed 3 times (plus a reminder to non-responders)
around major proposal deadlines from May- Jul '12.
• Five Likert scale, 2 multiple choice and 2 open-end questions about
service delivery satisfaction
• Email request sent from the RMS Director, Marge O'Halloran
• Phase 1B response rate to this survey was 18%, or 41 faculty
respondents. Results are not valid because of low response rate.
26
Client Satisfaction‘How's My Driving?’ Phase 1B Faculty Quick
Survey Questions
1. Related to your most recent proposal submission, please rate the following:
–
–
–
–
–
My RSC is responsive to my proposal development service needs.
My RSC is knowledgeable about pre-award.
My RSC is resourceful.
My RSC utilizes my time efficiently.
Overall I'm satisfied with the pre-award services provided by my RSC.
2. Do you feel like you spent the appropriate amount of time on the non-scientific
development of your proposal?
3. What was the most positive aspect of this proposal development process?
4. What was the most challenging aspect of the proposal development process?
5. This survey is linked to your email address. Would you like us to contact you to
follow-up on your survey response?
27
Client satisfaction
Department Managers & Post-award Staff
Methods
• Shortly following the six-month mark, two focus groups were
conducted for each Team-- one with Department Managers and
another with Post-award Staff.
– At the focus group sessions, 5-question Likert scale satisfaction surveys
were distributed.
– Phase 1B staff unable to attend the focus group were invited to complete
an open-ended questionnaire.
• Of 18 Phase 1B Department Managers & Division Managers:
– 10 attended the focus group session
– 1 completed a questionnaire
• Of 23 Phase 1B Post-award Staff:
– 11 attended the focus group session
– 2 completed a questionnaire
28
Membership of the Pre/Post Think Tank
Think Tank Co-Chairs: Eunice Chang and Christine Razler
• Department members: Joanne Dang, Bridget
Fleming, Susan Lau, Brandi Moretz, Bill Neely,
Snow Nguyen, Christine Razler, and Jeannie
Wong.
• RMS members include: Catherine Chan, Eunice
Chang, Matty Gilreath, Lani Pettersen, Rejina
Sincic, Lei Lei Win, Sam Yee and one TBN.
29