RMS' SIX-MONTH PROGRESS
REPORT PLAN
JANUARY 2012
1
TABLE OF CONTENTS
•
•
•
•
•
•
•
Introduction
Timeline
Purpose of the six-month progress report
Client satisfaction & quality of work measures
Internal processes measures
Costs measures
Implementation progress measures
2
INTRODUCTION
• The six-month progress report will measure and
evaluate the implementation, operations and
performance of 1 RMS team (out of 10 teams
total in end-state).
• Six-month progress report audience includes
Deans, MSOs, Chairs, Faculty, OECC, OE Faculty
Oversight Committee, RMS internal purposes,
and the implementation project team.
3
PURPOSE
• Guide for ensuring successful implementation
• Establishing an accurate record of development as
it occurs
• Guide for decision-making in on-going operations
• Benchmark progress toward achievement of goals
4
TIMELINE
8/15/11- 2/14/12: RMS six-month
internal evaluation period
February 2012: Data collection
conducted by the project team
March 2012: Data analysis and
discussion of conclusions
April 2012: Final progress report
distributed to the campus
5
CLIENT SATISFACTION & QUALITY OF
WORK MEASURES
Descriptive or
Operationalized
Evaluative Question Measure Description
Data Collection
Method
Evaluative Criteria or
Performance
Measurement Target
Percent of submitted
Team Manager and
What is the quality of proposals rejected by
Contracts & Grants
proposals developed the sponsor due to
records and Team
by RMS?
critical errors within
Manager interview.
the control of RMS
Acceptable quality=
0%
Unacceptable
quality= more than
0%
Percent of proposals Team Manager and
What is the quality of not submitted to
Contracts & Grants
sponsor by deadline records and Team
RMS' work?
due to RMS' errors Manager interview.
Acceptable quality=
0%
Unacceptable
quality= more than
0%
6
CLIENT SATISFACTION & QUALITY OF
WORK MEASURES, CONT'D
Operationalized
Descriptive or
Measure
Evaluative Question
Description
Data Collection
Method
Evaluative Criteria
or Performance
Measurement
Target
• For Feb 5 deadline
proposals, both
How many proposals
Number/percent of
RMS and C&G
• Compare RMS to
are sent to the
C&G results.
proposals returned to
record which
authorized
proposal developer by
proposals are
• N/A: Developing
institutional signatory
baseline data.
proposal reviewer.
returned to
with errors?
administrators by
reviewers.
7
CLIENT SATISFACTION & QUALITY OF
WORK MEASURES, CONT'D
Descriptive or Evaluative
Operationalized
Measure Description
Question
Data Collection Method
1.5 hour focus group
Department Manager's
facilitated by PMO
How do Department
self-reported experience:
project team during
Manager clients feel
last 2 weeks of Feb
 what was successful
about RMS? What was
2012
 what needs
their experience over the
improvement
• short Likert scale
six-month Phase 1A
survey distributed at
 general attitude
period?
towards the RMS
the start of focus
group
• 1.5 hour focus group
Post-award Analyst's
facilitated by PMO
project team during
How do Post-award self-reported experience:
last 2 weeks of Feb
Analyst clients feel about  what was successful
2012
RMS? What was their  what needs
improvement
experience over the six• short Likert scale
survey distributed at
month Phase 1A period?  general attitude
towards the RMS
the start of focus
group
Evaluative Criteria or
Performance
Measurement Target
•
During last 30 minutes of
focus group, facilitate to
find consensus on most
important messages to
report in the 6-mth
progress report to the
community.
During last 30 minutes of
focus group, facilitate to
find consensus on most
important messages to
report in the 6-mth
progress report to the
community.
8
CLIENT SATISFACTION & QUALITY OF
WORK MEASURES, CONT'D
Descriptive or
Evaluative Question
What is the average level
of satisfaction among
faculty clients over the sixmonth Phase 1A period?
And, is that level of
satisfaction acceptable?
Operationalized Measure Data Collection
Description
Method
Faculty clients' post-proposal
submission and self-reported
satisfaction attitude of RMS preaward services as measured by
on-line survey 5-point Likert scale
response to: "Overall I'm satisfied
with the pre-award services
provided by my RSC."
Did faculty satisfaction
levels significantly change Differences in faculty satisfaction
survey responses over the sixover the six month period?
month Phase 1A period.
• Are there patterns of
• Inductive content analysis of
types of customer
open-ended questions to
service issues?
determine patterns in issue
• Are there patterns of
types.
satisfaction levels across • Frequency distribution of
departments?
respondents by department.
Evaluative Criteria or
Performance
Measurement Target
82% average satisfaction level
• Distribute five
performance target from the pretimes during the
Phase 1A campus-wide
six-month Phase
satisfaction questionnaire results.
1A period, a
• Acceptable= greater than 82%
"How's My
satisfied
Driving?" on-line
• Unacceptable= less than 82%
survey to faculty
satisfied
that recently
submitted
Measure change of average
proposals via
customer satisfaction levels
RMS.
during six-month period.
• Pre-Phase 1A
period Summer
2011 campus- • Same issue type repeated by
wide Faculty Predifferent respondent more
award
than once.
Satisfaction
• Significant department
Questionnaire
representation is greater than
results.
25% of responses from one
9
department.
INTERNAL PROCESSES MEASURESSTAFF SATISFACTION
Descriptive or Evaluative
Operationalized
Measure Description
Question
What is the job
satisfaction level of RMS RMS staff satisfaction
Associate RSC?
overall and with specific
components of the RMS
model:
What is the job
satisfaction level of RMS  training
 workload
RSC?
 operations
 team environment
What is the job
 challenges
satisfaction level of RMS  positives aspects
 expectations met
Team Manager?
 implementation
process
What is the job
 resources availability
satisfaction level of RMS
Director?
Staff attrition
Number and reason for
Phase 1A staff attrition
Data Collection Method
Evaluative Criteria or
Performance
Measurement Target
• ARSC group interview facilitated
by project team;
• short Likert scale survey
distributed at the start of focus
group
• Two RSC small focus groups;
facilitated by an external Meeting
Facilitator;
• short Likert scale survey
Exploratory, inductive
distributed at the start of focus
group
content analysis of interview
• Interview the Team Mgr
responses
facilitated by project team;
• short Likert scale survey
distributed at the start of focus
group
• Interview the Director facilitated
by project team;
• short Likert scale survey
distributed at the start of focus
group
Interview with Team Manager;
Natural attrition= good;
short Likert scale survey distributed Termination or resignation due to
at the start of focus group
job dis-satisfaction= concern
10
INTERNAL PROCESSES MEASURESPERFORMANCE METRICS
Descriptive or
Operationalized
Evaluative Question Measure Description
Data Collection
Method
Evaluative Criteria or
Performance
Measurement Target
Compare progress
Proposal Express and
toward end state target:
TM records
83 proposals/RSC
N/A: monitoring data,
Efficiency
TM records and
not evaluating or
RSC to assigned PI ratios
proposal express
performance targets
targeting a specific goal
Team Managers and
Five days or less= good
other certifiers'
RMS internal review and
More than 5 days=
Cycle times
approval time average estimates via interview
inadequate
questions
Pre-award activity
Reports submitted
reports sent to each
monthly to all
Accountability
Count shared drive
Department client
departments= good
records
monthly
less than monthly=
inadequate
Efficiency
performance targets
Average annualized
number of proposals
developed per RSC
11
INTERNAL PROCESSES MEASURESTRAINING PROGRAM
Descriptive or
Operationalized
Evaluative Question Measure Description
Data Collection
Method
Participant self-reported
What were the
opinions on training
participants' reactions to
• On-line survey
program experience on
the Ph1A and Ph1B
• RSC focus group
the last day of program
question
Immersion Training
and six months after for
Program?
Ph1A
Did the participants' pre- Gains in subject matter
award knowledge
knowledge and
Learning objectives
increase after their
confidence as evidenced survey administered on
first and last days of
participation in the Ph1A by accurate answers to
questions in a learning Ph1A and Ph1B programs
and Ph1B Immersion
objectives survey
Training Program?
Participant on-the-job
Are Ph1A participants'
applying information application of Immersion Progress report from
Team Managers via
training curriculum as
taught during the
interview questions
observed by their Team
Immersion Training
Managers
Program on-the-job?
Evaluative Criteria or
Performance
Measurement Target
N/A: monitoring data,
not evaluating or
targeting a specific goal
Increase in knowledge
and confidence=
adequate
Decrease in knowledge
and confidence=
inadequate
Exploratory, inductive
content analysis of
responses
12
COST MEASURES
Descriptive or
Operationalized
Evaluative Question Measure Description
Data Collection
Method
Proposal Express,
What is the cost per
Proposal Activities
proposal at this point Phase 1A annualized
report from Team
in the implementation costs of RMS divided
Manager, Weblinks,
process?
by annualized number
salary agreements for
And, is it on track to
of proposals
staff not yet rolled in,
meet the end state
developed by RMS
other financial records
cost target?
kept by the EVCP
Evaluative Criteria or
Performance
Measurement Target
• Compare to end
state target=
$2,300/proposal
• Compared to Ph1A
departments' preimplementation
baseline cost per
proposal range
• If within range=
adequate, above
range= inadequate
13
IMPLEMENTATION PROGRESS MEASURES
Descriptive or
Evaluative Question
Operationalized Measure
Description
Data Collection
Method
Evaluative Criteria or
Performance
Measurement Target
Implementation
schedule
Extent to which implementation is
following original schedule
Project team interviews
N/A: monitoring data to
understand impacts of scheduling
timelines, not evaluating or
targeting a specific goal
Implementation
environment
List of emergent challenges or
unexpected events that occurred during
the six-month implementation period
and the resulting short-term impacts to
the implementation process and the RMS
model, and possible speculation on longterm impacts.
Interviews with Director,
Team Mgr, Project Team
N/A
Were any of the components of the
operations model not implemented?
Model integrity
Were any of the components of the
operations model not implemented?
Any new activities or processes not
originally planned in model? short and
long term impacts?
Determine if changes from original
model are:
RSC focus group and TM
interview question; project • Planned or natural variations
team report
• Vision or implementation
failures
14