Mark Tregar, MS
Real World Consulting
Organizational Performance
Measurement
© 2013 IBM Corporation
Agenda
1.
2.
3.
4.
5.
Introduction and Bio
Career Path
What is Consulting Really?
Example of Organizational Performance Management Project
Questions?
© 2013 IBM Corporation
Mark Tregar - Bio
© 2013 IBM Corporation
Levels Across Firms: An example of my career path
IBM
Deloitte
CNA
Partner/Principal/Director
Leaders
Sr. Managing Consultant (B9)
Manager
Associate
2
Specialist
1
Assistant
Associate Partners (B10)
Sr. Manager
Analysts
Senior Consultant
Consultant
Analyst
Partners (B10+)
3
Managing Consultant (B8) 4
Senior Consultant (B7)
Consultant (B6) – CBD Prog
Analyst/Assistant
• My career trajectory has spanned 3 organizations in the past 8 years.
• Possible UB graduates entry point: Between consultant and senior consultant!
© 2013 IBM Corporation
Why Companies Hire Consultants
1)
2)
3)
4)
5)
6)
Staff augmentation
External change force aka “political cover”
Best practices across industries and functions
Analytical horsepower
Fresh perspective
Training and skillset growth of staff
Management consulting is the practice of helping organizations to improve their
performance, primarily through the analysis of existing organizational problems and
development of plans for improvement. Organizations may draw upon the services of
management consultants for a number of reasons, including gaining external (and
presumably objective) advice and access to the consultants' specialized expertise.
http://en.wikipedia.org/wiki/Management_consulting
© 2013 IBM Corporation
USAID HR LOB CHRRPS
Technical Integration Support
Project
A Review of Performance Measures Work at USAID
November 21, 2013
Table of Contents
1. Background
2. Purpose and Approach
1. Conceptual Design
2. Detailed Design
3. Data Gathering
4. Data Reporting and Analysis
3. Next Steps and Lessons Learned
8
9
9
10
13
14
16
© 2013 IBM Corporation
The purpose of the Performance Measure effort was to aid
the measurement of the overall health of HR operations.
 The team identified a series of specific Key Performance Indicators
(KPIs) and then selected the appropriate KPIs to provide an accurate
measure of current HR operations for USAID.
– The identified KPIs were reported and measured in a one-time scorecard using
Balanced Scorecard-based categories
– Our approach to selecting KPIs was adapted from the 2009 OPM Shared Service
Center (SSC) Human Resource Line of Business (HR LOB) Benchmarking Process,
which was piloted with 8 agencies and expanded to 12 in 2010
Steps
Figure 1. Our 4-Step Approach Adapted from OPM’s SSC HR LOB Benchmarking Steps
Conceptual
Design
Deliverables
Nov
Data
Gathering
Data
Reporting
and Analysis
Dec-May
May-Jul
Jul-Sep
Scorecard
Template
ID of 10 OHR Process and
HR LOB System KPIs
Initial Performance
Report
Delivered:
12/12/12
Delivered:
5/24/13
Delivered:
8/9/13
Detailed
Design
© 2013 IBM Corporation
In Conceptual Design, IBM developed a conceptual model
Scorecard Template and gathered feedback on inclusions.
We envisioned a future state report/scorecard with two levels and four
Balanced Scorecard Categories:
 Level 1 - Top OHR USAID
Management Report/
Scorecard reports highlevel results (i.e., Customer
Service) with an overall
status for a given time
period.
 Level 2 - HR Management
and Staff Report/Scorecard
reports at a lower level of
results (i.e., results for a
specific metric) for a given
time period.
© 2013 IBM Corporation
In Detailed Design, the team refined our original list of 135 KPIs,
to a smaller set that were most relevant to USAID.
 We cut our list to 49 preliminary KPIs* based on initial performance
measure research and a series of meetings with key stakeholders.
– Our original list of 135 was an agency list with additional potential KPIs from
IBM’s Benchmarking wizard
– In addition, we added KPIs based on our first focus group which included HR
Staff members and meetings across HR.
 To gather feedback we conducted workshops with the following
groups: HR leaders, HR staff (Customer Service Group), 3 HR
teams, and one contractor team.
Original 135
Baseline Key
Performance
Indicators (KPIs)
135
KPIs
Refined set of
49 KPIs
following initial
stakeholder
meetings
45
KPIs
Set of 45 KPIs
following initial
research
* The umber of KPIs ranged between 26 and 55
throughout the stakeholder engagement process.
49*
KPIs
54
KPIs
Modified list of 54 KPIs
following additional
stakeholder
discussions
End
Result:
Final KPIs
selected
and then
measured
© 2013 IBM Corporation
In Detailed Design, we used a series of questions to help
leadership stakeholders cut KPIs
Discussion Items:
1. Which KPIs are most valuable for selection and use? If so, why?
2. Which KPIs should be removed from our potential list?
Example KPI: HR Servicing Ratio - This metric measures the number of employees receiving HR services
from Agency HR employees. The metric provides insight into the size and support level of HR.
UB Discussion: What are some others that could be important to an agency?
Figure 1. Items to consider in reviewing KPIs:
Is the potential
measure
Actionable?
The data is under
control of the
agency (or other
provider) currently
available and
practical to collect.
Is the potential
measure
Applicable?
The measure is
relevant to
government
processes and
supports USAID
HR goals.
Is the potential
measure
Meaningful?
Is the potential
measure
Detailed?
Is the potential
measure Easily
Understood?
The data will be
meaningful to
decision makers and
the derived value is
greater than cost of
collecting.
The scope of the
measure is defined
and is quantifiable
to the extent
possible.
The data measured
is clear and is easy
to understand.
© 2013 IBM Corporation
Following Detailed Design discussions, the IBM team
rated on Relevance and Data Availability
 The feedback we received from the HR Director and targeted HR
groups helped us understand the key priorities for OHR.
•
•
•
Based on this understanding of USAID,
we rated each KPI on Relevance and
Data Availability
The sum of these two ratings forms the
Overall KPI Usage Score
Each KPI was also cataloged into one
of 12 categories (e.g., Time to Hire,
Processing) to help understand KPI
type
• The final KPI determination for use
was based on Overall KPI Usage
Score and representation in 12
categories.
•
This allowed for targeting KPIs with
available data, while avoiding
oversampling in categories
•
Relevance Rating: (Likert scale of 1-5)
–
–
–
–
–
•
1: Not Relevant to OHR Priorities
2: Little Relevance to OHR Priorities
3: Some Relevance to OHR Priorities
4: Moderate Relevance to OHR Priorities
5: Very Relevant to OHR Priorities
Data Availability Rating: (Likert scale of 1-5)
–
–
–
–
–
1: Data is not available in any format
2: Data is available sporadically and is maintained
less than monthly
3: Data is available but in a static spreadsheet format
4: Data is fully available in a database and available
to the team indirectly
5: Data is fully available in a database and available
to the team directly
Relevance Rating + Data Availability Rating =
Overall KPI Usage Score
© 2013 IBM Corporation
In Data Gathering, the team met with groups such as HR
Training and Education (HR T&E) to locate and collect data
• In USAID, each KPI had a specific HR owner who had access to raw
data and could provide data to the team.
•
•
•
•
•
•
•
The team interviewed the data owners and tracked the status of data collection,
the data source (identification of specific data base), and who could help us
collect the data.
The team also tracked the specific parameters of each data element (e.g., the
date range, and specific queries used in WA, Peoplesoft, or other databases) in
order to validate the metrics at a later date
Sample KPI Discussion Questions:
In our current list of KPIs (right), which KPIs do
you currently track?
If you track this information, where is it
currently stored (e.g., Excel, a database)?
What metrics do you measure monthly?
Annually?
Which KPIs are most important to your team?
Least important?
In your ideal scorecard, which KPIs would be
reported?
List of Potential T&E KPIs Used in
Discussion:
Automated ILTP Metrics, for example:
- Percentage of staff currently entering ILTPs
Post-Course Training Evaluation Ratings
Percentage of students who feel the knowledge
they gained from the training will be applied on
their job
Percentage of students who feel training will
improve their job performance
Percentage of staff taking USAID University
Training
© 2013 IBM Corporation
In Data Reporting and Analysis, the team successfully
collected and analyzed all data
• Each KPI was reported across 3 periods of time
• We compared each result to IBM’s Benchmark in terms of median and a
specific public sector median.
• The data was shared with the HR director in the form of the Initial
Performance Report. The HR Director reported our findings in multiple
sessions with her Agency Leadership Council (ALC) and the USAID
Administrator, Dr. Rajiv Shah
• Similar data analysis can be found in annual HR LOB Reports
– http://www.opm.gov/services-for-agencies/hr-line-of-business/benchmarking/payrollbenchmarking/2011report.pdf
– http://www.opm.gov/services-for-agencies/hr-line-of-business/benchmarking/hrbenchmarking/2010report.pdf
© 2013 IBM Corporation
Sample of results in 2010 OPM HR LOB Report
– Agency HR Servicing Ratio: This metric measures the number of employees receiving
HR services from Agency HR employees. The metric provides insight into the size and
support level of HR.
– Source: MAESC 2010 HR LOB Benchmarking Report - http://www.opm.gov/servicesfor-agencies/hr-line-of-business/benchmarking/hr-benchmarking/2010report.pdf
© 2013 IBM Corporation
HR Leaders are moving forward with a new BI Tool Deployment
Next Steps
•
•
HR leaders are moving forward with BI Tool Deployment based on our ideas
We hope to work with USAID later this year
Lessons Learned
Do:
– Get leadership’s input and approval before discussing the effort with other leaders and
managers.
– Use multiple ways of gathering information. Even if you are promised data, it may not
be available as needed.
– Create ratings based on your understanding of KPIs, and work to validate this
understanding throughout the process.
– Encourage transparency with the initiative’s goals, expected outcomes, and level of
accountability.
Don’t:
– Hold up your progress for one person.
– Look for one person to have all the answers.
– Commit to more KPIs or data collection than necessary for your primary objective.
© 2013 IBM Corporation
Thanks!
Mark Tregar
mrtregar@us.ibm.com
© 2013 IBM Corporation
PBC 2012 General Information
Picture
Here
Utilization
Name:
Mark Tregar
Title:
Managing Consultant
Band/Year in Band:
SAM Name:
8 / 9 Months
Stephen Hutton
Are you a SAM?
Yes
No x
Client Delivery
PA Name
Chargeable:
59%
USAID
O
O
O
O
O
Total
(110% Target):
112%
WellPoint
VG
VG
O
O
VG
Results
Overall, I was very successful in my first 9 months including solid client contributions,
Comments:
as well as B&P and MOS project success. At USAID, I received “O” ratings and high
client satisfaction. At WellPoint, I stabilized a challenging client environment which
supported our win. At CPAC, I completed all deliverables on time and on budget.
Business Development
Comments:
• Led USAID’s Performance Measures development task,
while providing competency management expertise in the
launch of the USAID LMS.
• Received positive client feedback for USAID performance
such as: ”I've been impressed with the quality of his work
products and his ability to dig right in…and figure it out.“
• Provide project management expertise at USAID including
Financial and labor planning, and creating Project Launch
documentation.
• Led WellPoint communications team in a complex and
competitive environment.
• Led CPAC team delivering all deliverables on time and on
budget, and developed executable plans for future work
• Analyzed and presented performance management
statistics to NASA HR leaders and received high client praise.
Lead or Mentor People
Client Proposal
(Type Work)
Individual
Contribution
VA HR LOB (New)
Section Lead (Cost
Estimate, Admin)
Win
OPM TMA (Recompete)
Section Lead
TBD
TCV
Result
USAID
Role (months)
Team Lead / DPM
# of resources led/mentored
Dept. of Energy (New)
Contributor
TBD
DHS S&T SETA IDIQ
Section Lead
TBD
Comments:
Current Client
Provided expert input across Federal B&P efforts: Led the VA
HR LOB cost proposal input development by interviewing proposal leaders, and
facilitated large proposal editing sessions. I authored a section of the OPM TMA
proposal and co-authored the VA Social Media RFI response. I provided research
and analysis for the Dept. of Energy solution and DHS S&T SETA IDIQ response.
2nd Client
Role
1-2
WellPoint
Team Lead
Role (months)
# of resources led/mentored
1
Comments:
3rd Client – VA CPAC, Role: Team Lead; Resources led/mentored: 1
Provided mentorship and leadership to all lower level staff, as well as
clear and consistent feedback on all products and deliverables.
Practice Giveback
Giveback
Bus &
FM
90%
Contributed to IBM
with work on over 10 proposals
including VA HR LOB, OPM TMA, DHS
S&T SETA IDIQ, Dept. of Energy, VA
VISN 5, and State of New York.
Recommended PBC Rating:
Relation
Prof Eff Leading
-ships
Chargeable Target:
Comments:
SAM Area: Administrative Compliance:
PA Score Average:
PA Results
Thought Leadership
Date
Thought Leadership/Strategic
Contribution
Role
Date
Developed an improved Public Sector
Competency Development Methodology
Lead
Sep-Jan 2013
Social Media for Human Capital White
Paper
Lead & Author
Nov-2012 (In Review)
Project 517/Career Pro – Competency Team Lead
Team Lead
Apr-Aug 2012
Department of Energy (DoE) Vforce
Contributor
April 2012
Comments:
costs) at DoE HQ.
Comments:
© 2013 IBM Corporation
Overview of Competency Based Management
Competencies embody an individual’s total experience, including both training
and education.
A competency is an observable, measurable pattern of skills, knowledge, abilities,
behaviors and other characteristics that an individual needs to perform work roles or
occupational functions successfully.
Knowledge - What you know about
the work.
Skills - Individual know-how and
techniques related to the work.
Ability - Your capability and capacity to
accomplish the work.
Behaviors & Characteristics - Your
attitude and state of mind.
© 2013 IBM Corporation
19
Relationship between KSAs and Competencies
A competency is a “bundle” of the KSAs associated with a given area of work. It is the
description of the observable behaviors that makes it different from a single K, S, or A.
Competency: Internal
Audit and Control
Knowledge
Understand
accounting and
administrative
controls
Review and validate
accounting and administrative
controls to safeguard the
integrity of programs and
achieve compliance in
accordance with government
auditing standards
Abilities
Able to audit
standards when
dealing with
financial
information
Skills
Operate internal
audit software
programs to
validate
compliance
A competency is more than just the sum of knowledge, skills, and abilities
© 2013 IBM Corporation
20
Competency Development Process
Plan
Plan, envision, and
communicate
competency effort
• Define Objectives
• Build business case for
competency model
• Link model to strategic
plan
• Collect Background
Materials
• Request existing
competency models,
position descriptions, etc.
• Review previous related
IBM competency models
• Research and Review
• Review strategic plan, and
organizational background
• Review related
occupational models
• Develop and Execute
Communications Plan to
Build Sponsor Support
• Deliver kick-off brief
• Collect feedback and
create additional briefings
21
Develop and
Refine
Initiate
Select and initiate specific
competency modeling
approach
• Identify data collection
methodology to meet
project objectives
Develop and refine
competency models
• Develop/refine
competency model
using Resource Panels
1
• Identify proper mix of
interviews, focus groups,
and surveys
• Identify & Select SMEs
• Work with sponsor to
identify Resource Panel
SMEs and Incumbent
SMEs
• Identify required SME
population details (i.e.,
numbers of staff,
occupations, levels)
• Initiate effort by inviting
attendees to Resource
Panel
• Support effort with the
use of an executive
memo or message of
support for effort
• Hold 2-3 facilitated
sessions with SMEs
• Output: Straw man
competency model
• Refine competency
model using Incumbent
SMEs via focus group or
survey [initial validation]
2
• Refine model using SME
input (i.e., ratings,
behavioral examples)
and factor analysis (100+
SMEs required)
• Output: Interim
Competency Model
• Develop final
competency model
3
• Analyze data and
develop final model
• Present final model to
sponsor
• Output: Final
Competency Model
Validate and Apply
Validate competency
models and apply models
in various applications
• Conduct final validation
survey with Incumbent
SMEs for preparation in
deeper applications
• Develop and refine
competency survey
behavioral anchors
• Execute competency
validation survey
• Note: This can
double as a
competency
assessment
(including
proficiency items)
• Output: Validated
Competency Model
• Apply model in
workforce applications
• Includes workforce &
succession planning,
selection*, training
needs assessment)
© 2013 IBM Corporation
An Example Of Data Collection for Competency Modeling
• Behavioral Event Interviews (BEI) are a type of structured interview
use for collecting competency data from SMEs
• BEIs can be used to collect additional qualitative information for
use in competency model development.
• BEIs often use a Situation-Task-Action-Reasoning/Results format
in response to a prompt (i.e., Tell me about a time when…)
Situation/Task
Action
Reasoning
What was the
situation or
context? What
were you doing?
What task were
you working on?
What did you do?
What were the
steps you took to
get to that
effective
outcome?
What was the
reasoning/
rationale that
led to the
action?
Results
What was the
result/
outcome of
the key
situation?
© 2013 IBM Corporation
22
Class Discussion
1. Why do IBM consultants exist? What is their job – what
is the end result for consultants?
• What types of competencies are useful?
• What are lacking in the workforce today?
 Think of what this looks like on the job: Think of a recent situation
where you helped a client, where you saw this happen (i.e., at work, in
a movie). What types of behaviors helped? How would you classify
those behaviors?
© 2013 IBM Corporation
23
As you move up the career leader there is an expectation of deeper
involvement, independence, and relationship building.
Band 6
Band 7
Band 8
Project Planning
Do research to understand project context
Review project documents (plans, deck)
Ask questions of project manager to clarify
Review project plans and provide
feedback on issues proactively
Volunteer for specific tasks of interest
Lead specific tasks independently
Produce project plans and build task plans
to help customer and staff understand
plans
Act on customer inputs to change plans
Communicate issues promptly to your
project manager, team lead, or customer
Communicate in a clear and professional
manner at all times
Listen to concerns of team members and
change plans and priorities accordingly
Communicate plans directly to customer
and stakeholders
Communication
Communicate issues promptly to your
project manager or team lead
Communicate in a clear and professional
manner at all times
Work Product Development
Create deliverables based on project
plans asking questions as needed
Provide deliverables and products that
have been thoroughly reviewed and QAed by team
Create deliverables based on project
plans, asking questions as needed
As needed Review Band 6 deliverables
and products for an extra layer of QA
Evaluate, Review, and provide feedback to
B6 and B7
Customer Management
Listen to customer concerns and provide
feedback to upper level staff
Listen to customer concerns and provide
feedback to upper level staff
Listen and respond to customer concerns
Create proactive strategies to respond to
problematic ‘themes’
What is this chart for?
Many beginning practitioners do not have a solid idea as to what is expected of them
based on their particular level. This chart make the expectations more clear.
© 2013 IBM Corporation
Applying Competency Models in Recruiting/Hiring
Competency: Financial Management
Interview Question: “In regards financial management, can you tell me a time when you
dealt with a large budget in a complex project that was delivered on time and on budget?
Please be specific as to size of budget, team and any constraints.”
1-Not
Capable
Behavioral Interviewing:
Once a validated model is
created with a full set of
behavioral anchors (i.e., a
model with 1, 3, 5
benchmarks) you can
create behaviorally-based
interview questions.
2-Minimally
Capable
•
•
•
•
3-Capable
4-Fully
Capable
5-Exceeds
Capability
25
•
•
•
•
•
•
Understand asset management principles to budgeting and programming
projects
Know about available tools (unit price books, other contract estimate
tools) in order to conduct complex end-to-end cost estimates
Understand asset management principles for budgeting and
programming projects and able to determine rules for use
Apply available tools in basic situations (unit price books, other contract
estimate tools) in order to conduct complex end-to-end cost estimates
Apply asset management principles to budgeting and programming
projects
Interpret and apply available tools (unit price books, other contract
estimate tools) in order to conduct complex end-to-end cost estimates
Apply asset management principles to budgeting and programming
projects
Selects the best available tools (unit price books, other contract estimate
tools) in order to conduct complex end-to-end cost estimates
Evaluate asset management principles and execution in budgeting and
programming projects
Evaluate the use of available tools (unit price books, other contract
estimate tools) in order to conduct complex end-to-end cost estimates
© 2013 IBM Corporation
All Top Consulting Firms Use Competencies in Hiring
• Companies are looking for examples of your past work. Remember,
history repeats itself!
• What you did in the past, chances are you will continue – good and
bad
• Think about what they are hiring for (look at the job posting) then think
about:
1. What competencies are do they need in their hires?
2. What BEHAVIORS can I talk about?
A pro-tip: Use the STAR response method. It will keep you from going
beyond what people want.
Situation/Task
Action
Reasoning
What was the
situation or
context? What
were you doing?
What task were
you working on?
What did you do?
What were the
steps you took to
get to that
effective
outcome?
What was the
reasoning/
rationale that
led to the
action?
Results
What was the
result/
outcome of
the key
situation?
© 2013 IBM Corporation