Conducting High-Profile Studies in
Public Policy Settings
Chaitra Hardison
About
RAND
Slide 2
What is RAND?
• An independent, nonprofit research institution
committed to advancing the public good
• Known for quality and objectivity
• Trusted source of expertise, analysis, and ideas
• Focused on the critical issues of our time
• Not a university and not a management
consultant—but with the capabilities of both
RAND's mission is to help improve policy and
decisionmaking through research and analysis
Slide 3
Our research agenda is broad
Science
and
Technology
Public
Safety
Population
and
Aging
Terrorism
and
Homeland
Security
Children
and
Families
Education
and
the Arts
Providing practical
solutions to
complex problems
Health and
Health Care
Infrastructure
and
Transportation
National
Security
Law
and
Business
Energy
and
Environment
International
Affairs
Slide 4
RAND’s key resource is its
~800 professional staff
Social
sciences
Political
sciences
11%
7%
Policy
analysis 10%
Arts &
letters
MD
2%
Behavioral sciences
6%
8%
6%
6%
Bachelor’s
9%
Business & law
Computer
sciences
Doctorate
Master’s
31%
57%
Physical 5%
14%
sciences
Economics
7%
Math, operations
8%
4%
7%
research, statistics
Engineering
Life sciences
International
relations
Supplemented by ~500 adjunct staff
Slide 5
RAND’s presence is global
Cambridge
Boston
Pittsburgh
Santa
Washington DC
Monica
Jackson
New Orleans
Brussels
Moscow
Erbil
Doha
Sydney
Headquarters
Other offices
Field sites
Slide 6
A wide range of high-level policy makers
sponsor RAND work
• US Military and Department of Defense Officials
• Congress
• Local and State Government, Public Services, and
Education Officials
• Non-US Government Officials
• And more…
Slide 7
Research sponsored by high-level policy
makers often faces unique challenges
• High-profile interest
• Unclear (or overly narrow) research questions
• Lay audiences
• Short timelines
• Complex policy landscapes
• Broad dissemination of the work
We consider it part of our job to navigate these issues
Slide 8
Two Research Examples
360-Degree Assessments:
How Useful Are They For
Military Officers?
credit: alexmillos/fotolia
Chaitra Hardison
OSD asked RAND NDRI to provide a study on the
advisability of using 360-degree assessment
The study was requested in response to a
requirement in Section 571 of the FY14 NDAA
“…the Secretary of Defense shall submit to the
Committees on Armed Services of the Senate and the
House of Representatives a report containing the
results of an assessment of the feasibility of including a
360-degree assessment approach, modeled after the
current Department of the Army Multi-Source
Assessment and Feedback (MSAF) Program, as part of
performance evaluation reports”
Slide 11
We were responding to a congressional request for
a report on 360-degree assessment
• The question and scope was vague
• The broader policy backdrop was important
• Lay understanding of 360s mattered
• The timeline was tight
• Congress and DoD leadership were very interested
We shaped our approach to help leaders decide
how best to implement 360s
Slide 12
RAND’s goal
• Provide OSD with
– Information to respond to the NDAA request about using 360degree assessments for evaluation
– Information on 360 more broadly, such as for development
• Explore the following questions
– What is known about 360-degree assessments?
– Are the military services already using 360s? If so, how are they
using them?
– Would it be advisable to implement 360s for development or
evaluation purposes for all officers in the military? Why or why not?
– What implementation challenges should the services be aware of
to ensure success?
Slide 13
RAND’s report includes information
from three main sources
• Research and expert advice on the use of 360
assessments in other settings
• Interviews with military subject matter experts and
stakeholders
• Review of documentation, policies, and statutes on
service performance evaluations and promotions,
including current use of 360 assessment
progress
All three sources inform the interview In
results
presented in the following slides
Slide 14
360 is already currently being used in many
forms in the services
•
•
•
•
Center for Army Leadership Multi Source Assessment Feedback (MSAF)
Commander 360
General Officers 360
Unit 360
•
•
•
•
•
360 at Navy Leadership and Ethics Center
Surface Community Pilot 360 Program
New Flag Officer and Senior Executive Symposium (NFLEX)
Naval Post-Graduate School Flag Officer 360
Center for Creative Leadership Flag Officer 360
•
•
Center for Army Leadership MSAF 360 (pilot program)
General Officer/Flag Officer Joint Staff 360 (pilot program)
•
•
•
•
Leadership Mirror
General Officer 360
Center for Creative Leadership Executive Dimensions
Center for Creative Leadership Benchmarks
Slide 15
Our interviews showed that the spirit of 360
assessments clearly resonate with the services
• Interviewees agreed that achieving the mission at the
expense of people and values is not acceptable
– They want to eradicate toxic leadership and unethical behavior
– They value developing good leadership behaviors
• The literature shows military personnel are interested in 360.
Many service members have chosen it as a thesis topic.
• Each service has implemented 360 (some more than others),
and all have other ways to solicit input from subordinates on
these topics
Slide 16
Interviewees believe 360 is not right tool to
eliminate toxic leaders and misconduct
• It might help if used for development
– For some leaders the feedback would be an eye opener
– For others, it would have no impact
• Evaluation is not ideal
– distrust
– toxic leadership is a training issue
• Expensive way to find a few bad eggs
• Other avenues are already in place for reporting
Slide 17
Is 360 feasible for evaluation purposes?
• Maybe, but there are many unsolved logistical concerns...
– Promotions, due process issues, comments would need scrubbing
– Retaliation, peer sabotage, whitewashing for friends
– Personnel time costs could be exponential
– Culture change supporting acceptance would be needed first
Is 360 advisable for evaluation purposes?
• Consensus says no.
– Would ruin developmental applications
– Incentivizes likeability at the expense of the mission
– Too many context differences to interpret across ratees
– Not at least without lots of extensive small pilot studies first
Slide 18
Is 360 worth the time, effort, and resources for
developmental purposes?
• Consensus from interviews is that it may be useful for some people, under
some circumstances
–
–
–
–
Use on an as-needed basis
Use only for those in leadership roles
Use in leadership courses with dedicated coaches
Could save rater’s time by soliciting subordinate perspectives only
• Existing literature and expert advice supports these sentiments
–
–
–
–
Best when combined with one-on-one coaching (but coaching is costly)
Useful when participant is open to the feedback
Time consuming when done well
However, actual performance improvement is not well supported
Slide 19
We offered three broad recommendations
• Do not incorporate 360-degree assessments into the
officer performance evaluation system at this time.
• Continue to utilize 360-degree assessments for
development purposes within the higher grades and for
people in leadership positions.
• Allow the services to continue on their current paths,
expanding the use of 360s in a way that is tailored to
individual service needs and goals.
Slide 20
Applicants Meeting Minimum Preliminary
Requirements
Improving the
Recruiting and Hiring of
Los Angeles Firefighters
Not Listed
5%
Native
American
1%
Asian
6%
Black
15%
White
42%
Hispanic
31%
Chaitra Hardison
The Los Angeles Mayor's Office asked RAND for
help in improving diversity of LAFD trainees
• In 2014, the LA times raised questions about low
minority selection rates and concerns about nepotism
in the LAFD hiring process
• Mayor Garcetti responded by halting the hiring of Los
Angeles fire fighters
We were asked to make recommendations for
improvements in three months time
Slide 22
We responded to a request for an independent
review of the LAFD hiring process
• The question was overly narrow
• The public safety backdrop was important
• The process involved very unique selection issues
• Lay understanding of fairness was important
• The timeline was tight
• The city and the public were very interested
We shaped our approach to help leaders
understand whether changes were advisable
Slide 23
There were several major
limitations to what we could accomplish
• Due to the timeline, no data could be collected to
validate the process
• The selection process had been halted so none of it
could be observed firsthand
• No archival selection data was available on job
incumbents or trainees
– Elements of the process were recently revised and no
selection had occurred for five years
Slide 24
Research questions and approach
• How can Los Angeles
–
–
–
–
Identify applicants most likely to be successful firefighters?
Ensure equal opportunity throughout the firefighter hiring process?
Increase the demographic diversity of new firefighter hires?
Minimize costs of the hiring process for the city and for applicants?
• Sources of information
– Interviews with stakeholder groups
– Detailed review of the official documentation on the selection
processes used by the city
– Interviews with city officials in charge of the processes
– Focus groups with firefighters to confirm key KSAOs
– Review of existing documentation and research on the LAFD
Slide 25
Key issues in the selection process
• The city needs a better way to narrow the applicant pool
– 13,000 applicants for fewer than 100 positions
– The city asked us to look into using random sampling approach
• Nearly all applicants will be rejected
– This inevitably leads to perceptions of injustice
– The city needs to take steps to reduce that sense of injustice
• The city is recruiting more black applicants than one
would expect, given LA city census information
– Increasing minority numbers is not the answer
– Targeted recruiting of highly qualified applicants is needed
Slide 26
Recruitment of black applicants was
higher than would be expected
LA County Population
(Census Data)
Native
American
0%
Asian
14%
Black
9%
Hispanic
49%
White
28%
Applicants Meeting Minimum
Preliminary Requirements
Native
American
1%
Not Listed
5%
Asian
6%
Black
15%
White
42%
Hispanic
31%
Slide 27
Adverse impact analyses for
the 2013 hiring process
• Most candidates were eliminated by either the written exam or the
requirement to submit their Candidate Physical Ability Test (CPAT)
certification within a specific time window
• Hispanic, black, and female applicants were less likely to take and
pass the written test
• Hispanic, black, Asian, and female applicants were less likely to
submit their CPAT certification within the required time window
• According to data from the organization that administers the CPAT,
94 percent of the people who take the test pass it. Hence, the CPAT
itself does not eliminate many applicants
Slide 28
Recommendations
• Overarching recommendations:
–
–
–
–
Begin a new citywide firefighter outreach and recruiting campaign
Validate the selection criteria
Explore options for reducing the applicant pool to a manageable size
Set aside funding and resources for a robust appeals process for
applicants who believe that they have been wrongly passed over
• We also provided detailed recommendations for changes to
each stage in the process
• We discussed random sampling as an option, but advised
against it in favor of other validated measures
Slide 29
How the work is disseminated
depends on the study
LAFD Study
• Mayor restarted selection before
our deadline and city instituted a
new process
• Radio, TV and print media
reported on our findings
• The press continues to track
reactions to the process
– Minority and female selection
rates are still low
360-Degree Feedback Study
• Sponsor delivered it wholesale to
congress
• Sponsor distributed it widely to
DoD and military decisionmakers
Other Studies
• Sponsors ask us to brief top
officials and key stakeholders inperson
• Key officials ask us to brief them
All studies undergo peer-review; nearly all are published on
the RAND website or in academic journals
Slide 30
Many RAND studies on personnel issues face
these research challenges
The Challenges
Our Responses
Unclear (or overly narrow) questions
Dig to uncover the real problem
High-profile interest
Press releases, interviews, briefings
Lay audiences
Explain the jargon and science
Short timelines
Solutions that fit within the timeline
Complex policy landscapes
Explore the bigger picture, understand
the policy maker’s thought process
Broad dissemination of the work
Presentations, meetings, publications,
1-pagers, research briefs
We don’t just do research, we do research to have impact
Slide 31
Slide 32