MLD-610
MLD-610
Generating and Using Evidence to Improve the Management of your Organization
Spring 2016
Meets Tuesdays and Thursdays 11:45am-1:00pm
HKS 1 Brattle St. Room 401
Julie Boatright Wilson
Taubman – 440
Julie_Wilson@harvard.edu
Assistant: Nicole Grenier
Taubman – 459A
Nicole_Grenier@harvard.edu
Nearly every organization is under pressure to produce evidence that the programs it runs are effective.
And many organizations are under pressure to demonstrate that they are making good use of public
funds or public trust. But how can you as a manager use the same tools and strategies that external
evaluators employ to improve the performance of your organization? What information on current
performance and operations would you need? How would you gather that information? How would you
assess it? And how would you feed it back to those in the organization so that they could work more
effectively to improve overall performance? Finally, what are the risks to you, as a manager, of
undertaking such an initiative?
Gathering and using data to more effectively manage an organization presents some unique challenges.
In this course students will develop a framework for thinking about evidence and apply this framework
through case discussions and group projects to a range of organizations and smaller programs. Students
will learn about different strategies for gathering information and employ them in their group project.
Audience: This course is designed for one-year and two-year students who are interested in developing
the skills and intellectual frameworks for generating and using evidence to improve the performance of
their organizations. Students will develop the skills for gathering and analyzing data on organizational
strategy and performance.
Course Outline: The course begins by laying out a framework for thinking about evidence. We will then
work through each part of the framework in depth: question zero, theories of change, logic models,
output and outcome measurement, performance measurement, implementation analysis and
management and measures of effectiveness. In the process, students will learn how to conduct indepth interviews and focus groups, undertake systematic observation, and construct and administer
surveys. We will discuss sampling strategies and students will apply them to their group projects.
Finally, we will spend time on the ethics of research and data gathering as well as strategies for
managing organizational change based on the evidence generated.
Class Meetings: This course will meet Tuesdays and Thursdays from 11:45am to 1:00pm in HKS 1 Brattle
St. Room 401. Class sessions will include lectures, discussions, and presentations. We may supplement
the class session with optional outside speakers.
Text and Readings: All readings are on line or on the Canvas website.
Specific Requirements: Students will be responsible for completing the assigned readings before class
and coming to class prepared to discuss the materials. Because a major focus of this course is skills
MLD-610
Page 1
MLD-610
development, we will also be practicing many of the techniques in class and discussing what lessons we
draw from our experiences.
Students will work in small groups for a unit within an organization, applying the class materials to
assess the unit’s performance. Students will have both individual assignments, primarily reflecting on
experiences applying materials, and group assignments reporting on the progress in getting to
understand and make recommendations for your organizational unit.
A number of units in the Kennedy School have volunteered to have students help them develop a plan
for measuring performance and effectiveness. The advantage of using internal HKS units is that the
turn-around time for getting to know the organization, setting up interviews and focus groups,
observing activities and measuring performance will be very quick. I am happy to entertain the idea of
using organizations external to the Kennedy School, but we should discuss that in advance.
Required assignments include:

Individual assignments: Personal reflections on what you learned from each of the following
exercises:
o Conducting in-depth interviews
o Conducting focus groups
o Clarifying a question zero and theory of change
o Using administrative data
o Observation research
o Implementing a survey
o Complete the CITI on-line training in research ethics

Group assignments
o Memo laying out question zero and theory of change for your project.
o Memo laying out implementation assessment, including sampling strategy
o Memo laying out performance metrics
o Final group presentation
o Final group paper for your client group
Class Participation: Class sessions will involve discussions of cases and readings, primarily focused on
applying the lessons from the readings to the cases. Class sessions will also provide time for group work
and opportunities for groups to report on what they are learning.
Determination of Grade: The final grade will be determined as follows:
 Participation in class discussion – 25%
 Individual memos reflecting on exercises – 25%
 Group project – 50%
Expectations of Professionalism:
You are expected to abide by the University policies on academic honesty and integrity as given in the
Student Handbook. Violations of these policies will not be tolerated and are subject to severe sanctions
up to and including expulsion from the university.
MLD-610
Page 2
MLD-610
While study groups are encouraged, their proper purpose is not to do the homework assignments, but
to help you learn the material. Each student is responsible for writing up and submitting assignments.
Separate copies of a group-constructed assignment are not acceptable.
*
*
*
*
*
Class 1: Introduction: Challenges for Managers: How to Generate and Use Evidence
Jan. 26
This class will lay out the goals and structure of the course. In particular, we will raise the issues of (1)
strategies by which management can question the effectiveness of their organization’s operations and
demand data to assess it, (2) designing and implementing strategies to effectively gather information on
performance and feed it back into organizational operations, and (3) strategies for changing the culture
of an organizations around gathering and using data, information and evidence.
Reading:
 Nadler, David, “Confessions of a Trusted Counselor,” Harvard Business Review, September,
2005, pp. 68-77.
 Case: Measured Approach: TEGV Assesses its Performance and Impact
Questions:
1. What is TEGV trying to accomplish? How did its goals change over time?
2. What questions were the managers/leaders of TEGV asking about the organization? In your
view, were these the right questions? What additional questions would you have wanted to
ask?
3. There are many ways we might define an organization’s performance. What are some of the
ways TEGV might frame the concept of performance?
4. What are the advantages of hiring outside experts to assess your organization’s performance?
The disadvantages? What are the advantages of relying on internal assessments of your
organization’s performance? The disadvantages?
Class 2: Laying out the Model
Jan. 28
We will use this class to begin developing our basic framework for thinking about data and other forms
of evidence.
Reading:
 Our Piece of the Pie: From Data to Decision-Making:
http://www.bridgespan.org/getdoc/a0e779be-3725-44de-b330-2e92f9978047/Our-Piece-ofthe-Pie-From-Data-to-Decision-Mak.aspx#.VIXBTsmAMmQ
 Jeri Eckhart-Queenan and Matt Forti, “Measurement as Learning: What Nonprofit CEOs, Board
Members, and Philanthropists Need to Know to Keep Improving,” Bridgespan
http://www.bridgespan.org/Publications-and-Tools/Performance-Measurement/Measurementas-Learning-What-Nonprofit-CEOs,-Board.aspx#.VkoHQL-VnhU
Demographics of Hartford: Hartford, Connecticut is an old New England industrial town between
Boston and New York City that over the last several decades has become the “headquarters” for many
MLD-610
Page 3
MLD-610
major insurance firms. The city’s population of 125,000 is 43% Hispanic, 35% black, 16% white and 6%
“other.” Three-quarters of the Hispanics were born in Puerto Rico, which is a territory of the United
States. One-third of the adults have less than a high school education and another 30 percent have a
high school education only. Only 14% are college graduates. About half the households have an annual
income of less than $25,000. The United States government sets the poverty level for a family of four at
$23,850, which means that Hartford is a poor city. The city is surrounded by higher income suburbs.
Situation: Many organizations are focused on strategies help youth navigate, particularly youth who are
poor and living in low income neighborhoods. Our Piece of the Pie is an organization that has changed
its focus from one of general support for everyone living in poor neighborhoods in Hartford to focusing
solely on adolescents and their passage to adulthood. The issues they face include how to keep youth
on track, develop education and workforce skills, deal effectively with personal and family challenges,
and avoid delinquent or criminal activities.
Challenge: We are only now beginning to learn about adolescent brain development and its
implications for education and labor force policies. Our Piece of the Pie, struggling with its mission to
support these young people, is trying to figure out what works for adolescents in Hartford and how their
organization can make sure that it is always performing at its best. We will use their organization and
efforts to lay out a framework for how you can lead your organization to better generate and produce
evidence about its effectiveness.
Study questions – Our Piece of the Pie: We will cover these questions over two class sessions, but
thinking about how you would answer them all is useful in preparing for the first class discussion.
How effective a program do you think Our Piece of the Pie is now? What are the criteria by which you
made your decision about its effectiveness? Along what dimensions do you think effectiveness should
be measured?
1. What should the relationship be between an organization’s theory of change and performance
measurement and between performance measurement and program evaluation?
2. What are the challenges to collecting, analyzing and presenting data in ways that inform
decisions from the front line to the CEO? To what extent and in what ways does collecting data
and analyzing performance enhance and detract from pursuing of a service provider’s mission?
3. What does the experience of Our Piece of the Pie suggest for how you might write contracts?
What performance measures you might request? What data you might provide and what other
supports might be useful?
4. Should contract renewals be based on performance? If so, what measures of performance do
you care most about? How might an organization like Our Piece of the Pie “game” your
assessment system?
Class 3: Laying out the Model -- continued
Feb. 2
See materials for class 2. We will continue the discussion of Our Piece of the Pie, focusing in particular
on the following questions.
1. What strategies did Our Piece of the Pie employ to try to change the culture of the organization
around the generation and use of data? What are the particular challenges they faced in trying
to do this?
MLD-610
Page 4
MLD-610
2. What are the general challenges organizations are likely to face when they try to change the
culture of an organization to become one more focused on the types of performance
measurement Our Piece of the Pie focused on?
Class 4: Clarifying Question Zero: In-Depth Interviews
Feb. 4
Many organizations operate under complicated mission statements or legislation, often added to in
response to emerging issues or crises but never really thought through. This is why clarifying what an
organizations Question Zero is is so important. For what do they want to be – or should be – held
accountable? Is there agreement on this?
One way to begin to get an understanding of what an organization’s Question Zero is and the extent to
which it is shared among top management is to conduct a series of in-depth interviews. In this class we
will spend time thinking about how you might phrase the questions to get a sense of what your
respondent thinks the goal of the organization is and
Readings:
 Measured Approach: TEGV Assesses its Performance and Impact
 Weiss, Robert S., Learning from Strangers: The Art and Method of Qualitative Interview
Studies. New York: Free Press, 1995. Chapter 4: Interviewing (pp. 61-119)
Study Questions:
1. What is TEGV’s Question Zero? Does it have more than one Question Zero? If so, what are
the other Questions Zero?
2. What are the implications of having more than one Question Zero? It might be useful to
think about this question from the perspective of top management, front-line management,
and those who used the services of your organization. It might also be useful to think about
this question from the perspective of a broader set of stakeholders.
3. How would you go about figuring out what various people in your organization think your
Question Zero is? Or your Questions Zero are?
Note: We will spend some time in the first three classes on identifying and selecting a group project. By
class 4 you should have organized your group and selected a project. All assignments will relate to this
project.
Class 5: Building a Theory of Change
Feb. 9
Most organizations or units within organizations do not take the time to think through their theory of
change. Doing so turns out to be hard work – but also very important. We will use this class to revisit
TEGV and develop a strategy for thinking about how to identify a theory of change. You should leave
the class with an idea of how you want to begin gaining an understanding of the theory of change for
the group you are working for.
Readings:
 Measured Approach: TEGV Assesses its Performance and Impact
 Weiss, Carol Hirschon, “Nothing as Practical as Good Theory: Exploring Theory-Based
Evaluation for Comprehensive Community Initiatives for Children and Families,” New
MLD-610
Page 5
MLD-610
Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts, ed.
James P. Connell, et al, The Aspen Institute, 1995
Questions:
1. What is the theory of change for TEGV? Does it have more than one theory of change? If
so, what are the additional theories of change?
2. What strategies might you use to develop a good understanding of the theory of change for
an organization?
3. At the end of this class, we will select a topic for the next class on focus groups. Your
assignment for that class will be to develop a set of focus group questions.
Class 6: Focus Groups
Feb. 11
Sometimes it is most productive to interview a small group of people together so that they build on one
another’s comments and push one another to think through the specific issue of concern to you. If you
want to run a focus group, how do you select the group members? How do you structure the
conversation? What do you do with the information they provide?
Readings:



Understanding Focus Groups with Author Thomas Greenbaum,
http://www.i5publishing.com/understanding-focus-groups-with-author-thomas-lgreenbaum/
Thomas Greenbaum, 10 Tips for Running Successful Focus Groups,
http://www.groupsplus.com/pages/mn091498.htm
U. S. AID Center for Development Information and Evaluation, Performance Monitoring and
Evaluation TIPS: Conducting Focus Group Interviews
Questions:
1. What are the advantages and disadvantages of focus groups as a strategy for gathering
information about the effectiveness of an organization’s operations?
2. Based on the topic selected in our last class, write a set of focus group questions. In doing
so, think about your opening questions, your strategies for getting focus group participants
to build on one another’s comments, how you will switch from question to question, how
you will conclude the discussion, and how you would take notes on the discussion if this
were part of an ongoing project.
Class 7: Sampling in Qualitative Interviewing
Feb. 16
We hear a lot about random sampling, but sometimes random samples of respondents are difficult, if
not impossible, to obtain. We only need to think about the demise of land lines and spread of cell
phones to realize that it may be impossible to realize the “statistical ideal.” This is why major telephone
pollsters like Pew are focused on getting representative samples. We will talk about different sampling
strategies in class.
MLD-610
Page 6
MLD-610
Readings:
 Kuzel, Anton J. “Sampling in Qualitative Inquiry,” Doing Qualitative Research. Crabtree,
Benjamin F. and William L. Miller, eds. Newbury Park, CA: Sage Publications, 1992. pp. 3145.
 Sanford Labovitz and Robert Hagedorn, “The Population and Sampling,” Fist-fights in the
Kitchen: Manners and Methods in Social Research. 1975, pp. 87-93.
Questions:
1. What is your strategy for sampling individuals to be interviewed on your project? How does
your sampling strategy vary with the type of information you want to gather?
2. What is your strategy for selecting focus group respondents for your project? How might
this sampling plan differ from that for gathering information from individuals?
Class 8: Testing a Theory of Change
Feb. 18
An organization’s theory of change is important. Most organizations have not thought carefully about
their theory of change so don’t know if they are explicitly following one? More important, they don’t
know if their theory of change is “right.”
Readings:
 Scared Straight: Freeport City Council Takes On Juvenile Delinquency: Part A
 Parts B and C will be handed out in class
 Ron Haskins and Greg Margolis, Show Me the Evidence: Obama’s Fight for Rigor and
Results in Social Policy. Washington, D.C.: Brookings, 2015. Chapter 1:Chapter 1: The
Obama Strategy for Attacking Social Problems, pp 1-30. (Note: This chapter was written
by Ron Haskins, a Republican, on the Obama strategy for funding evidence-based
interventions.)
 If you want to get a better understanding of the Scared Straight Program, you may want
to watch part of all of this video:
http://www.bing.com/videos/search?q=scared+straight+programs&FORM=VIRE2#view
=detail&mid=014C3B8036A0D9FD4931014C3B8036A0D9FD4931
Questions:
 What is the Question Zero for the Scared Straight Program?
 What is the Theory of Change for the Scared Straight Program?
 If you were on the city council, how would you vote?
 What are the ways you can assess an organization’s theory of change?
Class 9: Logic Models
Feb. 23
Our theories of change should lead us into the development of our logic models. And our logic models
will feed into our performance management system, so we need to “get it right.”
Readings:
 Measured Approach: TEGV Assesses its Performance and Impact
 Carolina Buiitrago, “Framing Program Evaluation: Tinkering with Theories of Change and
Logic Models,” November, 2015.
MLD-610
Page 7
MLD-610

W. K. Kellogg Foundation, Logic Model Development Guide. (This is somewhat long, so you
should feel free to skim it for class. But it will be a very useful “how to” guide for your
project.)
Questions:
1. What is the logic model for TEGV?
2. Does TEGV have two logic models? One for recruiting and sustaining volunteers and
another for its after-school initiatives?
Class 10: Implementing a Program: Structuring Implementation
to Gather Evidence on Theory of Change
Feb. 25
Getting a program or initiative up and running is normally a great deal of work. But when you can be
systematic about implementation, you can use it to your advantage to test your theory of change and
identify the systematic challenges in implementation. This may allow you to make changes along the
way to improve the program’s overall effectiveness. One only needs to think about the roll-out of
federal health care reform to understand the benefits of taking advantage of a roll-out to ensure that
your program will operate effectively.
Readings:
 Deworming Kenya
 Note: a readying may be added depending on the issues raised in class to date and the
issues emerging from your work on your projects
Questions:
1. What are the pieces of evidence on which the Deworming Kenya program is based?
How strong is this evidence? Note that there is evidence from a number of different
sources and fields.
2. Although the proposed intervention is quite simple as far as interventions go, actually
getting the initiative up and running is very challenging. What are the challenges and
what types of resources do the protagonists need to muster to implement the program?
3. Those implementing the intervention used some of the challenges to design their
implementation in a way that would generate even more evidence about the program’s
effectiveness. How did they do this? What lessons do you draw from their experiences?
Class 11: Process Evaluation
Mar. 1
A program with a robust theory of change will not be effective if it is not implemented with fidelity. We
often start our implementation or process analysis by mapping – mapping how clients will flow through
the program, how information will flow through the program and how funds will flow through the
program. Sometimes this process is quite straight-forward and sometimes it is complex. In the case of
the Jamaica PATH Program, it is complex.
Readings:
 Jamaica PATH case
MLD-610
Page 8
MLD-610

Peter Rossi, et al, Evaluation: A Systematic Approach, Seventh Edition, SAGE Publications,
2004. Chapter 6: Assessing and Monitoring Program Process, pp. 169-201
Questions:
1. The Jamaica PATH Program has a lot of moving parts. It is dependent on a lot of different
people who report through different chains of command doing their part. Try to sketch out
how clients, information and money flow through the process.
2. Where do you anticipate this information to flow smoothly? Where do you think the
greatest challenges to accurate and efficient movement of data are likely to be? Why?
What problems do you anticipate?
3. What do your responses to questions 1 and 2 suggest for how you want to organize your
process/implementation assessment?
Class 12: Process Evaluation: Using Administrative Data
Mar. 3
Administrative data that are collected routinely in administering a program are very useful for assessing
program operations. But these data are not without their problems. First, organizations have many
different reasons for collecting data, some of which are dictated by outside organizations, confidentiality
restrictions, agreements with other agencies, or historic practice. Second, those capturing and
recording these data as part of their job are often overworked and, in some cases, under trained. This
sometimes results in inaccurate or incomplete data collection. Often only those specific pieces of data
necessary for eligibility determination are reliably captured. Thus, there are many challenges to using
administrative data to assess program performance.
Readings:
 Jamaica PATH case
 Joseph Wholey, et al, Handbook of Practical Program Evaluation. San Francisco: Josey-Bass
Publishers, 1994. Chapter 16: “Collecting Data from Agency Records, pp. 374-385.
 David Pitt, Using Historical Sources in Anthropology and Sociology. New York: Holt, Rinehart
and Winston, Inc., 1972. Chapter 4: The Critical Analysis of Documentary Evidence, pp. 4662.
Questions:
1. As you think about your assessment of the implementation of the Jamaica PATH Program,
what administrative data would you want to collect and routinely review? What questions
do you want these data to answer?
2. Where do you think you will be able to obtain the most reliable data? The least reliable
data?
3. If you wanted to assess the reliability of the data collected, how might you do this? What
would you be looking for?
Class 13: Process Evaluation: Observation/Participant Observation
Mar. 8
As that famous social scientist, Yogi Berra, once said, “You can observe a lot by just watching.” This
insight is very useful for process/implementation evaluation. There are some processes that you can
most accurately assess by observing them happen. And, there is nothing quite like being on site to get
an understanding of what it feels like to participate in a program. But there is nothing casual about
MLD-610
Page 9
MLD-610
unobtrusive observation or participant observation. You need to be clear about your questions and you
need to think hard about sampling.
Readings:
 Jamaica PATH case
 Eugene Webb, et al, Unobtrusive Measures. Thousand Oaks, CA: SAGE Publications, 2000.
Chapter 5: “Simple Observation,” pp. 113-126.
 Family Health International, Qualitative Research Methods: A Data Collector’s Field Guide.
Module 2: Participant Observation, pp. 14-28.
Questions:
1. What questions about the implementation of the Jamaica PATH Program could best be
answered – or only be answered – by observing day-to-day activities?
2. Given your response to question 1, in what ways would observation and participant
observation lead to insights you might not gain through the use of administrative data,
interviews or focus groups?
3. How would you structure your observation? Your participant observation? When would
you go to each site? How long would you stay? What would you do? What
data/information would you be able to collect?
Class 14: Measuring Program Outcomes
Mar. 10
If your theory of change is correct and you have implemented your program with fidelity, you should
expect to produce your expected outcomes. But, since there is never enough time and there are never
enough resources, you need to be parsimonious in selecting the outcomes to measure and in measuring
them.
Readings:
 Measured Approach: TEGV Assesses its Performance and Impact
 Peter Rossi, et al, Evaluation: A Systematic Approach, Seventh Edition, SAGE Publications,
2004. Chapter 7: Assessing and Monitoring Program Outcomes, pp. 203-232.
Questions:
1. Think back to our discussion of TEGV’s Theory of Change. What outcomes should TEGV be
measuring? This would include short term and long-term outcomes.
2. Since TEGV offers students a number of different programs, how should they think about
the effectiveness of any individual afterschool program? The combination of afterschool
activities in which students are involved?
3. What time period should they consider?
4. What strategies would you consider using to measure these outcomes?
Spring Break – No Class
Mar. 15
Spring Break – No Class
Mar. 17
MLD-610
Page 10
MLD-610
Class 15: Gathering Data on Outcomes: Developing Surveys
Mar. 22
Often the most reliable way to get information from individuals who participated in an activity is asking
them directly. We have all completed surveys – or refused to complete surveys. Sometimes the surveys
seem thorough and make us think. Other times they are sloppy, incomplete, or have a clear political
agenda. In this class we will discuss the basics of writing a good survey.
Readings:
 Floyd Fowler, Jr., Improving Survey Questions: Design and Evaluation. Thousand Oaks, CA:
SAGE Publications, 1995. Chapter 4: Some General Rules for Designing Good Survey
Instruments, pp. 78-103.
 Joseph Wholey, et al, Handbook of Practical Program Evaluation. San Francisco: Josey-Bass
Publishers, 1994. Chapter 11: Designing and Conducting Surveys, Thomas I. Miller, pp. 271292
 John D. McCarthy and M. Barbara McCarthy, “Power and Purpose in Survey Research (If You
Got the Money, Honey, I Got the Time), in George H. Lewis, ed., Fist-fights in the Kitchen:
Manners and Methods in Social Research. Santa Monica, CA: Goodyear Publishing
Company, Inc., 1975.
Questions:
1. You should be thinking about the information you want to gather through a survey for your
group project. Come to class with a list of the general areas of information you would like to
address and specific pieces of information you would like to gather.
2. One important set of categories to consider are the following: What background behavioral
information do you want to gather? What attitudes or orientations would you like to
measure? Do you want to ask questions about anticipated future behavior?
Class 16: Gathering Data on Outcomes: Using Surveys
Mar. 24
Survey research, particularly rapid-turnaround political polling by telephone, is becoming increasingly
difficult to connect. And while the web provides opportunities to reach out to a wide audience, it too
has challenges.
Readings:
• Mick P. Couper, “Web Surveys: A Review of Issues and Approaches,” Public Opinion
Quarterly, Winter 2000 (Vol. 64, No. 4), pp. 464-494.
• Pew Research Center, “Advances in Telephone Survey Sampling: Balancing Efficiency and
Coverage Using Several New Approaches,”
http://www.pewresearch.org/2015/11/18/advances-in-telephone-survey-sampling/
• Check the Pew site for the most updated information on new strategies for insuring valid
sampling and coverage in surveys: http://www.pewresearch.org/methods/
• For an example of a web-based survey that produced data that had previously been
unavailable, you may want to skim the following PAE: Helena Pylvainen, “Women Veterans
Transitioning to Civilian Life: Assessing Needs and Priorities.” On canvas web site.
MLD-610
Page 11
MLD-610
Questions:
1. In our last class we discussed various aspects of survey construction and developed a sense
of what constitutes a high quality survey. But we should also be worried about who answers
our survey questions and who does not. What are the key questions we should ask about
the sample of individuals or groups who receive a survey? About the response rate?
2. What are some of the strategies you might employ to improve your sampling strategy? Your
response rate?
Class 17: Changing Organizational Practice Culture with Data
Mar. 29
Measuring performance is useful, but what do you do with the data when you have it?
Readings:
• Our Piece of the Pie
• Robert Behn, “Why Measure Performance? Different Purposes Require Different
Measures,” Public Administration Review, Vol. 63, No. 5, September/October 2004
Questions:
1. How would you describe the culture of Our Piece of the Pie? What are the likely
characteristics of the people who work there? What is their training likely to have been?
What aspects of their performance are particularly highly valued?
2. How did Our Piece of the Pie want to use data? What were the questions they wanted to
address?
3. What was the management strategy Our Piece of the Pie used to begin thinking about using
data
Class 18: Performance Management: Measuring Outcomes and Mapping Back
Mar. 31
Readings:
 Case: NYPD New
 Robert Behn, The PerformanceStat Potential: A Leadership Strategy for Producing Results,
Washington, D.C.: The Brookings Institution Press, 2014, Chapter 19, “Analyzing and
Learning from the Data,” pp. 145-171.
Questions:
 When Bratton became commissioner, what was the public perception of the major
problems facing the Police Department?
 Bratton took a new approach to assessing the problems facing the NYPD. How did he go
about identifying the internal operating challenges? What did he find? How did he go
about identifying the crime challenges? What did he find?
 What lessons can we take from this case for managers interested in developing a strategy
for gathering and using data to assess and change organizational performance?
 How did Bratton use the data to change the culture of the NYPD?
 What are the potential pitfalls of Bratton’s strategy?
MLD-610
Page 12
MLD-610
Class 19: Changing the Organizational Culture around Data
April 5
It’s one thing to have the top management team or a few managers focus on data and performance.
But how do you change the culture of your entire data so that everyone begins thinking differently
about how to assess current processes and performance and take on responsibility for trying new
strategies and measuring their effectiveness?
Readings:
 Cincinnati Children’s Hospital
 Amy Edmondson, “Strategies for Learning from Failure,” Harvard Business Review, April
2011, pp. 49-55.
 Edgar Schein, Organizational Culture and Leadership, 1992. Chapter 1:Defining
Organizational Culture and Chapter 2:Uncovering the levels of culture, pp 1-27.
Questions:
1. Uma Kotagal and her team are trying to get physicians, nurses and other health care
providers to both use evidence-based techniques and generate evidence about what leads
to and supports higher quality health care. In what way is the first part of this challenge –
getting front line health care providers and their managers to use evidence based
techniques particularly difficult in health care? In what was is this challenge similar to and
different from the challenge of getting front line staff and their managers to use evidencebased practices in other fields?
2. The Cincinnati Children’s Hospital team developed a “theory of change” for improving the
quality of care it provided and the quality of its overall operation. How would you describe
this theory of change? What were its strengths in this particular situation? Its weaknesses?
3. What were the key domains of resistance to change? How did the Hospital’s Leadership and
the Quality Improvement team address this resistance? Do you think they were effective?
What would you have done differently?
4. Changing the culture of an organization takes both leadership and followership. What does
the Cincinnati Children’s Hospital case tell us about both leaders and followers? What does
it suggest leaders consider as they initiate and try to sustain the type of cultural change that
totally infuses and organization?
5. What enables Cincinnati Children’s Hospital to sustain the change? If you were Jim
Anderson’s replacement, what would you be concerned about?
Class 20: Management Strategies for Sharing Data to Solve Problems
April 7
Often as part of the process of accomplishing our mission, we need to share information with other
organizations about an individual, a family, a program or a situation. This turns out to be extremely
difficult for a number of reasons.
Readings:
 Wraparound Milwaukee case
MLD-610
Page 13
MLD-610

Stephen Goldsmith, et al, The Power of Social Innovation: How Civic Entrepreneurs Ignite
Community Networks for Good, San Francisco: Jossey-Bass, 2010. Chapter 6: Turning Risk
into Reward. (On Canvas website)
Questions:
1. What is the Wraparound Milwaukee model? What is its goal? How does it work?
2. A key factor in the Wraparound Milwaukee model is the capacity to share data about one
child and his or her family across a range of service providers. Why is this advantageous?
What are the concerns about sharing data of this sort?
3. What lessons could other organizations draw from Wraparound Milwaukee about strategies
for sharing data or information on specific individuals or situations?
Class 21: Generating and Using Data across Organizational Boundaries:
Collective Efficacy
April 12
Most of the issues our organizations are trying to address are complex. Often this means that we are
focused on only a small part of the overall issue or problem and are contributing a small, though often
important, part to the overall solution of the problem. How might we gather and use evidence to show
that we are
Readings:
• Strive Together: Reinventing the Local Education Ecosystem
• John Kania, et al, “Strategic Philanthropy for a Complex World,” Stanford Social Innovation
Review, Summer 2014, pp 26-37
• John Kania, et al, “Embracing Emergence: How Collective Impact Addresses Complexity,
Stanford Social Innovation Review, 2013, pp. 1-7
Questions:
1. STRIVE is structured as a “backbone” organization. What does that suggest for the roles it
plays? What responsibilities does it have to the other organizations contributing to the
educational ecosystem?
2. How effective was STRIVE?
3. How did they use evidence to assess effectiveness? To set goals? To encourage behavior on
the part of others?
Class 22: Ethical and Legal Issues around Gathering, Sharing and Using Data
April 14
Institutional review boards are important, though they have a reputation in some circles for making
research difficult. How do you know when you should be getting institutional review board approval for
your work?
Readings:
• Each student should complete the CITI on-line ethical training for the IRB if you have not
already done so. Once you complete this, you will be certified for three years.
• Mary Roach, Stiff: The Curious Lives of Human Cadavers. New York: W. W. Norton &
Company, 2003. “Crimes of Anatomy,” pp. 39-57.
MLD-610
Page 14
MLD-610
•
Herbert Kelman, “The Rights of the Subject in Social Research: An Analysis in Terms of
Relative Power and Legitimacy,” The American Psychologist, November 1972, pp. 989 –
1005.
Questions:
1. As a manager, what are some of the guidelines you should abide by in collecting data on the
performance of your organization’s employees? Those your organization serves?
2. What do these guidelines suggest for strategies about sharing information inside your
organization? With other organizations?
3. What do these guidelines suggest about restrictions you might place on outside researchers
who want access to data your organization generates?
Class 23: What do we mean by Evidence?
April 19
We began the course by laying out a model for evidence and spent our time figuring out how we as
managers would use that model to better understand and improve our organization’s performance as
well as change the culture of our organization around the generation and use of data for management.
Each of you has applied the model to a specific program or organizational unit, which has given you a
chance to practice using the techniques and skills the course covered. This is our chance as a class to
assess the model.
Class 24: Presentations
April 21
Class 25: Presentations
April 26
Class 26: Wrap Up
April 28
MLD-610
Page 15