Research and Analysis for Public Policy and Public Management:
Principles and Practices from Active Learning
Dahlia K. Remler
School of Public Affairs, Baruch College, City University of New York
Department of Economics, The Graduate Center, City University of New York
National Bureau of Economic Research
Draft
Do not cite without the author’s permission
November 2, 2011
Acknowledgements: Gregg G. Van Ryzin and I developed many of these ideas and approaches
jointly. All mistakes are my own.
1. Research and Analysis in Public Policy and Management
Most Masters of Public Administration (MPA) and Masters of Public Policy (MPP)
programs have some kind of statistics, quantitative data analysis or research requirement,
although the specific courses’ content and names vary (NASPAA, n.d.). Such requirements seem
essential, since public sector managers, non-profit managers and policymakers at all levels are
increasingly called upon to use evidence-based practice and data for both management and
accountability purposes. Yet students often find their research and analysis (R&A) courses
irrelevant—as well as difficult. In this paper, I describe principles and practices for MPA1 R&A
courses that improve student learning, interest and motivation.
Most of the principles and practices I describe are relevant for a wide variety of courses:
statistics, research methods, evaluation, data analysis, research in public administration and so
on. Although I emphasize quantitative methods, qualitative methods are also included and
many of the principles and practices apply equally to both. While the practices and principles
apply widely, they are particularly focused on some general learning objectives, which I expect
most MPA R&A courses share to some extent. Specifically, the objectives are that after the
course(s) students will be able to:



Critically consume research
o Spot weak or invalid conclusions in formal and “informal” research
o Extract relevant and valid conclusions from research
Perform research in policy and practice capacities at a basic level
Deal effectively with the quantitative aspects of public affairs
Since most MPA students do not become researchers or analysts, I recommend that primary goals
for all programs be that graduates are effective consumers of research and analysis and are
quantitatively literate. (Steen (2001, 2004) defines quantitative literacy, often also referred to as
quantitative reasoning, and which I discuss extensively below.)
1
The principles and practices described here are equally relevant to MPA and MPP students.
Hereafter, in the interests of brevity, I will only refer to MPA students. The only relevant
difference is that MPP students are more likely to consider research and analysis training of
interest to them and therefore require less motivation for R&A courses (Infield and Adams,
2011).
1
I developed the practices and principles described in this article through two main
means: Trial and error in my own teaching2 and in collaboration with colleague Gregg Van Ryzin
as we wrote a research methods textbook (Remler and Van Ryzin, 2011); The teaching and
learning literatures on active learning and quantitative literacy.
The paper is organized as follows: In section 2, I describe those aspects of the diversity
and context of MPA students most relevant to R&A courses. In section 3, I describe the active
learning and quantitative literacy literatures, extracting the parts relevant for MPA R&A
courses, providing some examples and concluding with general principles. In section 4, I provide
descriptions of good practices for MPA R&A courses and many examples. In section 5, I briefly
conclude.
2. Diversity and context of MPA students
MPA students are diverse in many ways: age, ethnicity, other demographics, but one
form of diversity has a tremendous effect on R&A courses: prior quantitative training. I will
characterize (or caricature) that variation with three made-up composite bios, characteristics of
students I have taught. Alice hated math in high school, avoided quantitative courses in college,
majored in Ethnic Studies and is now a community organizer with a small not-for-profit. Brenda
was a math major in college who now, under the supervision of a senior researcher, does data
management and some analysis for a headquarters of a national religious organization. She
wishes to become an analyst in her own right. Carlos majored in history, is now a human
resources manager in a Federal government agency and wishes to move up as a manager in the
Federal government. Brenda is the only student with a strong quantitative background and an
interest in research per se, but Alice also is motivated to learn about research, because her
organization’s funders want evidence of its effectiveness. Carlos is only interested in
2For
over seven years, I have taught the two-semester research and analysis sequence to MPA
students at Baruch College of the City University of New York. My students included full-time,
part-time and executive MPA students. For the five years prior to that, I taught research
methods to Masters of Public Health students in health policy and management at Columbia
University.
2
management and sees no relevance of the MPA course to his career, beyond meeting a
requirement. The ideal class would be useful, interesting, and challenging but not impossible
for all three students. As the sample composite bios illustrate, a second form of MPA diversity is
also important: the research and analysis requirements of students’ planned careers.
Which stages students are at in their careers are also important for designing and
implementing R&A courses. Students who have substantial career experience, particularly
“executive” students, have a wealth of examples and potential applications. Such students also
tend to be impatient with any material whose relevance they cannot see. Their math skills may
be weak since it may be many years since they last took a math or quantitative course in college
or even high school. In many public affairs careers, students well along in their careers have
probably seen the emphasis on data and evidence, giving them particular motivation for R&A
courses. With a few careers, however, students well advanced in their careers may have not
missed quantitative and analysis skills, and resent being forced to jump through a hurdle they
see as irrelevant. For many MPA students, relevance to policy and practice, and particularly
relevance to a job they have or would like to have, is essential for motivating and engaging
them.
Other students, however, have less career experience and some even start an MPA
program directly after an undergraduate degree. Yet other students may have experience in a
very different career and plan to make a career change. Therefore, when crafting assignments
and exercises, it is important to be aware of these students also. Nonetheless, since they
choose to do an MPA, they should at least have public affairs interests to draw upon.
Some MPA students attend full-time with little work outside school, while others attend
part-time and continue to work. Working while attending school can provide examples and
motivation in the same way that career experience does, but it also can make students only
interested in material they see as relevant. Working also takes time away from studies and
reduces the flexibility of students’ time, as can family obligations or other constraints.
Attending class at night after working makes it harder for students to pay attention, increasing
3
the need for methods for keeping their attention. The long, intense sessions common in
executive programs cause similar problems.
For all students, it is important to keep the learning goals relevant for MPA students. A
minority of MPA students, and even of MPP students, become researchers or analysts. We turn
next to principles that can help the achievement of R&A learning goals.
3. Active Learning and Quantitative Literacy: Principles for Research and Analysis Courses
Today most instructors have heard the idea that they should not just lecture but rather
employ active learning. But what exactly is active learning? Bonwell and Eison (1991) say that
active learning requires that “students must do more than just listen: They must read, write,
discuss or be engaged in solving problems. Most importantly… [they] must engage in such
higher-order thinking tasks as analysis, synthesis and evaluation.”
Of course, almost all courses require active learning tasks outside the classroom. What
is the evidence about the effectiveness of active learning in the classroom? Pascarella and
Terenzini (2005) summarize the experimental and quasi-experimental literature on active
learning in higher education, saying that, “studies either report better mastery of course
content when actively engaged in learning…or no significant learning differences or mixed
learning effects when comparing active to passive lecture instructional approaches” (p. 102).
Although not all studies provide enough data to determine an effect size, among those that do,
Pascarella and Terenzini estimated an average effect of .25 standard deviations. Studies using
observational data and control variables were consistent with the experimental and quasiexperimental findings.
A critical feature of active learning approaches, as with any teaching strategy, is to
clearly define the learning objectives and then ensure that the approaches further those
objectives. Bonwell (1996) suggests several active learning techniques suitable for a
predominantly lecture class. Some that are suitable for R&A classes include: Short Writes, which
could be short interpretations or simple problem-solving; Think-Pair-Share, in which a short
problem or question is given and two students discuss it for two or three minutes, prior to a full
4
class discussion; Formative Quizzes, with a few short questions that allow students to test their
own understanding and practice applying the material. Bonwell and Eison (1991) describe many
lecture-substitute activities, including case studies, debates, and peer teaching.
Many active learning approaches described in the literature do not fit well with students
trying to learn essential quantitative skills that they could not learn on their own, particularly
when skills build on one another. For example, a debate using empirical evidence to support
positions would motivate students and is a wonderful way to synthesize and reinforce
application and critical analysis of evidence. But it is not a good method for first teaching
statistical significance to students who do not yet understand it. The literature on quantitative
literacy is therefore particularly useful, since it is focused on the right sort of learning for R&A
courses.
Steen (2001, p.8) describes several components of quantitative literacy, of which the
most relevant for MPA R&A courses are:
Confidence with Mathematics…Individuals who are quantitatively confident routinely use
mental estimates to quantify, interpret, and check other information…
Interpreting Data. Reasoning with data, reading graphs, drawing inferences, and
recognizing sources of error…
Logical Thinking. Analyzing evidence, reasoning carefully, understanding arguments,
questioning assumptions, detecting fallacies, and evaluating risks…
Making Decisions. Using mathematics to make decisions and solve problems…
Number Sense. Having accurate intuition about the meaning of numbers, confidence in
estimation, and common sense in employing numbers as a measure of things.
Practical Skills. Knowing how to solve quantitative problems that a person is likely to
encounter at home or at work…
I suspect that MPA programs all require some quantitative analysis or research course as
much to enhance their students’ general quantitative literacy as to teach statistics or any other
particular skills or knowledge. Despite the variation in MPAs’ career goals, some tasks should be
doable by all MPAs, both immediately upon graduating and long after finishing their degree.
Consider an example: An MPA’s organization has implemented a new policy and wants to see if
it “worked” to improve an outcome. The organization has individual level outcome measures
over time. (For example, an educational not-for-profit has added math majors as tutors and
5
wants to know if their centers have more students in the community coming for help with
math.) An MPA should be able to take the data in a spreadsheet or statistical package, graph
averages over time and interpret the results. More importantly, any MPA graduate should be
aware of the problems of statistical significance (that a change could be a fluke), even if they
cannot implement the correct statistical significance test, and be aware of the problems in
assuming that any change in outcomes was caused by the program change. Finally, he should
consider the outcome measure’s validity and reliability. Another example, which is both
qualitative and less formal, would be that when seeking feedback or thoughts on an issue, the
MPA graduate will always be attuned to the issue of the representativeness of those providing
feedback, even when time and resources constraints prevent obtaining a representative
sample.
Steen (2001, 2004) and references therein describe a variety of teaching practices to
increase quantitative literacy. I have selected the most important relevant recommendations
for MPA R&A courses from Collison et al’s (2008) adaption of Steen’s recommendations. They
are listed in Table 1.
Table 1: Strategies for Quantitative Literacy



Rule of Four: All applications and concepts presented as:
o Words
o Numbers
o Graphs
o Symbols
o Translate from any one to the other
Practice
o Interpreting and writing about numbers
o Explaining equations in words
o Reading, interpreting and applying technical writing
Assignments and tests that require students to apply skills in applications that are
meaningful to the students
o Examples involving familiar concepts are more effective than examples that
require extra learning
o Examples which motivate and interest students are valuable
6


A variety of different applications
o Increasing student role in framing the problem and in abstracting
Spreadsheet and/or statistical software exercises integrated into course content
throughout the curriculum
The first strategy, the “Rule of Four,” means that concepts and applications should be
approached in four forms: equation, graph, numbers (data), and words, and that students
should be able to translate between any two forms. Although the Rule of Four was developed
for teaching calculus (Hughes-Hallett, Gleason and Flath, 2008), I have found it useful in all
quantitative subjects. While all four forms are useful, not all MPA students will become
comfortable with equations in some contexts, and therefore, to some extent, rationing
equations can be useful.
Consider a simple regression example, with data from a representative sample providing
individuals’ annual earnings and the number of years of education. The equation is
Earnings = const + B* Education.
(I recommend using the names of variables: Earnings, rather than “y”; Education, rather than
“x.”) A useful introduction or application of regression would include a scatterplot of the data,
with the best-fit ordinary least squares regression line, and a table (in software) showing the
numerical data. Students should also interpret the coefficient and constant in words, “For each
additional year of education a person has, we expect him to earn $B more annually” and “We
expect that a person with zero years of education would earn $const.”3 (They should use the
actual numbers, rather than B or const, of course.) Students should also describe other aspects
of the relationship in words, including tightness of fit. All four approaches together should be
used for introducing the concept and when students apply it, using statistical software to
estimate the constant and coefficient. It is relatively common to teach regression using the Rule
of Four, but the approach can be applied to hypothesis testing and many forms of data analysis.
3
The interpretation of the constant, of course, should be immediately followed with a
discussion of how no one has anything close to zero years of education and the problems of
out-of-sample predictions.
7
While all four parts of the Rule of Four are valuable, communicating in words about
quantitative content is particularly important. First, it is an extremely important skill in its own
right. MPA students, even those who become researchers or analysts, will work in the worlds of
policy or practice. Sentences like, “the logit results were an odds ratio for X1 of 1.6 with a pvalue of .007 and an odds ratio for X2 of 1.1 with a p-value of 0.26.” will not serve them well in
their careers even if they understand such sentences well. MPAs will need to be able to express
quantitative and technical information in the most accessible and meaningful manner possible.
Miller (2004, 2005) has written two books on writing about numbers and about multivariate
analysis, which include sections on writing for relatively broad audiences.
Second, describing numbers, equations and analytical concepts in words is an effective
way for many to learn. Students should practice interpreting entries in tables and statistical
package output in words. Being able to express the results in words both requires and aids
understanding. I recommend repeated exercises in which students interpret both individual
numbers within a table of results and the overall picture from the table. Sometimes non-native
English speaking students resent an emphasis on words in a quantitative class, having expected
that their weak English would matter little, if at all. For such students, suggest to them that they
start by practicing explaining in their native language and note that being required to explain
well in English will make them more effective at explaining well in their native language.
The third bullet in Table 1 refers to the importance of relevant applications. For MPA
students, who may already work or be well advanced in their careers, such applications are
extremely important, particularly for motivation. But applications are critical for all students
and they should be rich and compelling. The fourth bullet refers to the importance of many
different applications. Difficult concepts and methods take repeated efforts to learn. No matter
how motivating and useful a single example, students may not learn to generalize to other
situations unless they have practice generalizing. I return to this issue in later sections. Finally, it
is important for students to work with data in spreadsheets and/or statistical packages. Even
students who will do little data work of their own gain a far fuller understanding of the meaning
of statistics and data analysis by doing it themselves. I return to this issue later also. However, it
8
a class should not be dominated by either data cleaning or the details of a particular statistical
package, because most MPA students will be primarily consumers of research and analysis, not
producers.
The quantitative literacy literature also stresses that the practices and skills of
quantitative literacy must be reinforced across the curriculum. Ideally, other MPA classes in
subjects such as policy analysis, management, budgeting, communications, economics, politics,
and of course a capstone, would all reinforce these skills. While that is a good idea, from
hereon, I restrict myself to suggestions implementable in R&A courses alone.
In summary, I recommend the following four principles for MPA R&A courses:
1)
2)
3)
4)
Students should write and speak in words, interpreting the results or studies.
Always teach using applications relevant to public policy and management.
Whenever possible, let students pick applications, particularly for longer assignments.
Use active learning.
In the next section, I flesh out these principles, particularly the active learning, with specific
practices and examples of their implementation.
4. Practices
In this section, I focus on those practices I have found effective in R&A, starting with
those that take the shortest amount of time and moving on to those who take the longest. In
every case, I include specific examples from my own teaching.4
4.1 One to Two Minute Individual Writing Exercise
Many of us ask our students questions and ask for volunteers to answer. While I confess
that this is my usual practice, it suffers from several problems. Students whose minds are
wandering or those who feel that they don’t understand will not actively try to answer the
question. Even students who do try may only have time to make an initial start before another
student answers the question, depriving the first student of a real opportunity to learn.
4
Many of these examples were developed working with Gregg Van Ryzin and he is the primary
creator of some examples.
9
Therefore, I recommend making questions into very short writing exercises. Ask the question
and ask every student to answer it in writing. When most students are done, bring the class
together and ask for volunteers or call on students. In cases where there is a correct answer,
make sure that it is provided, clearly labeled as the correct answer and fully explained.
This approach can work both in the middle of a lecture and while discussing a study. This
approach has several advantages. It makes students realize if they do not understand or do not
know how to apply, potentially prompting students to ask useful questions. These exercises
make it much harder for students to tune out. When teaching skills that build on one another,
this approach can make sure the foundations are there before moving on. For example, if a
student cannot use the basic tools of path diagrams (circles for variables and arrows for causal
relationships), he will not understand subsequent material on mechanisms or common causes.
Example short writing exercises I have found effective include:







Identify independent and dependent variables in a particular application
Describe the population sampled for a particular survey
Interpret in words one number in a descriptive statistics table
Determine if a particular result is statistically significant in a results table
Calculate the margin of error for a poll
State the null and alternative hypotheses
Determine if a study is descriptive or causal; determine if a study is an experiment
This practice can also be used to help each student find relevance in the material for herself,
and thus become more motivated. Examples of such exercises include:



Describe a relationship between two categorical variables relevant to your work (or
interests)
Think of a program and an outcome it is designed to improve in your work (or an area of
interest). Do you have any estimates of the program’s effect on the outcome?
Think of a measure, such as a performance measure, used in your work. How valid do
you think it is?
Using the first exercise will make cross-tabs seem much more relevant and interesting. Using
my fictional students to illustrate, Carlos, the government human resources management,
might be interested in the relationship between region and ethnicity of applicants. When
discussion of cross-tabs continues, Carlos’s example could continue to be used. Alice, the
10
community organizer, will likely have little trouble finding examples that would interest her
funders.
One problem with this practice is the large variation in students’ backgrounds and skills.
Illustrating again with my fictional students: Brenda, the math major and data analyst, will often
have the answer almost immediately, while Carlos and Alice will need more time. Either Brenda
sits around bored or Carlos and Alice get cut off and don’t have a chance to learn. To solve this
problem, prepare an “extra task or question” that you describe as only for students who have
time for it. For example, if students are being asked if a particular result is statistically
significant, the extra task could be assessing its practical (substantive) significance. If the main
task is interpreting a result, the extra task could an interpretation clear and accessible to a
journalist.
4.2 Think-pair-share
The think-paid-share approach is similar to the above individual writing exercise but asks
each student to work with another, usually after first pausing to think for herself. Tasks are
similar to the above but should take slightly more time. If students differ in ability, this
approach gives the stronger student the opportunity to teach, reinforcing that student’s
understanding and communication, while preventing the weaker student from floundering
alone. However, if the pairs have similar abilities, the pairing can exacerbate the problem due
to ability variation, with some finishing instantly and others floundering together. For that
reason, as well as the additional time for pairing up, I prefer the short individual exercise above
or longer group exercises described below. Nonetheless, think-pair-share is a standard in the
teaching and learning literature and it can be useful, particularly for less technical material. For
example, a pair could be asked to write a survey question asking respondents about how safe
they feel at school.
4.3 Longer Group In-class Exercises
For larger group exercises, students form groups of three to four students to do a
substantive task or solve a multi-part problem. I find this approach essential for tasks or
11
concepts that students find very difficult and which require multiple tries, with feedback, for
mastery. Examples include:





Finding alternative causal explanations for a correlation (e.g., reverse causation,
common cause)
Creating a mechanism in a logic model of a particular program
Interpreting regression coefficients and other statistical package output in terms useful
for policy or practice
Predicting the direction of non-response bias for particular measures, and other biases,
in a particular survey
Evaluating the generalizability and quality of causal evidence (internal validity) of a
particular quasi-experimental study
The first example is shown in Appendix A, in which I would expect all groups to get through part
(c). I often have the same kinds of exercises for out-of-class assignments, since students require
repetition. All of these tasks take significant time, sometimes a half hour or even more,
representing significant opportunity cost for class time. Therefore, I reserve such group
exercises for topics that students struggle with and that I consider very important.
The variation in student preparation and ability is again a problem. One approach is to
pre-select groups based on prior background (e.g., undergraduate major) to attempt to get
balance, although it can take time for students to physically form these groups. However, again,
the main solution is to have core tasks and extra tasks. For the logic model example, every
group should come up with at least one mechanism, but some groups may have time to find
several mechanisms or include moderating variables. If students are interpreting multiple
regression output from a statistical package, an extra task might be to discuss what other
control variables would be wanted and the bias from omitting them.
Circulating is essential for uncovering significant problems or misunderstandings. Make
sure to interact at least once with each group. If not, students may go off on the wrong track for
a long period or even the entire exercise. When I initially started to do small group exercises, I
made them much too difficult—the kind of problem appropriate for outside of class with much
more time. Not only did students not learn but they often confused one another and spread
bad approaches. The tasks should be meaty enough to warrant such a significant chunk of class
12
time, but almost all students should be able to complete the core tasks. The core tasks should
be doable from material taught immediately before or required as reading for the class.
4.4 Discussions of Studies
All MPAs should be consumers of research and analysis in many forms: peer-reviewed
academic journal articles, government and foundation reports, stray numbers or anecdotes
bandied about, advocacy, media accounts and more. They should be able to spot weak or
invalid conclusions and extract relevant, valid conclusions. MPAs will often work in settings
where time and resource constraints necessarily limit the quality of data and analysis and they
must know how to make the best of what they have. They will also need to go to the existing
literature and try to extract what is useful and relevant. Therefore, it is essential that they
practice reading and interpreting studies of many kinds: mainstream media presentations of
studies, mainstream media examples not of formal studies but ostensibly evidence,
government reports, advocacy reports, journal articles for which students know most methods,
journal articles which are too advanced for them but from which they can learn to extract
useful information. Management examples, whether from case studies or real problems, are
also useful.
Due to the scarcity of class time, readings should be done outside of class and discussed
in class. Since the discussion can stray far from the main learning goals, I suggest providing
questions to go with each reading. (See Appendix B for examples.) For long and/or difficult
articles, tell students which sections and/or issues to focus on.
One difficulty is ensuring that students do the readings before class. A possible approach
is a brief reading quiz that asks simple factual questions about the article (such as which city the
study took place in). Another problem is that only a few students may participate in the
discussion. The think-pair-share practice be useful for discussing questions on the readings,
before coming together as a full class. On-line discussion boards, with students required to give
a certain number of comments is another way to ensure broad participation. A final problem is
that even with specific questions, the conversation can go astray, and therefore attention must
be paid to keep on topic.
13
Reading studies is essential for research courses but cannot be easily used in a pure
statistics course. (That very fact reveals why an MPA program should offer more than simply a
pure statistics course.) However, in pure statistics classes, it is valuable to give students tables
of results for interpretation, with some context and set-up provided. Having students interpret
both individual entries and the large picture from tables of results will help with both
motivation and application skills.
4.5 Out-of-class Assignments, Including Data Analysis
An important purpose of out-of-class assignments is to reinforce the most difficult
and/or important topics, as well as to synthesize and apply more deeply. Out-of-class
assignments can provide opportunities for students to choose something relevant to them,
increasing their motivation and often their learning. Examples of out-of-class assignments that
can be chosen by students are a logic model and designing a survey design. (Appendix C
contains these two example out-of-class exercises.)
As already discussed, analyzing quantitative data is an important component to
developing quantitative literacy, as well as an important skill in its own right. Some instructors
use labs with statistical software to develop such skills. Labs are another form of active learning
in which students implement what they learn during class. Therefore similar strategies are
needed to cope with student variability. Whether or not there is a lab class or class time is used
for labs, I recommend out-of-class data analysis exercises, to allow for much deeper skills and
individual investigation.
For practical reasons, it is difficult for students to choose their own individual data sets
and therefore, in most classes all students will analyze the same data provided by the
instructor. However, in some contexts, it may be realistic for students to assemble their own
datasets from publicly available data (Aguado, 2009; Remler and Van Ryzin, 2011, p. 204). Even
if students all use the same datasets, they may be able to pick their own analysis (Hill 2003). For
example, students could select variables for descriptive statistics, choose their own control
variables or address different research questions.
14
The teaching and learning literature recommends that instructors use rubrics to
evaluate student assignments (Andrade, 2005). A rubric is “an assessment tool that lists the
criteria for a piece of work or what counts and articulates graduations of quality for each
criterion,” not only a ”checklist of criteria” but also “gradations in quality” (Andrade, 2005 p.
27). Appendix C includes two sample rubrics I have used to grade the accompanying two
sample out-of-class assignments.5 6 Students should receive the rubric with the assignment.
Andrade (2005) describes rubrics’ multiple strengths. They push instructors to clarify major
goals and expectations about those goals and to focus instruction, assignments and assessment
on those goals. They communicate priorities, desirable qualities and common pitfalls to
students. They provide informative feedback to students with less instructor time than is usual
for such individual and contentful feedback. While much of the peer-reviewed empirical
literature about rubrics focuses on writing instruction, there is recent work on assessing
quantitative reasoning in written assignments (Grawe, Lutsky and Tassava, 2010) and the
principles appear relevant to R&A courses.
4.6 The Case Against One Big Project
Many MPA research and analysis courses have students do one large research project as
their main or only outside-of-class assignment. There are many advantages to such an
approach. Students can select their topic, making it meaningful, increasing motivation and
learning. The work is very applied and forces students to deal with many real world
complexities. While it is great if students can do such projects in a capstone class, I argue
against them in standard core R&A courses.
First, MPA students need to learn quite a few distinct skills and concepts. To name just a
few, all MPA students need to aware of issues like: the representativeness of those who choose
5
I have only recently begun to use rubrics in my teaching and have no doubt that these could
be improved substantially. These sample rubrics are revised versions of rubrics developed in
coordination with my practitioner co-instructor, Gregorio Morales.
6 The Association of American Colleges and Universities (2010) has created an extensive rubric
for undergraduate education, including quantitative reasoning. While it is far too general for a
specific class, it is an excellent illustration of rubric design as well as principles of quantitative
literacy.
15
to speak up about an issue, the idea of statistical significance (differences or trends possible
being a fluke), knowing when and why a proportion is needed, and not misinterpreting a
correlation as evidence of causation. Any given research project is likely to only touch on a few
of these skills. Moreover, students are quite likely to choose a qualitative research project,
which is a logical choice for many, since they are more likely to become producers of qualitative
evidence than producers of quantitative evidence. However, all MPA students should be able
consumers of at least basic quantitative evidence. They will not gain those skills by going a
qualitative research project. A further issue is that working with one example, even if it is rich,
makes students less likely to be able to generalize—to use their skills in a different context. If a
single project is to be used, it should be structured to ensure that a wide-range of skills are
included. (See Aguado (2009) for an example.) Of course, a big project has many advantages
and ideally students will also have the opportunity to complete one in a capstone course.
Instead of a single big research project, I recommend several out-of-class assignments,
along with shorter practice problems and at least one exam. This provides multiple chances to
learn with different applications. Otherwise, however, this recommendation is not based on the
teaching and learning literature, but on my own experience and my observations of others’
classes. In my experience, an exam focuses students’ minds on learning particular skills in a
manner that allows application to new problems. Of course, the short problems and exam
questions should still be applications and not just rote calculations or definitions. Appendix D
contains examples of sample problems and exam questions for research methods. Such
problems for a statistics course should include a great deal of interpretation in words of output
from statistical packages.
4.7 “Tell the story of the course”
In any course, but particularly in a technical course, students can lose sight of the big
picture: why they are doing this, what it is useful for, how it all fits together. To combat this
problem, periodically ask students to “tell the story of the course,” describing both
16
chronological and conceptual progression, and answering those three why, what and how
questions. They should be able to do this in a few sentences.7
For example, the story of the first semester statistics course that I teach would ideally
go something like, “We started by trying to describe one variable, such as gender or age,
graphically and with numerical summaries. We then moved on to the relationships between
variables, focusing in turn on the forms needed for different types of variables; for example,
cross-tabs describe the relationships between two categorical variables. We then turned to
inference, learning how precisely we estimate things from a sample and seeing if relationships
are statistically significant—or just a fluke.”
The story of the second semester research methods course that I teach would ideally go
something like, “We started with theories, which explain why variables vary, and with
qualitative research for initial explorations. We then turned to describing the world: how to
measure, how to sample, how to design surveys, how to find existing data. We moved next to
estimating and understanding causal effects: what happens to an outcome if we change
something—“what if” questions. Specifically, we covered the many causal explanations of a
correlation, trying to disentangle causal effects in observational data, randomized experiments,
and quasi and natural experiments.”
Obviously, these stories of the courses could be lengthened or shortened, include more
or less jargon, or include more or fewer examples. While students often struggle to do more
than list topics, it is useful for them to try and useful for the instructor to remind them of the
big picture.
4.8 Shortage of Time and Solutions to It
The literature on active learning lists several barriers to the use of active learning. As
already described, the biggest barrier for MPA R&A courses is not listed in the literature I have
read: the tremendous variation in student quantitative background and in interests in research.
7
This suggestion came from my colleague Neill Sullivan who asks students to tell the story of
the class in Introduction to Public Affairs. It is a valuable tool in any class.
17
However, the literature does focus on the second biggest barrier: time. As Bonwell and Eison
(1991) state, with active learning “one cannot cover as much content in the time available” (p.
59). Of course, there is no reason that all the content must be covered orally in class. They note,
“as has been pointed out countless times, the lecture was outmoded by the invention of
printing and by cheap and easy access to printed words” (quoted in Bonwell and Eison, 1991, p.
60).
In today’s world, reading is far from the only means for delivering content outside of
class. Software to teach statistics exists and more is being developed. Many statistics textbooks
have accompanying packages with simulations and on-line exercises. In addition to such
statistics tools for sale, the Open Learning Initiative provides a free interactive statistics course
on-line (Open Learning Initiative, n.d.). Such software has the advantage that each student can
go at his own pace. Free videos teaching statistics abound on the web, although sorting through
them to figure out which ones are good takes considerable time.8 With technology today,
professors or departments can create their own videos—or audio recordings to accompany
visual slides. All of this allows content to be covered outside of class and class-time to be used
for those tasks where physical presence is most useful and important, such as discussions,
sample problems and the exercises described previously.
Of course, many instructors, including myself, will feel obligated to lecture on most of
the required content, because we know that all students will not learn all assigned material.
That does not mean that active learning should be given up for lecturing on all the required
content. The unfortunate truth may be that given the backgrounds of entering students, the
time available for class, and the time students have available outside class, it will not be
possible for many or even most students to learn all the required content.
One option is to teach less content—but more thoroughly and with greater student
mastery of the material. While that can be a good approach to some extent, two factors argue
against it. First, some students are able to learn all the content and they want and need it.
(Think of Brenda who is already doing some data analysis professionally and wishes to develop
8
I have not done this.
18
those skills further.) Reducing the content ill serves those students and makes the program less
attractive to them. Second, different students will want or be able to reach higher levels for
different topics. For example, Alice may not reach an advanced level with multiple regression,
but her background with non-profit funders may make her able and willing to reach advanced
levels with logic models and survey design. Reducing content in some areas would help Alice
but in other areas it would hurt her.
The best approach is to prioritize some content and reinforce that content through
active learning. Additional easy-to-learn material can be covered through readings and other
outside of class forms. Additional hard-to-learn material can be covered outside of class, letting
students know that it is the harder material and less of a priority. Harder additional material
can also be covered in the “extra” or more advanced tasks of in-class active learning.
4.9 Summary of Practices
In summary, there are a variety of active learning practices that can promote
quantitative literacy, skills and knowledge useful for MPA students. They range from short
individual writing exercises that interrupt lectures, to longer group in-class exercises, to
discussions of studies, to meaty applied out-of-class exercises, to on-line software, to short
applied out-of-class problems.
5. Conclusions
Seeing relevance is key to motivating MPA students. Providing relevant applications
ensures that they learn the skills relevant to their fields. When research and analysis courses
are based on relevant learning objectives and are taught well, students see the relevance and
are motivated. They understand that it is about the culture of evidence, data and performance
measurement they see and hear about so often. They should not think that research is only the
stuff in academic journals, although they should learn how relevant academic journals are to
them. They should see that research and analysis is about things that all MPAs will see in their
careers: client satisfaction surveys, figuring out if a new program is working, being wary of the
representativeness of the “squeaky wheel.”
19
The tools of active learning can aid all forms of education; MPA research and analysis
courses are no exception. However, the tools must be selected and adapted appropriately. On
the one hand, there is an existing teaching and learning literature we should make use of. On
the other hand, some of that literature is based on poor research, particularly poor causal
research. Moreover, very little of that literature is based on studies in MPA programs or on R&A
courses in MPA programs. Schools of public affairs and public policy are filled with people who
can do good research, particularly good causal research in applied settings. Following on calls to
increase the scholarship of teaching and learning (Boyer, 1990; Carnegie Foundation, n.d.),
perhaps some of us should work on good evidence about what works for MPA and MPP
students.
20
References
Aguado, N. Alexander. 2009. “Teaching Research Methods: Learning by Doing” Journal of Public
Affairs Education 15(2): 251-260.
Andrade, Heidi Goodrich. 2005. “Teaching with Rubrics: the Good, the Bad and the Ugly”
College Teaching 53(1): 27-30.
Association of American Colleges and Universities. 2010. Quantitative Literacy VALUE Rubric.
Washington, DC: Association of American Colleges and Universities.
Ariguesta, Maria P. and Jeffrey Raffel. 2001. “Teaching Techniques of Analysis in the MPA
Curriculum: Research Methods, Management Science and ‘The Third Path’” Journal of Public
Affairs Education 7(3): 161-169.
Bonwell, Charles C. and James A. Eison. 1991. Active Learning: Creating Excitement in the
Classrom. ASHE-ERIC Higher Education Report No. 1, Washington, DC: The George Washington
University, School of Education and Human Services.
Bonwell, Charles C. “Enhancing the Lecture: Revitalizing a Traditional Format” in Using Active
Learning in College Classes: A Range of Options for Faculty, Tracey E. Sutherland and Charles C.
Bonwell, editors.
Boyer, Ernest. 1995. Scholarship Reconsidered: Priorities of the Professoriate San Francisco,
CA: Jossey-Bass.
Carnegie Foundation for the Advancement of Teaching. n.d. Carnegie Academy for the
Scholarship of teaching and Learning (CASTL). Retrieved October 22, 2011 from
http://www.carnegiefoundation.org/scholarship-teaching-learning
Collison, Joe, Catherine Good, Sonali Hazarika, Matt Johnson, Jimmy Jung, Anita Mayo, Will
Millhiser, Dahlia Remler, and Laurie Beck. 2008. “Report of the Provost’s Task Force on
Quantitative Pedagogy.” Retrieved August 1, 2010 from
http://www.baruch.cuny.edu/facultyhandbook/documents/TaskForceonQuantitativeSkills2008
-09-05.doc
Grawe, Nathan D. and Carol A. Rutz. 2009. “Integration with Writing Programs: A Strategy for
Quantitative Reasoning Program Development.” Numeracy 2(2): Article 2. Retrieved May 12,
2010 from http://services.bepress.com/numeracy/vol2/iss2/art2.
Hill, Carolyn J. 2003. “Can They Put it All Together? A Project for Reinforcing What Policy
Students Learn in a First-Semester Quantitative Methods Course” Journal of Policy Analysis and
Management 22(3): 473-481.
Hughes-Hallett, Deborah, Andrew M. Gleason and Daniel E. Flath. (2008) Calculus: Single and
Multivariable. John Wiley & Sons. 5th edition.
21
Infield, Donna Lind and William C. Adams. 2011. “MPA and MPP Students: Twins, Siblings or
Distant Cousins?” Journal of Public Affairs Education 17(2): 277-303.
Miller, Jane E. 2004. Chicago Guide to Writing about Numbers Chicago: University of Chicago
Press.
Miller, Jane E. 2005. Chicago Guide to Writing about Multivariate Analysis. Chicago: University
of Chicago Press.
NASPAA (National Association of Schools of Public Affairs and Administration) n.d. “The
MPA/MPP Degrees” Retrieved October 22, 2011 from http://gopublicservice.org/degree.aspx.
Open Learning Initiative n.d. Retrieved October 22, 2011 from
http://oli.web.cmu.edu/openlearning/forstudents/freecourses/statistics
Pascarella, Ernest T. and Patrick T. Terenzini. 2005. How College Affects Students: A Third
Decade of Research. 2nd edition. San Francisco, CA: Jossey-Bass.
Remler, Dahlia K. and Gregg G. Van Ryzin. 2011. Research Methods in Practice: Strategies for
Description and Causation. Thousand Oaks, CA: Sage.
Steen, Lynn Arthur. Editor. 2001. Mathematics and Democracy: The Case for Quantitative
Literacy. National Council on Education and the Disciplines.
Steen, Lynn Arthur. (2004) Achieving Quantitative Literacy: an Urgent Challenge for Higher
Education. Mathematical Association of America.
22
Appendix A: In-class group exercises
I. Correlation and causation
A research article reports that looking across schools, there is a correlation between mean test
score and whether or not the school library has a qualified librarian. The Association of School
Librarians picks up on the study and says that it shows that better librarians result in better
student learning and test scores and therefore funding for qualified librarians should be
increased.
(a) According to the librarians, what is the dependent variable and what is the independent
variable? What is the unit of analysis in the study?
(b) Describe a theory that is consistent with the librarians’ view of what causes what. Use both
words (a few sentences at most) and a path diagram. Make sure to include some intervening
variables (i.e., a mechanism)—at least a start at convincing a foundation to give money to
support qualified librarians.
(c) Describe an alternative theory that both contradicts the librarian’s position and explains the
correlation in the study. Use a path diagram and words (a few sentences at most).
(d) What is the relevant counterfactual question?
(e) Explain in a few sentences what all of this has to do with the idea of endogeneity.
23
Appendix B: Questions for In-class Discussions of Readings
Reading Discussion Example I: Peer-reviewed research article
Reading is Cattaneo, Matias D., Sebastián Galiani, Paul J. Gertler, Sebastián Martinez, Rocio Titiunik.
2009. “Housing, Health and Happiness” American Economic Journal: Economic Policy 1(1): 75-105.
Reading instructions: You do not need to read section VI (robustness checks) and you can skim
section III (data). Throughout, do not worry about the details but focus on the big picture of the
basic idea of the study.
Questions on Catteneo et al “Housing, Health and Happiness”:
(1) What is (are) the research question(s)?
(2) What type of study is this?
(3) What are the outcome measures?
(4) How valid is the causal evidence (internal validity)? Justify it. What weaknesses are there?
(5) What were the findings? Focus on Table 5.
(6) Remembering the 3 main things to look at—statistical significance, practical significance and
evidence of causation—how do you assess the findings?
(7) How generalizable are the results?
24
Reading Discussion Example II: Media articles
Read the stories on these web pages, in order
http://www.cnsnews.com/node/62812
http://www.nejmjobs.org/rpt/physician-survey-health-reform-impact.aspx
http://www.themedicusfirm.com/pages/medicus-media-survey-reveals-impact-health-reform
http://www.themedicusfirm.com/pages/survey
Questions on Medicus Survey of Physicians about Health Reform Readings
1) The New England Journal of Medicine (NEJM) is a peer-reviewed journal. Was the study peerreviewed? Was it published in the NEJM? What does the Medicus Firm do?
2) What is the sampling frame or method of obtaining sampling units? What population does
the sampling frame represent? What population does the survey interpreters claim that it
represents? Are the two populations different in any way? Is there coverage bias in the “would
try to leave medical practice” measure? Explain.
3) What is the size of the true sample? What is the size of the observed sample? (Explain the
difference between the true and observed sample.) Which documents report the true sample?
4) Who is likely to respond and who is not? How might the propensity to respond produce bias
in the “would try to leave medical practice” measure?
5) Discuss the relationship between stating in a survey that one “would try to leave medical
practice” and actually leaving medical practice in fact. Are the persons surveyed competent to
provide the desired information?
25
Appendix C: Examples of Out-of-class Assignments with Corresponding Rubrics
I. Logic model/mechanism assignment
Consider a policy or social program that actually exists, that you would like to propose or that
someone else has proposed. Choose something that interests you and that you know
something about. Prepare a description of the theory of the mechanism of how your program
works.
Write this up as memo to a boss or collaborator who is working with you to develop the
program. This is not someone you need to convince about the importance of the outcomes or
the program.
Make sure that you including the following:
(1) What is (are) the outcome(s) (dependent variable(s)) the program is designed to affect? If
there are many outcomes, restrict your analysis to one outcome or two closely related
outcomes. (For example, your program program’s goal might be to raise high school graduation
rates in urban areas and so the outcome is graduation rate.) Make sure that you state the
outcome(s) explicitly.
(2) Describe your program—what it is. This should be as explicit as possible, not vague
generalities. This section should be brief: a half double-spaced page at the most. Do not include
marketing or promotion: your reader does not need to be convinced of the importance of the
project. Write an objective and concrete statement of what the program literally does but do
include implementation details.
(3) Using a path diagram and a narrative description, describe your theory of how the program
is supposed to work. Both the path diagram and the narrative description should make clear the
mechanism(s) through which the program will affect the outcome. So, if a link is not obvious,
break it down into the steps along the way, illustrating the intervening variables.
This section should illustrate to your readers why they should believe that the program will
work—will affect the outcome(s). It should also make clear what the weak linkages are.
This part is the main focus of the assignment.
Hints: The circles represent variables and the arrows represent causal effects. Make sure that
you understand clearly the unit of analysis in your theory—the individuals to whom the variable
applies: For example, is the program working on students, on schools, on cities?
There can be many mechanisms through which a program works. If so, pick only a couple and
just note that there are other mechanisms. These should be more detailed than the logic
26
models you see in many grant proposals and papers. Each link should be spelled out and made
believable.
(d) Qualitative research: Suppose that you want to make sure that one part (link or links) of
your mechanism works how you think it does. Describe briefly (in two paragraphs or less) some
qualitative research you could do once the program is running to see if this part of your theory
is correct. What method (structured interviews, focus groups, observation, etc.) would you use?
What would you expect to learn from this?
Notes:
Do not include introductions, motivations, background, marketing and so on.
Do not include inputs, resources, or (detailed) activities. This logic model is not an
implementation-oriented one: It focuses on mechanism. Implementation is done more
effectively after you understand clearly the mechanism.
Check that each separate causal link makes sense isolated. Check that you are not missing
causal links between variables on the page.
27
Rubric for Logic Model Assignment
Component
A level work
B level work
C level work
F level work
Independent
variable
Clearly defined indep
var in narrative and
path diagram
Clearly defined indep
var in narrative and
path diagram
Some definition of
indep var in either
narrative or path
diagram
No clearly defined
indep var
All effects on
outcome (except
contextual vars) lead
ultimately from
indep var
Some effects on
outcome lead
ultimately from
indep var
Clearly defined Dep
var in narrative and
path diagram
Clearly defined dep
var in narrative and
path diagram
Some definition of
dep var in either
narrative or path
diagram
No clearly defined
dep var
No other unspecified
outcomes are de
facto outcomes
Almost no other
unspecified
outcomes are de
facto outcomes
Clearly defined Interv
variables in narrative
and path diagram
Clearly defined Interv
variables in narrative
and path diagram
Dependent
variable
Intervening
variable
The bulk of
preceding variables
are logical causes
and following
variables are logical
consequences
Does not confuse
process and
mechanism
Most of preceding
variables are logical
causes and following
variables are logical
consequences
Does not confuse
process and
mechanism
Other unspecified
outcomes are de
facto outcomes
Some definition of
interv variables in
either narrative of
path diagram
Some of preceding
variables are
logical causes, and
some following
variables are
logical
consequences
Intervening
variables are not
variables
Preceding
variables are not
logical causes and
following variables
are NOT logical
consequences
Confuses process
and mechanism
28
Narrative portion is
clear, concise and
Mechanics of avoids marketing in
Assg
favor of program
description.
Clearly describes
variables and
mechanisms.
The bulk of path
diagram has clearly
drawn variables,
explicit relationship
arrows, and includes
signs to show
positive or inverse
causal effects
Writing
quality
Narrative portion is
clear, concise and
contains little
marketing in favor of
program description.
Narrative portion
is confusing and/or
contains mostly
program
marketing
language.
Describes variables
and mechanisms,
Path diagram is
missing some
variables and/or
some arrows and
signs
Most of the path
diagram has clearly
drawn variables,
explicit relationship
arrows, and includes
signs to show
positive or inverse
causal effects
Narrative does not
explain the
program or
variables.
Path diagram
missing many
arrows and signs.
Writing is very clear
Writing is fairly clear
Writing is unclear
Writing is unclear
Arguments are
cogent and
persuasive
Arguments are fairly
cogent and
persuasive
Arguments are
cogent and
persuasive
Arguments are
cogent and
persuasive
Memo’s organization
is sensible and clear
Memo’s organization
is mostly sensible
and clear
Poor organization
No organization
Language is mostly
correct
Language has
mistakes
Some unnecessary
repetition
A lot of
unnecessary
repetition
Language has
significant
mistakes
Language is correct
and concise (with no
repetition)
Much repetition
29
Rubric for Qualitative component
Component
A level work
B level work
C level work
F level work
Qualitative
method
Appropriate
qualitative
method chosen
Qualitative
method chosen
Non-qualitative
method
No method
Description of
research
question
Clearly described
purpose to
qualitative
research
Research
question
described
No research
question
described
No research
question
described
Question clearly
relates to link in
mechanism
Question sort of
relates to link in
mechanism
30
II. Survey Design Assignment
In this assignment, you will design a survey, including a questionnaire, to gather information on
a population of interest. This population and the information that you gather about them
should be relevant to some policy or practice question of interest.
Purpose
What is the purpose of this survey? What information are you seeking and how will it be used?
Your purpose can be purely descriptive—the way the world is. Alternatively, your purpose can
be to obtain variables (dependent, independent and/or control) to address a causal question.
(Example: You survey adult immigrant English as a second language students and you want to
know if having an immigrant instructor influences satisfaction with the instruction program.)
Explain, as specifically as possible, how the results will be used for policy or practice.
Population
Define the target population of interest. For example, it could be clients of a program you work
on or a group, like parents of public school children.
Describe any differences between the ideal target population for your purpose and the actual
study population. (Usually, it is not possible to study the ideal population…)
Measures
Describe the characteristics that you would like to know about your population—the measures
or variables. Choose between four and eight substantive measures, plus any demographics (like
age and gender) that are relevant. Write out the conceptual definition—the construct-- of each
substantive variable. The construct is what you really want to capture.9
Identify which questionnaire item corresponds to each variable. Identify the level of
measurement (nominal categorical, ordinal categorical, or quantitative).
9
For example, if you want to learn about income, do you want to include in that measure, unearned income from
government benefits and income from investments? What is the deeper purpose or meaning of income in your
study?
31
Discuss how well the measures from your questionnaire do at capturing the construct that you
wish to measure. Describe any problems of answer bias that you anticipate and why.
Sampling Plan
Describe how you will sample and contact survey respondents. Will you survey everyone in the
population (a census) or will you sample? If you will sample, will it be a simple random sample?
How will you contact respondents?
Mode of data collection
Which mode of data collection will you use—in-person, telephone, self-administered
questionnaire, Internet survey or some other approach? Briefly justify your choice, stating its
pros and cons relative to alternative choices.
Questionnaire
Design and provide a questionnaire to measure your substantive variables and the needed
demographic variables.
You may use a few standard questions for existing questionnaires, but you must credit them.
Otherwise, the words of your questions must be your own.
Table shells and made up results
Illustrate the way that you will use the results of the survey by showing the shells of tables for
results. Then make up examples of numerical results that you might conceivably find. Use the
principles of good table construction described in chapter 15.
Critique survey, including predicting biases
What problems of coverage bias are likely? Describe what drives the differences between those
who are in the sampling frame and those who are not. Predict the direction and extent of
coverage bias for each of your variables. (For example, is the variable biased upwards? Is one
category much less likely due to bias?)
32
What problems of non-response bias are likely? Describe what drives the differences between
those who are more likely to respond and those who are less likely. Predict the direction and
extent of non-response bias for each of your variables.
More generally, critique your survey. Explain its weaknesses but explain why it is still useful.
Note 1: The entire assignment should be at most eight double-spaced pages, not counting the
questionnaire itself. This will require substantial editing to write all of the required information
succinctly.
Note 2: Make sure that you read through the assignment carefully to make sure that you have
answered each part.
Note 3: While you may get help from others both inside and outside this class, you need your
own individual topic
Hint: Do not start this assignment by thinking about “what could I ask?” Instead, start by
thinking about “what do I want to know?” What you can ask may fall short of what you want to
know. But you want to know how they differ and you might be able to get closer.
33
Rubric for Survey Design Assignment
Component
Purpose
Population
Measures
A Level Work
B Level Work
C Level Work
F Level Work
Survey explains
its purpose
clearly and
concisely. It
explains not only
its intent but
also how its
findings will be
used for policy or
practice.
Survey explains
its purpose
clearly. It
explains its
intent and
mentions how its
findings will be
used for policy or
practice.
Survey explains
its purpose in a
confusing or
incoherent
manner. It fails
to explain either
its intent or how
its findings will
be used for
policy or
practice.
Survey does not
explain its
purpose or how
its findings will
be used for
policy or
practice.
The target
population is
clearly defined
and the choice is
persuasively
justified given
the survey
purpose.
The target
population is
clearly defined
and the choice is
somewhat
justified given
the survey
purpose.
The target
population is not
clearly defined
or its choice is
illogical given the
survey purpose.
The target
population is not
defined.
The study
population is
juxtaposed with
the ideal target
population and
any differences
are clearly and
concisely
articulated.
Makes mention
of some
differences
between the
study population
and the ideal
target
population.
Describes clearly
and concisely 4
to 8 substantive
variables as well
as the constructs
they are
Describes 4 to 8
substantive
variables as well
as the constructs
they are
intended to
Makes little
mention of the
differences
between the
study population
and the ideal
target
population.
Describes fewer
than 4
substantive
variables or
describes them
in a confusing
Makes no
mention of the
differences
between the
study population
and the ideal
target
population.
Fails to describe
the variables
used or makes
no mention of
the constructs
intended to be
34
intended to
measure.
Identifies
questionnaire
questions
associated with
each measure.
Sampling Plan
Clearly,
concisely, and
systematically
explains how the
sample will be
constructed and
how
respondents will
be contacted –
including callback strategies.
Makes
appropriate use
of technical
terminology.
Mode
Clearly and
concisely
explains the
mode of data
collection used
Persuasively
measure.
manner and
makes little
mention of the
constructs they
are intended to
measure.
measured.
Identifies
questionnaire
questions
associated with
each measure.
Does not clearly
identify
questionnaire
questions
associated with
each measure.
Does not identify
questionnaire
questions
associated with
each measure.
Clearly and
concisely
explains how the
sample will be
constructed and
how
respondents will
be contacted.
Explains in a
confusing
manner how the
sample will be
constructed and
how
respondents will
be contacted.
Does not explain
how the sample
will be
constructed
and/or how
respondents will
be contacted.
Makes
appropriate use
of technical
terminology.
Does not make
appropriate use
of technical
terminology.
Clearly explains
the mode of data
collection used
Explains in a
confusing
manner the
mode of data
collection used
Justifies the
choice of mode
by mentioning its Cursorily justifies
Does not explain
the mode of data
collection used
Does not justify
the choice of
mode
35
justifies the
choice of mode
by clearly
articulating its
pros and cons
relative to
alternative
modes
pros and cons
relative to
alternative
modes
the choice of
mode used
and/or does not
mention its pros
or cons relative
to other modes
Has clearly
worded
questions that
are suitable for
the study
population.
Meets high
standards of
question
wording.
Questions are
suitable for the
study
population.
Questions are
confusing but
suitable for the
study
population.
Many questions
are not clear or
well worded.
Questions are
inappropriate for
the study
population.
Creates clear and
correct table
shells
Creates table
shells that are
not fully clear or
not fully suitable
Creates table
shells that are
not clear or not
suitable
No table shells
Table shells
somewhat
address survey
purpose
Table shells do
not address
survey purpose
Most questions
meet standards
for question
wording.
Most questions
are not clear or
well worded.
Questionnaire
Shell Tables
Table shells
clearly address
survey purpose
Puts reasonable
and clear
numbers in
tables
Puts mostly
reasonable
numbers in
tables
and/or no
numbers in
tables
Puts
unreasonable
numbers in
tables
36
Critique
Clearly,
concisely, and
thoughtfully
explores the
likely coverage
and nonresponse biases:
describing the
differences
between the
respondents and
the study
population and
prediction of the
direction and
extent of the
biases
Clearly,
concisely,
thoughtfully, and
realistically
assesses the
weaknesses and
value of the
survey
Explores the
likely coverage
and nonresponse biases:
describes the
differences
between the
respondents and
the study
population
But does not
correctly predict
the direction and
extent of the
biases
Assesses the
weaknesses and
value of the
survey
Explores the
likely coverage
and nonresponse biases
but does not
describe the
differences
between the
unreached
population and
those who
responded and
fails to give a
prediction of the
direction and
extent of the
biases
Does not explore
likely biases
Does not asses
the weaknesses
and/or value of
the survey
Cursorily
assesses the
weaknesses and
value of the
survey
37
Writing quality
Writing is very
clear
Writing is fairly
clear
Arguments are
cogent and
persuasive
Arguments are
fairly cogent and
persuasive
Essay’s
organization is
sensible and
clear
Essay’s
organization is
mostly sensible
and clear
Language is
correct and
concise (with no
repetition)
Language is
mostly correct
Writing is
unclear
Writing is
unclear
Arguments are
often not cogent
Arguments are
not cogent
Poor
organization
No organization
Much repetition
Language has
mistakes
A lot of
unnecessary
repetition
Some
unnecessary
repetition
38
Appendix D: Examples of short problems for practice and exams 10
1) You want to conduct survey to learn how safe New York City public school students feel, on
average, while at school.
(a) Describe in 1-2 sentences the sampling frame(s) you could use for this study.
(b) If your survey is limited to one question, write a good (closed-ended) survey question for
the purpose described.
(c) Explain in 1-2 sentences why it would be better to have more questions to assess how safe
students feel at school.
(d) If you wanted to do a qualitative study to explore students’ feelings of safety in school,
which form of qualitative study would you choose? Explain why in 1-2 sentences.
2) A charter high school surveyed its alumni, from a list it maintains for fund-raising purposes,
to find out how well they were doing in the job market. Nearly 50 percent responded, and the
average (mean) salary calculated from the survey was $79,000.
(a) Predict the direction and extent of non-response bias of the mean salary, explaining your
prediction.
(b) Recall that the sampling frame was a list maintained for fundraising purposes. Predict the
direction of coverage bias of the mean salary, explaining your prediction.
3) In several observational studies, children who eat nutritionally balanced meals were found to
get higher grades in school, on average. Some suggest that providing nutritionally balanced
lunches to students in school will improve their learning.
(a) In the suggestion, what is the independent variable? What is the dependent variable?
(b) Create a path model showing one mechanism consistent with the suggestion above.
(c) Create a path model showing an alternative theory that is both consistent with the
correlation described and contradicts the causation implicit in the suggestion. Briefly explain
your path model in words.
10
Some of these problems are in Remler and Van Ryzin (2011).
39
(d) Design a quasi-experiment, using difference-in-differences, to determine the effect of
providing nutritionally sound lunches on student learning. Imagine that you have an assessment
that accurately describes students’ knowledge and skills. Describe briefly how you would design
the quasi-experiment and analyze the data.
(e) In 1-2 sentences, discuss the generalizability of the results of the study you designed.
5) You are principal of a school and want to learn how well parents feel the school
communicates with them. You send home a survey with students and then have each teacher
follow up with a phone call to parents about how important the survey is. What kinds of nonresponse or coverage problems will you have? What bias (including direction of bias) do you
think would result in your measure of school communication? Explain.
6) Suppose two manufacturing companies merge. Prior to the merger, company A had only one
health plan option: a restrictive closed panel HMO while company B had several health plan
options, ranging from the very restrictive to plans without any restrictions at all. After the
merger, all employees have only the restrictive option that A had for all of this time. Luckily, the
merger did not cause any lay-offs.
You have access to all of the health care utilization data for all employees from both A and B for
several years before and after the merger. You want to estimate the effect of health plan
restrictiveness on health care costs (spending on health care for an individual from all sources).
(a) What is the independent variable of interest? What is the relevant outcome (dependent
variable)?
(b) Explain how this event can be seen as a natural experiment. Discuss the quality (or lack of
quality) of the causal evidence. Explain.
(c) Describe how you would use this natural experiment and the available data in a differencesin-differences study to determine the effect of restrictive health plans on health care utilization
and health care costs. Specifically, describe:



What data you would use
How you would analyze that data (including a shell table)
How you would interpret the results
40
7) Your very large home meals service serves mostly elderly people but also a few younger
disabled people. Your boss wants to learn about the satisfaction of all your clients but is
particularly worried about the younger disabled clients and wants to make sure that you have
precise information about them. What kind of sampling should your satisfaction survey do?
Explain.
8) Imagine that the subway line cleanliness measures are being revised. One proposal is to have
raters count the number of pieces of trash in the center-most car of the train that arrives at the
end of the line on Wednesday evenings at 8pm. Another proposal is to have raters ride the line
several different times a week at different locations and say how dirty they think it is generally.
Which measure is more reliable? Which measure is more valid? Explain.
9) A large randomized field experiment of treatment for alcoholics was conducted with 10,000
subjects. They were randomized to either an intensive short-term inpatient treatment program
or a less intensive long-term outpatient treatment program with equal total costs. The main
outcome measure is long-term (5-year) sobriety. Results showed that 35% of those in the
inpatient program remained sober for 5 years while 55% of those in the outpatient program
remained sober for 5 years.
(a) The p-value associated with a test of the difference between the programs was less than
.001. What is the statistical significance of these results?
(b) What is the practical significance of these results?
(c) The study was performed according to standard ethical principles of informed consent:
subjects were fully informed about what the study would entail and voluntarily chose to
participate. What kinds of alcoholics would agree to participate in the experiment? What kinds
would not?
(d) Someone reading this impressive study concludes that long-term sobriety rates of all
alcoholics would be 20 percentage points higher if all alcoholics received long-term outpatient
treatment than if all alcoholics received short-term inpatient treatment. Do you think their
conclusion is valid? Explain.
(e) The study as described does not shed light on why the outpatient program had better
results. Describe how the use of one qualitative method (e.g., open-ended interviews,
41
structured interviews, focus groups, observation) during the experiment could have shed light
on this issue. Briefly describe which method you think would be best and why.
10) Excerpts from an Op-ed contribution from the New York Times Sunday City section, by E.S.
Savas, Baruch School of Public Affairs Professor:
“TRANSIT officials are decentralizing the subway system to improve service, cleanliness and ontime performance by appointing individual managers for each of the 24 lines. …
The authority plans to start with the No. 7 and L lines and evaluate the pilot program by
surveying riders after about three months. This implies only modest goals, as that time is too
short for major improvements.
Moreover, more money and manpower are to be allocated to those lines, making it impossible
to figure out whether any improvements result from better management or more spending. The
plan seems loaded to elicit favorable comments in the short term from riders on those particular
lines, which unlike the other 22 lines are isolated: they have separate tracks. …”
Evaluators of the present plan could use a differences-in-differences framework to evaluate
the impact of their decentralization program. Savas notes three problems that undermine the
ability of such an evaluation to allow generalizations to the long-term effects of creating
individual managers for each of the 24 lines.
(a) Briefly describe the differences-in-differences framework that could be used to do an
evaluation of the decentralization program.
(b) Explain the three problems Savas describes. Explain how they would undermine the desired
generalization from your differences-in-differences study. (Make sure to briefly define
generalization.)
42