The Paradox of
Policy-Relevant Natural Experiments
Gilles Chemla
Christopher A.Hennessy
Imperial College, DRM/CNRS, and CEPR.
LBS, CEPR, and ECGI
November 2015
Abstract
Conventional wisdom holds that natural experiments represent especially credible bases for
econometric inference, facilitating evidence-based policymaking. We demonstrate an inherent
fragility in policy-relevant evidence generated by exogenous involuntary …rst-stage randomizations, examining inference in parable economies. In the …rst (second) economy, econometric
analysis is policy-relevant (irrelevant) as the government is able (unable) to alter future policy.
In the …rst economy, but not the second, evidence is contaminated by endogeneity after-the-fact,
with measured treatment responses contingent upon (unknown) parameters of the government
objective function into which evidence is fed. A paradox arises: The irrelevant experiment is
bias-free, while the relevant experiment is biased after-the-fact. Similarly, a perceived-credible
experiment is contaminated by Hawthorne E¤ects while a non-credible experiment is not. Since
we consider anonymous measure-zero agents, Hawthorne E¤ects are shown to result from rational expectations, as distinct from behavioral biases. Together, our results imply the methodology
advocated by the randomization school will become unreliable when/if it ultimately succeeds in
convincing governments to engage in experiment-based policymaking.
JEL Codes: D04, H00, Q58, G18. We thank seminar participants at Stanford, U.C. Berkeley, Carnegie Mellon,
Boston College, LBS, Imperial College, Washington-Seattle, Miami, and Nova-Lisbon. We also thank Antoinette
Schoar, Manuel Adelino, Taylor Begley, and Douglas Gale for early suggestions. Hennessy acknowledges funding
from the European Research Council. Address correspondence to chennessy@london.edu.
1
1
Introduction
The randomized controlled trial (RCT) is viewed as the gold standard of empirical evidence in
medical science. Increasingly, this view is being carried over to the social sciences. For example,
Greenstone (2009) writes, “The gold standard for estimating the causal impact of a regulation
is the randomized trial.” In recent years, applied micro-econometricians have shown considerable
ingenuity in attempts to identify and exploit sources of independent variation in policy variables
arising in natural experiments. Angrist and Pischke (2010) herald the new methods as a “credibility
revolution”arguing empirical evidence delivered via random assignment represents a credible standalone product. As stated in the introduction to their textbook (2009), “A principle that guides our
discussion is that most of the estimators in common use have a simple interpretation that is not
heavily model dependent.”
Ingenuity notwithstanding, the credibility revolution has been criticized by some for allegedly
placing means before ends, elevating clever identi…cation over economic relevance. For example,
Rubinstein (2006) argues there has been some abandonment of economic questions in favor of novel
application of economic methods to other academic disciplines, what he terms “economic imperialism.” Similarly, Rust (2010) expresses concern that the reduced-form empirical research program
could devolve into an “elite infotainment”industry, with novelty of the identi…cation strategy trumping correct economic conclusions.
In response to such criticisms, empirical researchers have redoubled their e¤orts to prove the
policy relevance of their work. In fact, the asserted credibility of econometric evidence derived from
natural experiments has recently led to a greater desire and willingness to make policy decisions
based directly upon them. For example, Dhaliwal and Tulloch (2015) note the existence of an
“increasing trend towards considering rigorous evidence while making policy decisions.” Consistent
with this view, the mission of the Abdul Latif Jameel Poverty Action Lab (J-PAL) is to promote
what it labels “evidence-based policymaking” with “evidence” and “rigorous evidence” apparently
being equated with econometric estimates derived from random assignment. Writing for the Tobin
Project, Greenstone (2009) makes a call “to move toward a culture of persistent regulatory experimentation.”1 Du‡o (2004) argues, “Creating a culture in which rigorous randomized evaluations are
promoted, encouraged, and …nanced has the potential to revolutionize social policy during the 21st
century, just as randomized trials revolutionized medicine during the 20th.”
As illustrated by the preceding quote, in the area of applied micro-econometrics, the appeal
of evidence obtained from random assignment derives in no small measure from the intuitively
plausible premise that randomization will prove to be equally e¤ective in the social sciences as it
1
Spatt (2011) discusses general challenges in using experiments to inform …nancial regulation.
has been in medical clinical trials. In both medicine and economics, concerns over self-selection
are a central concern. In fact, Angrist and Pischke (2009) use medical treatment e¤ects as their
initial example of the confounding e¤ects that can arise from self-selection. Apparently, they view
selection/endogeneity as the problem standing in the way of causal inference, stating, “The goal of
most empirical research is to overcome selection bias, and therefore to have something to say about
the causal e¤ect of a variable.” They go on to argue that, “The most credible research designs are
those that exploit random assignment.” Most often, in applied micro-econometric work, credibility
is established (lost) by demonstrating exogeneity (endogeneity) of the studied policy event.
In this paper, we provide a detailed exposition of an internal inconsistency at the heart of
the natural experiment research program. Our argument implies that the research methodology
advocated by the randomization school will become invalid when/if it succeeds in convincing the
government to engage in evidence-based policymaking. To illustrate, suppose an empiricist is able
to convince us that his identi…cation strategy is “clean,” making a compelling argument that, say,
Nature herself forced an exogenous change in policy. He is next likely to be questioned regarding
the relevance of his work. But suppose that here too the empiricist can easily meet the challenge,
demonstrating that the experiment has a direct bearing on policy, e.g. he was just in Washington
last week advising the relevant agency about his …ndings. However, at this stage the empiricist must
concede that, in establishing the relevance of his study, he has also demonstrated that the statistical
distribution of the policy variable is being determined using the outcome of his experiment as a
source of information. As we show, this type of feedback from empirical …ndings to policy-setting
clouds the interpretation of once-clean experimental evidence. Importantly, we demonstrate the
failure to account for such feedback, as is presently the norm in the applied micro-econometrics
literature, can lead to faulty inference.
In situations like the one just described, the problems of endogeneity and model-dependence
are actually just pushed back in time. While the initial treatment may well be independent of the
(unknown) deep parameters of the government’s objective function, the subsequent policy variable
path will depend on both the experimental evidence and these deep parameters. But with forwardlooking agents this gives rise to endogeneity-after-the-fact. In particular, measured responses to the
initial randomization are dependent upon subsequent endogenous government decisions.
We begin by contrasting econometric inference in two parable economies. In both economies,
there will be a …rst-stage natural experiment featuring an exogenous change in a policy variable,
pollution regulation, with …rm investment responses being analyzed by an econometrician. Each …rm
is measure zero and has no ability to in‡uence inference, ruling out a trivial source of bias. In the …rst
economy, econometric analysis is policy-relevant as the government has the ability to alter regulation
at a future date. In the second economy, econometric evidence is irrelevant as the government has no
2
ability to change the policy variable post-experiment. In the …rst economy, econometric evidence is
contaminated by endogeneity after-the-fact, with treatment responses contingent upon parameters
of the government objective function into which the empirical estimates will be fed. In the second
economy, the evidence is free of such contamination. Thus, we have the following paradoxical
situation: The irrelevant natural experiment is bias-free, while the relevant experiment is biased
after-the-fact.
We next examine the link between the perceived credibility of natural experiments and Hawthorne
E¤ects (see Levitt and List (2011) and Zwane et al. (2011)). A Hawthorne E¤ect is said to arise if
agents change their behavior under observation. The organizational behavior and psychology literatures have postulated a range of behavioral rationales for such e¤ects such as self-consciousness, peer
e¤ects, or a desire to in‡uence study outcomes. Importantly, our model abstracts from such e¤ects
since …rms are measure zero and anonymous. Here again we contrast econometric inference in two
parable economies. In both economies the government has the ability to choose the policy variable
post-experiment. The economies di¤er in that in one of them the government views the natural
experiment as providing credible evidence and in the other the government views the experimental
evidence as non-credible. In the former economy, …rm behavior changes according to whether or
not they are being observed, while in the latter economy there are no observer e¤ects. Intuitively,
if agents know experimental evidence is viewed as credible by the government, they postulate a
di¤erent distribution of the policy variable according to whether or not observation is taking place.
Thus, we have the following paradoxical situation: The perceived-credible experiment is Hawthornecontaminated, while the perceived-non-credible experiment is Hawthorne-uncontaminated.
Operationally, our arguments imply it is invalid to treat the estimates from natural experiments
as separable plug-in quantities that can be fed into arbitrary government objective functions. Rather,
we show that policy-relevant empirical analysis must account for the government objective function
at the inference stage. Of course, this directly contradicts the argument that evidence drawn from
randomized assignment is “not heavily model dependent.”
Essential to our argument is that the agents exposed to the …rst-stage randomization make decisions during the experiment (as opposed to crops, cells, or particles), and that those decisions are
functions of the future probability distribution of the policy variable the randomized experiment
will in‡uence. Thus, the choice of an investment decision as the basis for our exposition is not
coincidental. However, it is apparent that our argument applies to a broad range of decision problems involving the accumulation of stock variables, for example o¤spring, health, household/…rm
debt, …rm capital structure, reputation, or human capital. Further, our argument applies to many
super…cially “one-o¤” decisions once one accounts for life-cycle considerations, e.g. labor supply.
We turn now to related literature. Our paper imposes the discipline of rational expectations
3
on the micro-econometric random assignment literature. Lucas (1976) imposed the discipline of
rational expectations on the macro-econometric literature, showing it is invalid to obtain econometric
estimates regarding agent decision rules under one government policy rule (e.g. in‡ation targeting)
and to then treat the same agent rules as being operative as one contemplates a di¤erent government
policy rule (e.g. output stabilization). In Lucas’argument, econometricians sit outside the model
in that their estimates are not part of the information set of agents inside the model. In contrast,
critical to our argument is that econometricians sit inside the model, with our focus being on the
link between econometric estimates and government policymaking. It is the anticipated interaction
between econometricians and the economy they study that is the underlying source of the paradox
we raise, upon which Lucas (1976) is silent. Viewed from this perspective our point is a simple
one: The goal of an econometric estimation is to change the government’s information set, but if
this is the case, applied econometricians cannot simply act as if the information will have no e¤ect.
In general, an alteration of information sets alters actions, with concomitant implications for the
quantities the econometrician measures.
The macro-econometric literature has focused on the implications of rational expectations for
the interpretation of vector autoregressions. Sargent (1971, 1973, 1977) and Taylor (1979) showed
that rational expectations implies restrictions on distributed lags of vector autoregressions. Sims
(1982) and Sargent (1984) pointed to an asymmetry in rational expectations econometrics practice in
postulating optimizing behavior on the part of households and …rms while assuming non-optimizing
behavior by the government. In our paper, we discuss the correct interpretation of experimental
data assuming all agents, including the government, behave optimally and make optimal use of the
information available to them.
Discussion of rational expectations and its implications for randomized experiments has been
limited. In this regard, our paper is most closely related to that of Hennessy and Strebulaev (2015)
who analyze the meaning of econometric evidence derived from an economy hard-wired with an
in…nite sequence of exogenous natural policy experiments, with zero endogeneity bias at any stage.
In contrast, we here consider an economy with only two policy changes. The …rst policy change
arises from an exogenous natural experiment. The second policy change is an optimal response
to evidence derived from the …rst. It is this second-stage optimization that is the source of the
endogeneity problem and paradox we discuss.
The rest of the paper is as follows. Section 2 describes technology and timing. Section 3 derives
the optimal …rm policies. Section 4 uses the two models as the basis for discussing econometric
inference for economy-wide experiments. Section 5 discusses Hawthorne E¤ects. Section 6 discusses
measured di¤erences between treatment and control groups.
4
2
The Model
We begin by analyzing two parable economies endowed with identical real technologies and identical
natural experiments supplied by Nature. The only di¤erence between the two economies is whether
or not the empirical evidence coming from the natural experiments can be utilized subsequently in
setting government policy. In the Exogenous Policy Economy, the natural experiment is irrelevant in
that parameters estimated by econometricians do not feed into government policy decisions. In the
Endogenous Policy Economy, econometric evidence derived from the natural experiment is relevant,
being used to select a socially optimal long-term regulation policy.
2.1
Technology
We build on the q-theory of investment as formalized by Abel and Eberly (1997). There is no government policy in their model and so they are silent on the issue of interpreting natural experiments.
Time is continuous and the horizon in…nite. All agents are risk-neutral and discount at a common
rate r > 0: There is a measure one continuum of …rms. The fact that each …rm is measure zero
implies that no individual …rm has an incentive to distort its behavior with the goal of in‡uencing
econometric inference.
We describe the optimal control problem confronting an arbitrary …rm suppressing time and …rm
identi…ers where obvious to conserve notation. The law of motion for the …rm-level capital stock is:
dkt = (it
kt )dt
k0 > 0:
(1)
The variable i denotes instantaneous gross investment and
0 is the depreciation rate.
For simplicity, we assume the total instantaneous ‡ow cost to investing is:2
i
with
> 0 and
2 f2; 3; 4; :::g: Assuming
=(
1)
(2)
> 1 is su¢ cient to ensure convexity of the investment
cost function. Restricting attention to integer values of
allows us to derive closed-form solutions
for the growth option portion of the value function, as de…ned below.
Firm cash-‡ow is denoted
and is equal to operating pro…ts less investment costs. We assume
the following functional form with state variables as de…ned below:
(x;
2
S ; bT ; k; i)
(x + bT
S )k
i
=(
1)
:
(3)
One could add direct costs of capital goods and irreversibility at the cost of complexities extraneous to our
arguments. See Abel and Eberly (1997).
5
The operating pro…t factor x in the preceding equation is a geometric Brownian motion with the
following law of motion:
dxt =
xt dt + xt dwt
(4)
x0 > 0:
Above w denotes a standard Wiener process. To ensure bounded …rm value, it is assumed r >
+
1 2
2
(
1):
To …x ideas, we shall think of all …rms as facing a common x process. This assumption is not
necessary, but serves the expositional purpose of generating the type of real-world macroeconomic
risk to which a government might be expected to respond, with endogeneity issues being a potential
concern to the econometrician. By construction, our natural policy experiments will allow the
econometrician to avoid this standard source of endogeneity bias.
The term b entering the cash-‡ow identity captures government policy in a parsimonious way.
The variable b measures private bene…ts to polluting. As described below, the state variable
depends on the stage S 2 fP; E; Ig, with
S
2 f ; g at each stage. Stage P refers to time periods
prior to the arrival of the natural experiment. Stage E refers to time periods during the natural
experiment. Stage I refers to periods after the natural experiment when a long-term policy is implemented. In reality, an empiricist might be concerned that the policy variable under consideration,
in our model, is correlated with x; creating a potential endogeneity bias. Such an e¤ect is ruled
out in our analysis, as the processes
and x are assumed to be independent.
Firms are heterogeneous in terms of whether they cause pollution. There are two types. A
high pollution type (T = H) creates k units of pollution while production by a low pollution type
(T = L) creates zero pollution. To allow for both possibilities, the cash-‡ow equation (3) expresses
the instantaneous ‡ow bene…t to pollution as bT
S k;
with:
bH > bL = 0:
(5)
That is, a type H …rm generates pollution and derives a private bene…t from this. A type L …rm
does not derive any such bene…t since it does not pollute. The …rm knows its own type but not
other …rm types. To create a rationale for government regulation, we assume polluters impose a
negative externality on non-polluters. Speci…cally, each unit of pollution generated by type H …rms
imposes a cost c on each of the non-polluting …rms.
Complicating the determination of the optimal regulatory policy is the fact that the true measure
f of type H …rms is not known. Rather, agents know types were randomly drawn by Nature at date
0 from one of N possible distributions, with N
3. For each possible distribution n 2 f1; :::; N g,
6
agents enter the model with a correct prior pn that the true distribution is n. If the true distribution
is n, the measure of Type H …rms is fn . The indexing convention is:
1
f1 > f2 > ::: > fN
0:
(6)
A detailed description of each stage follows in the next subsection.
2.2
Timing
The model opens at date 0 with a Pre-Experiment Stage P . During Stage P …rms face a common
pollution factor
P,
with
P
pre-determined by Nature. That is,
P
is the economy’s initial tech-
nological endowment and does not represent the outcome of a government decision. This rules out
econometric issues arising from pre-experiment policy endogeneity.
During Stage P …rms optimally choose their instantaneous investment rates (i). They make
decisions knowing that a one-time natural experiment will arrive at a random future date
instantaneous transition rate into the Experiment Stage E is
E
1
E
:
At the start of the Experiment Stage E, Nature randomly assigns a fraction
=
, with remaining …rms assigned
E
=
The
> 0: It follows that at any instant
prior to such a transition, the expected remaining duration of Stage P is
E
E:
of …rms to
. The assignments last throughout Stage E: We
consider two types of experiments. In the …rst experiment type,
2 f0; 1g, meaning all …rms are
assigned the same economy-wide regulatory treatment during Stage E: In the second experiment
type,
2 (0; 1), allowing us to consider the behavior of treatment versus control group data. In
the model, “unexpected” experiments can be handled parametrically by setting
E
to a very small
number. This would have no e¤ect on treatment versus control group di¤erences, the key measured
variable in the second experiment type. For the …rst experiment type, featuring economy-wide
regulation, unexpected experiments simply feature larger responses, with no e¤ect on the substance
of our arguments. That is, as can be seen below parametrically, invoking the notion of an experiment
being “unexpected” does not provide a defense against our critique.
During Stage E …rms choose gross investment rates optimally taking into account the fact that
the economy will transition into the Implementation Stage I at a random future date
instantaneous transition rate into the Implementation Stage is
to the transition, the expected duration of Stage E is
1
I
I
I:
The
> 0: Thus, at any instant prior
:
By construction, random assignment during the Experiment Stage o¤ers the econometrician
information free of endogeneity before-the-fact. After all, Experiment Stage treatments are independent of the stochastic process x and independent of …rm-speci…c characteristics. Further, as we
show below, a sophisticated econometrician will be able to correctly infer the relevant unknown information based upon a single simple summary statistic. Speci…cally, in the case where di¤erent …rms
7
are assigned di¤erent treatments during Stage E, the di¤erence between the aggregate investment
of treatment versus control groups su¢ ces to infer f: In the case of an economy-wide experiment,
where all …rms are assigned the same treatment during Stage E, the aggregate investment response
su¢ ces to infer f: Since f will be correctly inferred, other empirical summary statistics would be
redundant.
At the very start of the Implementation Stage I econometricians observe the empirical evidence.34 A long-term regulatory policy is then implemented. For simplicity, it is assumed no further
investment is possible during Stage I. Since adjustment costs are independent of the capital stock,
ruling out investment during Stage I has no e¤ect on optimal …rm-level investment decisions during
stages P and E.
The only di¤erence between the Exogenous and Endogenous Policy economies is in the determination of
value
EX
I
I.
In the Exogenous Policy Economy,
I
is set equal to some exogenously stipulated
2 f ; g and the government is powerless to alter this fact. In this economy, evidence
derived from the natural experiment is policy-irrelevant.
In the Endogenous Policy Economy, the government is free to choose
I
2 f ; g: It does so
utilizing econometric evidence derived from the natural experiment. Since the aim here is to focus
on econometric issues, the setup is designed to generate a simple government decision rule. Given
that investment is impossible after the Implementation Stage, the government can determine the
socially optimal policy by comparing total private bene…ts and total social costs of pollution on a
‡ow basis. Letting KH denote the total capital stock of the polluting H-types, aggregate pollution
is equal to KH : The instantaneous ‡ow of private bene…ts to pollution exceeds social costs if:
bH KH
(1
f )cKH :
(7)
Ex post it is socially optimal for the government to allow high pollution levels ( ) if the fraction of
…rms bene…ting from pollution is su¢ ciently high. We have the following socially optimal policy:
f
< 1
f
1
bH
)
c
bH
)
c
I (f )
= :
I (f )
= :
(8)
To make the public policy problem interesting, assume neither regulatory policy is optimal for all
possible values of fn . In particular, assume there is an n 2 f1; :::; N
optimal threshold for switching from
to ; with:
fn
3
4
1g representing the socially
1
bH
> fn
c
+1 :
Observation during Stage E adds an uninteresting limbo phase where I is inferred, with
Letting …rms make the same observation as econometricians has no e¤ect.
8
(9)
E
still in e¤ect.
Implementation of the socially optimal regulatory policy described in equation (8) requires correct econometric inference regarding the true fraction of …rms bene…ting from pollution. Before
turning to the econometric challenges, it is necessary to …rst characterize optimal investment policies in the two economies, the subject of the next section.
3
Optimal Firm Policies
This section analyses optimal …rm investment policies. For each stage, we begin with an analysis of the Endogenous Policy Economy. From there, the Exogenous Policy Economy is simple to
characterize.
3.1
Implementation Stage
Superscripts ST denote stages (S) and types (T ) for value and policy functions. Optimal …rm
policies can be determined via backward induction. We …rst pin down the type-contingent value
function during the Implementation Stage, denoted V IT . We next determine optimal investment
policy during the Experiment Stage as well as the type-contingent value function V ET associated
with this stage. Finally, knowledge of the Experiment Stage value function allows us to pin down
optimal investment during the Pre-Experiment Stage.
The expected instantaneous holding return for each asset must be equal to the discount rate r:
The expected holding return on stock in the …rm is equal to the sum of cash-‡ow plus expected capital
gains. Using Ito’s Lemma to compute expected capital gains we have the following equilibrium
condition:
rV IT (x; k) = (x +
I bT )k
+ xVxIT (x; k) +
1
2
IT
x Vxx
(x; k)
2 2
kVkIT (x; k):
(10)
Given that cash-‡ow is linear in the capital stock, we conjecture the following linear value function:
V IT (x; k) = kq IT (x):
(11)
In the preceding equation q IT represents the shadow value of capital during the Implementation
Stage for a …rm of type T . Substituting the conjectured solution into the pricing equation and
simplifying we obtain the following ordinary di¤erential equation (ODE):
(r + )q IT (x) = (x +
I bT )
+ xqxIT (x) +
9
1
2
2 2 IT
x qxx (x):
(12)
This ODE has the following no-bubbles solution:
q IT (x) = x +
I bT
IT
r+
1
r+
IT
(13)
:
Throughout the analysis upper bars denote value and policy functions if
stage S. Lower bars denote value and policy functions if
S
=
S
=
during the current
during the current stage S. We
have:
IT
(x; k) = k x +
IT
with
IT
V IT (x; k) = k x +
IT
with
IT
V
bT
r+
bT
:
r+
The shadow value of installed capital during the Implementation Stage is increasing in
(14)
I
for
only those …rms deriving a positive private bene…t from pollution, and we recall that bH > bL = 0.
Thus, we anticipate that during Stages P and E only the investment of high types will be in‡uenced
by expectations regarding
3.2
I:
Agent Beliefs
In order for a …rm in the Endogenous Policy Economy to formulate its optimal investment policy,
it must form a probability assessment over the Implementation Stage regulatory policy that will
be chosen by the government on the basis of the experimental evidence. In the posited rational
expectations equilibrium, the government’s econometrician will make the correct inference regarding
the unknown parameter f so that the socially optimal regulation policy described in equation (8) will
be implemented. In this setting, we will contemplate the inference problem facing econometricians
sitting outside the government.
Recall that for each n 2 f1; :::N g; the …rm enters the model with a correct prior pn that the true
density is fn . The …rm will then revise its probability assessment based on its privately-observed
type. Intuition suggests that high (low) types should assign a relatively high probability to those
distributions featuring a higher fraction of high (low) types.
Let
T
n
denote the type T probability assessment that the true density is fn : From Bayes’rule
10
it follows:
H
n
= Pr [fn jT = H] =
Pr [fn \ bH ]
pn fn
= N
Pr [bH ]
X
pj fj
(15)
j=1
L
n
Pr [fn \ bL ]
pn (1 fn )
= Pr [fn jT = L] =
= N
:
Pr [bL ]
X
pj (1 fj )
j=1
Next, let
ment
I
T
denote the probability assessment of a type T …rm that the government will imple-
= : We have:
H
L
Pr [
Pr [
I
I
= jT = H] =
= jT = L] =
n
X
n=1
n
X
H
n
(16)
L
n:
n=1
The following lemma, demonstrated in the appendix, will be useful in the analysis.
Lemma 1 In the Endogenous Policy Economy, relative to prior beliefs, high (low) type …rms revise
upward (downward) their assessment of the probability of deregulation during the Implementation
Stage, with
H
>
n
X
pn >
L:
n=1
The intuition for the preceding lemma is as follows. A high type …rm assigns a relatively high
probability to those densities with a high fraction of high types. And the …rm also knows the
government will …nd it optimal to implement
for f su¢ ciently high. Thus, high types assess a
higher probability of deregulation than low types. Note, it is the conjunction of private information
regarding type and aggregate uncertainty regarding type that gives rise to heterogeneity in beliefs
regarding government policies.
3.3
Experiment Stage
Consider next the Bellman equation for the Experiment Stage in the Endogenous Policy Economy.
Here we must account for the possibility of transitioning to the Implementation Stage, which arrives
at rate
I.
Further, in the Endogenous Policy Economy the Implementation Stage regulatory policy
is not known. Rather, …rms have type-contingent beliefs regarding
11
I,
as described in the previous
subsection. The implied Bellman equation for the Experiment Stage is:
rV ET (x; k) = max
i
+(i
(x +
E bT )k
=(
i
k)VkET (x; k) +
I T
1)
h
V
1 2 2 ET
x Vxx (x; k)
2 i
IT
V ET (x; k) + I (1
(x; k)
T) V
+ xVxET (x; k) +
IT
(x; k)
(17)
V ET (x; k) :
We conjecture the value function is separable between the value of assets in place and growth
options. For each type T and each possible assigned value of
E,
the conjectured value function
takes the form:
V ET (x; k) = kq ET (x) + GET (x):
(18)
In order to pin down the optimal investment policy we con…ne attention to terms in the Bellman
equation involving the control i: Since the same form of investment program will recur throughout
the analysis that follows, it will be convenient to reference the following lemma.
Lemma 2 The optimal policy for the investment program
max
iq
i
i
=(
is
1
1
i (q)
1)
1
q
:
(19)
If q(x) = x + , the instantaneous gain accruing under the optimal control policy is
i [q(x)]q(x)
[i (q(x))]
=(
1)
=( x+ )
=
X
hx
h
h=0
where
(
h
1)
1
h h
h
1
:
The …rst part of Lemma 2 follows from the …rst-order condition. The second part of the lemma
follows from the binomial expansion formula. It follows from the lemma that the optimal investment
policy during the Experiment Stage is i (q ET ); with the function i being de…ned in equation (19).
It is readily veri…ed that the function i is strictly increasing, with i (q) > 0 for all q > 0 as will
be the case throughout our analysis. Figure 1 plots the investment policy function i for di¤erent
assumed values of the adjustment cost parameter :
12
We turn next to computing the shadow value of capital. Substituting the Implementation Stage
value functions from equation (14) into the Bellman equation we obtain:
(r +
I )[kq
= (x +
ET
(x) + GET (x)]
E bT )k
i
=(
k)q ET (x) +
+(i
I
(20)
1 2 2 ET
1)
+ x[kqxET (x) + GET
x [kqxx (x) + GET
x (x)] +
xx (x)]
2
x
(
+ (1
T ) )bT
k
+ T
r+
r+
In order for the Bellman equation to hold at each point in the state space, the terms scaled by k
must be equal. Isolating these terms we obtain the following ODE for the shadow value of capital:
(r+ +
I )q
ET
(x) = (x+
E bT )+
xqxET (x)+
1
2
2 2 ET
x qxx (x)+ I
x
+
r+
(
T
+ (1
r+
T)
)bT
:
(21)
We conjecture the following functional form for the shadow value of capital:
q ET (x) = x
ET
ET
+
:
(22)
Substituting the conjectured solution into equation (21) and solving one obtains the shadow
value for the Experiment Stage. We summarize the results for the Endogenous Policy Economy as
follows:
ET
i
(x) =
iET (x) =
q ET (x) = x +
ET
=
ET
=
1
1
1
1
ET
[q ET (x)]
1
[q ET (x)]
1
q ET (x) = x +
;
(r + ) + I ( T
(r + )(r +
(r + ) + I ( T
(r + )(r +
+ (1
+ I)
+ (1
+ I)
(23)
ET
T)
)
T)
)
bT
bT :
Consider next the Exogenous Policy Economy. It is readily veri…ed that in such an economy,
beliefs regarding
I
must be equal to the true value in this economy, which was denoted
EX :
I
That
is, we can simply take our earlier solution and make the following substitution:
T
+ (1
T)
!
EX
I :
(24)
It follows that in the Exogenous Policy Economy, the only change relative to equation (23) is that
we now have:
ET
=
(r + ) + I EX
I
bT ;
(r + )(r + + I )
ET
=
(r + ) + I EX
I
bT :
(r + )(r + + I )
(25)
Expressions for the respective growth option values are not required for the empirical analysis,
so we relegate their derivations to the appendix.
13
3.4
Pre-Experiment Stage
We consider next the Pre-Experiment Stage value function for the Endogenous Policy Economy.
During this stage the …rm knows that in the event of a transition to the Experiment Stage, which
arrives at rate
E,
it will be assigned
E
with probability : It follows that the Bellman equation
=
for Stage P is:
rV P T (x; k) = max
i
+(i
(x +
P bT )k
i
k)VkP T (x; k) +
=(
E
1)
h
V
+ xVxP T (x; k) +
ET
(x; k)
1 2 2 PT
x Vxx (x; k)
2i
V P T (x; k) +
(26)
) V ET (x; k)
E (1
V P T (x; k) :
We conjecture a value function separable between the value of assets in place and growth options:
V P T (x; k) = kq P T (x) + GP T (x):
(27)
Under the conjectured value function, the instantaneous control problem to pin down the optimal
investment (i) is identical in form to that described in Lemma 2. It follows that the optimal control
policy during Stage P is i (q P T ):
We conjecture the following linear functional form for the shadow value of capital:
q P T (x) = x
PT
+
PT
:
(28)
We turn now to pinning down the shadow value of installed capital. Substituting the conjectured
solution from equation (27) into the Bellman equation (26) we obtain:
(r +
E)
kq P T (x) + GP T (x)
(29)
1 2 2 PT
= (x + P bT )k
i =( 1) + x[kqxP T (x) + GPx T (x)] +
x [kqxx (x) + GPxxT (x)] + (i
2
h
i
ET
) kq ET (x) + GET (x)
+ E kq ET (x) + G (x) + E (1
k)q P T (x)
Equating the terms in the Bellman equation scaled by k, which must match, we obtain the following
ODE for the shadow value of capital:
(r + +
E )q
PT
(x) = (x +
xqxP T (x) +
P bT ) +
1
2
2 2 PT
x qxx (x) + E
q ET (x) +
E (1
)q ET (x): (30)
Substituting in the expressions for q ET and q ET from equation (23) into equation (30) and grouping
terms we obtain:
(r + +
E )q
P T (x)
= (x +
+
P bT )
x
E
r+
1 2 2 PT
x qxx (x)
2
(r + )( + (1
) ) + I(
(r + )(r + +
+ xqxP T (x) +
+
14
(31)
T
I)
+ (1
T)
)
bT :
We again conjecture the shadow value is linear. Substituting the conjectured linear solution into
equation (31) and solving we …nd:
q P T (x) = x
PT
PT
PT
PT
+
:
(32)
1
=
r+
P bT
r+ +
=
+
E
E
(r + )( + (1
) ) + I ( T + (1
(r + )(r + + I )(r + + E )
T)
)
bT :
Consider next the Exogenous Policy Economy. It is readily veri…ed that in such an economy,
beliefs regarding
I
must be equal to the true value in this economy, which was denoted
EX :
I
We
thus have:
PT
4
=
P bT
r+ +
+
E
E
(r + )( + (1
(r + )(r + +
) )+ I
I )(r + +
EX
I
E)
bT :
(33)
Econometric Inference I: Economy-Wide Experiment
This section considers the following economy-wide experimental setting. At each point in time,
all …rms are assigned the same pollution regulation. The initial technological endowment during
the Pre-Experiment Stage is
…rms are regulated, facing
…xed at
EX
I
=
E
P
=
. At the random arrival date of the Experiment Stage all
= . In the Exogenous Policy Economy, the policy variable remains
and the government is powerless to alter this fact. Although stylized, this setting
approximates the frequently invoked assumption made by empiricists that the studied policy change
is permanent. We note that if a policy change is indeed permanent, then the econometric evidence
generated by studying it is irrelevant in the sense that it is not possible to change the policy in the
future even if the evidence in favor of making a change is overwhelming.
Assume that in both economies econometricians look back at the data and measure the change in
aggregate investment that occurred at the start of Stage E. Implementation of the socially optimal
policy is as described in equation (8), requiring correct inference regarding the fraction of high
type …rms (f ). A sophisticated econometrician can correctly infer f by observing the instantaneous
investment response of …rms at the start of the experiment. Intuitively, it is only the investment
of high types that is responsive to changes in
; so the decline in investment at the start of the
experiment should be directly informative about f .
4.1
Numerical Illustration
Since the model has analytical solutions, no simulation is required. However, an initial understanding
of the econometric issues is perhaps best gained via speci…c examples. Let us then assume the
15
following parameter values: r = :05;
= :15;
= 0;
E
= 1;
I
= 1;
= :90;
= :20;
= 2;
= :20; and x = 1. There are 101 possible f values ranging from 0 to 1 in equal-sized steps. Agents
have uniformly distributed priors over the potential distributions. The private bene…t to pollution
is bH = :25 and the externality cost to pollution is c = :50. The socially optimal regulatory policy
for the government to set in the Endogenous Policy Economy is
I
=
if f
1=2. Here high and
low types do indeed form di¤erent beliefs regarding the Implementation Stage policy. In particular,
from equation (16) it follows that under the current parameterization
H
= :757 and
L
= :252.
Consider now Figure 2. On the horizontal axis is the fraction f of high types, the quantity the
econometrician wants to infer. On the vertical axis is the change in aggregate investment at the
start of the Experiment Stage. The solid line maps realized f values to investment responses in the
Endogenous Policy Economy and the dotted line maps realized f values to investment responses
in the Exogenous Policy Economy. The dashed line considers the Endogenous Policy Economy
parameterized with a higher pollution cost of c = :75: Notice, in all economies there will be no
investment response if f = 0 since no …rm is a¤ected. Further, in all economies, the investment
response is monotonically increasing in f: It follows that so long as the econometrician understands
the economy in which she is operating, she will be able to correctly infer the true realized value of
f based upon the observed investment response.
Figure 2 allows us to contrast the inference that will be made by a sophisticated econometrician
versus a naïve econometrician. Consider an econometrician working in the Endogenous Policy
Economy (c = :50), featuring evidence-based policymaking. The solid line in Figure 2 re‡ects
reality in this economy. If the econometrician is sophisticated, she will account for the link between
policymaking and empirical evidence and use the solid line in performing inference, resulting in
correct estimation of f . If the econometrician is naïve, she ignores the link between evidence and
policy and instead uses the dotted line to perform inference.
In the present example, the naïve econometrician will understate the true f: To see this, suppose
for example that the true realized f = :75; resulting in an observed investment response equal to
:10: Here the naïve econometrician will work along the dotted line and infer that this response
resulted from f = :28: Note, this is precisely the problem the …rst-stage random assignment was
supposed to solve. Recall, in the introduction the econometrician’s initial concern was that …rststage endogeneity would result in the understatement of negative program impacts. Here instead,
second-stage endogeneity causes the same form of bias to emerge.
The source of the bias here is obvious. Recall, the experiment consists of a period during which
Nature imposes regulation, with
remains …xed at
EX
I
=
E
=
. In the Exogenous Policy Economy, the policy variable
; with the government powerless to alter this fact. In contrast, in the
Endogenous Policy Economy, the government has the power to reverse Nature’s course and set
16
I
=
at its discretion. Consequently, the negative investment response to the introduction of
the regulation will be less severe in the Endogenous Policy Economy. The naïve econometrician
incorrectly interprets the less severe negative response as indicative of lower f values.
Consider next policy recommendations. The naïve econometrician would advise a policymaker
as follows. “It is socially optimal to deregulate if more than half the …rms in the economy bene…t
from pollution. And if indeed more than half the …rms bene…t from pollution, investment will fall
by at least 0.18 (dotted line evaluated at f = 1=2).” But turning attention to the solid line, which
re‡ects reality here, we see that in the Endogenous Policy Economy investment never falls by this
amount. Thus, the econometrician would always recommend the imposition of regulation. The
probability of an incorrect policy recommendation is 50%, and we are no better o¤ here than had
the econometrician made her recommendation by ‡ipping a coin. The bias towards regulation here
is the inevitable consequence of understating negative program impacts.
Next, let us return to the argument made in the introduction that econometric inference for
policy-relevant natural experiments must be predicated upon the deep parameters of the government
objective function into which they will be fed. To see this, suppose that government policy is set
endogenously, but that the true cost of pollution is c = :75: Here the government will only …nd it
optimal to deregulate during the Implementation Stage if the fraction of high types is greater than
2/3. Consequently, the negative investment response to the onset of the Experiment Stage will be
stronger in this economy than in an economy in which the government bases its decision on a cost
parameter of c = :50:
To see the potential for error here, suppose now that the econometrician accounts for feedback,
but performs inference under the incorrect premise that c = :50 while the true c = :75: In this
case, reality is re‡ected by the dashed line while the econometrician incorrectly performs inference
based upon the solid line. In the present example, the econometrician will overstate f . To see this,
suppose for example that the true realized f = :40; resulting in an observed investment response
equal to
:075: Here the econometrician will work along the solid line and incorrectly infer that this
large response resulted from f = :60:
In general, if the pollution cost is indeed high, the government, if advised by a sophisticated
econometrician, demands to see a much stronger negative investment response in order to recommend
deregulation. This is due to two factors. First, as discussed above, if the pollution cost is high,
the government sets a higher threshold f value for implementing deregulation. The second factor,
related to the …rst, is that for any given realized f value, the investment reduction is more severe in
the high pollution cost economy. This is revealed by comparing the solid and dashed lines in Figure
2. The stronger investment response in the high pollution cost economy is due to the fact that here
rational agents attach a higher probability to the econometric evidence supporting the continuation
17
of the regulation long-term.
Of course, the preceding argument also shows that the external validity of even ideal exogenous
…rst-stage randomizations must also be understood as predicated upon an assumption of equality
of parameters of government objective functions into which the evidence will be fed. In fact, it can
also be shown that external validity also demands equality of agent priors regarding the distribution
of the unknown parameter, here f , to be estimated. These are severe limits to external validity that
go beyond the standard caveats.
4.2
Formal Analysis of Economy-Wide Experiments
The objective of this subsection is to pinpoint the source of the inference problem discussed in the
previous section. This serves two purposes. First, one may want some assurance that the e¤ects
illustrated were not an artifact of speci…c parameter assumptions. Second, a formal analysis better
illustrates the subtleties of the underlying mechanism.
Our objective is to deconstruct the measured investment reaction to the introduction of the
randomized regulation. To begin we recall that only the investment of high types is a¤ected by
changes in : Therefore, the measured change in aggregate investment at the start of the Experiment
Stage (
E
= ) will be given by:
RESP ON SE = f
i
q EH (x)
1h
1
= f
i
q P H (x)
q EH (x)
1
(34)
q P H (x)
1
i
From the model analysis above we know the Pre-Experiment shadow values di¤er as follows:
Endogenous
:
Exogenous :
qP H = x +
qP H = x +
P bH
r+ +
P bH
r+ +
+
(r + )(
E
E
+
E
E
+ (1
) ) + I ( H + (1
(r + )(r + + I )(r + + E )
(r + )( + (1
) ) + I EX
I
bH :
(r + )(r + + I )(r + + E )
H)
)
(35)
And the Experiment-Stage shadow values also di¤er:
Endogenous:
Exogenous:
(r + ) + I ( H + (1
(r + )(r + + I )
(r + ) + I EX
I
q EH (x) = x +
bH :
(r + )(r + + I )
q EH (x) = x +
H)
)
bH
(36)
Since the shadow values di¤er across economies, it is apparent that the regulation response
function di¤ers across the two economies. To see this formally, note that the decline in shadow
18
bH
values at the introduction of the experimental regulation will di¤er across the economies, with:
q EH (x)
q P H (x) = q EH (x)
q P H (x) +
Exogenous
Endogenous
I bH [ H
+ (1
H)
(r + + I )(r + +
EX ]
I
E)
:
(37)
Notice, the source of the shadow value wedge across the two economies is the di¤erence between the
high type’s belief regarding the Implementation Stage policy in the Endogenous Policy Economy,
versus the Implementation Stage policy in the Exogenous Policy Economy. It follows that accounting
for heterogeneity in agent beliefs is critical to understanding the nature of potential biases.
We have the following proposition.
Proposition 1 For all
2 f2; 3; 4; :::g; the economy-wide regulation reaction functions di¤ er be-
tween the Exogenous Policy Economy and the Endogenous Policy Economy.
5
Econometric Inference II: The E¤ect of Credibility
This section utilizes the modeling framework to consider a di¤erent issue, the link between the
perceived credibility of evidence from random assignment and the nature of the evidence itself.
Again, it is most convenient to initially illustrate the arguments by way of a contrast between two
parable economies.
5.1
Parable Economies Redux
Consider two economies where …rms are hard-wired with the same real technologies as described in
the preceding section, while the government retains the same objective function. Further, assume
the same sequence of events as in the previous section. In particular, there is an exogenous
the Pre-Experiment Stage followed by an exogenous transition to
Stage followed by an Implementation Stage
I:
E
6=
P
P
during
during the Experiment
And, as in the preceding section, assume
is common
across all …rms.
In contrast to the preceding section, assume now that in both economies the government has
the ability to choose
I
2 f ; g: That is, the di¤erence between the two economies is no longer due
to di¤erences in the innate ability of the government to choose
I:
Rather, the di¤erence between
the two economies is in the government’s perception of the credibility of evidence generated by
randomized experiments. In one economy, Economy NC, the government views such evidence as not
credible. In Economy C the government views evidence arising from randomized policy assignment
as credible.
Finally, we assume that observation of the …rms by econometricians may or may not be feasible,
and that all agents know whether or not such observation is taking place.
19
5.2
Hawthorne E¤ects
To begin our analysis recall equation (8) in which
I (f )
denoted the ex post socially optimal
Implementation Stage policy given that the true value of f is known, e.g. based upon a correct
inference of f through a sophisticated econometric analysis. In contrast, let
IP
denote the optimal
Implementation Stage policy based upon prior beliefs. We have:
N
X
pj fj
< 1
bH
)
c
IP
= :
pj fj
1
bH
)
c
IP
= :
j=1
N
X
j=1
(38)
With this notation in-hand, consider government policy in Economy NC. In this economy, policy
is invariant to whether or not an econometric study takes place. If no study has taken place, the
government has no choice but to rely on its prior beliefs in setting long-term policy, so
I
=
IP :
If instead a study has taken place, Government NC does not view it as credible and so disregards
the evidence and so again
I
=
IP :
Proceeding via backward induction it follows that …rms in this
economy will base their decisions on the rational conjecture that
I
=
IP
regardless of whether or
not there is an econometric study taking place. Technically, all …rms in Economy NC behave just as
in the Exogenous Policy Economy, but with
EX
I
now set to
IP .
It follows there is no Hawthorne
E¤ect in Economy NC.
Consider next Economy C. In this economy, government policy during the Implementation Stage
will be contingent upon whether or not an econometric study has taken place. If no study has taken
place, the government has no choice but to rely on its prior beliefs and so
I
=
IP :
If a study has
taken place, the government views it as credible and uses the evidence supplied by the econometrician
to infer f implying
I
=
I (f ):
Proceeding via backward induction it follows that …rms in Economy
NC behave di¤erently depending on whether or not they are being observed. If they are not being
observed, they set investment policies during stages P and E just as in the Exogenous Policy setup
with
I
set to
IP :
If they are being observed, they set policies just as in our Endogenous Policy
Model since they rationally anticipate the government implementing
I
=
I (f ).
Returning to the derivation of equation (37), we can express the Hawthorne E¤ect via shadow
values. Recall, in Economy C, …rms behave just as they do in the Exogenous Policy Economy,
treating
EX
I
=
IP ,
if they are being studied, and otherwise behave just as in the Endogenous
Policy Economy. It follows that in Economy C, the jump in shadow value at the onset regulation
depends upon whether the …rms are being observed or not. In particular:
q EH (x)
q P H (x) = q EH (x)
Observed
q P H (x) +
Not Observed
20
I bH [ H
+ (1
H)
(r + + I )(r + +
Hawthorne E¤ect
IP ]
E)
:
(39)
We thus have the following proposition.
Proposition 2 If the natural experiment is viewed as credible (non-credible), the evidence is (not)
contaminated by Hawthorne E¤ ects.
5.3
Hawthorne E¤ect: Numerical Illustration
The preceding argument is readily illustrated by way of the example depicted in Figure 3. Figure 3 is
based on the same parameter values as Figure 2. In contrast to Figure 2, Figure 3 models economies
in which the government is always free to alter the policy variable once the Implementation Stage
is reached and will do so using either econometric evidence or its prior beliefs. Here, based on prior
beliefs, the optimal policy is
IP
= : Thus, in an economy in which the experimental evidence is
viewed as non-credible, …rms invest knowing the Experiment Stage regulation is transitory and will
be reversed with probability one. If instead econometric evidence is viewed as credible, then the
Implementation Stage policy is contingent upon whether observation takes place or not. Without
observation, the government relies on its prior beliefs and sets
I
=
IP
=
: With observation,
sophisticated analysis allows the government to implement the ex post …rst-best policy
I
=
I (f ).
Figure 3 allows us to contrast the inference of an econometrician who properly takes into account
observation e¤ects, inherent in credibility, as distinct from an econometrician who naively ignores
them. To …x ideas, suppose the econometrician inhabits an economy in which econometric evidence
is viewed as credible. Here, a naïve econometrician will overstate negative program impacts (f )
in the event of observation taking place. To see this, suppose the true realized f = :40: In the
event of observation, this results in an experimental investment response equal to
:055 (solid line).
Ignoring observation e¤ects the naïve econometrician will work along the dotted line and infer that
this response resulted from f = :85: Notice, this is the opposite of the problem the …rst-stage random
assignment was intended to solve. Recall, in the introduction the econometrician’s initial concern
was understatement of negative program impacts arising from …rst-stage policy endogeneity. Here
instead, second-stage endogeneity combined with observation e¤ects causes the econometrician to
drastically overstate the percentage of …rms negatively a¤ected by pollution regulation. Of course,
this would lead to an incorrect recommendation to deregulate.
The source of the bias here is obvious. Recall, the experiment consists of a period during which
Nature imposes regulation, with
back to its pre-experiment value
E
= . In the absence of observation, the policy variable reverts
IP
=
with probability one. In contrast, if observation takes
place there is a …fty percent chance of the government continuing with regulation. Consequently, the
negative investment response to the introduction of the regulation will be more severe if observation
takes place. The naïve econometrician incorrectly interprets a severe negative response as indicative
21
of higher f values.
Consider next policy recommendations. Again, let us assume that observation takes place. The
naïve econometrician would advise the policymaker as follows. “It is socially optimal to deregulate
if more than half the …rms in the economy bene…t from pollution. And if indeed more than half the
…rms stand to bene…t from pollution, the randomized regulation will cause investment to fall by at
least 0.03 (dotted line evaluated at f = 1=2).”But turning attention to the solid line, which re‡ects
reality here, we see that the investment will fall by 0.03 or more if the realized f exceeds :22: It follows
that the naïve econometrician would incorrectly recommend deregulation for f 2 (:22; :50): Thus,
the probability of an incorrect policy recommendation is 28%. There is bias towards deregulation
here that is the inevitable consequence of overstating negative program impacts as captured by the
estimated f parameter.
The preceding discussion raises another important concern regarding the robustness of policyrelevant natural experiments. In particular, suppose that instead of knowing whether or not they
are being observed, the …rms in our economy simply know that they will be observed with some
probability ' where 0 < ' < 1: In Economy NC, …rm behavior and econometric inference can
proceed independently of ': After all, the …rms in Economy NC know the government will implement
EX
I
=
IP
regardless of whether an econometric study takes place. However, in Economy C …rm
behavior will be predicated upon ' implying that econometric inference must be made contingent
upon this parameter. To see this, note that in Economy C agents will form the following typecontingent expectations regarding the Implementation Stage policy:
'[
T
+ (1
T)
] + (1
')
IP :
(40)
It follows that, in such a setting, in order for the econometrician to correctly recover f from the
observed aggregate investment response, she must correctly postulate the parameter ' in addition
to the technological parameters entering the government’s objective function, here (c; bH ): In reality,
one might imagine that there will tend to be a good deal of uncertainty regarding the true magnitude
of ' implying that the resulting inference will be noisy as well.
6
Econometric Inference III: Treatment-Control Di¤erences
This section illustrates some of the econometric issues that surface in experiments involving di¤erences between treatment and control group behavior. We begin with numerical examples before a
more formal analysis.
22
6.1
The Random Assignment Experiment
Consider now the following experimental setting. During the Pre-Experiment Stage …rms enter the
economy free to pollute at a high level, with
are assigned to the two possible
P
= . During the Experiment Stage, half the …rms
values. In the Exogenous Policy Economy, once the Experiment
Stage ends, the government is powerless to prevent the reversion of
I
to its initial value, implying
= : Thus, the Exogenous Policy Economy is one in which …rms are generally unregulated, with
a random sampling of one-half the …rms facing a restriction on pollution during the Experiment
Stage.
For the sake of numerical illustration, let us assume following parameter values: r = :05;
= 0;
E
= 1;
I
= 1;
= :90;
= :20;
= 4;
= :15;
= :20; x = 1; bH = :25; and c = :50: Again,
there are 101 possible f values ranging from 0 to 1 in equal-sized steps, and agents have uniformly
distributed priors over the potential distributions. Under these parameters optimal policy for the
government to set in the Endogenous Policy Economy is
I
=
if f
1=2.
Consider now Figure 4. On the horizontal axis is the realized fraction of high types. On the
vertical axis is the di¤erence between aggregate investment by the control (
(
E
=
E
= ) and treatment
) groups. The solid line measures the investment di¤erence in the Endogenous Policy
Economy and the dotted line measures the investment di¤erence in the Exogenous Policy Economy.
The dashed line considers the former economy parameterized with a higher cost of pollution c = :75:
The investment policy for the Exogenous Policy Economy is obtained from equation (25), while the
investment policy for the Endogenous Policy Economy is obtained from equation (23).
Figure 4 allows us to contrast the inference that will be made by a sophisticated econometrician
versus a naïve econometrician. Consider an econometrician working in the Endogenous Policy
Economy (c = :50), featuring evidence-based policymaking. The solid line in Figure 4 re‡ects
reality. If the econometrician is sophisticated, she will account for the link between policymaking and
empirical evidence and use the solid line in performing inference, resulting in correct estimation of f
given the observed di¤erence between treatment and control group investment. If the econometrician
is naïve, she ignores the link between evidence and policy and instead uses the dotted line to perform
inference.
In the present example, the naïve econometrician will understate f . To see this, suppose for
example that the true realized f = :60; resulting in an observed control-treatment investment
di¤erence of 25: Here the naïve econometrician will work along the dotted line and infer that this
di¤erence resulted from f = :42: Notice, this is precisely the sort of bias that motivates the search for
randomized policy assignment in the …rst place. For example, as discussed in the introduction, an
econometrician might be concerned that the estimate of …rms negatively a¤ected by environmental
23
regulation might be biased down if the …rst-stage assignment is endogenous. Here the …rst-stage
assignment is exogenous, but we still have a downward bias in estimated negative program impacts.
The correct recommendation here is deregulation but the naïve econometrician would recommend
regulation.
Next, let us return to the argument made in the introduction that econometric inference for
policy-relevant natural experiments must be predicated upon the deep parameters of the government
objective function into which they will be fed. To see this, suppose that government policy is set
endogenously, but that the true cost of pollution is c = :75: Here the government will only …nd it
optimal to deregulate during the Implementation Stage if the fraction of high types is greater than
2/3. To see the potential for error here, suppose now that the econometrician accounts for feedback,
but performs inference under the incorrect premise that c = :50 while the true c = :75: In this case,
reality is re‡ected by the dashed line while the econometrician incorrectly performs inference based
upon the solid line. In the present example, the econometrician will understate negative program
impacts (f ). To see this, suppose for example that the true realized f = :82; resulting in an observed
control-treatment investment di¤erence equal to 30. Here the econometrician will work along the
solid line and incorrectly infer that this small di¤erence resulted from f = :62:
Of course, the preceding argument also shows that the external validity of even ideal exogenous
…rst-stage randomizations must also be understood as predicated upon an assumption of equality of
parameters of government objective functions into which the evidence will be fed, as well as equality
of agent priors regarding the distribution of the unknown parameter, here f , to be estimated.
Figure 5 illustrates the importance of accounting for heterogeneity in agent beliefs. The …gure
returns to our original analysis of the Endogenous Policy Economy (true c = :50), but considers a
somewhat less naïve econometrician who accounts for policy endogeneity but fails to account for
heterogeneity in …rm-level beliefs. The solid line in Figure 5 represents the treatment response
that would be correctly postulated by our sophisticated econometrician. The dashed line shows
the treatment response that would be postulated by our naïve econometrician who incorrectly
assumes that all …rms will cling to their prior belief regarding the random variable f . The naïve
econometrician would advise the policymaker as follows. “It is socially optimal to allow high levels
of pollution if more than half the …rms in the economy bene…t from pollution. And if more than half
the …rms do indeed bene…t from pollution, the control-treatment investment di¤erence will be at
least 17.” But under the correct stipulation of beliefs (solid line), the investment di¤erence exceeds
this cuto¤ for any f exceeding :38. Thus, an incorrect recommendation to allow high pollution
would occur for all realized f 2 (:38; :50):
In fact, the preceding also shows that the econometrician would make the wrong inference
(dashed line) even if she stipulates the correct unconditional distribution of the policy variable. To
24
see this, note that the prior beliefs of …rms in the economy is actually equal to the true unconditional
distribution of the policy variable.
6.2
Problem Diagnosis
This section considers the underlying source of the errors in inference discussed in the preceding
subsection. To begin, we recall our basic heuristic argument: In general correct inference based
upon evidence from even ideal natural experiments requires correctly specifying the government’s
response function for each possible realization of the test statistic, as well as correctly specifying
agent beliefs regarding the distribution of the government policy variable. Let us consider this
argument more closely.
To begin, it will be useful to compute the di¤erence in the shadow value of capital across
Treatment (
E
=
) and Control (
E
=
) groups. In actuality, the di¤erence between q ET
and q ET is the same for the Exogenous Policy Economy as for the Endogenous Policy Economy,
regardless of the …rm type. To see this, note that the di¤ erence between q ET and q ET is invariant
to the speci…cation of policy during the Implementation Stage. After all, in each economy …rms are
ultimately subjected to the same
I
(although
I
di¤ers across the economies). Thus, di¤erences in
the shadow value of capital between Treatment and Control groups must be due only to di¤erences
in the marginal product of capital during the Experiment Stage.
Formally, the di¤erence between q EH and q EH is just equal to the present value of a claim to
)bH accruing during the Experiment Stage. From
the di¤erence in the ‡ow of private bene…ts (
equations (25) and (23) it follows that in either economy the wedge between shadow values across
Control and Treatment groups is:
q ET (x)
q ET (x) =
ET
ET
=
(
)bT
:
(r + + I )
(41)
And it follows from the preceding equation that in both the Exogenous and Endogenous Policy
economies, low type …rms (bL = 0) have the same shadow value of capital across the two assignment
categories. Thus, the di¤erence in aggregate investment between Control and Treatment groups is
equal to the di¤erence in high type investment. We have:
DIF F EREN CE = f
= f
= f
i (q EH (x))
i
i (q EH (x))
(
)bH
q EH (x) +
i (q EH (x))
(r + + I )
"
1
1
1
(
)bH
q EH (x) +
(r + + I )
(42)
q
EH
(x)
1
#
:
In general, the preceding test statistic will di¤er across the two economies. To see this, recall
from equations (25) and (23) that the level of the shadow values di¤ers across the two economies.
25
To illustrate the source the inference problem in the present example, it is instructive to consider
the exception that proves the rule. To this end, suppose instead that adjustment costs are quadratic,
with
= 2: In this speci…c case, for both the Exogenous and Endogenous Policy economies the
investment di¤erence between control and treatment groups is as follows:
= 2 =) DIF F EREN CE =
f
2
(
)bH
:
(r + + I )
(43)
And so we have the following proposition.
Proposition 3 The functions mapping f to the di¤ erence between treatment and control group
investment are the same across the Exogenous Policy Economy and Endogenous Policy Economy if
and only if
= 2:
Thus, as suggested by the speci…c numerical examples in the preceding subsection, as a general
matter statistical inference will go awry if the econometrician fails to account for the way in which
her estimate will be used in the policy process, as well as heterogeneous agent expectations regarding
this policy mapping. However, the proposition reveals that in the present example, even a naïve
econometrician who ignores the link between estimation and policy setting will draw the correct
inference in the special case of quadratic adjustment costs. Of course, this would probably be viewed
as little comfort for those who claim that randomization allows one to avoid “strong functional form
assumptions.”
7
Conclusion
This paper illustrates an inherent tension between the perceived credibility of the quasi-experimental
micro-econometric methodology and its actual credibility. In particular, once this econometric
methodology becomes su¢ ciently credible, and perhaps it already has passed this threshold, estimates derived from it will actually be used in setting policy. But this contaminates the original
econometric estimates by exposing them to endogeneity after-the-fact, with treatment responses
dependent upon the (unknown or unspeci…ed) parameters of policymaker objective functions. Two
closely-related paradoxes are shown. Policy-relevant analyses are contaminated by second-stage endogeneity while policy-irrelevant analyses are not. Similarly, a credible study is contaminated by
Hawthorne E¤ects while a non-credible study is not. In light of our critique, there is then a clear
trade-o¤ between using contemporary experiments that are tailor-made for an upcoming government decision versus historical experiments that generate less directly relevant information, but are
less exposed to endogeneity after-the-fact.
26
Although the preceding results can be viewed as negative, we show that valid policy-relevant empirical inference can nevertheless be performed provided the econometrician correctly and explicitly
models how the future path of policy variables will be in‡uenced by observed treatment responses,
and how observed treatment responses are shaped by heterogeneous expectations of the future
policy variable path. There is little controversy in maintaining that in order to utilize empirical
estimates in a quantitative analysis of policy alternatives an objective function must be stipulated.
The disagreement then amounts to one of timing. The proponents of randomization apparently
favor eschewing discussion of government objective functions at the econometric inference stage.
Our argument implies that the objective function must be treated as part of the inference problem.
27
Appendix: Proofs and Derivations
Lemma 1
Proof.
We claim
explicitly, we claim:
Pn
H
n=1 n
>
Pn
L
n=1 n
for n < N: Expressing the terms in this inequality
n
n
X
X
pn fn
pn (1 fn )
>
N
N
X
X
n=1
n=1
pj fj
pj (1 fj )
j=1
j=1
The preceding inequality holds if:
F (n )
Pn
Pn
n=1 pn fn
n=1 pn (1
fn )
>
N
X
pj fj
j=1
N
X
pj (1
= F (N )
fj )
j=1
It is su¢ cient to show that F is decreasing in its positive integer argument. We claim that for any
n0 < N :
F (n0 ) > F (n0 + 1):
Or:
2
3 2
n0
n0
X
X
4
pj fj 5 = 4
pj (1
j=1
2
32
n0
n0
X
X
4
pj fj 5 4
pj (1
j=1
j=1
fj ) + pn0 +1 (1
j=1
2
3
n0
X
4
pj fj 5 (1
j=1
3
2
nX
0 +1
fj )5 > 4
3
j=1
3 2
nX
0 +1
pj fj 5 = 4
pj (1
j=1
3
fj )5
2
32
n0
n0
X
X
fn0 +1 )5 > 4
pj fj + pn0 +1 fn0 +1 5 4
pj (1
j=1
2
n0
X
fn0 +1 ) > fn0 +1 4
pj (1
j=1
3
j=1
3
fj )5
fj )5
3
2
3 2
3
2
n0
n0
n0
X
X
X
(1 fj ) 5
fj 5
4
> 4
pj 5 > 4
pj
:
pj
fn0 +1
(1 fn0 +1 )
j=1
j=1
j=1
The …nal equation follows from the fact that fn is decreasing in n:
Growth Option Values
We begin by determining the growth option value function for the Experiment Stage in the
Endogenous Policy Economy. Recall that the terms in the Bellman equation (20) scaled by k
28
cancel. Utilizing the second part of Lemma 2 it follows that the growth option value must satisfy
the following ordinary di¤erential equation:
(r +
I )G
ET
1
2
xGET
x (x) +
(x) =
ET
h
ET
(
h
x GET
xx (x) +
2 2
ET h
h x
(44)
h=0
h h
)
X
:
Since the preceding form of growth option value function will recur, it will be convenient to
reference the following lemma.
Lemma 3 The growth option value function satisfying
rbG(x) = xGx (x) +
has solution
X
G(x) =
1
2
2 2
x Gxx (x) +
X
hx
h
h=0
h!hx
h
h=0
!h
Proof.
rb
1
1 2
h(h
2
h
1)
:
The function G represents the value of a claim to a sum of geometric Brownian motions
to successive powers. The value gh of a claim to an arbitrary constituent ‡ow payment
hx
h
must
satisfy the di¤erential equation:
rbgh (x) = xgh0 (x) +
1
2
2 2 00
x gh (x)
+
hx
h
(45)
We conjecture this value function takes the form:
gh (x) =
h!hx
h
Substituting the conjectured solution back into equation (45) one obtains the stated expression for
!h:
From equation (44) and Lemma 3 we obtain the following expressions for the growth option
value functions for …rms respectively assigned
G
ET
(x) =
X
E
ET E h
h !h x ;
=
and
GET (x) =
h=0
ET
h
!E
h
h
=
X
during the Experiment Stage:
ET E h
!h x
h
(46)
h=0
(
ET
)
h h
;
ET
h
h
1
r+
E
I
h
1 2
h(h
2
29
1)
:
(
ET
)
h h
We turn next to determining growth option value during the Pre-Experiment Stage in the Endogenous Policy Economy. We now con…ne attention to the remaining terms in the Bellman equation
(29) that are not scaled by k. This yields the following ODE:
(r +
E )G
PT
xGPx T (x) +
(x) =
+
E
h
G
ET
1
2
x GPxxT (x) +
2 2
X
h=0
(x) + (1
)G
ET
h
PT
(
)
h h
xh
(47)
i
(x) :
Substituting in the expressions for GET from equation (46) and grouping terms one obtains:
(r+
E )G
PT
(x) = xGPx T (x)+
1
2
x GPxxT (x)+
2 2
X
h=0
h
(
PT
)
h h
+
E
E !h
ET
h
+ (1
)
ET
h
(48)
Again, the growth option value is a linear sum of the geometric Brownian motion x to successive
powers. From Lemma 3 it follows that growth option value during the Pre-Experiment Stage is:
GP T
=
X
PT P h
h !h x
(49)
h=0
PT
h
! Ph
h
(
PT
h h
)
+
1
r+
E
h
ET
h
E
E !h
1 2
h(h
2
1)
+ (1
)
ET
h
:
The growth option value for the Exogenous Policy takes the same form with the exception that
the respective
values for the Experiment and Pre-Experiment stages must be taken from equations
(25) and (32).
30
xh :
References
[1] Abel, Andrew, and Janice Eberly, 1997, An exact solution for the investment and market
value of a …rm facing uncertainty, adjustment costs, and irreversibility. Journal of Economic
Dynamics and Control 21, 831-852.
[2] Angrist, Joshua D. and Jorn-Ste¤en Pischke, 2009, Mostly Harmless Econometrics: An Empiricist’s Companion, Princeton University Press.
[3] Angrist, Joshua D. and Jorn-Ste¤en Pischke, 2010, The credibility revolution in economics: How
better research design is taking the con out of econometrics, Journal of Economic Perspectives
24, 3-30.
[4] Dhaliwal, Iqbal and Caitlin Tulloch, 2015, From research to policy: Evidence from impact
evaluations to inform development policy, Working Paper, Abdul Latif Jameel Poverty Action
Lab.
[5] Du‡o, Esther, 2004, Scaling up and evaluation, in Annual World Bank Conference on Development Economics: Accelerating Development, ed. François Bourguignon and Boris Pleskovic,
341–69. Washington, D.C.: World Bank; Oxford and New York: Oxford University Press.
[6] Greenstone, Michael, 2009, Toward a culture of persistent regulatory experimentation and
evaluation, in D. Moss and J. Cisternino, eds. New Perspectives on Regulation, Cambridge,
MA., The Tobin Project.
[7] Hennessy, Christopher A. and Ilya A. Strebulaev, 2015, Beyond random assignment: Credible
inference of causal e¤ects in dynamic economies, Working paper, London Business School.
[8] Levitt, Steven D., and John A. List, 2011, Was there really a Hawthorne e¤ect at the Hawthorne
plant? An analysis of the original illumination experiments, American Economic Journal:
Applied Economics 3(1), 224-38.
[9] Lucas, Robert E., Jr., 1976, Econometric policy evaluation: A critique, in K. Brunner and A.
Meltzer Eds., The Phillips Curve and Labor Markets, Amsterdam: North Holland.
[10] Muth, John, 1961, Rational Expectations and the Theory of Price Movements, Econometrica
29, 315-335.
31
[11] Rubinstein, Ariel, 2006, Freak freakonomics, Economist’s Voice, Berkeley Electronic Press.
[12] Rust, John, 2010, Comments on Michael Keane’s “Structural versus atheoretic approaches to
econometrics,” Journal of Econometrics.
[13] Sargent, Thomas J., 1971, A note on the accelerationist controversy, Journal of Money Credit
and Banking 3 (3), 721-725.
[14] Sargent, Thomas J., 1973, Rational expectations and the dynamics of hyperin‡ation, International Economic Review 14 (2), 328-350.
[15] Sargent, Thomas J., 1977, The demand for money during hyperin‡ations under rational expectations, I, International Economic Review 18 (1), 59-82.
[16] Sargent, Thomas J., 1984, Autoregressions, expectations and advice, American Economic Review 74, 408-415.
[17] Sims, Christopher A., 1982, Policy analysis with econometric models, Brookings Papers on
Economic Activity 1, 107-164.
[18] Spatt, Chester, 2011, Measurement and policy formulation, working paper, Carnegie Mellon
University, Tepper School of Business.
[19] Taylor, John B., 1979, Estimation and control of a macroeconomic model with rational expectations, Econometrica 47, 1267-1286.
[20] Zwane, Alix, Jonathan Zinman, Eric Van Dusen, William Pariente, Clair Null, Edward Miguel,
Michael Kremer, Dean S. Karlan, Richard Hornbeck, Xavier Giné, Esther Du‡o, Florencia
Devoto, Bruno Crepon, and Abhijit Banerjee, 2011, Being surveyed can change later behavior
and related parameter estimates, Proceedings of the National Academy of Science (February).
32