Defending the Causality of Rainfall as an IV
One of the main challenges of proving causality in a relationship between two variables is
unobserved variables because, as the name says, they're unobserved. In the paper "Do Political
Protests Matter? Evidence from the Tea Party Movement," the authors struggle with this
problem. In estimating the effect of protests, unobservable political preferences will likely
determine the number of protesters and policy outcomes. They combat this by exploiting
variations in rainfall during the day of the protest. Under the assumption that the absence of
rainfall affects policy and voting behavior only through the number of protesters, the authors can
estimate the effect of protest size using an instrumental variables approach. We can break this
assumption into four parts: randomization, monotonicity, first-stage relationship, and exclusion
restriction. Each of these assumptions must hold for rainfall to be a suitable IV. Many other
authors have used rainfall as an IV, like Yanagizawa-Drott, who used daily rainfall to generate
variation in outdoor participation on July 4 to study the effect of celebrating Independence Day
(Mellon). Another example is Collins and Margo's work on the impact of the riots following the
assassination of Martin Luther King Jr. on income, labor, and housing market outcomes for
African Americans, which exploited rain during the month of April 1968 as an instrument for
riot severity (Mellon). However, in the paper, "Rain, Rain, Go Away: 195 Potential
Exclusion-Restriction Violations for Studies Using Weather as an Instrumental Variable,"
Jonathan Mellon explains that widely used instruments can potentially be flawed because
alternate causal routes from the IV to the dependent variable are exclusion-restriction violations.
While the overall point Mellon is trying to make is significant, the variables that Mellon lists as
potential exclusion restriction violations, based on the studies Mellon cites that generated these
variables, are irrelevant and, therefore, do not threaten the validity of Madestam et al. analysis.
Before delving into the potential exclusion restriction violations, defending the other
assumption Madestam et al. make by using rainfall as an IV is essential. Rainfall fulfills the
requirement of "randomization" because it is determined independently of the study's
independent and dependent variables and is as good as randomly assigned, irrespective of other
covariates. By employing rainfall, an inherently random and exogenous factor, the study adheres
to the principle that an effective IV should mirror random assignment within the research
context. The study details that rainfall's unpredictability ensures its independence from political
motivations. Rainfall is determined by weather patterns and climatic conditions, specifically in
the short term, independent of the political processes and rally attendance. The study also uses
historical weather data from the National Oceanic and Atmospheric Administration from 1980 to
2010 to construct a baseline measure of rainfall (Madestam et al). This historical perspective
supports the argument that rainfall patterns, especially on a specific day like Tax Day, are
random and not systematically related to other variables of interest in the study. This
independence is crucial, as it isolates the rally's impact on political outcomes from other
confounding variables.
Rainfall is also a good IV because it has a first-stage relationship and is monotonic with
political rally turnout. The study demonstrates this relationship by showing that rainfall has a
direct and measurable impact on the number of people attending political rallies. The study's
analysis, drawing on the same historical weather data, reveals a clear pattern: inclement weather,
specifically rainfall, leads to decreased rally attendance (Madestam et al). The study also
demonstrates a consistent decrease in voter turnout with increased rainfall, illustrating this
relationship's one-directionality. This pattern is not merely coincidental but is statistically
significant, indicating a direct causal link between rainfall and turnout. Furthermore, the study
meticulously controls for various factors such as demographic characteristics, past voting
behavior, population size, regional differences, economic conditions, and the general probability
of rain (Madestam et al). Establishing that rainfall, a variable external to the political process and
independent of human influence, directly affects rally attendance upholds the "First Stage
Relationship" and "Monotonicity" assumption for a good IV. This relationship is essential, as it
underpins the study's ability to isolate the effect of rally size on political outcomes, ensuring that
the observed impacts are attributable to changes in turnout rather than other external factors.
When understanding the last assumption, exclusion restriction, it is important to note that,
theoretically, it is untestable as there is no guarantee that we have enumerated through all the
variables that could create other pathways between our IV and our dependent variable. To
combat this, Mellon reviewed 289 studies and found 195 variables previously linked to weather
(Mellon). Just because these variables are linked to the weather does not mean they violate the
exclusion restriction assumption for Madestam et al. specifically. Knowing this, Mellon first
filters through these variables, sorting by relevance to time period, country income level, and
short-term versus long-term rain analysis. So, in this case, studies done using rainfall in the
long-term sense or studies in lower-income countries are not relevant to Madestam et al.: a study
conducted analyzing rainfall on the day of a protest in a high-income country like the United
States whose economy is diverse enough not to be affected by rains in the short-term. After
assessing each variable using "theory and literature" and sorting through exceptions where both
results are compatible, Mellon estimated the potential bias of any variables left through
sensitivity analysis. The seven variables mentioned as potential exclusion restriction violations
for Madestam et al. were COVID-19, social distancing, violent protests, property and violent
crime, mood, and traffic (Mellon). The two obvious variables that slipped through the cracks
after assessing theory and literature are COVID-19 and social distancing, as the time period of
studied protests was in 2009. However, after further delving into the variables and the studies
behind them, the variables listed fit into one of the exceptions listed, or the study's scope or
results are irrelevant to Madestam et al.
The study that concluded a decrease in rainfall leads to an increase in violent protest
results, and scope is irrelevant to Madestam et al. The study, "Weather effects on social
movements: Evidence from Washington, D.C., and New York City, 1960-95," found that good
weather, combining climate and precipitation, increases the likelihood of a protest and violence.
However, these results are irrelevant to Madestam et al. due to the time period of the analysis and
the cities studied. The study examines protests from 1965 to 1995 in New York and Washington,
D.C. Not only does the time period not overlap with the Tea Party protest in 2009, but the study
even states, "In the first half of the 36 years, 1960–77, when social movements were generally
more active, they were also more sensitive to weather conditions. In contrast, in the second half,
1978–95 events seem to have been somewhat indifferent to weather conditions" (Zhang). This
implies that the results they found were only relevant in a specific time period, and as time went
on, their results became less relevant. On top of this, the study emphasizes that the relationship
of violence is only related to warm temperatures, not precipitation (Zhang). Lastly, the study
explains that further research is needed, and caution must be taken in extrapolating the results
found in this study as it only looks at two cities. In contrast, the Tea Party protests occurred
throughout the country in various places, from cities to smaller towns. In totality, it is clear that
this variable is not relevant as an exclusion restriction violation because this study focuses on a
variety of weather conditions, limited geographical scope, and distinct historical periods that do
not align with the more specific conditions (rainfall), nationwide scope and more recent
timeframe of the Tea Party protests.
Multiple studies concluded that weather affected crime; however, none of these results
apply to Madestam et al. The first study cited, "Crime, weather, and climate change," concluded
that increases in temperature have a strong positive effect on criminal behavior; however, rainfall
is not mentioned once in the study, which Mellon incorrectly cites. Another study, "Sinning in
the Rain: Weather Shocks, Church Attendance and Crime," found that one more Sunday with
precipitation at the time of church increases yearly drug-related, alcohol-related, and white-collar
crimes. However, despite what Mellon directly claims in his paper, the study explicitly states, "I
do not find an effect for violent or property crimes." Even just looking at the crime increases,
they found that April 15, 2009, was a Wednesday, meaning that no churchgoers' experience was
affected by the protest. The last study they cited on this, "The effect of weather on crime: an
investigation of weather and annual crime rates," initially finds that higher temperatures are
associated with fewer robberies and homicides and more burglaries, while precipitation is
initially found only to be associated with lower robbery levels. However, after adding in
economic and demographic controls along with country-specific fixed effects, both the
temperature and precipitation results become statistically insignificant, according to the study
itself (Reichhoff). The only relationship found with the controls was that increased rainfall and
snow were associated with fewer burglaries. Still, the scope of the study was limited to specific
counties in California. Even the researcher admits he had limited data because of these counties'
lack of rain and snow, and to therefore be hesitant to extrapolate results (Reichhoff). Even taking
the relationship found at face value, it is difficult to believe that a day's worth of burglars', who,
according to the study's logic, would have been convinced out of committing their crime that day
due to rainfall or emboldened to rob someone due to the lack of precipitation, would exhibit
enough bias through some unidentified connection to more Republican votes in the 2010
midterm elections to violate the exclusion restriction assumption.
Mood and traffic, in the few relatively relevant studies, actually fall under one of the
exceptions Mellon states. Mellon has eight expectations where an appropriate alternate variable
will not cause exclusion restriction violations, and, in this specific case, the relevant findings
about rainfall's effect on mood and traffic would make up the independent variable rally
attendance. It is still important to note, though, that many mood studies are not actually relevant
to Madestam et al. For example, "Weather, Risk, and Voting: An Experimental Analysis of the
Effect of Weather on Vote Choice" and "Swingin' in the Rain: The Impact of Inclement Weather
on Voting Behavior in U.S. Presidential Elections" look at how rainfall affects risk aversion on
election day claiming bad weather significantly depresses risk tolerance making voters less likely
to vote for risky candidates. The applications towards elections are irrelevant as the rain being
studied did not occur on an election day. Another, more relevant, cited study, "Rain or Shine:
Happiness and risk-taking," finds opposing evidence, claiming that lack of rain leads to happier
people who are less risk averse and are more idealistic about the long term. The one study
analyzing traffic, "Let it rain: Weather effects on activity stress and scheduling behavior," finds
inconclusive evidence about how weather affects stress levels but reinforces the general idea that
rainfall worsens traffic. It's rain's effect on mood and traffic that makes up the change in rally
attendance. Traffic increases are most likely part of why Madestam et al. find conclusive
evidence that rainfall decreases rally attendance. The same logic can be applied to the change in
mood (though it's harder to explain as Mellon cites contradicting evidence). This path is much
more likely than a day of rainfall affecting the general mood and traffic of the country enough to
sway a midterm election a year later.
Mellon's overall claim that a good instrument should have a "certain ridiculousness" is
essential. Until the causal pathway is explained, the link between the instrument and outcome
should seem absurd. However, Mellon's analysis needs to be more precise and thoughtful in its
analysis of Madestam et al. Mellon tries to apply pathways from other studies that simply are
irrelevant to the casual relationship Madestam et al. establishes. Mellon needs to consider the
external validity of the studies that generate the variables he performs sensitivity tests on,
sometimes even going so far as to misrepresent the results of the studies he looks at blatantly.
When trying to find alternate pathways from the IV to the dependent variable, it is essential to
find variables tailored explicitly towards the parameter and characteristics of the study,
especially since Madestam et al. take great lengths to defend rainfall as a great IV. Madestam et
al. demonstrate meticulousness through robust checks, such as using various precipitation
thresholds and considering rainfall data from a broader geographical scope. Madestam et al. use
of placebo tests, employing rainfall data from different historical dates, further solidifies the
causal link they propose, ensuring that their findings are not mere artifacts of random variation.
Mellon admits in his conclusion that "nothing in this paper disproves particular empirical claims.
Many weather-IV papers provide independent sources of evidence, and robustness checks (e.g.
placebo tests) and some results will be true by chance." Madestam et al. robustness compared to
Mellon's allows me to conclude that the rainfall IV design in this study can identify an effect of
protests.
Bibliography
Madestam, Andreas, Daniel Shoag, Stan Veuger, and David Yanagizawa-Drott. "Do political
protests matter? Evidence from the tea party movement." The Quarterly Journal of Economics
128, no. 4 (2013): 1633-1685.
Mellon, Jonathan. "Rain, Rain, Go Away: 195 Potential Exclusion-Restriction Violations for
Studies Using Weather as an Instrumental Variable." Available at SSRN 3715610 (2023).
Bassi, Anna. 2019. “Weather, Risk, and Voting: An Experimental Analysis of the Effect of
Weather on Vote Choice.” Journal of Experimental Political Science 6 (1): 17–32.
Chen, Roger, and Hani Mahmassani. 2015. “Let it rain: Weather effects on activity stress
and scheduling behavior.” Travel Behaviour and Society 2 (1): 55–64.
Duhaime, Erik, and Taylor Moulton. 2018. “Swingin’ in the Rain: The Impact of Inclement
Weather on Voting Behavior in U.S. Presidential Elections.” SSRN.
Reichhoff, Martin. 2017. “The Effect of Weather on Crime: An Investigation of Weather and
Annual Crime Rates.” MSc Thesis.
Zhang, Tony. 2016. “Weather effects on social movements: Evidence from Washington, D.C.,
and New York City, 1960-95.” Weather, Climate, and Society 8 (3): 299–311.
Ranson, Matthew. 2014. “Crime, weather, and climate change.” Journal of Environmental
Economics and Management 67 (3): 274–302.
Moreno-Medina, Jonathan. 2021. “Sinning in the Rain: Weather Shocks, Church Atten- dance
and Crime.” The Review of Economics and Statistics, 1–46.
Guven, Cahit, and Indrit Hoxha. 2015. “Rain or shine: Happiness and risk-taking.” Quarterly Review of Economics and Finance 57: 1–10.