The Right Side? Under Time Pressure, Approach Motivation Leads to Right-Oriented Bias
Mathias Alfransedes
Frank Bais
Mattis van der Berg
Sanne Boertien
Sara Boxhoorn
Marie Deserno
Monique Duizer
Barbara van Heijst
Rachel King
Universiteit van Amsterdam
Department of Psychology
Course: Good Science, Bad Science
Supervisor: Marjan Bakker
Date: October 2012
Vera van der Molen
Sam Prinssen
Daan van Renswoude
David Scholz
Anja Sommavilla

Background
In their article “The Right Side? Under Time Pressure, Approach Motivation Leads to Right-
Oriented Bias” Marieke Roskes, Daniel Sligte, Shaul Shalvi and Carsten K.W. De Dreu
(2011) present two studies that they conducted in order to find out if there is an interaction effect between motivation (approach- vs. avoidance) and time pressure (high vs. low) on the rightward bias that is attributed to human beings. Therefore they carried out one experimental and one observational study. We will focus the observational study.
In the observational study Roskes et al. (2011) investigated the right-oriented bias of goalkeepers under different conditions. They argued that goalkeepers who are behind in penalty shoot-outs are more approach motivated than goalkeepers who are ahead or tied.
Because they are more approach motivated and they have to decide quickly where to dive, they fall back on this right-oriented bias. Their results supported the hypothesis that goalkeepers display the right-oriented bias when behind, but not when tied or ahead.
The paper of Roskes et al. (2011) received a lot of attention in science and the media.
It is an interesting finding that goalkeepers tend to dive more to the right when they are behind in a penalty shoot-out. This finding is relevant for soccer and the chance on winning or losing important tournaments. Replication is an important way to accumulate support for effects found in research. Therefore a replication study will be conducted to see if we can support the hypothesis that goalkeepers display the right-oriented bias when behind, but not when tied or ahead. To replicate this study in a confirmatory way, this document contains the methods and analysis that will be used in this replication study.
Methods
Sample
The sample of penalties was obtained by first checking internet archives to identify games that included penalty shootouts in the following international competitions: European
1

Championship, Copa America, Champions League, and Europa League. Videos of the penalty shootouts identified in this way were then searched for on the online video platform YouTube.
This resulted in a sample of 50 penalty shootouts, which are listed in Appendix A. These videos are put in a random order for rating purposes.
Power analysis
Both φ and Cohen's w indicated an effect size of .417, which indicated a power of .53 for Roskes et al.'s (2011) study. This informs us that if we seek to replicate their study with a desired power of .8, we will need at least N=46 penalty situations in which the goal-keeper's team is behind. The sample of 50 penalty shootouts should be large enough to contain 46 penalty situations in which the goal-keeper's team is behind, so there is enough power to replicate the effect.
Rating
Two researchers will independently rate whether the goalkeepers are ahead, tied or behind when the penalty is taken. Six research assistants who are not aware of the purpose of the study will rate the games. Each assistant will watch and rate half of the penalties, the same half of the penalties as two other raters. They will rate whether the penalty shooter shoots to the left, the middle or the right and whether the penalty resulted in a goal, a save or a miss.
They will also assess whether the goalkeeper stays in the middle or jumps to the right or to the left. The raters will focus on the initial movement of the goalkeepers. This is important because we are interested in the automatic initial reactions. If a goalkeepers first moves to the left, but then decides to dive to the right this counts as a dive to the left. Furthermore, the raters will name the color of the shooter’s jersey to check whether they rated the correct penalty. This information is not included in the analyses.
All penalties, on which these three raters do not agree, will be assessed again by seven new raters who are also not aware of the purpose of the study. For these contentious penalties,
2

we will follow the vote of the majority of all the 10 raters. If there is a 5-5 tie on the issue, the penalty will be excluded from the analysis.
Procedure
All raters will rate the penalties individually. The videos will be played without sound.
The raters will be instructed to watch one penalty and decide in which direction the penalty taker shot (right, left, or middle), in which direction the goalkeeper dived (right, left, or middle) and whether the penalty was scored, missed or saved. Before starting the ratings, the raters will be shown two practice trials (penalties that were taken during the course of a normal game, not during a penalty shootout). Then, they will begin by watching the first penalty. They can watch the penalty repeatedly, if they wish. Next, they rate the direction of the shot and the dive and the outcome of the penalty. When the rating of one penalty is finished, they start the same process again with the next penalty. The raters will be instructed to take the perspective of the penalty taker in all left/right judgments. The instructions can be found in appendix B.
Analysis
Analysis plan
Our analysis will be twofold: First, we will analyze the new data in the same fashion as Roskes et al. (2011) analyzed their data of archival footage. All null-hypotheses refer to the absence of differences between the respective cells.
We will use a first set of separate χ²-tests to see whether

Goalkeepers dive to the right more often when their team is ahead

Goalkeepers dive to the right more often when their team is tied

Goalkeepers dive to the right more often when their team is behind.
3

We will correct for multiple comparisons (3) by using the Bonferroni procedure, resulting in
α=.017 for each test. If the effect Roskes et al. found is real, we would expect to find that only the last test turns out significant.
In a second set of separate χ²-tests we will see whether

Goalkeepers are less likely to save shots when their team is behind than when their team is not behind, using a 2x2 χ²-test.

On-target shots are scored more often when the goalkeeper's team is behind than when the teams are tied or ahead, using a 2x3 χ²-test.
Again, we will correct for multiple comparisons (2) by using the Bonferroni procedure, resulting in α=.025 for each test. Based on Roskes et al.'s results, we expect to find no significant differences. A final χ²-test will show whether penalty takers kick the ball to the right more often than they kick it to the left. We expect to find no significant difference at
α=.05.
Second, we will perform a multilevel log-linear analysis (GLMM) of a three-way frequency table (advantage x kick-direction x dive-direction) to test for interaction between these variables while accounting for the inherent dependencies in the data. The data is dependent because the same goalkeepers participate in different penalties. The design is therefore (3 x 3 x 3) unless the count drops below 5 for the kick-direction or dive-direction cells, in which case we will only compare right vs. left (3 x 2 x 2 or 3 x 2 x 3 design) . Based on Roskes et al.'s (2011) results, we expect to find no significant interaction other than
(advantage x dive-direction). As full correction for multiple comparisons would reduce the power of our analysis too much, we will use a liberal α=.05. To compensate, we will report effect sizes for each effect, significant or not.
In addition, we will perform a Bayesian analysis of the frequency table to determine the size of the effects pertaining to Roskes et al.'s (2011) main hypothesis. For this purpose,
4

we will use the program PostPM (Klugkist, Laudy & Hoijtink, 2010). To judge the magnitude of the effects, we will use the guideline provided by Jefferys (1961).
The models for this Bayesian analysis are as follows:

M1 = There is no difference between the proportions dives to the right vs. left

M2 = There is no difference when the goal-keeper's team is tied or ahead, but goalkeepers are more likely to jump to the right (from their perspective) when behind.
Results of the reanalysis of Roskes et al. (2011)
We have conducted several analyses with the original data of Roskes et al. (2011). A log-linear analysis of the three-way frequency table (advantage x kick-direction x divedirection) showed no significant interaction; in other words, the diving direction of the keeper does not depend on the (dis-)advantage of his team.
A Bayesian analysis by means of PostPM showed that for models M1=There is no difference between the proportions in right and left and M2=There is no difference when the goal-keeper's team is tied or ahead, but goal-keepers are more likely to jump to the right (from their perspective) when behind, model 2 fits the data better than model 1 (BF = 94.29/26.60 =
3.54). According to Jefferys (1961), this would be on the high end of 'scarce' and the lower end of 'substantial' evidence against the null.
If the 'behind' data are analyzed separately (comparable to the analysis in the article), the equal model yields BF=0.54 and the unequal model yields BF=1.98. Comparing those models gives a BF of 1.98/.54 = 3.64, similar to the above result. Accordingly, we will choose the first approach, as it provides a more complete translation of our hypotheses.
Finally, Roskes et al. (2011) reported that goal-keepers were 3 times less likely to save the shot when behind, however, they did not report that this difference is non-significant with
χ²(1) = 2.34, p = .126. The authors also compare percentages of on-target shots and report
5

only percentages, even though the differences were similarly non-significant as the analysis of both 2x2 (tied and ahead combined) and 2x3 contingency tables show (
χ ²(1) = 1.92, p = .166, and
χ²
(2) = 1.56, p = .459, respectively).
6

Literature
Jeffreys, H. (1961). Theory of probability. Oxford, UK: Oxford University Press.
Klugkist, I., Laudy, O., & Hoijtink, H. (2010). Bayesian evaluation of inequality and equality constrained hypotheses for contingency tables. Psychological Methods , 15 , 281-299.
Roskes, M., Sligte, D., Shalvi, S., & De Dreu, C. K. W. (2011). The right side? Under time pressure approach motivation leads to right-oriented bias. Psychological Science, 22 ,
1403-1407.
7

Competition Year
European Championship 1996
European Championship 2000
European Championship 2004
European Championship 2008
Copa America 1993
Copa America
Copa America
Copa America
Copa America
1993
1993
1995
1995
Copa America
Copa America
Copa America
Copa America
Copa America
Copa America
Copa America
1995
1995
1997
1999
1999
2001
2004
Copa America
Copa America
Copa America
Copa America
Copa America
Champions League
Champions League
Champions League
Champions League
Champions League
Champions League
Champions League
Champions League
Champions League
Champions League
Champions League
Champions League
Champions League
Champions League
Champions League
Champions League
Champions League
Champions League
Champions League
Europa League
Europa League
Europa League
Europa League
Europa League
Europa League
Europa League
Europa League
Europa League
Europa League
2004
2007
2011
2011
2011
1970/1971
1971/1972
1977/1978
1983/1984
1985/1986
1985/1986
1986/1987
1987/1988
1988/1989
1988/1989
1990/1991
1990/1991
1995/1996
2000/2001
2002/2003
2004/2005
2007/2008
2011/2012
2011/2012
2007/2008
2007/2008
2008/2009
2008/2009
2009/2010
2009/2010
2009/2010
2010/2011
2011/2012
2011/2012
Appendix A
Team 1
Germany
Italy
Netherlands
Spain
Colombia
Argentina
Argentina
Colombia
Brasil
USA
Uruguay
Mexico
Mexico
Uruguay
Honduras
Brasil
Brasil
Brasil
Argentina
Brasil
Paraguay
Everton
Celtic Glasgow
Juventus Turin
Liverpool
Barcelona
Steaua
Juventus Turin
PSV
Nechatel
Red Star Belgrad
Malmö
Red Star Belgrad
Ajax Amsterdam
Bayern Munich
Juventus Turin
Liverpool
Manchester United
Real Madrid
Bayern Munich
Everton
PSV
Dortmund
AaB
Sarajevo
Dinamo Bucuresti
Lech
Grasshoppers
Mainz
Lokomotiv Sofia
8
Real Madrid
Benfica Lissabon
Larissa
AC Milan
Dresden
Marseille
Juventus Turin
Valencia
AC Milan
AC Milan
Chelsea London
Bayern Munich
Chelsea London
Fiorentina
Tottenham
Udinese
Manchester City
Helsingborg
Liberec
Club Brugge
Steaua
Gaz Metan
Slask
Team 2
England
Netherlands
Sweden
Italy
Uruguay
Brasil
Colombia
Paraguay
Argentina
Mexico
Brasil
Peru
Peru
Paraguay
Uruguay
Uruguay
Argentina
Uruguay
Uruguay
Paraguay
Venezuela
Mönchengladbach
Internazionale Milano
Ajax Amsterdam
AS Rome
Göteborg
Barcelona

Appendix B
Thank you for helping us with our research.
We will show you videos of penalty shoot-outs from 26 different soccer matches. We would like you to watch closely and report the following on the report sheet:
The colour of the shooter’s jersey. Sometimes a penalty is replayed once or multiple times.
To make sure that you do not code the replay, the colour of the shooters jersey is coded (this should alternate).
The direction of the ball. Was the ball shot to the left, right or middle (from the perspective of the shooter)
The direction of the goalkeeper. Did the goalkeeper go to the left, right or middle to try to save the ball (also from the perspective of the shooter)? Please fill out the keeper’s initial direction. For example, if the goalkeeper first moves right, but then dives left, please code
‘right’.
The outcome of the penalty. Was the goal: scored (the ball went in) saved (the goalkeeper stopped the ball) or missed (the ball went outside the goal)
After you have seen the penalty, pause the video and fill in the report sheet. If you are not sure of any of these things you should write down, you can watch the penalty again – just rewind the video and watch it again.
If you feel you need to take a break, we encourage you to do so.
We will start with two examples.
If you have any other questions, please ask.
Good luck!
9