Cognitive Bias in Public Procurement
Omer Dekel* and Yoav Dotan**
* Senior Lecturer, College of Law & Business, Ramat Gan, Israel
** Edwin A. Goodman Professor, Faculty of Law, Hebrew University, Jerusalem,
Israel
Thanks for very useful comments to: Jeffery Rachlinski, Ilana Ritov, Amos Schurr,
Eyal Zamir,
© Draft. Please do not cite without authors permission.
1
ABSTRACT
To what extent decision-making by administrative officials is subject to cognitive
biases? And how can a change in the regulatory framework contribute to debias such
influences? In the current paper we investigate these questions by referring to
decision-making in the field of competitive bidding (CB). In CB procedures there is a
substantial concern that when bid evaluators are exposed to the bid price, it may affect
their judgment, even with regard to decisions that should not be affected by this
information. More specifically, there is a concern that in the first stage of the CB
process bid evaluators would qualify faulty bids when they are aware of the fact that
that bid is the lowest. Similarly, there is a concern that at the second stage of the
process, the evaluation and scoring of the qualitative criteria of the competing bids
(such as experience, recommendations, etc.) would be affected by the knowledge of
the price, in a way that will give an undue advantage to the lower bidder. We term this
failure – "the lower bid bias" (LBB). The purpose of the current study is to examine
whether and under what conditions such bias does influence decision-making in CB
processes, and whether it can be debiased by regulatory change. To this end, we
conducted a series of experiments, with the participation of real decision makers, in
situations that resemble much their daily work. Our main finding is that LBB has a
significant effect in favor of the lower bidder in the scoring stage of the CB process;
however, we also found that bid evaluators have the ability to overcome that bias in
the qualifying stage of the process. We cautiously ascribe the difference in the
susceptibility to the LBB to differences in the decision type and suggest a normative
regulatory change.
2
1. INTRODUCTION
Each year governments purchase goods, services and work in hundreds of billions of
dollars. Much of it is done through competitive biddings (CB). The CB mechanism
should ensure selecting the best bidder, while maintaining integrity, fairness and equal
opportunity for the bidders. Given the centrality and importance of government
procurements it is not surprising that CB procedure have been subject to ample
research by economists and public policy researchers. However, as far as we know,
there is no research that examined CB procedures from behavioral economics point
of view. The current study aims to fill this gap.
Ample research in psychology has demonstrated that decision-makers often
rely on mental shortcuts or "heuristics" to make complex decisions. While ordinarily
quite effective, in certain situations these heuristics also result in systematic errors
(e.g. Kahneman & Tversky 1974; Chapman & Bornstein 1996; Hastie, Schkade &
Payne 1999). Accordingly, a growing body of academics is focusing on how human
behavior deviates systematically from what would be predicted by the expected utility
theory underlying traditional economic analysis (e.g., Jolls, Sunstein, & Thaler 1998;
Korobkin & Ulen 2000; Jolls & Sunstein 2006). Various studies demonstrate that
official decision-makers such as criminal investigators, juries, and even professional
judges are susceptible to various cognitive biases, much like laypeople. For example,
Guthrie, Rachlinski & Wistrich (2001; 2007) demonstrated that professional judges
who took part in experiments proved susceptible to heuristics such as anchoring,
when asked to decide the amount of damages in tort claims. Likewise, Wistrich,
Guthrie & Rachlinski (2005) established that in many cases judges fail to disregard
inadmissible information that was brought to their notice (see also Rachlinski, Guthrie
& Wistrich, 2006; Guthrie, Rachlinski & Wistrich, 2009). The impact of cognitive
biases on professional administrative decision-makers in the field of public
procurement has yet to be addressed. The current study aims to fill this gap.
The CB process inherently requires decision making, which its peak is
choosing the CB winner. These decisions are made routinely by professional officials
that specialize in this field and it is highly regulated. Considering the vast scope of
competitive public procurement, it is of great importance to establish whether these
decisions are exposed to cognitive biases, and if so whether debiasing is feasible.
Observing the CB procedure reveals that it can often be divided into two major stages.
3
At the first stage of the process CB officials are called to decide which of the
submitted bids comply with the CB prerequisites and which do not and should be
rejected (qualification stage). At the second stage of the process CB officials are
called to conduct evaluation comparison and ranking of the qualified bids and select
the best qualified offer as the CB winner (evaluation stage).
As previous research suggests, there is a substantial concern that when bid
evaluators are exposed to the bid price, it may affect their judgment, even with regard
to decisions that should not be affected by this information (Dekel & Schur 2013).
More specifically, there is a concern that in the qualification stage bid evaluators
would qualify faulty bids when they are aware of the fact that that bid is the lowest.
Similarly, there is a concern that at the evaluation stage, the scoring of the qualitative
criteria of the competing bids (such as experience, geographic location, number of
employees, recommendations, etc.) would be affected by the knowledge of the price,
in a way that will give an undue advantage to the lower bidder (Dekel & Schur 2013).
We termed this failure – "the lower bid bias" (LBB). The purpose of the current study
is to examine whether and under what conditions such bias does influence decisionmaking in CB processes, and whether it can be debiased by regulatory change. To this
end, we conducted a series of experiments, with the participation of real decisionmakers, in situations that resemble much their daily work.
Our main finding is that LBB has a significant effect in favor of the lower bidder
in the evaluation stage of the CB process; however, we also found that bid evaluators
have the ability to overcome that bias in the qualification stage of the process. We
cautiously ascribe the difference in the susceptibility to the LBB to the differences in
the decision type and suggest a normative regulatory change.
The paper is arranged as follows: Part 2 contains a short introduction to the world
of competitive public procurement, and explains why there are good reasons to
assume that bid evaluators are susceptible to cognitive bias in their decision-making
process. Part 3 presents the experiments and their results, while Part 4 provides
several alternative explanations for these results. Parts 5 include normative aspects of
the study results, and Part 6 presents our conclusions.
2. THE POTENTIAL FOR COGNITIVE BIAS IN COMPETITIVE PUBLIC
PROCUREMENT
4
2.1 The Decision-Making Process in Competitive Public Procurement
In most western countries, public procurement is a highly regulated field.1 In order to
insure efficiency, fairness, and integrity in a sphere that deals with the allocation of
hundreds of billions of public dollars annually,2 the law generally3 dictates that public
procurement will be conducted through a competitive mechanism that seeks
efficiency but at the same time maintains fairness and integrity. Even though public
procurement deals with a huge range of transactions and with many alternative
models of competitive mechanisms,4 all those mechanisms generally share a similar
pattern: the competitive bidding designer stipulates prerequisites that a bidder must
fulfill in order to participate in the CB, and evaluation criteria by which the best
bidder will be chosen. These stipulations must be presented as clearly as possible in
the solicitation documents, cannot be altered retroactively, and binding the bidders
and the bid evaluators as well. After the solicitation documents are drafted, they are
brought by publication to the knowledge of potential bidders. At that point, entities
that are interested in the transaction and consider themselves as fulfilling the
1
For the regulation of the U.S. Federal Government acquisition see the Federal Acquisition Regulation
(FAR) (48 CFR § 1). For the European Community regularization see: Directive 2004/18/EC of the
European Parliament and of the Council of 31 March 2004 on the coordination of procedures for the
award of public works contracts, public supply contracts and public service contracts, Official Journal L
134 , 30/04/2004 P. 0114 – 0240. For the regularization by the World Trade Organization (WTO) See the
Government Procurement Agreement (GPA). Similar regularizations exist also in the state of Israel,
where the experiments that are used for the current research took place. See the Public Tenders Act, 1992
(Israel).
2
For example, in 2012, the U.S. federal government procured approximately $536 billion worth of goods,
services, and infrastructure.
Of this, $334.5 billion were expended through more than 3 million
competitive procedures (Federal Procurement Data System – Next Generation. Available at:
https://www.fpds.gov/fpdsng_cms/index.php/reports.last updated June, 2013). Moreover, millions of
additional competitive procedures, worth billions of dollars, are conducted each year by administrative
agencies, state and local governments, and this is at the U.S. alone.
3
There are some exceptions for that rule in certain circumstances, like urgency, confidentiality or lack of
competition as a result of a single supplier (FAR §6.302).
4
For example Request for Proposals (RFP) which allows the consideration of various evaluation criteria
in order to choose the best bidder (FAR §15.2); Invitation for Bids (IFB) which allows the consideration
of price alone as a criterion for the choosing best bidder (FAR § 14.2); Request for quotations (RFQ)
which designed to simplify the procurement process as to small acquisitions (FAR § 13).
5
solicitation demands, prepare their bids in accordance with the solicitation documents
and submit them with the aim of winning the CB. Now the reins pass back to the
hands of the procurement officials. Authorized bid evaluators verify whether the
submitted bids comply with the CB's prerequisites, reject unqualified bids; and from
the remaining bids choose as the CB winner the one who complies with the evaluation
criteria set in the solicitation in the optimal way.5 The focus of our research is on the
decision-making process of bid evaluators. It is worth noting that according to the
law, the bid price should not play any role neither in deciding whether a bid complies
with the CB demands nor in scoring the bids qualitative parts. This means that while
the price is normally an important factor within the evaluation process its relative
weight is defined by the solicitation documents and hence procurement officials are
prohibited by law to ascribe to it weight beyond the conditions set in the solicitation.
An important trait of the CB decision process is its segmentation into two
separate differently structured stages. As can be observed, the first stage requires a
decision – which of the submitted bids are compatible with the CB prerequisites and
which are not, and as a result of that decision – rejecting the non-compatible bids.
This process is binary, since it is always a "yes" or "no" decision, without any
intermediate alternative. Additionally, each bid is inspected discretely, without any
reference to its competitors, and the decision is made according to clear rules (the
solicitation prerequisite). The second stage contains comparison, evaluation, and a
decision how to rank the remaining bids on a scoring scale (usually from "1" to "100")
and as a result declaring the highest scored bid as the CB winner. 6 Decision making at
this stage is usually much more discretionary and subjective in nature. It is
characterized as a scale decision and is often controlled by vague standards.
5
In the federal level, the person who is responsible for that operation to function is the "Contracting
Officer"(48 CFR 1.602-2; FAR § 2.101) with the assistance of the Contracting Officer Representatives
(COR), Contracting Officer Technical Representatives (COTR), a Technical Evaluation Panel (TEP) (See
FAR § 2.101.) and more.
6
The only exception is CBs that contain competition on price alone, by which the actual scale is the
"price" criterion
6
2.2 The Vulnerability of the Competitive Public Procurement Decision
Making to Cognitive Bias
In a CB, the government obviously wishes to achieve an optimal transaction.
That desire may exposes bid evaluators to cognitive bias in favor of the lower bidder.
For example, in a situation that the best bid slightly deviates from the CB's demands,
the bid evaluators may be motivated to disregard that flaw even though rejection of
that bid in such circumstances is obligatory; or when competing bids differ in their
price and quality, since price is an attribute that is more salient and easy to evaluate
than quality, bid evaluators may tend to overvalue the lower bid and by that giving it
an unjust advantage (Dekel and Schurr 2013). These tendencies rest on two wellknown phenomena in the literature of cognitive psychology – Halo Effect and
Confirmation Bias.
2.2.1
Halo Effect
Cognitive bias in favor of the lower bidder can be attributed to the halo effect. This
refers to decision-makers’ tendency to focus on the most prominent attribute of the
subject they are evaluating (be it a person, an object, or a bid); their impression of this
attribute affects their assessment of its other attributes, even though there is no
rational relation between them. The first to observe the halo effect was Thorndike
(1920), who showed that people tend to ascribe success, happiness, and higher social
status to people with an attractive external appearance. The halo effect was thereafter
found to be present with regard to other subjects of evaluation as well (Beckwith and
Lehmann 1975; Nisbett and Wilson 1977; Felton, Mitchell and Stinson 2004; Smith,
Read and Lopez-Rodriguez 2010). In the CB context, a low bid price is one such
prominent attribute that impacts the overall perception of the bid in a positive way.
2.1.1
Confirmation Bias
Confirmation bias relates to the human tendency to overvalue information or evidence
that support a desired result, and respectively undervalue - and sometimes even ignore
- information or evidence contradicting that outcome (for a comprehensive review of
the literature, see Nickerson 1998). This bias is present not only in situations in which
7
the decision-maker has an interest in obtaining a particular result (a motivated bias),
but also in situations where no such interest exists but the decision-maker has already
formulated an opinion on the matter (unmotivated bias) (Ross, Lepper and Hubbard
1975; Lord, Ross and Lepper 1977; Edwards and Smith 1996). However, in the
former situation the confirmation bias has stronger impact (Kunda 1987, 1990). It
should also be noted that "[T]he line between deliberate selectivity in the use of
evidence and unwitting molding of facts to fit hypotheses or beliefs is a difficult one
to draw in practice…" (Nickerson 1998, 175).
Confirmation bias can also have legal implications. For example, it was found
that confirmation bias causes criminal investigators and prosecutors to overestimate
evidence that supports their hypothesis as to the suspect's guilt, and neglect evidence
that contradicts it (Ask and Granhang 2007; O'Brian 2009; Rassin, Eerland
and Kuijpers 2010; Eerland and Eric Rassin 2012). Confirmation bias can also be
found in the courtroom. As a result of confirmation bias, jury members tend to
interpret new evidence in a manner supporting their prior beliefs as to the defendant's
guilt (Carlson, and Russo 2001), and judges might make erroneous decisions as well
(Rachlinski 2012). In the context of the public CB process, we suggest that in order to
obtain a good bargain and to use public funds efficiently, bid evaluators may identify
the lowest bid as the desirable one and as a result be susceptible to confirmation bias.
The positive impression caused by the low price (halo effect) and the desire
that the lower bid will win (confirmation bias)can have two main effects: first, they
may cause bid evaluators to underestimate or even ignore deficiencies in the lowest
bid in order to avoid its rejection at the qualification stage of the CB procedure.
Naturally, this tendency is not without limits; it is unlikely to take place when the
lowest bid is completely incompatible with the solicitation requirements. However,
when the lowest bid slightly, though clearly, deviates from the solicitation
prerequisites, or when the compatibility of the lowest bid to the solicitation demands
is questionable due to ambiguity in its terms, it is possible that this deficiency will be
disregarded due to halo effect and confirmation bias.
Second, in their pursuit of a good bargain for the public entity, bid evaluators
may subconsciously identify the lowest bid as the desirable one, even though its
quality is inferior to other bids. That desire may then be translated into a preference
8
for that bid, through evaluating its other attributes too highly, or underestimating its
weaknesses (Dekel and Schurr 2013).
As has been mentioned, we term these biases – The Lower Bid Bias (LBB).
As has been said, from a normative viewpoint bid evaluators are expected to
overcome the LBB by disregarding the bid price whenever the rules require them to
do so.7 However, from the psychological point of view their ability to do so is
questionable. We will now present our study concerning this question.
3 THE PUBLIC PROCUREMENT STUDY
3.1 Research Hypotheses and Design
Following the above discussion, our hypothesis was that procurement officials would
be susceptible to LBB at both stages of the CB process decision-making. Accordingly,
we expected that knowledge of the bid price would have a significant effect on their
decision-making in a way favoring the lower bidder.
In order to determine that, we conducted a study consisting of an opinion
survey and five controlled experiments, with the participation of Israeli procurement
officials8 who are involved in government procurement on a daily basis.9
7
It may be argued that when procurement officials over-value the price factor (beyond the requirements
of the solicitation documents) they behave rationally since low price by the bidder serves as a signal for
better quality (that is, that the bidder is more efficient than its competitors etc.). This argument should
however be rejected for a number of reasons: First, one should assume that if lower price is a proxy for
better quality in a given set of CB, then this fact is already contemplated by the CB designers and is
reflected in the relative weight already ascribed to the price within the CB structure. Secondly,
procurement officials are bound by law to follow the CB stipulations. Lastly, if procurement officials
ignore the CB original structure it may also push the bidders to set their bids in a way that put more
weight on the bid price than stated in the solicitation document (Dekel and Schurr 2013).
8
"Procurement officials" refers to any official in a public entity that is subject to a mandatory CB
process, e.g., government offices, administrative agencies, and government-owned companies.
9
Note that the Israeli legal regime governing public procurement in general, and bid evaluation and
comparison in particular, closely resembles the U.S. scheme. Thus, as in the U.S., Israeli bid evaluators
are restricted to the CB evaluation criteria specified in the CB records when scoring the bids (see FAR §
15.305(a) for the American provision), and the bid receiving the highest rank is the winner. However, in
Israel, decisions are made by a contracting committee, composed of three to five members authorized by
law, rather than by a contracting officer.
9
First, we performed a survey designed to explore the views of procurement
officials and frequent bidders on bid evaluators' vulnerability to the LBB during the
two stages of the CB process - the qualification stage, and the evaluation stage.
The experiments were designed to explore the actual impact of cognitive bias
on bid evaluators during these two stages of the CB process. In experiments 1 and 2
we checked the effect of the influence of exposure to the bid price on bid evaluators
during the CB evaluation stage (stage 2), while in experiments 3, 4 and 5 we
examined that influence during the qualification stage (stage 1).
3.2 A survey of procurement officials and frequent bidders' views on
procurement officials' vulnerability to the LBB
We first tested our hypothesis by an opinion survey of procurement officials and
bidders who frequently participate in public procurements. The survey sought to
uncover the respondents’ views on how awareness of the bid price impacts on bid
evaluators' decision-making.
The survey was conducted with two different groups of respondents: (1) 91
procurement officials from various Israeli public entities, whose jobs entail
involvement in procurement procedures, either as members of contracting committees
(analogous to contracting officers in the U.S.) or as assistants to those committees (the
counterparts to contracting officer representatives and technical evaluation panels);
and (2) 96 employees of corporations that are frequent bidders in public CBs. The
survey was conducted during a series of professional training courses. It comprised
twenty questions relating to a variety of issues regarding the procurement process,
with one question (Question 10) referring to the effect of the knowledge of bid price
on the decision whether to reject a faulty bid, and one question (Question 17)
referring to the effect of knowing the bid price on the bid evaluation process. The
respondents were asked to indicate their position regarding the following statement on
a five-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree):
Q10:“When a contracting committee member is aware of the bid price, this
knowledge influences her judgment regarding the question of whether she
should qualify or disqualify a faulty bid.”
10
Most respondents to that statement believed that officials' knowledge of the bid price
affects their judgment concerning qualifying a faulty bid (M =3.51, S.D. = 1.25). Yet
the concern was stronger among bidders than among public officials (M = 3.73, S.D.
= 1.10, vs. M = 3.31, S.D. = 1.36, t(181) = 2.303, p < 0.05, respectively).
Q 17: “Contracting committee members should not be exposed to the bid
prices until after the bid evaluation process has been completed, so that
knowledge of the bid prices does not affect their judgment.”
Here again, most respondents on this statement indicated agreement with it (M = 3.71,
S.D. = 1.33). And here also, the frequent bidders expressed stronger concern
regarding the impact of knowledge of the price on the evaluation process (M = 3.93,
S.D. = 1.25 and M = 3.49, S.D. = 1.37, t(176) = 2.22, p < 0.05, respectively).
Altogether, in line with our research hypothesis, the survey participants
intuited that procurement officials are subject to the LBB both in their qualifying
decisions and their evaluation decisions. The survey results served as a background
for the next stage of our study. It consisted of five experiments, two of which were
designed to examine whether bid evaluators' awareness of the bids' price would
influence their judgment concerning the evaluation of competing bids, and three
experiments testing the same question with regard to the qualification of a cheap but
faulty bids. We begin by presenting the 2 experiments regarding the evaluation stage.
3.3 The effect of knowing the bid price on the evaluation process – controlled
experiments with procurement officials
Experiments 1 and 2 were designed to determine whether bid evaluators are affected
by the exposure to the bid price when they are asked to evaluate, compare, and score
the qualitative evaluation criteria of competing bids, in order to announce the best
bidder as the CB winner (the second phase of the CB process).
Our hypothesis was that procurement officials’ decision making would be
susceptible to the LBB when they are evaluating and scoring competing bids. To test
our hypothesis, we designed a within-subject experiments that tested the influence of
awareness of the bid price on the bids' evaluation process.10 This was done by
10
We used a within-subject design due to the fact that those experiments were done with professionals in
an environment that is as close as possible to their routine decision-making setting, the number of
11
presenting the same case to the same participants twice, and asking them to score the
competing bids' qualitative criteria in both cases: at first, when they were unaware of
the bid price, and after some time, when the bid price was revealed. We checked
whether the exposure of bid evaluators to the bid price would diminish the gap
between the competing bids' qualitative score in favor of the lower bidder.
3.3.1 Experiment 1: Eight evaluation criteria
(1) Method
The experiment participants were 36 Israeli procurement officials. The experiment
was conducted in the framework of a seven-week professional training course that
met for a six-hour session once a week, sponsored by the Israeli Ministry of Finance.
Employing a within-subjects design, the first part of the experiment was conducted
during the course's first session, and the second part was conducted four weeks later at
the beginning of the fifth session, before the topic of "bid evaluation" was discussed
in class.
Participants were presented with a CB case for the provision of school
transportation services, conducted by a municipality, in which two bids had been
submitted. Participants were given a table detailing eight qualitative criteria to be
considered (geographical location, references, number of buses available to the bidder
etc.). The table presented the maximum score that could be given for each criterion, as
well as the information each of the two bidders provided regarding these criteria.
Participants were asked to score the bids on all eight evaluation criteria, with the
maximum combined score allotted for these components being a total of 400 points of
the overall 1000 points. The remaining 600 points were allotted to the price criterion.
potential participants is obviously limited. A within-subject design enabled us to conduct the experiment
with sufficient number of participants. In addition a within-subject design enabled us to examine and
measure the impact of the LBB on each and every participant in person (See Appendix A). It should be
noted that balance design was impossible in the circumstances, because after being exposed to the bid
price at the first run it was unlikely that subjects will not remember which bid was the lowest at the
second run (when price should be unknown).
12
In the first part of the experiment, the case was set up as a two-stage CB – by
which the bid price is submitted in a separate sealed envelope that is opened only after
the evaluation process is completed. According to this, participants were asked to
evaluate the qualitative components of the bids without knowing their prices. In the
second part, the same case was presented to the participants, but the bid price was
included along with the qualitative information, as well as the points that each bid had
received for its price—i.e., a single-stage CB scenario. Thus, the lowest bidder price
was about 800,000$11 and accordingly it won the maximum score for the price
criterion – 600 points; and the more expensive bid price was about 875,000$ and its
relative scoring for that criterion was 550 points (
.
To minimize the possibility that the participants would recall their previous responses,
the order of the two bids within the table was switched, and the order in which the
qualitative components were presented was altered randomly. In order to determine
the impact of their exposure to the bid price, participants were asked to provide
identification on the experiment form filled out at both stages of the experiment (see
Appendix A for the experiment 1 form).
(2) Results
The results of the experiment are summarized in Table 1 below (for detailed results,
see Table A1 in Appendix A). As can be seen, it emerges that when participants
assessed the qualitative components of the two bids without being aware of the
proposed prices, the mean evaluation of the higher-priced, qualitatively-superior bid,
was higher than that of the lower-priced, qualitatively-inferior bid, by 84.24 points,
where the maximal evaluation was 400 points (M = 357.86, S.D. = 30.04, versus M =
273.61, S.D. =43.00, respectively). After factoring in the bid price, which gave the
lower bid a 50-point advantage, the final difference between the bids was 34.25 points
in favor of the qualitatively-superior bid. However, when participants assessed the
bids' qualitative components while knowing their prices, this difference was
significantly decreased. The mean qualitative evaluation of the higher-priced bid was
then 367.08 (S.D. = 22.32), whereas the mean qualitative evaluation of the lower11
The price was presented by New Israeli Shekels (NIS), 1 NIS = roughly 0.25 USD. This is also true
with regard to the other experiments.
13
priced bid was 297.42 (S.D. =36.08). This amounted to a mean difference of only
69.66 points in favor of the more expensive and qualitatively superior bid before
factoring in bid price, and a difference of only 19.66 points after the bid price was
factored in, yielding, again, a 50-point advantage for the lower-priced bid.
A Repeated Measures ANOVA, testing the difference between the two-stage
CB and the single-stage CB in terms of the average qualitative bid scores, confirmed
that the above observations were significant. Most importantly, and in line with our
prediction, there was significant interaction between type of procedure (single-stage
versus two-stage) and the gap between the evaluations of each bid (F(1,35) = 5.909,
p<0.05, partial η2 = 0.144). Orthogonal to our research question, the analysis also
revealed a significant effect of procedure (F(1, 35) = 11.19, p<0.01, partial η2 =
0.242).
The same trend also emerged in our analysis of the participants' specific
choices. All 36 participants awarded a higher score to the qualitative part of the more
expensive and qualitatively superior bid than to the qualitative part of the lowerpriced, qualitatively inferior bid. However, in the two-stage procedure, the difference
in the qualitative scores between the two bids was less than 50 points in 6 cases out of
36 (16.67%). Thus, once the bid price was factored in, the lower bid emerged as the
CB winner in these 6 cases. However, in the single-stage procedure, the lower-priced
bid was the winner in 13 of the 36 cases (36.11%), more than double than in the twostage CB procedure.12
Taken together, these results indicate that the bid evaluators’ knowledge of the
bid prices led to: (1) a significant reduction in the difference between the qualitative
scores awarded to each bid, from difference of 84.25 points to difference of 69. 66
points, representing a decree of about 20%, which becomes 40% decree after
factoring in the bid price and (2) a considerable increase in the number of cases in
which the lower bid won the CB, as a result of awareness of the bid price, from 6
12
Because of the small group a Z test for proportions between the two dependent groups did not find
that change as a significant one (Z = 1.87, p = 0.06). However, a joint analysis of this experiment
together with Experiment 2 below, revealed a significant difference between the two bidding
procedures (Z = 2.39 p<0.05) see the discussion of experiment 2 at p. 18 below and note 13 id.).
14
cases
to
13
out
of
a
total
of
36,
amounting
to
116%.
15
Table 1
Summary of Results of Experiment 1
Mean score of qualitative
components of the lower bid (out
of a possible 400 points)
Mean score of qualitative
components of the higher bid (out
of a possible 400 points)
Difference in scores (before
weighting of bid price)
Final difference in scores, after
weighting of bid price (50-point
advantage for the lower bid)
Number of participants (out of
36) choosing the lower bid as the
winner
Two-stage CB (36
participants)
273.61
(43.00)
Single-stage CB (36
participants)
297.42
(36.08)
Variance resulting from
shift to single-stage CB
An additional 23.81 points
for the lower bid
357.86
(30.04)
367.08
(22.32)
An additional 9.22 points
for the higher bid
84.25 points in
favor of the higher
bid
34.25 points in
favor of the higher
bid
69.66 points in favor
of the higher bid
Difference in scores
reduced by 14.59 points
19.66 points in favor
of the higher bid
Final difference in scoring
reduced by 42.59%
6/36 (16.67%)
13/36 (36.11%)
Chances of the lower bid
winning the CB increased
by 116%
3.3.2 Experiment 2 – Five evaluation criteria
(1) Method
Experiment 2 was designed to corroborate the findings of Experiment 1. Its
participants were 29 Israeli procurement officials who were participating in a threeweek professional training course that met once a week for a six-hour session. The
same experimental design was used as in Experiment 1, with only two alterations: (1)
the CB procedure included five evaluation criteria rather than eight; and (2) the time
that elapsed between the two evaluations was shorter than in the first experiment (two
weeks instead of 4), since the course ran for only three weeks (see Appendix B for the
experiment form).
(2) Results
16
Corresponding with the findings of Experiment 1, and as demonstrated in Table 2
below, when participants assessed the qualitative components of the two bids without
knowing the bid price, the mean evaluation of the higher-priced, qualitatively superior
bid was greater by 87.36 points than the mean evaluation of the lower-priced,
qualitatively inferior bid (M = 370.93, S.D. = 21.99, versus M = 283.57, S.D. =34.41,
respectively). After factoring in the bid price, the difference between the bids' score
was 37.36 points, giving the lower bid a 50-point advantage. However, when the
participants assessed the bids' qualitative components while informed of the bid
prices, the difference decreased significantly: the mean qualitative evaluation of the
higher-priced bid was 371.38 (S.D. = 25.27), whereas the mean qualitative evaluation
of the lower-priced bid was 295.52 (S.D. = 47.62). In other words, a mean difference
of 75.86 points emerged in favor of the more expensive bid before factoring in the bid
price, which dropped to only 25.86 points after the bid price was factored in (see
Table B 1 in Appendix B for the detailed results).
Repeated Measures ANOVA, testing the difference between the two-stage CB
procedure and the single-stage procedure in terms of the average qualitative bid
scores, confirmed the significance of the above observations. Most importantly, and
again as expected, this analysis revealed significant interaction between the type of
procedure (single-stage versus two-stage) and the gap between the bids’ scores
(F(1,28) = 4.782, p<0.05, partial η2 = 0.146). Orthogonal to our research question, the
analysis did not reveal a significant effect of procedure (F(1,28) = 1.253, p = 0.272,
partial η2 = 0.043).
This trend was also apparent in the analysis of the participants' choices. In the
two-stage procedure, 5 out of the 29 participants (17.24%) chose the lower-priced bid
as the CB winner (without being aware of this result, since the price was concealed).
However, in the single-stage procedure, where the price was known from the outset,
10 out of the 29 participants (34.48%) chose the lower-priced bid as the winner - a
100% increase.13
Taken together, the results corroborate that the bid evaluators’ knowledge of
bid price produced: (1) a significant narrowing of the difference in the scoring of the
13
Although a Z test did not reveal a significant difference when examining this experiment separately (Z
= 1.50, p = 0.13), a joint analysis of the two experiments revealed a significant difference (Z = 2.39
p<0.05).
17
two bids, from a gap of 87.36 points to a gap of only 75.86 points, which amounts to a
decrease of about 13%, 30% decree after factoring in the bid price; and (2) a
substantial increase in the number of cases in which the lower bid won the CB, as a
result of the LBB, from 5 cases out of 29, to 10 cases. These findings corroborate our
research hypothesis above.
Table 2
Summary of the Results of Experiment 2
Two-stage CB (29
participants)
283.57
(34.41)
Single-stage CB (29
participants)
295.52
(47.62)
Variance resulting from
shift to single-stage CB
An additional 11.95
points for the lower bid
370.93
(21.99)
371.38
(25.27)
An additional 0.45 points
for the higher bid
Difference in scores (before
weighting of bid price)
87.36 points in favor
of the higher bid
75.86 points in favor
of the higher bid
Difference in scores
reduced by 11.50 points
Final difference in scores, after
weighting of bid price (50-point
advantage for the lower bid)
37.36 points in favor
of the higher bid
25.86 points in favor
of the higher bid
Final difference in scores
reduced by 30.78%
Number of participants (out of
29) choosing the lower bid as the
winner
5/29 (17.24%)
10/29 (34.48%)
Chances of the lower bid
winning the CB increased
by 100%
Mean score of qualitative
components of the lower bid (out
of a possible 400 points)
Mean score of qualitative
components of the higher bid (out
of a possible 400 points)
3.3.3 Discussion
The results of experiments 1 and 2 corroborate the research hypothesis and establish
that when conducting qualitative evaluation of competing bids, decision-makers are
susceptible to cognitive bias in favor of the lower bid. The exposure to the bid price
has a significant effect on the qualitative assessments of the competing bids and this
effect is strong enough to double the chances of the lower priced (but qualitatively
inferior) bid to be selected as the CB winner.
Given the fact that both experiments were conducted with actual procurement
officials and that their platform was almost identical to the routine, real-life procedure
of competitive bids, the results provide considerable support for the research
hypothesis.
18
3.4 The effect of knowing the bid price on the decision whether to qualify
problematic bids – controlled experiments with procurement officials
We now turn to the question whether the same effect would be demonstrated with
regard to procurement official decisions made during the qualification stage of the CB
procedure. The next three experiments were designed for this purpose.
3.4.1 Experiment 3 – Clinic CB
The object of the experiment was to test whether knowing the bid's price affect
procurement officials' decision whether to qualify or disqualify a problematic bid.14
(1) Method
To perform the experiment, we used a between-subject design with 71 procurement
officials who are employees of various government offices. The participants were
asked to respond to a case dealing with a CB for the construction of a pediatric
oncology clinic, for which four bids had been submitted. The CB prerequisites and the
bids' details were described in a tabular form. Bid A was slightly but clearly noncompliant with one of the CB requirements. Thus, the solicitation required that “the
bidder employs a construction engineer with 8 years of experience” and the table
indicated that “according to the certification attached, the engineer has 6 years and 8
months of experience.” Bid B compliance with one of the prerequisites was doubtful.
Thus, the CB demand was that "The bidder employs a physician with 10 years of
seniority, who completed a residency in pediatric oncology", while bidder B
employed a physician with "[r]esidency in general oncology (completed in 1993),
because there were no pediatric oncology residencies in Israel at that time. Since then
has worked as a pediatric oncologist in a hospital." Bids C and D were described as
being in full compliance with the CB prerequisites (The experiment form is attached
in Appendix C).
One group of participants (n=35) was shown a scenario that did not include
the bid price (two-stage CB) and were asked to mark the bids that they would pass on
to the next stage (when the bid price would be revealed). The other group (n=36) were
14
By the term "problematic bid" we mean a totally defective bid or a bid that its compatibility with the
CB’s prerequisites is questionable and subject to interpretation of the CB’s terms..
19
faced with the same scenario, but with the bid price indicated, and were asked to mark
the winning bid. The experiments' forms were assigned randomly. The bids' prices
were the following: bid A (the clearly flawed one) was the lowest, having a bid price
of 17.8 million dollars; bid B (compliance was doubtful) was the next lowest, with a
bid price of 18.9 million dollars; and bids C and D were more expensive, and had bid
prices of 20.3 and 20.8 million dollars, respectively. Our attention was focused again
on the participants’ judgments regarding bids A and B.
(2) Results
The results are summarized in line 1 of Table 3 below. When the participants were
unaware of the bid price, 8.5% of them (3 participants out of 35) passed bid A
onwards, and 48% (17 out of 35) passed bid B. Total – 51.4% (18 out of 35) passed
on any of the problematic bids.
However, when the participants were exposed to the bids' price, only 2.7% of
them (1 out of 36) chose bid A as the CB winner, and only 27% (10 out of 36) chose
bid B. A total of 29.7% chose one of the problematic bids.
As can be seen again, the awareness of the bid price not only did not increase the
impact of the LBB, but in fact significantly decreased its effect (p<0.05).
3.4.2 Experiment 4 – Cables CB
In order to reinforce the results of the former experiment, we designed another
experiment that replicated Experiment 3's traits, but this time with governmental
company's employees and with a different scenario.
(1) Method
The experiment was conducted among 99 employees of the Israeli Electricity
Corporation (IEC), a government-owned company that manufactures and supplies
electricity throughout Israel, and is subject to mandatory CB regulations. The
20
experiment was conducted during a professional training program for the company's
employees who engage in public procurement in their everyday work.
This experiment employed a between-subject design as well. Participants were
presented with a case involving a CB, conducted by the IEC itself, for the acquisition
of high voltage electricity cables, in which four bids had been submitted. Participants
were told that they were facing a real dilemma, from the desk of the IEC's contracting
committee. The case was presented on a sheet with the IEC's logo.
Here too, the CB demands and the bidder responses to it were presented in a
tabular form that included four CB prerequisites: (1) "Valid affidavit concerning
bidder's proper management;" (2) "Prior supply of 20 km of similar cables in each
year of the years 2009-2011;" (3) "The manufacturer operates a quality control
department;" and (4) "Obligation to start supplying the cables within 30 days from
notification of winning." The table also indicated that: [1] Bidder A "supplied 18 km
of the required cables in 2009, in 2010 - 21 km, and in 2011 - 27 km.," i.e., bid A was
slightly but clearly incompatible with the solicitation demands; [2] As for bidder B,
the table indicated that "The manufacturer has a contract with a lab that supplies them
with quality control services," i.e., bid B's compatibility with the solicitation
prerequisites was questionable and subject to the interpretation of the bid’s
prerequisite (3) above.
As in the former experiment, one group of participants (n=47) was presented
with the case without knowing the bids' price (two-stage CB) and asked to mark all
the bids that they would pass on to the next stage – in which the price envelopes
would be opened and the CB winner will be declared. The second group (n=52) were
faced with the same case as a single-stage CB, so the table in their case also indicated
that bid A is the cheapest (1.3 million dollars), bid B is next, with a 1.4 million dollar
offer, and bids C and D were more expensive – 1.6 and 1.8 million dollars
respectively (the experiment form appears in Appendix D). That group of participants
was asked to mark the winning bid. The experiments' forms were randomly assigned.
Our attention was focused on the participants’ judgments regarding bids A and
B when they did not know the bid price, as compared to their decision regarding these
bids when they were aware of the price.
21
(2) Results
As can be seen in line 2 of Table 3 below, when the participants were not aware of the
proposed prices, 55.3% of them (26 out of 47) passed either bid A or bid B (or both)
on to the next stage. 23.4% (11 participants) passed bid A onwards; 38% (18
participants) passed bid B, and 6% (3 participants) passed both of them.
However, when the participants were aware of the bid price, only 36% of the
participants (19 out of the 52) chose bid A or bid B as the CB winner. 13% (7
participants) chose bid A and 23% (12 participants) chose bid B. In other words,
when participants were aware of the bid price, not only did their willingness to
qualify the problematic bids not increase, it in fact transpired to be marginally
significantly reduced (p=0.097).
3.4.3 Experiment 5 – Generators CB
The goal of the fifth experiment was to test the generality of the results of
Experiments 3 and 4. Accordingly, it was designed with the following twists: First,
we designed Experiments 3 and 4 in a way aimed to resemble as much as possible a
real life CB. Therefore, subjects that responded to the price concealed case were
asked to mark the bids that they will pass on to the next stage of the CB; while
subjects responding to the case with the revealed price were asked to mark the
winning bid. While such a design provided us with a setting that effectively
represented real-life environment for CB procedures it was methodologically
constraint by the somewhat different nature of the task assigned to each group of
participants. Thus in Experiment 5 both groups were assigned exactly the same task,
i.e. both groups were asked to mark the bids that they would pass on to the next
stage.) In addition, in order to control for the possibility that small number of bids
restrain bid evaluators from rejecting problematic bids, we increased the amount of
bids from four bids to nine. Lastly, in order to control for the possibility that the order
of presentation of the bids may affects bid evaluators attitude toward the problematic
bids, we moved the problematic bids from the A and B position, to the F and D
position respectively.
22
(1) Method
We conducted this experiment among 73 IEC employees, during a professional
training program for the company's procurement employees. Some of the participants
in this experiment took part in Experiment 4 as well. However, between the
experiments was a gap of 4 weeks and the experiments' results and purpose were not
revealed to the participants.
The experiment employed a between-subject design. Participants were
presented with a case involving a CB conducted by the IEC, this time for the
acquisition of six generators in which nine bids had been submitted. The CB's
demands and the bidders' responses to it were presented in a tabular form that
contained four prerequisites: (1) "ISO 9001, 2010 edition;" (2) "…five years
experience in the field;" (3) "Five years guarantee for the products;" and (4)
"Obligation to supply at least two generators within 60 days from the notice of
winning." The table also indicated that: [1] Bidder D "… has 42 years of experience
in manufacturing turbines, but only 4.5 years of experience in manufacturing
generators." That is, bid D's compatibility with the solicitation prerequisites was
questionable; and [2] Bidder F "… does not have ISO certification, but declared that
he has finished the certification process and expected to receive it in a week." In other
words, bid F was slightly but clearly not in compliance with the solicitation demands.
One group of participants (n=34) were presented with the case without
knowing the bids' price and were asked to mark the bids that they would pass on to
the next stage – in which the qualitative part would be scored and the price envelopes
would be opened. The second group (n=39) was shown the same table, but it
additionally indicated the bids' price. Thus, this group of participants was aware of the
fact that bid F (the faulty one) was the cheapest (20.750 million dollars), bid D (the
questionable one) was next, with 20.877 million dollars, and the remaining seven
bids, that fully complied with the CB prerequisites, were more expensive. (The
experiment form is attached in Appendix E). The forms were randomly assigned.
Our attention was focused on the participants’ judgments regarding bids F and
D (the problematic ones) comparing the decisions of participants who were aware of
the bid price, to those of the participants who did not know it.
23
(2) Results
As can be seen in line 3 of Table 3 below, there was no significant difference between
the responses of the two groups: among the participants who did not know the bid
price (the first group) 20.5% (7 participants out of 34) passed bid F and 20.5% (7
participants out of 37) passed bid D. A total of 29.7% (11 out of 37) passed the
problematic bids (F or D) while not knowing their price. Among the participants that
knew the bids' price (the second group) 17.9% (7 out of 39) passed bid F and 23% (9
out of 39) passed bid D. At total 33.3% of the participants (13 out of 39) passed at
least one of the problematic bids (F or D) while knowing their price. Like in the case
of Experiments 3 and 4 above, the participants' knowledge of the bid price did not
increase their vulnerability to the LBB, and actually it did not affect their attitude
toward the problematic bids.
The results of Experiments 3-5 are summarized in the following table:
Table 3:
Passed on / chose the problematic bids while the bid price was Unknown
Experiment 3
(Clinic)
Experiment 4
(Cables)
Experiment 5
(Generators)
Faulty
8.5%
(3/35)
Vague
48.5%
Known
Faulty
2.7%
(1/36)
Vague
27%
(10/36)
Total
29.7%
(17/35)
Total
51.4%*
(18/35)
23.4%
38.3%
55%*
13%
23%
36%
(11/47)
(18/47)
(26/47)
(7/52)
(12/52)
20%
(7/34)
20%
(7/34)
29.7%*
(11/34)
17.9%
(7/39)
23%
(9/39)
33.3%*
(13/39)
* Some participants passed on both bids.
3.4.4 Discussion regarding Experiments 3-5
The conclusion we derived from the fact that procurement officials' awareness of the
bid price did not result in a more lenient approach towards the problematic (but lowpriced) bids, is that they seemed able to resist LBB at the qualifying stage of the CB
procedure. Furthermore, their compliance with the legal rules and resistance to LBB
seemed to unexpectedly increase due to the exposure to the bid's price. These results
contradicted our research hypothesis and the survey results. While the survey's
24
participants and we ourselves assumed that knowing the bid price would cause
decision-makers to express a lenient attitude toward problematic bids, the opposite
happened.
3.5 General Discussion
Our study, consisting of the survey and the five experiments, was constructed to test
whether procurement officials can overcome a cognitive bias in favor of the lower
bidder when they decide whether to reject a faulty or questionable bid at the
qualifying stage of the CB procedure, or asked to evaluate the bids at the CB
evaluation stage. Our research hypothesis and the opinion survey results were that
procurement officials are subject to that bias both in their qualifying and evaluation
decisions. However, the results of the experiments showed that this intuition was only
partly correct.
With regard to evaluation decisions, we indeed corroborated our hypothesis
that bid evaluators were subject to LBB. Yet, with regard to qualification decisions
the experiments’ results demonstrated that we and the survey participants got it
wrong. We unexpectedly found that CB officials were able to resist the LBB, and that
the exposure to the bid price not only did not increase the tendency to qualify faulty
bids but possibly even decreased that tendency. Further research is needed in order to
account for this latter finding.
These findings raise two important questions: first, what causes the differences
in vulnerability to the LBB, between the two stages of the CB process. Second, what
are the normative implications, if any, that these results entail.
4 THEORETICAL EXPLANATIONS FOR THE STUDY RESULTS
One of the basic principles of rational choice theory is procedure invariance. Under
this principle, preferences should not be influenced by the way in which they are
solicited. A person who prefers jazz over classical music is expected to maintain that
preference whether she is asked to choose between the two, or rate them on a scale.
25
"A person who prefers chicken over pasta should not change this preference on
learning that fish is also available" (Kelman, Rottenstreich and Tversky 1996).
In contrast with this assumption, however, decision-making research has
revealed that procedure invariance does not always hold. A substantial body of
psychological research suggests that in many situations preferences are sensitive to
the procedure by which decisions are made. Preference has been found to be a
constructed, context-dependent process, largely influenced by the framing of a
problem, by the method in which preferences are elicited, and by the context in which
a decision is made (Lichtenstein and Slovic 1971;Tversky and Kahneman 1986;
Simonson and Tversky 1992; Fox and Tversky 1995; Hsee 1996; Simonson 2008;
Hsee and Zhang 2010; Kahneman 2011, at 160).
To demonstrate, Nowlis and Simonson (1997) showed that when people are
asked to choose between products, precise and easy-to-evaluate attributes (such as
price) obtain more weight than ambiguous and hard-to-evaluate attributes (such as
brand name or country of manufacture) in a choosing procedure. The opposite occurs
in a rating procedure.
In our case, observing the decision-making process of bid evaluators reveals that
the qualifying decision of the CB first stage is structured as a binary one, while the
evaluating decision at the CB second stage is structured as a scale one. In addition,
the qualification stage is controlled by clear-cut rules; questions at this stage receive
right (or wrong) answers; each bid is analyzed on its own independently of the other
bids, and the bids are approved or disapproved but not evaluated. The evaluation
stage, on the other hand, is controlled by more ambiguous rules, there is no right (or
wrong) decision, the decision-maker's judgment regarding each bid is relativistic (i.e.
it is made by comparing the bids), and the bids are evaluated and ranked rather than
approved or disapproved. We now address each of these differences in more detail.
4.1 Binary versus scale: The impact of task type
One potential reason for the different influence of the price awareness on bid
evaluators in the qualifying stage, as opposed to the evaluation stage, rests on the
different types of decision-making. Taleb and Tetlock (2013) point to the differences
26
between binary and scale decisions (which they term "vanilla"), and argue that
"vanilla exposures are sensitive to Black Swan effects, model errors, and prediction
problems, while the binaries are largely immune to them." That distinction can serve
as a potential explanation to the differences in exposure to the LBB in our case as
well.
The first stage of the CB process entails an approved or disapproved binary
decision. At this stage, the only question that procurement officials have to answer is
whether the bid comply with the CB prerequisites. This is a simple "yes" or "no" task,
even if reaching the right answer for that task is not always an easy one. The opposite
is true with regard to the CB evaluation stage. At that stage, the bids' attributes are
assessed, compared to each other and ranked on a scale, which is meant to elicit a
judgment as to the best bid that should win the CB. As opposed to the binary decision,
that has only two valid options (qualify or not), the scale one has endless options,
ranging from 0 to 100.
It is reasonable to assume that when it comes to a binary decision task,
disregarding the bid price, and by that avoiding the LBB, is feasible. However, when
the decision task requires attributes' evaluation and consideration, and the bid
evaluator has endless legitimate options it is much harder to avoid the price attribute,
and to resist the LBB.
4.2 Clear-cut rules versus ambiguous standards
Another difference between the two phases of the CB process that may account for
the different vulnerability to the LBB relates to the character of the norms that govern
each phase of the decision-making process. The qualification stage is governed by
specific and concrete rules that define the threshold requirements. Thus, there is a
right answer that bid evaluators are expected to provide, and a wrong one that they are
expected to avoid. The right answer is to reject any faulty bid when they identify one,
regardless of its attractiveness, and the wrong one is to qualify such a bid by
undermining its defect or completely ignoring it. This clear rule is expected to assist
decision-makers to obey the rules and resist cognitive biases. Admittedly, providing
the right answer is not always a simple task even under a regime of rules since rules
27
may sometimes be ambiguous and require the decision-maker to conduct a process of
interpretation (as was the case with the ambiguous bids in Experiments 3-5 above).
While interpretation of rules may, however, be somewhat difficult in the case of
interpretative ambiguity, the decision-maker is still required to choose between the
right and wrong answers. Our findings suggest that in such a process decision-makers
prove resistant to the LBB despite this relative complexity of the decision.
In contrast, the evaluation stage of the CB is governed by scoring principles,
which are much more open-ended, ambiguous, and leave room for wide discretion in
most circumstances. Usually, there is no precise scale for the measurement of the
differences between qualitative criteria of competing bids, such as experience,
seniority, distance from the purchaser headquarter, or past performance. How many
points will best represent the differences between five years and seven years of
experience, or between two excellent recommendations and three very good ones? In
such cases any reasonable judgment will work, and as a general rule it is almost
impossible to draw a clear line between a right decision and a wrong one. In this
respect, the distinction between a binary and a scale decision resembles, to some
extent, the one between rules and standards (Feldman and Harel 2008).
Even though at both stages bid evaluators are well aware of the fact that, from
the legal perspective, they are not allowed to take the bid price into consideration,
they may still be influenced by the inclination to choose the bid which would be the
most cost-saving for the organization they work for. In this respect, this tendency
functions as a social norm that may influence agents' decision-making, despite the
imperatives of the legal norms (Feldman and Harel 2008).
In light of the above, it is likely that obeying the rules by disregarding the bid
price, is much easier for decision makers at the CB qualification stage than at the
evaluation one, since the former is controlled by clear-cut legal rules, while the latter
is controlled by ambiguous standards. As a result, bid evaluators would be more
vulnerable to cognitive bias at the evaluation stage than at the qualification one.
Additional support for this can be drawn from research on motivational reasoning,15
which demonstrates that confirmation bias increased in ambiguous situations
15
The tendency to evaluate information in a way beneficial to the decision maker self-interest.
28
(Thompson and Lowenstein 1992) like the one characterizing the evaluation phase of
the CB process.
4.3 Complexity and relativity of decisions
Another seemingly relevant distinction between the two phases of the CB process is
the relative simplicity of the first task - rejecting faulty bids - as opposed to the
complexity of the second - evaluating and scoring the bids' qualitative evaluation
criteria.
Research on decision-making reveals that people experience difficulties in
reaching complex decisions, while simple ones are much easier to handle
(Tversky 1972; Payne 1976; Wood 1986; Swait and Adamowicz 2001). One of the
methods that decision makers implement in order to deal with complex decision tasks
is concentrating on the most prominent attribute and choosing the option that is
superior on that dimension – the prominence effect (Tversky, Sattath and Slovic 1988;
Fischer and Hawkins 1993; Hawkins 1994; Fischer, Carmon, Ariely and Zauberman
1999).16
In our case, the first CB stage needs a relatively simple decision: whether a bid
is flawed and, if so, to reject it. In border-line situations that decision can be difficult,
but this does not make it a complex one. By complex decision, we mean a multidimensional decision that requires many factors to be considered in order to reach the
final judgment. On the other hand, the second stage is much more complicated and
demanding, since decision-makers must take into consideration multiple attributes and
assess them all, one against the other, in a multidimensional comparison.
Additional distinction between the CB two phases relates to the perception
that people react differently when they have to judge an option by itself, as opposed to
judging it in a relativistic context (Bazerman, Loewenstein and Blount White 1992;
16
While there is some resemblance between the prominence effect and the halo effect, discussed in
Section 2.2.2 above, they have different characteristics. The halo effect points to the fact that decisionmakers tend to irrationally attribute from certain known characteristics of an object to its unknown
characteristics. The prominence effect means that in complex situations people tend to focus on one
prominent consideration on the account of others.
29
Hsee 1996; Bazerman at al. 1999). At the first stage of the CB, bid evaluators are
required to concentrate on one bid at a time and decide its fate, disregarding the other
bids' status. In such a process there is absolutely no relevancy whether the other bids
are flawed or comply with the CB demands. Every bid is judged by itself and on its
own attributes. Hypothetically, even if all the bids suffer from the same flaw (for
example: failed to meet the submission deadline) they must all be disqualified, one
after the other. However, at the evaluation stage, the decision making process is
relativistic. Bid evaluators have to consider, weigh, and compare many factors during
the assessment process, and by that taking into account all the qualified bids
attributes. For example, a bidder with three excellent recommendations may get
substantially different score if its competitor has five recommendations, or only two.
Even though bid price should not affect the decision whether to qualify or
disqualify a faulty bid, or how many points deserve each of the qualitative evaluation
criteria, price is a prominent factor in most CBs. Presumably, while decision-makers
can resist its unconscious influence when arriving at a simple and absolute decision
(the qualification stage) they might be unable to do so when a complex and relativistic
decision is at stake (the evaluation stage).
4.4 Noticeable vs. Unnoticeable deviations from the rules
Research on dishonesty behavior shows two phenomena that have relevancy to
our case as well. First, it shows that people are willing to act in dishonest way to the
extent that allows them to maintain their self concept of integrity (Mazar, Amir and
Ariely 2008). Therefore, they may allow themselves small deviations from obliging
rules but not a large and noticeable one.
Second, it demonstrates that the willingness to accept small deviations from
ethical standards, even though these deviations aggregate (slippery-slope effect), is
greater than the willingness to accept one substantial deviation from the outset. This is
due to lack of noticeability of each of the small deviations (Gino and Bazerman 2005)
or the narrow perspective of decision makers in situations that include small ethical
deviations (Schurr, Ritov, Kareev and Avrahami 2012).
This phenomenon can provide another explanation for the differences in
vulnerability to cognitive bias between the two CB phases. In the qualification stage,
30
passing on a faulty bid is a clear and noticeable deviation from the law and from the
solicitation demands. Therefore, in such decision substantial resistance for law
infringement and cognitive biases like the LBB can be expected. In contrast, the
evaluation phase is composed by many small decisions as to the scoring of each of the
evaluation criteria. Small deviation in any of these decisions from the "right" scoring
is difficult to be bobserved and noticed either by the decision maker herself or by any
external observer. This lack of noticeability may results in greater willingness of
decision makers to "cut corners" in order to get what they identify as a good bargain
for the government. It may also results in a higher level of exposure to cognitive
biases like the LBB.
To sum up, there are several differences between the CB stages that may
explain the differences in bid evaluators' vulnerability to confirmation bias. While the
first stage contains a binary task, the second entails a scale one; while the first
decision is governed by relatively clear-cut "right and wrong" rules, the second is
governed by vague principles with no "right or wrong" answer; while the first stage
comprises a relatively simple and defined task, the second contains a complex
relativistic one; and while the first stage does not comprise an evaluation process, the
second does. In addition, deviation from the CB rules in the qualification stage is
noticeable both to the decision maker herself and to external observers, while in the
evaluation stage such deviation is substantially less observable. Any of these, standing
alone or combined, may account for the different levels of decision-makers'
vulnerability to the LBB in particular, and to cognitive biases in general.
Further research, however, is needed in order to indicate the relative importance
of various characteristics of the decision-making process (i.e. binary vs. multi-choice
decision; scoring vs. passing decision; clear-cut rules vs. ambiguous standards etc.) on
its vulnerability to cognitive bias.
5. NORMATIVE APPLICATIONS
Even without committing to concrete explanation for the differences in vulnerability
to the LBB between the two CB stages, our results seem to carry noteworthy policy
31
implications, some of which refer to the procurement process and others that are more
general.
5.1 Normative Applications regarding Public Procurement
With regard to the competitive public procurement process, two policy implications,
that are the flip sides of the same coin, seem to follow from our findings. First, the
existence of the LBB during the bids evaluation process provides justification for a
legal regime under which any CB that contains qualitative evaluation criteria (like
Requests For Proposals (RFP)) should be conducted through a two-stage CB, by
which the submitted price is revealed only after the evaluation process is ended.
Second, since bid evaluators are able to resist that bias in qualifying decisions, the
above conclusion does not refer to CBs by which the competition is on price alone
(like Invitations For Bids (IFB)).
The Federal Acquisition Regulations do not dictate whether to conduct a onestage or a two-stage CB. Actually, a two-stage CB is not even mentioned in the
regulations. The Federal Acquisition Regulations do refer to a closely-resembling
process - "Two-Step Sealed Bidding" (FAR § 14.5) - which is a voluntary procedure
by which the bids submitted at the first stage include only technical features. Only
after these technical components have been reviewed, are the bidders who have
passed that stage invited to submit their bid price. This procedure is reserved mainly
for cases involving complex technical issues, and is designed to enable the purchaser
to formulate its exact requirements over the course of the procurement process. Such
need is naturally rare. Our findings point to a need for reform so as to render a
regulatory presumption that any RFP will be conducted through a two-stage CB,
unless there is special reason not to.
5.2 Applications with regard to other legal fields
While the current study focuses on decisions in the field of public procurement, the
results carry potential implications for other fields of law and decision-making. In
various fields of law one can distinguish between two stages of decision-making
process in which the first (preliminary-qualifying) stage is binary in nature and
controlled by relatively clear-cut rules, while the second is a scale-type and requires a
32
more open-ended and relativistic evaluation of various factors to be weighted. For
example, in criminal law the court is first called to decide whether the accused is
guilty (or not) and only after guilt is determined, to decide the gravity of the
punishment (usually a scale-type discretionary decision). Similarly, in tort law one
can distinguish between the first stage, in which the court is required to decide
whether the defendant is liable, and the second stage, by which the principal issue is
estimating damages (again, in a scale-type, open-ended decision-making process).
Likewise, in evidence law, we often distinguish between the stage at which the court
is called to decide whether certain information is admissible (according to relatively
clear-cut rules), and then consider the relative weight of the evidence that comply
with the admissibility rules. Unlike the first stage, at which the decision is made for
each and every piece of evidence separately, in the second stage the court is called to
weight all pieces of evidence together, in a relativistic manner. At this latter stage, the
weight of each factor may influence and may be influenced by the evaluation of the
other pieces of evidence that qualify for this stage.
Our findings suggest that there is a reason to assume that the decisions at the first
stage of the above examples are more resistance to cognitive biases than those in the
second. Of course, the decision-making processes in the above examples are different
from those studied in our experiments; further research is required before one can
draw robust general conclusions in this respect. The principal finding of the current
study however, is that – in contrary with initial intuitions – there is a significant
difference in the vulnerability of different types of decision-making processes to
cognitive biases. Since different types of decisions often co-exist in various legal
processes the outcomes of our study carry potential implications for future regulation
of different types of decision-making processes in many other fields of law and
regulation.
6. SUMMARY AND CONCLUSIONS
Our study demonstrates that CB officials are subject to cognitive bias in favor of the
lower bidder when they asked to evaluate and score competing bids. However, when
they are required to decide whether or not submitted bids meet prerequisites that were
33
set in the solicitation documents, procurement officials proved impressively resistant
to the influence of the same cognitive bias.
In the context of public procurement these findings call for a regulatory change –
setting the two stage competitive bidding as the default rule in any CB that require
bids evaluation and scoring, namely – Request For Proposals. More generally, the
study results call for the attention of regulators and decision-makers in the legal field
to the distinction between binary decisions that are often made according to clear-cut
and relatively simple rules, and discretionary evaluation decisions based on more
complicated and inter-dependent scale-type comparison between competing options.
Our study demonstrates the potential of that distinction to serve as a useful tool for
explaining levels of vulnerability of decision-makers to cognitive biases, although,
admittedly, more research is required to substantiate such general inference.
34
APPENDICES
Appendix A. Experiment 1 form
The municipality of Ramat-Gan conducted a Request for Proposals (RFP) for the
transportation of the city's schoolchildren during the 2011-2012 school-year. A weight
of 60% was given to the bid price and 40% was given to quality criteria. Two bids were
submitted.
[1st version The bid prices were submitted in separate sealed envelopes, to be opened
after examination of the other parts of the bids (two-stage sealed bidding).]
[2nd version "The Carrier Ltd.” submitted the lowest bid, at $800,000, and received the
full 600 points (out of 1000) for the price criterion. The other bid, submitted by “The
Transporter Ltd.,” for $875,000, scored only 550 points for the price criterion.]
As a member of the contracting committee, you are asked to score the bids with respect
to the other criteria:
Criterion17
[2nd version only:
Price]
Years of operation
Bus model
Max.
points
600
"The Carrier"
$800,000
Points
600
"The Transporter"
$875,000
60
30
8 years
Mercedes "VIP" 2011
Recommendations
Number of buses
owned by the bidder
Average experience
of the drivers
Training hours per
driver
ISO certification
70
60
2 good, 1 moderate
12 buses
12 years
Mercedes "Dolphin"
2010
5 good
20 buses
70
12 years
20 years
30
35 hours
20 hours
50
Has certification
Distance from R.G.
Total
30
1000
Not certified, but in the
final stages of the
certification process
Bnei-Brak (5 km)
17
Points
550
Ra’anana (15 km)
The order of presentation of the evaluation criteria was randomly altered between the two rounds of the
experiment, to minimize the possibility of recalling responses from the first round of the experiment.
35
Table A1. Experiment 1 results: The mean scores for the bids’ components at each stage
Criterion
Max.
points
"The Carrier"
2nd version only:
Price
600
$800,000
Years of operation
60
8 years
Bus model
30
Mercedes
2011
Recommendations
70
Number of buses
owned by the
bidder
Average points
"The Carrier"
2-stage
1-stage
CB
CB
"The
Trans."
Average points
"The Trans."
2-stage
1-stage
CB
CB
----
600
$875,000
----
550
43.33
(12.31)
49.72
(8.53)
12 years
55.41
(9.81)
58.47
(4.28)
28.56
(4.07)
29.31
(2.44)
Mercedes
"Dolphin"
2010
24.39
(5.12)
25.72
(4.31)
2 good,
1 moderate
37.64
(12.15)
44.58
(11.29)
5 good
67.03
(6.00)
66.81
(7.85)
60
12 buses
36.56
(10.90)
39.28
(9.24)
20 buses
54.25
(8.00)
56.25
(9.88)
Average
experience of the
drivers
70
12 years
49.58
(12.56)
56.94
(12.21)
20 years
67.03
(6.12)
68.19
(5.09)
Training hours per
driver
30
35 hours
28.50
(3.28)
28.42
(3.29)
20 hours
20.42
(5.53)
21.81
(3.72)
ISO certification
50
21.94
(18.02)
20.69
(19.71)
Has
certification
49.72
(1.67)
46.66
(10.95)
Distance from R.G.
30
Not certified, but
in the final stages
of the certification
process
Bnei-Brak (5 km)
400
28.75
(3.85)
297.42
(36.08)
Ra’anana
(15 km)
Total (quality)
27.50
(3.68)
273.61
(43.00)
19.61
(6.48)
357.86
(30.04)
23.17
(7.76)
367.08
(22.32)
Total
1000
873.61
897.42
907.86
917.08
"VIP"
36
Appendix B. Experiment 2 form
The preface is exactly the same as in Appendix A.
Criterion
[2nd version only:
Price]
Years of operation
Recommendations
Number of buses
owned by the bidder
Average experience
of the drivers
Distance from R.G.
Total
Max.
points
600
"The Carrier"
Points
$800,000
600
"The Trans."
Points
$875,000
550
75
80
70
8 years
2 good, 1 moderate
12 buses
12 years
5 good
20 buses
75
12 years
20 years
Bnei-Brak (5 km)
Ra’anana (15 km)
100
1000
Table B1. Experiment 2 results: The mean scores for the bids’ components at each stage
Criterion18
2nd version only:
Price
Max.
points
"The
Carrier"
Average points
"The Carrier"
2-stage CB
1-stage CB
----
600
"The
Trans."
2-stage CB
1-stage CB
----
550
600
$800,000
80
2 good,
1 moderate
42.96
(8.83)
47.65
(11.86)
5 good
78.79
(3.18)
78.97
(5.57)
Number of buses
owned by the
bidder
70
12 buses
44.41
(11.63)
47.52
(10.33)
20 buses
66.89
(8.06)
69.14
(2.70)
Average
experience of the
drivers
75
12 years
49.95
(13.78)
53.73
(14.59)
21 years
72.59
(5.77)
72.97
(5.09
Distance from R.G.
100
Bnei-Brak
(5 km)
91.72
(12.48)
90.86
(22.83)
Ra’anana
(15 km)
79.31
(16.73)
76.38
(21.99)
Years of operation
75
8 years
56.59
(9.68)
55.57
(9.68)
12 years
73.34
(3.30)
73.93
(2.45)
Total (quality)
400
283.57
(34.41)
295.52
(47.62)
370.93
(21.99)
371.38
(25.26)
1,000
883.57
895.52
920.93
921.38
Recommendations
Total
18
$875,000
Average points
"The Trans."
The order of presentation of the evaluation criteria was randomly altered between the two rounds of the
experiment.
37
Appendix C. Experiment 3 form
The following case is based on a real competitive bidding conducted by the Ministry of Health. Together
with the opening of the new medical school in the city of Safed [a city in northern Israel] the Ministry of
Health decided to publish an IFB for the building of an advanced pediatric oncology clinic. The clinic
was meant to be a part of the hospital in Safed, which did not yet have those facilities, and sick children
were therefore required to travel to distant hospitals in order to receive the treatment they required.
[1st version: The bidders were asked to submit the bid amount in a separate sealed envelope, to be opened
after the examination of the other parts of the bids. Upon the completion of this examination (except for
the price) you received the following document:]
[2nd version: After the completion of the examination you were asked to determine who should win the
IFB, based on the following document:]
2nd version
only:
The bid
price.
(In millions
of dollars).
The bidder
has built 2
projects, at a
cost of 12.5
million each,
during 2010 –
2011.
The bidder
employs a
construction
engineer with
8 years
experience.
The bidder is
officially
registered as
"unlimited".
The bidder employs
a physician with 10
years of seniority,
who completed a
residency in
pediatric oncology.
The bidder
must
submit a
$25,000
bid bond
valid until
6/1/2012.
Bid
A
17.8
V
See remark
V
V
V
Bid
B
18.9
V
V
V
See remark
V
Bid
C
Bid
D
20.3
V
V
V
V
V
20.8
V
V
V
V
V
Remarks
According to the
certification attached,
the engineer has 6
years and 8 months of
experience.
Residency in general
oncology (completed
in 1993), because
there were no pediatric
oncology residencies
in Israel at that time.
Since then has worked
as a pediatric
oncologist in a
hospital.
The bid complies with
the requirements.
The bid complies with
the requirements.
[1st version: Mark the bids that will proceed to the next stage (the opening of the price envelopes):]
[2nd version: Mark the winning bid:]
 Bid A
 Bid B
 Bid C
 Bid C
38
Appendix D. Experiment 4 Form
Invitation for Bids No. A041024210 for the supply of high voltage electric cables
The company's contracting committee wants to share with you a dilemma recently laid on its table. Please
read the following and mark/indicate what would be your decision as a contracting committee member.
For the construction of new high voltage electricity line the Israeli Electric Company published an
international Invitation for Bids (IFB) for the acquisition of 30 km of high voltage copper cables, 30 mm
diameter. The completion of the line construction is urgent and needed to assure regular electric supply in
the coming summer. The prior assessment as to the transaction cost is 50,000 dollars per Km, a total of
1.5 million dollars.
[1st version: The IFB was operated in a two-stage version (bidders were asked to submit their bids in two
envelopes. One envelope contains the technical proposal and the second contains the proposed price. The
IFB records determined that the price envelopes will be opened at the end of the technical review
process).
At the end of the bids' review process (not including the price) the following table was presented to you.]
[2nd version: After the completion of the review process the following table was presented to you in order
to determine the IFB results:]
2nd version
only: Prior
assessment:
1,500,000$
The bid
price.
Valid
affidavit
concerning
bidder's
proper
management
Prior supply
of 20 km of
similar cables
in any of the
years 20092011
The
manufacturer
operates a
quality control
department.
Obligation to start
the supplying of the
cables within 30
days from the notice
of winning
Bid
A
1,300,000$
V
See remark
V
V
Bid
B
1,400,000$
V
V
See remark
V
Bid
C
Bid
D
1,600,000$
V
V
V
V
1,800,000$
V
V
V
V
Remarks
In 2009, supplied 18 km of the
required cables, in 2010 - 21
km, and in 2011 - 27 km.
The manufacturer has a
contract with a lab that
supplies them quality control
services
The bid complies with the
requirements.
The bid complies with the
requirements.
[1st version: Mark the bids that will proceed to the next stage (the opening of the price envelopes, and
choosing the IFB winner):]
[2nd version: Mark the bid that in your opinion should win the IFB:]
 Bid A
 Bid B
 Bid C
 Bid C
39
40
Appendix E. Experiment 5 Form
Invitation for Bids No. A041024211 for the supply of 45 megawatt generators
The company's contracting committee wants to share with you a dilemma recently laid on its table. Please
read the following and mark/indicate what would be your decision as a contracting committee member.
Due to expected manufacturing deficiency/shortage in electric supply in the coming summer the company
has published an international Request for Proposals (RFP) for the acquisition of six generators each one
with 45 megawatt load/power.
The prior assessment as to the transaction cost is 3.5 million dollars for each generator, a total of 21
million dollars.
[1st version: After the completion of the prerequisite review (and before scoring the technical part of the
bids and opening the price envelopes) the following table was presented to the contracting committee.]
[2nd version: After the completion of the prerequisite review (and before the scoring of the technical part
of the bids) the following table was presented to the contracting committee.]
Bid A
2nd version only:
Prior assessment:
$21,000,000
The bid price
$23,000,000
ISO
9001,
2010
edition
V
The bidder
has a 5 years
experience in
the field
V
Five years
guarantee
for the
product
V
Obligation to supply
at least 2 generators
within 60 days from
the notice of winning.
V
Bid B
$21,550,000
V
V
V
V
$25,250,000
V
V
V
(See remark)
V
$20,875,000
V
V
See remark
V
Bid E
$24,750,000
V
V
V
V
Bid F
$20,750,000
See
remark
V
V
V
Bid G
$23,875,000
V
V
V
V
$21,675,000
V
V
(See remark)
V
V
$23,200,000
V
V
V
V
Bid C
Bid D
Bid H
Bid I
Remarks
The bid complies with the
requirements.
The bid complies with the
requirements.
The bid complies with the
requirements. The bidder has
37 years of experience.
The bidder has 42 years of
experience in manufacturing
turbines, but only 4.5 years of
experience in manufacturing
generators.
The bid complies with the
requirements.
The bidder does not have ISO
certification, but declared that
he has finished the
certification process and
expected to receive it in a
week.
The bid complies with the
requirements.
The bid complies with the
requirements. The bidder gave
10 years guarantee.
The bid complies with the
requirements.
[1st version: Circle the bids that you would pass on to the next stage (in which will be the scoring of the
technical parts and the opening of the price envelopes):]
[2nd version: Circle the bids that you would pass on to the next stage (in which will be the scoring of
the technical parts and the choosing of the CB winner):]
Bid A
Bid B
Bid C
Bid D
Bid E
Bid F
Bid G
Bid H
Bid I
41
REFERENCES
Ask, Karl and Pär Anders Granhag 2007. Motivational Bias in Criminal Investigators'
Judgments of Witness Reliability. Journal of Applied Social Psychology 37:561591.
Bazerman, Max, George F. Loewenstein and Sally Blount White 1992. Reversals of
Preference in Allocation Decisions: Judging an Alternative versus Choosing
among Alternatives. Administrative Science Quarterly 37:220-240.
Bazerman, Max, Don A. Moore, Ann E. Tenbrunsel, Kimberly A. Wade-Benzoni and
Sally Blount 1999. Explaining How Preferences Change Across Joint versus
Separate Evaluation. Journal of Economic Behavior & Organization 39:41-58.
Carlson, Kurt A. and J. Edward Russo 2001. Biased Interpretation of Evidence by
Mock Jurors. Journal of Experimental Psychology: Applied 7:91-103.
Dekel, Omer and Amos Schurr 2013. The Lower Bid Bias (working paper). Available
at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2270773.
Edwards, Kary and Edwards E. Smith 1996. A Disconfirmation Bias in the Evaluation
of Arguments, Journal of Personality and Social Psychology 71:5-24.
Eerland, Anita and Eric Rassin 2012. Biased Evaluation of Incriminating and
Exonerating (Non) Evidence. Psychology, Crime and Law 18:351-358.
Feldman, Yuval and Alon Harel 2008. Social Norms, Self-Interest and Ambiguity of
Legal Norms: An Experimental Analysis of the Rule vs. Standard Dilemma.
Review of Law and Economics 4:81-126.
Fischer, Gregory W. and Scott A. Hawkins 1993. Strategy Compatibility, Scale
Compatibility, and the Prominence Effect. Journal of Experimental Psychology,
Human Perception and Performance 19:580-597.
Fischer, Gregory W., Ziv Carmon, Dan Ariely and Gal Zauberman 1999. Goal-Based
Construction of Preferences: Task Goals and the Prominence Effect. Management
Science 45:1057-1075.
Fox, Craig R. and Amos Tversky 1995. Ambiguity Aversion and Comparative
Ignorance. The Quarterly Journal of Economics 110:585-603.
42
Goldin, Claudia and Cecilia Rouse 2000. Orchestrating Impartiality: The Impact of
"Blind" Auditions on Female Musicians. The American Economic Review 90:715741.
Greenwald, Anthony G. and Linda Hamilton Krieger 2006. Implicit Bias: Scientific
Foundations. California Law Review 94:945-967.
Guthrie, Chris, Jeffrey J. Rachlinski and Andrew J. Wistrich 2001. Inside the Judicial
Mind. Cornell Law Review 86:777-830.
--- 2007. Blinking on the Bench: How Judges Decide Cases. Cornell Law
Review93:1-44.
--- 2009. The Hidden Judiciary: An Empirical Examination of the Executive Branch
Justice. Duke L.J. 58:1477-1530.
Johnson, Eric. J. and Daniel Goldstein 2003. Do defaults save lives? Science
302:1338-1339.
Hawkins, Scott A. 1994. Information Processing Strategies in Riskless Preference
Reversals: The Prominence Effect. Organizational Behavior and Human Decision
Processes 59:1-26.
Hsee, Christopher K. 1996. The Evaluability Hypothesis: An Explanation for
Preference Reversals between Joint and Separate Evaluations of Alternatives.
Organizational Behavior and Human Decision Processes 67:247-257.
Hsee, Christopher K. and Jiao Zhang 2010. General Evaluability Theory. Perspectives
of Psychological Science 5:343-355.
Kahneman, Daniel 2011. Thinking Fast and Slow (Farrar, Straus and Giroux).
Kahneman, Daniel and Amos Tversky 1979. Prospect Theory: An Analysis of
Decision under Risk. Econometrica 47:263-292.
Kelman, Mark, Yuval Rottenstreich and Amos Tversky 1996. Context-Dependence in
Legal Decision Making. The Journal of Legal Studies 25:287-318.
Kunda, Ziva 1987. Motivation and Inference: Self-Serving Generation and Evaluation
of Evidence, Journal of Personality and Social Psychology 53: 636-647.
Kunda, Ziva 1990. The Case for Motivated Reasoning. Psychological Bulletin
108:480-498.
43
Lerner, Jennifer S., and Philip E. Tetlock 1999. Accounting for the Effects of
Accountability. Psychological Bulletin 125:255-275.
Lichtenstein, Sarah and Paul Slovic 1971. Reversal of Preferences between Bids and
Choices in Gambling Decisions. Journal of Experimental Psychology 89:46-55.
Lord, Charles G., Lee Ross and Mark R. Lepper 1977. Biased Assimilation and
Attitude Polarization: The Effects of Prior Theories on Subsequently Considered
Evidence, Journal of Personality and Social Psychology 37:2098-2109.
Meloy, Margaret G. and J. Edward Russo 2004. Binary Choice under Instructions to
Select versus Reject. Organizational Behavior and Human Decision Processes
93:114–28.
Nickerson, Raymond S. 1998. Confirmation Bias: a Ubiquitous Phenomenon in Many
Guises, Review of General Psychology 2:175-220.
Nowlis, Stephen M. and Itamar Simonson 1997. Attribute-Task Compatibility as a
Determinant of Consumer Preference Reversals. Journal of Marketing Research
34:205-218.
O'Brian, Barbara 2009. Prime Suspect: An Examination of Factors that Aggravate and
Counteract Confirmation Bias in Criminal Investigations. Psychology, Public
Policy and Law 15:315-334.
Park, C. Whan, Sung Y. Jun, and Deborah J. MacInnis 2000. Choosing What I Want
versus Rejecting What I Do Not Want: An Application of Decision Framing to
Product Option Choice Decisions. Journal of Marketing Research 37:187–202.
Payne, John W. 1976. Task complexity and contingent processing in decision making:
An information search and protocol analysis. Organizational behavior and human
performance 16:366-387.
Rachlinski, Jeffrey J. 2012. Judicial Psychology. rechtstreeks 2:15-34.
Rachlinski, Jeffrey, Chris Guthrie and Andrew J. Wistrich, 2006. Inside the
Bunkruptcy Judge’s Mind, B.U.L.Rev 86: 1227-1265.
Rassin, Eric, Anita Eerland and Ilse Kuijpers 2010. Let's Find the Evidence: An
Analogue Study of Confirmation Bias in Criminal Investigations. Journal of
Investigative Psychology and Offender Profiling 7: 231-246.
44
Ross, Lee, Mark R. Lepper and Michael Hubbard 1975. Perseverance in Self
Perception and Social Perception: Biased Attributional Processes in the Debriefing
Paradigm, 32 Journal of Personality and Social Psychology 32:880-892.
Samuelson, William and Richard Zeckhauser 1988. Status Quo Bias in Decision
Making. Journal of Risk and Uncertainty 1:7–59.
Shafir, Eldar 1993. Choosing Versus Rejecting: Why Some Options Are Both Better
and Worse Than Others. Memory & Cognition 21:546-556.
Simonson 2008. Will I Like a "Medium" Pillow? Another Look at Constructed and
Inherent Preferences. Journal of Consumer Psychology 18:155-169.
Simonson, Itamar and Peter Nye 1992. The Effect of Accountability on Susceptibility
to Decision Errors, Organizational Behavior and Human Decision 51:416-446.
Simonson, Itamar and Amos Tversky 1992. Choice in Context: Tradeoff Contrast and
Extremeness Aversion. Journal of Marketing Research 29:281-295.
Swait, Joffre and Wiktor Adamowicz 2001. The Influence of Task Complexity on
Consumer Choice: A Latent Class Model of Decision Strategy Switching. Journal
of Consumer Research 28:135-148.
Thompson, Leigh and George Lowenstein 1992. Egocentric Interpretations of
Fairness and Interpersonal Conflict. Organizational Behavior and Human Decision
Processes 51:176-197.
Tversky, Amos 1972. Elimination by Aspects: A Theory of Choice. Psychological
Review 79:281-299.
Tversky, Amos and Daniel Kahneman 1986. Rational Choice and the Framing of
Decisions. The Journal of Business 59:251-278.
Tversky, Amos and Daniel Kahneman 1991. Loss Aversion in Riskless Choice: A
Reference-Dependent Model. The Quarterly Journal of Economics 106:1039-1061.
Tversky, Amos, Shmuel Sattath and Paul Slovic 1988. Contingent Weighting in
Judgment and Choice. Psychological Review 95:371-84.
Wedell, D. H. 1997. Another Look at Reasons for Choosing and Rejecting. Memory
& Cognition, 25:873–887.
45
Wistrich, Andrew J., Chris Guthrie and Jeffrey J. Rachlinski 2005. Can Judges Ignore
Inadmissible Information? The Difficulty of Deliberately Disregarding. University
of Pennsylvania Law Review 153:1251-1345.
Wood, Robert E. 1986. Task Complexity: Definition of the Construct. Organizational
behavior and human decision processes 37:60-82.
46