E
: P
- E
,
*
P
D
, C
,
F
David Rothschild
Yahoo! Research
David@ResearchDMR.com w ww.ResearchDMR.com
Abstract In this article I test a new graphical, interactive interface that captures bo th “best estimate” po in t - estimate s and probability distributions fro m non - experts. As in the previo us literature, re spondents are overconfident. My innovation is to show that in contrast to the st andard meth od of directly asking respondents to state their confidence, using my method, which
induc e s the respondents to reveal confide n ce, there is a sizable and statically significant positive rela tionship between the respondents’ co n f i d e n c e an d the accur a cy of their individua l- level expectations. This positive co rr elation between confidence and accuracy can be utilized to create confidence - weighted aggregated forecasts that are more accurate than the stan dard “consensus forecasts”. Th e payment of financial incentives does not affect these findings.
Rough Draft: Do NOT Cite or Circulate without the Author’s Perm ission
This draft: May 27, 2011
Latest draft: http://ResearchDMR.com Keywords: Polling, informat ion aggregation, belief heterogeneity JEL codes: C52, C53, C8
3, D03, D8
*
Thank you to Alex Gelber, Andrew Leonard, Katherine Milkma n, Neil Mal h otr a , Ja mes Rothschild Jr. , Uri Simons ohn, Jeremy Tobacman, Ju stin Wolfers, and the two anonymous reviewers for the Workshop on Social Computing an d User Genera te d Content. Fu r t he r tha nks to the Wharton Ri sk Center’s Ackoff grant program tha t helped me fu nd this project.

The main meth od tested in this article captures “best - estimate ” point - estimates and th e n probab ility distributions from non
-experts on upcoming events. Utilizing a graphical, interactive interface, my method
provides several innovations to ease the creation of probabi lity distributions by non - expert respondents. The point - estimate s and probability distributions are used to explore the nature of both individ ual- le vel expectations and aggregated forecasts, especially in regard to individ ual- le vel confidence, as revealed through the size and shape of the responde nts’ probability distribution.
1
The responses demonstrate a slight overconfiden ce. In testing over/under- confidence, I compare my method of gathering probabil ity distributions to the two most attractive alternatives: a simple four point scale of stated confidence an d the best calibrated/transparent method of co llecting a confidence range around a point - estimate. Where comparable, the probab ility distributions and confidence range provide a similar slight overconfide n ce; this overconfide n ce is consistent with the literature (Soll and Klayman, 2004; Teigen and Jo rg e n s e n , 2005; and Speirs - Bridge et al., 2010).
There is a sizable and statistically significant cor relation between confidence derived from the respondents’ probabi lity distributions and the accur a cy of the corresponding expectations that has not been previo usly docum e nted (where confidence is estimate d as the inverse of the variance). This correlat ion is true both within ques tion s (i.e., between respondents) and within respondents (i.e., between ques tion s )
. This correl a tion is weaker and/or non
-significant in the two simplified methods. Engleberg et al (2009) determines that th e distribution of probabi lit ies widens as the event horiz o n lengthens. Yet, that article does not address the correlation with the size of a distribution and the accuracy for a given event. Other
1
Over/under-confidence is determined by the percenta ge of answers tha t come to fruiti on within a confidence rang e ; a set of perfectly ca li br a t e d 80% conf idence ra ng es would ha v e 80 % of the answers occur wi th i n the range. Stated confidence cannot provide a mea sure and the conf idence ra ng e provides only the one ra ng e it asks, where the probability di s tr i bu ti on reveals an infini

te set of confidence ra nges.
Page 2 of 25

2
articles conc lude that tigh ter confidences ranges correlate wit h increased overconfide n ce (i.e., at best, confidence is uncorrelated with accuracy fo r non - experts).
3
This first finding addresses various literatures where Likert - type rating sca les are used to supplement estimates. Phycology, politica l science, and marketin g are jus t a few literatures where there are standard survey practices that ask respondents
to state their: confidence, likelihood, or agreement with their previo usly stated estimations. Generally these are ordered, one - dimensional scales that are designed to provide an extra dimension to the estimation.
In Likert - type rating scales some respondents gravitate towa rds or away from extreme choices; they are responding to varying ince ntives not related to their true state in regard to th e ques tion
. When I use a re ve a l e d meth od for confidence, respondents are unaware of the extremities of reasonable answe rs or provided any reference if they wish to manipulate the strength or weakness of their response. With Likert - type rating scales respondents see thes e ques tion s as separate follow - up ques tion s , rather than the main answer, provid ing different levels of attention and incentives. In this article I test a method tha t has the respondents reveal their confidence inside th e main ques tion
. Finally, the graphical nature of the method, fully utilizing a web- based sett ing, allows the researcher to subtly provide information that allows respondents to a nswer complicated ques tion s which would be very difficult to ask in a telephone or in - person setting. Thus, the revealed confidence from my method is more representa tive of the respondents’ information, rather than th e i r natural state of confidence or a range manipulated for any other incentive than
the “best estimate” of th e respondent.
Confidence can be used as a we igh t in the creation of aggregated forecasts of the level of the outcome; confidence - weighted fo re ca s t s are more accura te than forecasts from the point - estimates alone. There is consensus that aggregating expectations creates more accur a te
2
Kuklinski (20 00) concludes tha t for objective political questions confidence ca n actually be neg a ti v el y correlated with accura cy.
3
Barber and Odean (2 001 ) shows tha t me n are more overconfident tha n women, as demonstrated with more aggres si ve beha vi or in the stock ma rket, rega rdless of knowledge. My meth od reveals the confidence rang e , wi th o u t the other confounding fa ctors of utility in a decision su ch as investing. One reason for the results in Kuklinski (20 00) is tha t respondents perceive an incentive to be strategic about stated conf idence levels in political questi ons, but thi s is not an issue when people are revealing, ra the r tha n stating their conf idence.

Page 3 of 25

4 forecasts, on average, than picking individua l- le vel expectations; Pennock and Reeves (2007) demonstrates that with a self - selected group of respondents making NFL prediction s only 6 of the 2231 players are more accurate than the collec t ive average. Further, there are numerous studies show ing that the simplest aggregation te chniques are the most efficient (Bates and Granger, 1969; Stock and Watson, 2004;
Smith an d Wallis, 2009). The simplest weight considered in these articles, other than an a verage, is the inverse of the mean square error; this occasionally provides a slig htly smaller erro r than averages. The ques tio n s for this article are answered concurrently by non - experts, providin g no opportunity for historically derived weights; yet, if the inverse of the varia nce of their probab ility distribut ions is well calibrated, it could serve as a proxy for mean square error.
Prediction markets weigh their individual - level data by how much money people are willing to gamble, a proxy for conf id ence; this second finding illustrates the benefits of a new method of dire ct ly weighing individ ual- leve l pol ling data by confidence. There is growing consensus that prediction markets aggr egate individ ual- le vel expectations into more accurate forecasts than stan dard polling methods. Rothschild (2009) confirms the accuracy of prediction market - based forecasts an d outlines some of the differences between them and standard polls as meth ods of capturing and aggregating individual- level responses: the nature of their sample of respondents, th e ques tion asked to the respondents, the aggregation method of th e individ ual- level data, an d the incentives for the respondents. Rothschild and Wolfers (2011) addresses the q ues tion being asked of the respondents; that article conc ludes that eliciting expectations, as prediction markets do, captures more informatio n from the respondent than th e ques tion s asked in stan da rd polls. This article addresses anoth e r of the differences between prediction markets an d polls, the weighting of th e individual- level data. Traditional methods of gathering information from individ u al respondents fail to capture large quantities of the respondents’ relevant informatio n an d the quality of the information they reveal; this new method gathers more rele vent information and provides a new rubr ic fo r weighing its quality.
4
Without hi s to r i c al da ta, I cannot upda te on ri s k profile as in Chen et al (20
03) or more complex Bayesian methods li ke those cata loged in Clemen ( 198 9) or sugg

ested in Clemen and Winkler (1 993 ) .
Page 4 of 25

In this article I test a graphical, interactive interf ace to advance the ability of researchers to collect and utilize individual- le vel expectations. I capture both point - estimates and probab ility distributions from non
-expert respondents. The results provide insight into expectations, revealing the natu re of their: accura cy, confidence, and calibration of probabil ities. Researchers can use these expectations as individual- level data for aggregated forecasts and, in the future, to understand heterogene ity in revealed behavior under uncertainty. Better understanding of non - expert expectations will allow researchers to learn more about the absorption and t ransform ation of information. Better forecasts help researchers connect shocks with changes
in the underlying val u es of the world and investors make more efficie n t use of their time an d money. As telephone and in - pers on - based polling is supplanted by web- based polling, I hope that this article will inspire furth e r explo ration into graphical, interactive interfaces that utilize the advantages of web- based polling, that ask the qu e s t i o n s and consequently gather information not possible in standard telephone or in - person - based polling.
I build on the most recent methods of surveying expectations to create a
graphical, interactive interface that gathers expe ctations: point - estimates and probability distributions. One influence on my meth od is Dela vande and Rohwedder (2008), which asks respondents for a point - estimate an d then uses a visual screen that as ks th e respondents to distribut e 20 balls (each representing a 5% probab ility of the outcome) in a series of bins that signify possible value - ranges of the outcome. I enhance their method with lessons from th e literature involving the generation of confidence ranges around poin t - estimates. A few key innovations of my method: it dis t rib u t e s probabil ities as small as 1% into upwards of nine bins of valu e - ranges, forces the respondent to consider the ques tion from multiple angles, and uses new graphical tools to efficiently clarif y the procedure for the respondent. Further, after I collect these
Page 5 of 25

ay be corr m ate and
pro a te poi n t - esti m t m oion s ( i.e.,
th e o nde nts
th exp ecta ti the resp s t h bias es or vat rib relat ed wit ho babi lity
di

s of
T es ut ion)
I
p a ryin g i nfor mp robe
o ther
m atio n leve l
5 ch arac teris t i ls. ic ue s t re sp o ns e q th e ti m at e” n an y q ue sti o an

in te
, th e m t io ns in t - es ti d of at a
” p o pi ec
f ir st m
T he w hi ch s ot e c h fo r v

a p pr
T h e te re s. te in o as ks
, w it dj o pr ia a is ce m t er fa hi le at e.
W
I re m

ta n h or in g u n de rs s l to or su g s ha es a o ns us ex a ge sti ve u p g ta ki n g h o ut

rf ac e o u g te d en ni n g j us all o w s p m ai nt ai o m ve r fr ea sy
e co it sti o ak es re q ue

n d g es p o ea nt he
r h e m o ve ra ll co sh n d
t th at li de r fo r t h pl es uc h
t m
t o o m

th e o w s o n de n e
r es p t w o n in g ef fo h e an d i t m p o n d e re s en su re th n g h
t o

r o rt.
u n de en t t o
r e s d
F ur th er
, n ha t t h e re g e
t cy
n si st en or an y p de nt
f
o f t w o-

ar dl es s q ue g pe cif ic e s hs s p of
t h v ot p ar ty v ot e ar ty
- p a o an ex t.
A s

b o h ar fo r
g sh ar e e
, th e e.
e sti o n
i ea a m pl es m it
t ak ex ac t ta n d p r rs o n de nt s

av o g s
t o a su re ha t th e ra p hi ca l de ly w va li d t de
o ro vi sp ec ifi cs b he s a
“

o id o f d
s ig n
t a tb es t ca n di d s o th es
,
Examplee of Point- E sstim a - inte ractive poll ba
ate Question: te a distri ution of pr bi lit ies or e
T would a s could pro My meth distrib utT b u s o b r probabilit h he second p s k a series o f o vide a serie h pondent to distribute h od creates a e s “100 like lp
their point iece of data f qu estion s t e s of pre - - e a set a series of va lihoods” int oelicited is a t ch represe o the respon value range a lue ranges c n e e stimate. T h n ting a value e responden e range). Fur i t t ies. n t ther, o the 9 diffe rprobabil it y n dent to cre a e distribut io n n . A non e s for the re s centered on rent bins (e a nst verbal me t al secu ri ty wh
5
Delava n distribu ti o eligible agnde and Rohw h to test the e fthods of ga th e ons on questi g e; the paper wed de hen the respo fficiency of the r (2008) ons involving has no calibratested thei r m ring probabi ondents reach g potential re t a ted outcomemethod agai n he responses.i li ty h the turns on soci a e s wi th which
P aa ge 6 of 25

n g irr d a dj us ti t an s
10
0
% an e
p m us bi n, th
r a n if
t he ge t a er m to an s w el e va n

a n d
t he r e fu ll y w er ca n n g
t cr e a s a m p ro gr ve
t o s pr o r m ln ge d u ni fo

b a it hi n ly w ut i o n. o o w id e ri b d ist q ue sti o or e ol d s m th re sh
’ ra n t he bi ns

so
t s h a t a n ge s co nt ra an d an ch or be b e n ot n ot d ca n ne w
s n s a an d a te

i nt er na w i e bi ns d s et of ll y
i a di cti re sp o n
t he n.
d en t w bi
C o ns ist r, b e o ns
f as te

he
l it ed en t w it h
t
i ns te a pl ac es m it e ca us e de ri ve d n g e ce ri e y ar b t th n g pr o

d of d e a pr e - d
f ro m a se ri e s o n
t h nt er ed bi lit ie s),
t h e ist en t ba o in co ns hi n
t he
af

p o h e all
r es h us o v e p o in t - es
(i.
e.,
t he en ti o ne o re m
t ha n
5 u m or e
I as s ra tu re
,

g e
.
r a n p r d
% of
t he
T h e e ra ll t pr
0 e th a n u m q ue sti o s of ed
t h ti et er m in

e
i s
j u o ba s, th er
.
R n bi l iti es re sh ol ds th e m h at e, i m an be o ns es
c pr o ba b

o d eli m in
R m et h ed
.
B y o n es p de nt e q us m us t ili ty o n de n e re sp y
i n ar o ba bi lit

q u al n
n t ut eo ne ri b di st a re q ue s s us t o ne n e d s ti o ca n s at es
Examplee of Probabill ity Distrib uu tio n Questt ion:

here are two respondent e stion, the t w e s ervable ch a nts’ person a o vie s
, I wouo additional is asked a s ew o ma a question w in qu e a l partiality ld ask the resources of d e As an exa ries of que s e stions are f orelated to t h e spondent a bd ata from th s tions on obs o llowed by a he question . bout th e i r in m n tention to s e
The d were five Each res p categoryd ata for this e categories p ondent was
. Everythin garticle was g of question s s in just one g is randomig athered in t s
, with each of th e stud i zed so that et wo rounds category ha v ies an d answ e ach respon d
n ts
. F irst,
be f a f ore the firs t Seco nd, wit n es the uestions
wer e . t t hin re w e categories h xa rom each in a rando n m ple, if t he qu e e th e m ovie
P aa ge 7 of 25 of stu dies
. I vin g eith er
9 w ere d j ust
o n den t se es
th

order, is randomly assign ed to a unique ques tio n in a category, and, if there is variati o n in the information level for a given ques tio n that is also randomly assigned. Both studies are a mixture of two groups of
people, the Wharton Behavioral Lab (mainly students an d staff) and respondents from around the United States with Mechanical Turk.
6
Study I concentrates on comparing my method to the methods of stated confidence
and confidence intervals. Each respondent answered one ques tion in the following five categories: calories, con cert ticket prices, gas prices, movie receipts, and unemployment rate (for ques tion details, see Appendix A). Half of the respondents were randomly assigned to create the full probably distribution and half were asked sta t e th e i r confidence and create a confidence interval. The stated confidence is recovered
with the standard polling question: “How confident are you of your answer?” with a drop down menu of four choices: very confident, somewhat confident, not
very confi dent, an d not at all confident. The confidence range is recover e d in the most efficient meth od, asking the respondent th e combination of: “I am 90% sure answer is greater than ________ " an d then “I am 90% sure the answer is less than ________ "
. Soll and Klayman (2004) demonstrates that re s p o n d e n t s reduce ove r confidence when th e y are asked to formulate an upper and lower limit in separate steps . The the o ry is also demonstrated in Vul and Pashler (2008); that article shows that when r espondents are forced to think about the same ques tion in two different ways, th e y add new information that is not apparent when they just consider the ques tion from one angle. This method of creating the confidence range exis t s between havin g the respondents directly state it and ha vin g them unwittingly reve al it.
Study II concentrates on comparing my method under different incentives; this s tudy responds to ques tion s from Becker, Degroot, and Marscak (1964), which determines that any
6
Teigen and Jorgensen (2 005 ) concludes tha t overconfidence is decreased when re s p ond e nts assign probabilities to confidence ranges , rather tha n conf idence ranges for a set probability and si mila rly Speirs - Bridge et al . ( 201 0) argues tha t it is best to ask the respondent to re- ca li br ate his own rang e (i
.e., show them their 80 % ra ng e they just created and ask th e m how wide tha t really is). The full probability distribu tion method requests the respondent to input their own probability to set ranges
, rather tha n input a ra ng e to a set probability, but tha t is not possible with the co n f i d e n ce ra ng es. I do not want to ma ke ungrounded assu mpti ons on how to translate a 60% ra ng e into an 80 % ra ng e in order to create uniform ra nges. And, wi tho u t unif orm ra ng es

, there is no method of comparing the respondents’ on their calibration, as they would al l ha v e di ffer ent size ranges.
Page 8 of 25

7 elicitation of expectations should be incentive compatible. Each respondent answere d one ques tion in each of the following five categories: calories, gas prices, and unemploymen t rates (as in Study I), and then voter turnout and Senate election results (for qu estion details, see Appendix A). The voter turnout and Senate election results are
done with the responde nts’ home states, so due to lopsided draws, are not included in this article, but th e basis for further research. Movies and concerts are dropped as categories, because
Study II was done in the early fall, where th e time frame was too short for movies and concerts are not frequent enough. Whil e the first three categories are the sa m e, all of the ques tion s are new. Half of the respondents are paid with standard flat fe e and half of the respondents are pa id a flat fee and then incentivized with an incentive compatible scoring rule. T he five respondents with the lowest weighted square erro r over all of their responses were given the bonuses. Since Study II is unbalanced
in its categories, I only use it in the full data when I am making comparison s concerning the effect of the incentives.
Between - respondent disagreement is much larger than within - respondent uncertainty; this demonstrates both th e overconfidence of the responses an d issues invol ving the accuracy of the stated po in t - estimate s. This comparison is ill ustrated in th e coefficients of variation in Table 1, where betw een - respondent disagreement is th e coefficient of variation of the stated point - estimates for a unique question and within - respondent uncertainty is the average coefficient of variation of the individual- level pro b ability distributions for that unique ques tion
. Within any given category, coefficie n t of variation is a consis tent measure of the variability of th e responses. Gurkaynak and Wolfers (2006) studied an econom ic deri vative market an d show th at between - respondent disagreement of point - es timates (submitted by experts) is less dispersed than within - forecast uncertainty, illustrated by an efficient market. One reason that the betw een - respondent disagreement is relatively larger in this article is that am not studying t he
7
I note in parenthesis in the directions “i.e., the most accurate distributions”, beca use very few respondents are going to know what a mea n squ a re error is, or wh a t type of response minimiz es it. This inability to comprehend the scoring ru le is a problem for incentives noted in Artinger (20 10) .

Page 9 of 25

8 uncertainty in an efficient market, but within individ uals, where the variance of the individua l- level pro bab ility distributions reflects individual uncertainty of the outcome. In two paragraphs I will demonstrate that these probability distributions are too narrow (i.e., overconfiden t). The second reason is because the respondents are providing point - estimate s that extend all over their distributions, not jus t the mean or median. The stan dard deviation of the point- estimates for one ques tion should be similar to the standard deviation of the most likely outco m e perceived by the respondents of that ques tion . Yet, for some categories of ques tion s the average absolute log difference between the mean an d media n of a respondent’s distribution and their point - estimate approaches 10%. Table 4 provides further insi ght into the accuracy of the stated point - estimates.
Table 1: Coefficients of Variation of Individual-
Level Probability Distributions and Coefficients of Variation of Point-
Estimates
Study I Study II Incentivized / non- Incentivized
Category Uncertainty Disagreement Uncertainty Disagreement Calories 0.221 0.373 0.
175/0.183 0.392
C o n c e r t T i c k e t s 0 .
2 2 2 0
.
3 8 4 G a s P r i c e s 0 .
0 2 6 0 .
0 1 5
0 .
0 2 7 / 0 .
0 2 6 0 .
0 2 7 M o v i e
R e c e i p t s 0 .
3 1 4 0 .
5 4 9 -
U n e m p l o y m e n t 0 .
0 1 3 0 .
0 3 9 0 .
0 1 8 / 0 .
0 1 8 0 .
0 9 5
Note: Stud y I is 12
0 respondents. Study II
is 10
3 respondents non-ince ntivized and 99 re s p ond e nts incentiviz ed. Coefficient of va ria tion for uncertainty is ( sta ndard deviation)
/(mean of di s t ri bu tion) and for disagree ment is (sta ndard devi ation)/( mean of point- esti ma tes) .
The data from St udy II sh ow

s that the alignmen t of the incentives h as a negligib le influence the ind ividual - level distributions. Th e incentive compati ble pay rewarded re spondents extra for properly cali b rated distributions. T able 1 demonstrates
that the coefficient s of variation are very similar, regardl e ss of incentives. Reg ardless of the outco me, incentives are n ot ideal for this proj ect, so it is co m f o r t i n g they have a negl igib le impact. First, if the researcher outline s the type of respo nse that maximizes t h e payout rule, she is manipulating the response, but, in thi s project, I want th e response to be w hatever the responden t thin ks is
8
While very few p oint- estima tes tha t occur on the ta ils of the probability
distributions, for ca lories, concert ti ckets, and movie recei pts the mea

n or media n is la rger th a n the point- esti ma te by a statisti cally significant amount
.
Page 10 of 25

9 the “best estimate” not what I define as th e “best estimate
”. Further, in future articles in this project where I connect th e expectations to decisions, I want th e true expectations, not expectations created to fulf i l l my scoring ru le. Second, non - payment or flat fees are standard in polling and I want the re s u l t s of this project to be relevant.
10
The calibrati on of confidence for the individ ual- level proba bi lity distribut ion is very similar to the confidence ranges (where it is comparable); bot h are slightly overconfident. Utilizing the data from St ud y I, the an s w er lies wit h in the 80% confidence range 60 % of the time and within th e middle 80 % range of the probabil ity distribution 58% of the time, a statistically insignificant difference. These are both
in line with most comp arable studies, where non - experts were asked simple knowledge ques tion s (e.g., th e ranking of college or the winning % of NBA teams).
11
The probab ility distribution forces the respondent to consider the answer for more time, but the confidence range implores them to consider the ques tion from more angles. This systematic overconfiden ce is not a pro b le m for this article, as I care more about the relationship between confidence and accuracy.
Confidence derived fro m the probability distributions correl a tes positively with accuracy; th is correlatio n is more siza ble than the correlation from the confidence ranges, an d the correlation from the stated confid ence is positive, but statistically insi gnificant. In order to make comparisons across the different types of data an d the different categories, I simplify the data. Within each ques tion I rank the responses from the least to most confident according to their sta ndard deviation (for probab i lity distribut ion), width of confidence interval, or stated confidence from 0 to 1 on its posi tion relative to all of the responses to its unique question. The smallest stan dard deviation, narrowest confidence range, or most confident answer is ranked 0, where the highest, widest , and least co n f i d e n t is ranked 1. I do the sa m e fo r the error of the point - estimate. Figure 1 illustrates the relationship between co n f i d e n c e an d accuracy in the
9
There are othe r contexts where I would want to get the optimal information for a foreca st, but in thi s arti cl e I am as interested in learning about expectations as I am in crea ting the most efficient forecasts.
10
Soll and Klay ma n ( 200 4) , Teigen and Jorgensen (2 005) , and Speirs

-Bridge et al. (2 010 ).
11
Again, in fu tu re work the expectations will be used to inform decision ma ki n g
, so I prefer a uthenti c representation of the expectations re la ti ve to forcing mo r e accu ra tely calibra ted co n f i d e n ce ra ng es.
Page 11 of 25

probab ility distribution data. Relatively low confidence corre lates with a lower than average, average rank of error, and relativel y high confidence correl a tes with a higher than average, average rank of error. Table 2 shows that the c orrelations are positive and significant within unique questions for both the pro bability distribution and confidence range, but nearly twice as sizable for the p robability distribution. The correlation is positive, but not significant, for stated
confidence.
Figure 1: Correlations Between Confidence Derived From Probability Dis tributio ns and Accurac y of Point- Estimate
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Rank of Confidence, from 0 to 1
Note: Loca l linear regres sion esti ma tes, using Epanechnikov kernel and ru l e- of- th u m b bandwi dth. Shaded area shows 95 % confidence interval.
Page 12 of 25

Table 2: Correlations Between Confidence and Accu racy of Point- Estimate
Stated Con Confidence Range Probabil
(Within Qu estion) ixed- Effect
R
2 fidence ity Distributio n
0
*
) OLS
0.006
(0.038)
3 0
- - 0.231*** (0.040) 0.053
0
.
0
(
8
)
0
3
0.150***
(0.04
0
1
*
0
)
0
5
.
0
(
*
- - 0.000
- 0.151*** (0.040) - 0.023
- 0.023
1)
*
)
F

(Within Responden t)
Page 13 of 25
* (0.052) 0.053
Note: ***, **, and * denote statisti cally si gni f ica n t coef ficients at the 1%, 5%, and 10% level, re s p ec ti ve l y . (Sta
.
0
3
*
* h respondent answered 5 ques ti ons, one in each category.
(
.
0
Confidence is not only correlated with accuracy within ques tion s
, but within respondents. Respondents adjust th eir confidenc e ift her confidence between ques tion s when she is using a confidence scale, as th e values remain constant betw
.
-
0 ity distribution, the relative confidence is uncertain, as average range size s an d stan dard deviations vary widely between ques tion s categories. This inability to consciously gauge relatively confide n ce makes it difficult for respondents to manipulate their confidence response for an y other incentives than their truthful res ponse.
The wisdom of the crowd is working; the mea n or median of the ques tion
’s responses is more accura te than a random respondent for the majority of answers and has a significantly
.
0
2
.3% of respondents’
0.070
(0.050)
0.222***
(0.05
2)
- 0.022

12 point - estimates are more accurate than the median point - estimate of th e respondents. Further, the error for the median is less than that of th e mean, on average and for the vast ma jority of ques tion s
. It is likely that the outliers are bigger issue in non -expert point - estimates than in expert point - estimates; eit h e r way, th e median is a standard point - estimate to use for aggregating point - estimates into forecast.
Table 3: Individual- Level Point- Estimates Study I Study II
Categories
5 3 Questions per Category 9.6 10
Observation s per Question 25.8 20.1 % of Individual- Level Point- Estimate Absol ute Errors <Mean Point -Estimate of Question Absolute Erro rs 36.7 % 38.8 %
Note: Point- esti ma tes are all record ed prior to the probability di stri butions. Stud y I is ra nd omize d between probability di s t ri bu ti on method and confidence questi ons, with 24
9 respondents. Study II is randomized between flat pa y and incentive compatible pay for probability di s tr i bu ti o n method, with 20
2 respondents.
Point - estimates derived from the probabil ity distributions are more accurate than the stated point- estimate. I te s t three simple point - estimates from the probability distributions: mean, me dian, an d mode . On the top of Table 4 I run a Fair-
Shiller (1989 and 1990) re gression, which includes the point - estimate an d the probability distribution’s point - estimate s, with fixed - effects by category of qu es tion . When compared directly wit h the stated point - estimates, th e mean an d the median of the probabil ity distribution are both significant at the 10% level where the stated po in t - estimate is not significant, and th e coefficients for the mean and median are substantially larger than the stated po int - estimate’s. I cannot rule out that the mode provides no information that is not i n the sta t ed po in t - estimate . By switching to OLS with no cons tant and constraining the coefficients to sum to 1, I can determine the opt i m a l weights of th e different variables if I was forced to put them together for a best estimate. A gain, co m p a r e d directly, the mean an d the median are both significant and the stated point - estimate is not. There are two
Ga l ton ( 190 7) recommends the media n for non- experts gues si ng the weight of a cow (a point-esti ma te) and this is the inspiration shown repeatedly in Surowiecki ’s Wisdom of th e Crowd

. Engleberg, et al (2 009 ) also us es the media n for GD P and inflation with experts.
12 % of Individual- Level Point- Estimate Absol 24.3 % 27.9 % ute Errors < Median Point -
Estimate of Question Absol ute Er rors
Page 14 of 25

plausible explanations fo r this finding. First, I cannot rotate the order of the sta t ed po in t - estimate and the probability distribution, thus the respondent may be ma kin g a more accurate estimate in th e probability distribution versus the stated point - estimate
, because it is her second chance to consider the ques tion
. Second, the respondents ma y be ignoring long asymmetric tails of the probability distribution when they state t heir point - estimates, to the detriment of their accuracy of their stated point - estimate.
Table 4: Comparing the Point Accu racy of the Individual- Level Expe ctations in regression: Mean Median Mode Point-Estimate - - 0.187
1.423 (0.
- - 0.018 (0.142) 0.472*** (0.147)
- - 32.2% (22.3) - 61.9%*** (0.211) - 38.1%* (0.21
1) - -
0.763*** (0.241)
2
Note: ***, **, and * denote statisti cally si gni f ica n t coef ficients at the 1%, 5%, and 10% level, re s p ec ti ve l y . (Sta ndard errors in parentheses). There are fixed - ef fec t s by ca te g o ry in top regression. If I run separa te OLS regressions with a constant on ea ch category
: the mea n has a much la rger coefficient (a nd more significance) in 3 ca tegori es, the point- esti ma te in 1 ca te g o r y , and simi la r in 1 ca te g o r y
. If I run a regression for each expectation type by itsel f, the mean has the highest R. There are tota l of 59 0 observations in the fi ve ca te g o r i es .
Weighing the point - estimates by confidence produces more accurate forecasts of the outcome level th an other standard methods. Table 3 ill ustrates how with out a prob ability distribution the most accurate consensus forecast of the outcome level is th e median of the point - estimates for any qu e s t i o n
. Table 4 shows that on an individ ual- le vel, the mean of the probab ility distribution is the most informative of its point - estimates (and more informative
-
0.310 (0.
824)
67.8%*** (
22.3)
Page 15 of 2
5
0.824 (0.
840)
-
0.314 (0.
221)
-
0.228 (
0.179)
0.307 (0.
198)
1.228***
(0.17
9)

t s h a
, i
, s e o n s p r e
A n y t h o i g w e h e f
1
1 ∑ 1
divided by th e sum of all of the inverse variances of the responses to its unique question. This method is efficient only if the respon ses are efficient, but I have already shown they are overconfident. Yet, this is
the most transpa rent and universal method available, and it does provid e a more accurate answer.
Illustrated in Table 5, the confidence - weighted fo re ca s t s have more weight and/or significance more often than th e median of the point - estimates. There is certainly fluctuation between the categories, but the confide nce - weighted mean de monstrates itself to be meaningful in relation to the standard method.
Table 5: Comparing the Point Accu racy of the Most Promising Forecasts

r t n c e
T i c r i l o
C a
1 e s
C o than the media n of the sta t ed point
-estimates). Following the literature
’s use of the inverse of the mean square error, I use the simplest and mos t transpare nt comparable method of aggregating the means of the probability distributions t
Median o Confidence f Point- - Weighted
Estimate Mean o create a consensus forecast; I weigh each mean by the inverse of their
stan dard deviation squared (i.e., the variance):
(0.286) 1.146*** (0.281)
0.730 (0.822) 0.282 (0.677)
- 0.315 (0.398) - 0.021 (0.425)
0.805** (0.319) -
0.791* (0.348)
- 1.052 Calories 1
.
(1.786) 2.097 (1.808) 0.007 (0.286) 1.186*** (0.279) kets
1
Ga s Prices
1
Movie R eceipts 1
Unemploy ment
1
0.052 (
0.245)
0.390 (
0.564)
-
0.405 (1.
133)
0.458 (
0.453)
-
0.480 (1.
553)
-
0.020 (0.24
6)
C a t e g o r y
0.948***
(0.245)
0.610 (
0.564)
1.405 (
1.133)
0.542 (
0.453)
1.480 (
1.553)
1.020*** (0.
246)
Weight Median of Point-
Estimate
Confidence - Weighted 0.059 Mean
Page 16 of 25

0.342 (0.846) - 0.633 (0.823)
- 0.499 (0.829) 0.792 (0.773)
- 1.017 (1.653) 2.055 (1.666)
0.345 (2.
463)
1.947** (
0.729)
-
0.738 (1.
515)
0.655 (2.
463)
-
0.947 (0.
729)
1.738
(1.515)
Table 6 looks at t h e R
2
Page 17 of 25
Concert Tickets
1
G a s Prices
1
. .
Movie Receipts 1
.
Unemployment 1
.
0.726
(1.064)
0.303
(0.940)
Note: ***, **, and * denote statisti cally si gni f ica n t coef
0.272
(0.861) ficients at the 1%, 5%, and 10% level, re s p ec ti ve l y . (Sta ndard errors in parentheses). There are 48 questi on tota l : 10 for calori es, 10 for gas prices, and 10 for unemployment, 9 for concert ti ckets, and 9 for movie receipts.
0.728
(0.861)
In the second part of Table 5 I explo re more efficient weightin g for the confidence - weighted forecasts. Rather than raisin g the stan da rd deviation to the exponent two (i.e., squared), I allow the exponent to fluct uate to whatever generates the smallest root mean square error fo r the predicted forecasts from the confidence - weighted forecast. There would be a different set of exponents if I take the exponents that minimize the mean square erro r of the raw confidence - weighted forecasts or what minimize s the jointly predicted mean square error of the confidence - weighted forecast and th e median of the point - es timates. Yet, regardless of which set of efficient exponents I use to create the

confidence - weighted forecasts, the weighting between the median point - estimate and the confidence - weighted forecast are similar.
2
from the regression of the answer on jus t the median of the point - estimate, on just the confidence - weighted forecast, and on bo t h the median of the point - estimate and the confidence - weighted forecast. For two of the categories, the growth in R from median of point - estimate to having both forecasts is minimal, but in the other th re e there are substantial gains (i.e., th e r e is explanatory power in the confidence - weighted forecast).

Table 6: Comparing the Point Accu racy of the Most Promising Forecasts, with R from
* Category R
2
with only Median of Point-Estimate R
2
with only Confidence -
Weighted Forecast R
2 2
for Joint Forecast
Calories 0.585 0.884 0.884 Concert Tickets 0.880 0.873 0.882 Ga
s Prices 0.308 0.347 0.362 Movie Receipts 0.131 0.129 0.534 Unemployment
0.985 0.987 0.988
Note: The conf idence- weighted foreca st is optimi zed by category as in the lower half of Ta b l e 5. The ta bl e is nearly identical rega rdless of which effici ent weighting scheme I utilize.
13
Two categories of ques ti on s have follow - up ques tion s that can be used to better calibr a te individ ual- level responses. First, for the calorie question, I ask “How closely do you follow c alories?” with options that run from “not very closely” to “regula r ly, and I know th e item in the ques tion .” There is no correlatio n between self - reported information and the rank of absolute error within the ques tion
. This confor ms to results of Table 2, where similar to stated information, stated confidence levels
on a four point scale fails to gain significance in its correlation with accur a cy. Second, for the movie ques tion
, I ask “Are you interested in seeing ‘MOVIE’?” with options that run from “D efinitely not going to watch it” to “Have alrea d y been excited about it/Definitely watch it in a thea ter.” Thus, this ques tion is c ombining two dimensions: information and partiality. There is a positive and statist ically significa n t correlation between information/intent an d the point - estimate; if a respondent likes a movie, she projects it to earn more money. A fi xed - effe ct regression of information/intent (w h i ch runs from 1 to 5) on the point- estimate yields a coefficient of 19.47 (5.33) for information/intent (i.e., respondents e stimate the movie to earn $19.5 million more for each degree of intent to see t he movie). Wit h repeated da t a
, I would be able to inflate or deflate responses to debias th e m for the information and partiality of the respondents.
13
Study I is slightl y negative and insignificant and the non- trea ted Stu d y II is slightly positi ve and insignif icant. Together the coefficient from OLS of inte nt on rank of error is 0.00 1 (0.022 ).
Page 18 of 25

There are several reasons why a graphical, interactive interf ace, utilized in a web- based setting, can collect info r m ation that is difficult to attain in standard telephone or in - person settings. First, information can be revealed, rather than stated, which makes it much ha rder for the respondents to manipulate the an s w er to fulfill incenti ves other than their best estimate. Second, the interface makes the confidence revelati on part of th e main ques tion , wherea s asking a respondent to state confidence after she supplied the ma in point - estimate, may not seem as serious to the respondent or operate under different inc entives. Third, Ar iely, et al (2003) shows that people can incorpor ate new information into their understanding of th e world; the problem is that they sometimes appear arbitrary, because they are no t sure where with what baseline they should start. A graphical interface can prov ide some subtle baselines for the respondent without pro viding too much anchoring. For example, in this article, the ques tion reg a rding calories of fast food includes pictures and descriptions of a few different foo ds. Similarly, the presentation of the ques tion s themselves are providing subtle information about point - estimates and probabi lit y distribut ions that teach people how to provide inform ation th e y have, but do not know how to elucidate.
Polls and predication markets are just two methods for gathering individual- level information and aggregating it into forecasts; both methods ha ve benefits and nega tives, and my method is one attempt to harness the better aspects of bo t h of them. One of the key problems with polls is the reluctance of researchers to ask th e ques tion they are trying to answer, wh ich is usually the ques tion that gathers the most relevant information from the respondent. The graphical
and interactive nature of this method allows me to ask ques tion s that do not gather consistent and meaning ful responses in a telephone or in - person setting. Polls’ aggregation does not ta ke advantage of disparities in information of the respondent and prediction markets’ aggregation does not record massive am oun t s of information; participants in predictio n markets are creating constant subjective pro bab ilities, but only the aggregated price and a few bids and asks are recorded for r esearchers. Further, pred iction markets are
Page 19 of 25

susceptible to manipulation.
14
With my method I can capture all of the informatio n and aggregate it, transparently, with confidence. Further, I can not only creat e accurate forecasts of the level of the outcome, but also, I can explore full p robab ili ty distributions on both the individ ual and aggregate level.
The full method proves itself meaning ful in absolute terms and trumps simpler confidence ranges in information, but it
does take up more time, which can be important in polling. The mean (medi an) length of time from start to finish for the five ques tion s with the full method is 13
.1 (12.0) minutes, while the confidence range variation is 7.6 (6.7) minutes. Further,
while it is not nearly as significant or meaningful as the full probab ility distribution, the confidence range responses provide a small, but statistically
significant positive cor re lation between confidence and accuracy that can be utili zed for the cr e a t i o n of certainty - weighted forecasts. The goal of this article is to provide val idation of my method versus the best and most practical of the other possible optio n s on information and utility, but if time/cost is an issue, there will definitely be scenarios where the confidence range is the right opt i o n .
Turning to decision making, there is consensus in the literature on the im portance of expectations in decision making. Manski (2004) demonstrates that playing
a simple econ om ic game, a subject with one of three different expectations an d on e of two different utilit y functions will make the sa m e move (i.e., revealed behavior) in four out of six possible scenarios. He outlines
many empirical example s of subjects having faulty expectations, but emphasizes the gap in the literature in understanding expectations sepa rated from utility.
The follow - up question for the gas qu e s t i o n hints at th e usefulness of my method in decoupling expectation from util ity in revealed decisions. The main ques tion asks the respondent to imagine that sh e is driving down a major highway and she notes th e price of gas at last fe w consecutive stations. She is running low and can hold out only long enough to stop at one of the next three stat ions; she is asked to create a probability distribution of the lo w e s t price
14
There is evidence tha t the national popu la r vote prediction ma rkets ma y su ff er from ma n i pu la t i o n by people moti va ted to ga i n pu blicity for their chos en can d i da t e . The aggregation is over wi llingness to invest money, not confidence!
Page 20 of 25

of gas among these next three stat ions . The follow - up question asks what price would induce the driver to stop at th e firs t station sh e sees
, rather than keep going and try one of the following two stations. The median resp onse was at th e 30 % point of the probab ility distribution of what th e y expect the lowest price of the next th re e gas stations to be. That means that th e median driver would stop where they believe that there is only a 30% chance that one of the next tw o stations would be less. Just 6% of respondents said th e y would stop at station in the 80
% percentile or higher. Most importantly, there is a statistically sig n ificant positive correlation between the point - estimate expectation and price in which the driver would stop for gas. Thus, th e higher the driver expects the lowest price to be, the higher price the drive r will stop and pay. Further, confidence demonstrates a mean ingful ro le in the decision making; if two drive rs have the sa m e point - estimate, the driver with the larger standard deviation (i.e., less confidence) will stop
at a gas station with a higher price. Expectation matt ers, but so does confidence!
Page 21 of 25

Arie ly, Dan, George Lowenstein, and Drazen Prelec. 2003. “Coherent Arbitrariness.” Qu
arte rly Journal of Economi cs, 118(1):73 - 105.
Artinger, Floriean, Filip pos Exadaktylos, Hannes Koppel, and Lauri Sддksvuaori. 2010. “Applying Quadratic S coring Rule Transparently in Multiple Choice Settings: A Note.” Jena Economic Res earch Pape rs 2010 - 0 21.
Avery, Chri stopher, Judith Chevalier, and Richard Zeckhauser. 2009. “The “CAPS” Prediction
System and Stock Marke t Returns.”Working paper, Harvar d University .
Barber and Odean. 2001. “Boys Will Be Boys: Gender, Overconfidence, an d Common Stock Investment ,” Quarterly Journal of Economics, 116(1):261 -292.
Bates J., and Grange C. 1969. “The Combination of Forecasts.” Operational Rese
arch Quarterly , 20:45 1 - 468.
Becker, G.M., Degroth, M.H., Marashak, J. 1964. “Measuring Utility by a Singl e - Response Sequential Method.” Behavioral Science 9(2):226 -232.
Chen, Kay-Yut, Leslie Fine, Bernardo Huberman. 2003. “Predicting the Fut u re .
” Information Systems Fr on tiers, 5(1):47 -61.
Clemen, Robert. 1989. “C ombining Forecasts: A Review and Annotated Bibliography.” Internation
al Journal of Fo recasting . 5:55 9 - 583.
Clemen, Robert and Robert Winkle r. 1993. “Aggregating Poin t Estimate s: A Flexible Modeling Approa ch.” Management Science, 39(4):501 - 515.
Delavande, A. and Rohwedder, S. 2008, “Eliciti n g Subjective Probabilit i e s in Internet Surveys.” Public Opinion Quart erly , 72(5):886 -891.
Dominitz, Jeff and Charles Manski.. 2006. “Measuring Pension - benefit Expectations Probabil istic a lly.” Labour , 20:201 - 236.
Engleberg, Joseph, Charles Manksi, and Jared Williams. 2009. “Comparing the Point Pre diction and Subjective Probability Distributions of Professional Forecasters.” Journal of
Busin ess and Economics Statistics, 27(1):30- 41.
Fair, Ray, and Robert Shi ller. 1989. “The Informational Content of ex-
Ante Forecasts.” Review of Economics an d Statistics 71(2):325 - 31.
-----------
. 1990. “Comparing Information in Forecasts From Econometric Models.” Ameri
can Economic Review 80(3):375 - 89.
Gurkaynak, Refet and Justin Wolfers. 2006. “Macroeconomic Derivatives: An Initial A nalysis of Market-
Based Macro Forecasts, Uncertainty, and Risk.” CEPR Discussion Paper No. 5466.
Kahnema n & Tve rsky. 1979. “Prospect Theory: An Analysis of Decision Under Risk.
” Econo met rica , 47:263 - 291.
-----------

. 1992. “Advances in Prospect Theory: Cumulative Representation of Uncertainty
.” Journal of Risk and Unce rt ainty , 5(4):297 - 323.
Kaufman - Scarborough, Carol, Maureen Morrin, and Eric T. Bradlow. 2010. “Im p rov in g the Crystal Ball: Harnessing Consumer Input to Create Retail Prediction Marke ts.” Journal of Rese ar ch in Interactive Marketing, 4(1):30 - 45.
Page 22 of 25

Klayman, Joshua, Jack Soll, Cladia Gonzalez - Vale jo, and Sema Barlas. 1999. “Overconfidence: It Depends on How, What, and Whom Y ou Ask.” Organiz ation al Behavior and Human De cision Process. 79:216- 247.
Kuklinski, James, Paul Quirk, Jennifer Jerit, David Schwide r, and Robert Rich. 2000. “Misinform ation and the Currency of Democratic Citizenship.” The Journal of Politics
, 62(3) 790 - 816.
Manski, Charles. 2004. “Measuring Expectations.” Economet rica , 72(5):132 9 -
1376. Pennock and Reeves. 2007. “How and When to Lis t e n to th e Crowd.” http://www.overcom ingbias.com/2007/02/how_and_when_to.html. Rothschild, David. 2009. “Forecasting Ele ctions, Comparing Prediction Markets, Polls, and
Their Biases.” Public Opinion Quart er
ly 73(5):895-16. Rothschild, David and Justin Wolfers. 2011. “Forecasting Elections:
Voter Intentions versus
Expectations.” Working paper, Univ ersity of Penn sylvania, Available at: http://a ssets.w harton.upenn.edu/~rot hscdm/Roth schildExpectations.pdf.
Smith, Jeremy, and Kenneth Frank Wallis. 2009. “A Simple Explanation of t he Forecast Combination Puzzle.” Oxford Bulletin of Economics and Statistics
, 71(3):3331 - 355.
Soll, JB and Klayman J. 2004. “Overconfidence in Interval Es timates.” Journal of Experiment al Psychology Learning Memory and Cognition
, 30(20):299 - 314.
Sonnemans , Joep and Th e o Offerman. 2001. “Is th e Quadratic Scoring Rule Really Incentive Compatible?” Working paper, CREE
D, Univers ity of Amsterdam.
Speirs -
Bridge, Andrew, Fiona Fidler, Marissa McBride, Louisa Flander, Geoff Cumming, and
Mark Burgman. 2010. “Reduc in g Overconfiden ce in the Inte rval Judgm ents of Experts.” Risk Analysis, 30(3):512 -523.
Stock, James and Mark Watson. 20 04
. “Combination Forecasts of Output Growth in a Seven -
Country Data Set.” Journal of Fore cas t ing , 23:405- 430.
Surow iecki, James. 2004. Wisdom of the Crowds: Why the Many are Smarter Than the Few and Ho w Collective Wi sdom Shapes Business, e co n omics, Soci e t ies, and Nati
ons, Little, Brown.
Teigen, KH and M. Jorgensen. 2005. “When 90
% Confidence Intervals are 50% Certain: On the Credibi lity of Credible Intervals.” Applied Cognitive Psychology, 19:455- 475.
Vul, Edward and Harold Pashler. 2008. “Measuring the crowd within: Pro babil istic representations within in dividua ls .” Psychologica l Science, 19(7):

645– 647 .
Page 23 of 25

Sa mm ple Questtions can bee found on mmy websitee
: www.Pr ee di c t W i se . cc om
Study I && II (Caloriee s Count): HH ere is a full exa m ple. It starts with tthe po in t - es ttimate ques tt ion:
Calories question Co unt Pro b is centered b ability Qu e on 55 0 calo re stio n: Since ries: e I placed 55 00 calories at my estimat ee the proba bbi lity

F o n t u e
C o d o n s p r e e s o r i
C a l i f t o t e n o l l

: n i o s t
.
c e i o l l
A h s e
Q u p u w
o c h a l l

l e s e t t a t r
, t c t a s s n u e e m d o w n p o o d r

t h n g i i r c f o e t o h e h e t r r e a
B e l o w

i h h t i n e d s t s u s s n o n i t e s q u u r o f f r

: y u d ce to t h e note d upc e or con cert
f oOn ly (
Con sele cted
u tabl e sh o offic
Stu dy
I O
2 t icke ts omi ng coo ws t he low e
Stu bHu b, w o r a
r

b efor e t he f ull list n o frice
s):
Wh at day
o p ri ce
(incl u d hich ever
is
l m sele ctio rom
e st po ssibl e pw and ome rt
Tic ket
P r d con cert
t heo ncer t f

e stim ate
f o
T he top d be pur y perf orm erst
ta ble ' s t op selli n g tub
Hub
' d ing
all
ch ar coul bes t the sho w?
T o f S low er, that
c f c onc erts
b yis you r

and
1
d m e l ist. t
t ou cket s, f rom
e d ays poss ible
p r i o ne ra n e hig hlig hts
o g con cert sam o r th e l owe s t s ed
10 d s on the ges)
for
2
tic rcha

fro m the i ce of m ly t om d o m urs.
T he
bot t e ithe r th e b o day
o x
s ta gas
a t t h e hig
Stu dy
I
& nex t 3 hw ay
&
II
(G as
P r a

fro m
8/4
/is
y our gas
s tati on s a re
b est
d esc rib ed hw ay d
Wh at e hig pri ces
a ric es):
e e
3 pre vio usy
.
All tio ns
o n
th

nr the
LO
WE
S a
as su m m ajo r
Eas ter ble
?
Th e be l m he bel ow
t a esti on a me sti ma te
fo r t
10
o n d t he
qu
/ 20 in
t s an

g t h h ow s th e pri og as am on
9/1
6/2
010
f c ow n the
s h u con tin ue
d
( of g lo w tab le hig hw ay.
(S
T
p rice es t hat
y ou n

Stu dy h e ce of m sa m o r t he e y
II.) i t t a s f o n g i v i d r e i c
P r
G a s

m p
3 o t u p u e
Q o r f p u
o w l l
F o s e n
?

d h a o u y f s a
, s
I o n i t t a s e o r
I m
: i o n t s

o f e h u g n o s g a e t u b o e a c i r p t a w h

y o o e y e u c f u l e n d i l d e u e r w h s g a u h w o

t e l b t a o r f h t e a t m o p t s o t u o o m c t

n e x t ars
(i.
DE
SCR
IPTI o ws t he ase.
On ly
(Mo vie of doll ions
rel ease
)? t able
s h o rele
Stu dy
I
O mill do me
R ecei pts)
: . e. wh at will
I
ON

mes tic n wid e rele ase
0
0ur best
es tim o ss do wid e- rele am ate
Nat io n f or the last
3
MO
VIE
g ro
OVI
E
.
OF
M
Om esti c g ross
f
Wh at i s you l

gh thei rs for
we e
OF
R
ELE
AS
E h rou
MO
VI E e ken d o f w wee k g ros s h its
4th a se mo vies
t h4 thro ug h e on
DA
TE
O for
th e 4 call y

w r 4 th wee ke nE id E
.
The
b elo in
d e w
n d of l o l o o
F t s i p c e e
R e e i v
M o

e t n i u u y o e e s r t e
A r
: n i o t t s u e
Q u p
w

M
M g i n
V
I
O e e s i n d d
E
?
pl o be lo
U ne m
I
&
St u d y w ta b pe

p lo y m o y m
U ne m pl w s th e nt
R a e e sh o
R at e bl m en t m p o y n e
(
U
II ri o ds
.
&

lo w y m be n ot ed ha t is en t
R at e
r ra te
):
W te s a en t ra pl o y m
U ne m at e h e
i n
t he
st

se as o na l e sti d o m or a
r an m at e
f or
l he s l in
(u se
t be st
e w y o ur d fs e
a dj us te

y o u t w o sh o g us t / t at e in e
A u en ds
. th st tr ch os en m ly lo w
t id er be

va nt
T he re le an s w er
)?
T w be r ur
Se pt e m a
f e w
/
Unempl oo yment Rat e e Follow - Question: H ow familia rr are you witt up H h the state iin th e quest iion?
P age 25 of 25