<meta name="description" content="Researchers are using the tools of economics to help
businesses better predict quarterly revenues, expected demand for a product and other
future events.">
<meta name="keywords" content=" Behaviorally Robust Aggregation of Information in
Networks; brain; experimental economics; sales forecast; future pricing; harvest
knowledge; information harvesting; prediction.">
Paragraph for news index:
Researchers are using the tools of economics to help businesses better predict quarterly
revenues, expected demand for a product and other future events.
PULL QUOTE: Researchers are testing the service with a customer. It's also attracting
interest from major corporate players in communications, software and energy
exploration.
LINKS:
>> Related research: BRAIN /research/ssrc/competitive/brain
>> Research paper: Forecasting uncertain events with small groups
/research/idl/papers/future/index.html
>> Information Dynamics Lab
Predicting the future -- with games
How researchers are using the lab to forecast business futures
By Simon Firth, July 2006
Predicting the future isn't just the province of fortune tellers or crystal-ball gazers. A team
of researchers at HP Labs is doing just that – with the tools of economics.
Labs’ most advanced project in this area is called BRAIN.
“That stands for Behaviorally Robust Aggregation of Information in Networks,” says
researcher Leslie Fine, an experimental economist at HP Labs and part of a team that is
using laboratory experiments with human subjects to evaluate economic propositions.
The program, founded in 1994 by researcher Kay-Yut Chen, has been nationally
recognized in applying this kind of research to the problems faced by corporations.
Fine, Chen and HP Senior Fellow Bernardo Huberman developed BRAIN to tackle a
problem that troubles the corporate world – how to extract accurate information about
future events (such as predicted quarterly revenues or expected demand for a product)
from small teams of knowledgeable workers.
Market mechanisms
In the wider world the best mechanism for predicting the future is to run a market.
When large numbers of people speculate on stocks or commodities, for example, their
prices reflect everything that is currently known about how they might perform in the
future.
The University of Iowa’s famous Iowa Electronics Market has shown that large groups of
people can also predict electoral results and the future direction of major economic
indicators with almost unrivaled accuracy.
Markets don't work well with small groups of people, though. It’s not that such groups
aren’t knowledgeable, but outcomes from small markets can easily be skewed thanks to
the low number of players offering bids, or through deliberate manipulation.
Working with small groups
But without markets, the decisions made by small groups are equally open to
manipulation. Take an IT procurement team trying to predict the future price of a key
component. Typically, Fine explains, they’ll do this by “sitting around a room once a
month and yelling at each other until a number comes out.”
As a result, she says, “he who yells loudest wins.” Quieter but more knowledgeable team
members -- or team members too intimidated to contradict the boss -- don’t get heard and
the predictions suffer as a result.
BRAIN gets around these problems in three ways: It makes the prediction process
anonymous; it asks people to back up their predictions with real money -- essentially, to
make bets on where they think the numbers will land; and it makes the whole exercise a
game.
Determining risk tolerance
If you are asking people to make bets, you need to know their appetite for risk in order to
makes sense of the bets they place.
So BRAIN first has the team play a market game which requires them to put stakes on
various imaginary outcomes. From that, researchers calculate a behavioral coefficient for
each player that summarizes their risk attitude and their predictive power.
Then players are asked to make bets on the real question in play (such as 'what will be the
price of computer memory chips in one month’s time?'). According to how accurate their
bets turn out to be, the players receive a payoff -- typically in tens rather than hundred of
dollars.
Meanwhile, says Fine, “we weight the bets by the coefficients we have for each player,
aggregate them, and pass back the prediction.”
BRAIN beats traditional methods
Such a mechanism, it turns out, works extremely well. Because it gets people to divulge
what they know, BRAIN’s predictions consistently beat forecasts generated in more
traditional ways. It helps that people generally enjoy playing BRAIN, too. It means
they'll go back and use the tool again.
The work that went into BRAIN also suggests ways to mitigate against other business
problems -- such as sandbagging -- where individuals underestimate their future
performance so as not to lose out on their pre-agreed bonuses.
In the face of that, says Chen, a manager could simply pay a bonus to people who made
accurate forecasts of their performance. But offer such a deal to some people, he adds,
“and you'll get a very accurate forecast, but they won't sell anything!”
Getting accurate sales predictions
To solve this, researchers have proposed presenting salespeople with the option of
receiving more of their pay as fixed compensation and less as bonus, or vice versa.
“So then you say to the salesperson,” says Huberman, “How would you like to get paid?”
“If he knows he’s going to sell everything,” Huberman suggests, “he’ll want all bonus. If
he knows he won’t sell anything, he’ll want to bring a check home. So by the way he
chooses, you can figure a probability that he will actually sell a certain number of
pieces.”
As a result, the boss knows the true sales outlook and the salesperson is still well
compensated for being good at his or her job.
Tests in the lab
Huberman and Chen were able to successfully test this idea in their experimental
economics lab, where people played a series of games that modeled such a sales situation.
It’s unusual for a corporation to have such a research facility, but very useful.
Several groups within HP have already engaged Chen to design games to help them
reformulate the contracts they sign to realign them in the interests of both signatory
parties.
“Contracts are a bit like the rules of a game,” says Chen. “If you change the rules of the
game, people will play differently. And once you understand how, you can write better
contracts.”
Customer testing
BRAIN has shown that economic games have predictive power outside the lab, says Fine,
who ran a successful field test with a financing forecast team in HP Services.
“At the beginning of each month, we would predict operating profits and revenues for the
end of that month,” she says. “And we blew their numbers out of the water.”
Fine is running a pilot of BRAIN with its first external customer. The service is also
attracting interest from major corporate players in communications, software and energy
exploration.
BRAIN can be useful, says Fine, “any place you have a small team that has some kind of
specialized knowledge that you want to tap, and where you can take what you want to
know and rephrase it as a bet.”
Shorter time scales work best. “No one wants to place bets on things they’re going to find
out about in ten years,” Fine notes. It’s also important, she says, that “there’s no ability
to lose money here. So we’re not gambling.”
Other prediction mechanisms
"This effort," says Huberman, "is really about harvesting implicit knowledge in
organizations." He has pioneered mechanisms for identifying the best set of people to
poll for such knowledge -- groups that may not always echo formal organization charts.
Researchers have developed other tools for prediction as well. One technique is aimed at
helping risk-averse business managers make decisions and another helps businesses
better predict demand. (See 'Afraid to make risky business decisions?' news/2005/octdec/decision_insurance.html and 'Riding the peaks' /news/2005/julsep/reservations.html.)
Tools like these are growing increasingly important, Huberman argues, as we're faced
with burgeoning amounts of information.
“It’s very hard to attend to what is important and what is salient or not," he says. "This
offers us a way to know.”
---------------Simon Firth is a writer and television producer living in Silicon Valley.