CAPTCHA,
THE ESP GAME,
AND OTHER STUFF
LUIS VON AHN
CARNEGIE MELLON UNIVERSITY
CAPTCHA
A PROGRAM THAT CAN TELL WHETHER
ITS USER IS A HUMAN OR A COMPUTER
CAPTCHA
A PROGRAM THAT CAN GENERATE
AND GRADE TESTS THAT:
A. MOST HUMANS CAN PASS
B. CURRENT COMPUTER
PROGRAMS CANNOT PASS
EXAMPLE
PICKS RANDOM
STRING OF LETTERS
oamg
RENDERS IT INTO A
DISTORTED IMAGE
…AND GENERATES A TEST:
TYPE THE CHARACTERS THAT
APPEAR IN THE IMAGE
P STANDS FOR PUBLIC
ALL CODE AND DATA USED BY A CAPTCHA
SHOULD BE PUBLICLY AVAILABLE
COMPLETELY AUTOMATED
PUBLIC TURING TEST TO TELL
COMPUTERS AND HUMANS APART
UTILIZING THE POWER OF
HUMAN CYCLES
CAPTCHA
APPLICATIONS
EXAMPLES
DIRTY HACKS
THE ESP GAME
OTHER STUFF
APPLICATIONS OF CAPTCHAS
QUICKVOTE
Which is the best Computer Science
Grad School in the US?
Berkeley
MIT
CMU
Princeton
Cornell
Stanford
vote
(FROM WWW.SLASHDOT.ORG)
APPLICATIONS
FREE E-MAIL SERVICES
DATA COLLECTION
WORMS AND SPAM
PREVENTING DICTIONARY ATTACKS
(PINKAS AND SANDER ‘02)
APPLICATIONS
FREE E-MAIL SERVICES
DATA COLLECTION
WORMS AND SPAM
PREVENTING DICTIONARY ATTACKS
(PINKAS AND SANDER ‘02)
APPLICATIONS
FREE E-MAIL SERVICES
DATA COLLECTION
WORMS AND SPAM
PREVENTING DICTIONARY ATTACKS
(PINKAS AND SANDER ‘02)
APPLICATIONS
FREE E-MAIL SERVICES
DATA COLLECTION
WORMS AND SPAM
PREVENTING DICTIONARY ATTACKS
(PINKAS AND SANDER ‘02)
EXAMPLES OF CAPTCHAS
PIX
WHAT ARE THESE PICTURES OF?
PIX
WHAT ARE THESE PICTURES OF?
THE IMAGES NEED TO BE
RANDOMLY DISTORTED
PIX
WHAT ARE THESE PICTURES OF?
THE IMAGES NEED TO BE
RANDOMLY DISTORTED
PIX
WHAT ARE THESE PICTURES OF?
THE IMAGES NEED TO BE
RANDOMLY DISTORTED
WHAT ARE THESE PICTURES OF?
BONGO
VISUAL ANALOGY
PROBLEMS
(BONGARD 1951)
SOUND-BASED CAPTCHA
HUMANS ARE BETTER THAN COMPUTERS
AT UNDERSTANDING SPOKEN LANGUAGE
OPEN PROBLEM
CREATE A CAPTCHA BASED ON
LANGUAGE UNDERSTANDING
DIRTY HACKS
CAPTCHA SWEAT SHOPS
SPAM COMPANIES HIRE HUMANS TO
SOLVE CAPTCHAS ALL DAY LONG
$2.50 PER HOUR FOR EACH HUMAN
720 CAPTCHAS PER HOUR PER HUMAN
1/3 CENT PER ACCOUNT
Free Nude Photos
> NEXT
Free Nude Photos
Type the word in the box if you want
to see the next picture
UTILIZING THE POWER OF
HUMAN CYCLES
CAPTCHA
APPLICATIONS
EXAMPLES
DIRTY HACKS
THE ESP GAME
OTHER STUFF
LABELING IMAGES WITH WORDS
MARTHA STEWART
FLOWERS
SUPER EVIL
STILL AN OPEN PROBLEM
IMAGE SEARCH ON THE WEB
USES FILENAMES
AND HTML TEXT
ACCESSIBILITY
LESS THAN 10% OF THE WEB IS
ACCESSIBLE TO THE VISUALLY IMPAIRED
REASON: MOST IMAGES DON’T HAVE
PROPER CAPTIONS
DESIDERATA
A METHOD THAT CAN LABEL
ALL IMAGES ON THE WEB
FAST AND CHEAP
USING HUMANS CLEVERLY
THE ESP GAME COULD LABEL ALL
IMAGES ON THE WEB IN 30 DAYS!
THE ESP GAME
TWO-PLAYER ONLINE GAME
PARTNERS DON’T KNOW EACH OTHER
AND CAN’T COMMUNICATE
OBJECT OF THE GAME:
TYPE THE SAME WORD
THE ONLY THING IN COMMON IS
AN IMAGE
THE ESP GAME
PLAYER 1
PLAYER 2
GUESSING: CAR
GUESSING: BOY
GUESSING: HAT
GUESSING: CAR
GUESSING: KID
SUCCESS!
YOU AGREE ON CAR
SUCCESS!
YOU AGREE ON CAR
© 2004 Carnegie Mellon University, all rights reserved. Patent Pending.
THE ESP GAME IS FUN
4.1 MILLION LABELS WITH 23,000 PLAYERS
THERE ARE MANY PEOPLE THAT PLAY
OVER 20 HOURS A WEEK
LABELING THE ENTIRE WEB
5000 PEOPLE PLAYING SIMULTANEOUSLY CAN
LABEL ALL IMAGES ON GOOGLE IN 30 DAYS!
INDIVIDUAL GAMES IN YAHOO! AND MSN
AVERAGE OVER 5,000 PLAYERS AT A TIME
A FEW MILLION LABELS
CAN IMPROVE IMAGE SEARCH
CAN BE USED TO IMPROVE
COMPUTER VISION
SINGLE PLAYER GAME
A SINGLE PERSON CAN PLAY WITH PRERECORDED ACTIONS AS THEIR PARTNER
(0:12) CAR
(0:15) HAT
(0:21) KID
WE EMULATE PARTNER
BY PLAYING PRERECORDED MOVES
(0:08) BOY
(0:23) CAR
WHEN 2 PEOPLE PLAY, WE
RECORD EVERY ACTION
WITH TIMING INFORMATION
NOTICE THAT THIS
DOESN’T STOP THE
LABELING PROCESS!
WHAT ABOUT CHEATING?
IF A PAIR PLAYS TOO FAST, WE DON’T
RECORD THE WORDS THEY AGREE ON
WHAT ABOUT CHEATING?
WE GIVE PLAYERS TEST IMAGES FOR WHICH
WE KNOW ALL THE COMMON LABELS:
WE ONLY STORE A PLAYER’S GUESSES
IF THEY SUCCESSFULLY LABEL THE
TEST IMAGES
SEARCH RESULTS
SAMPLE LABELS
BEACH
CHAIRS
SEA
PEOPLE
MAN
WOMAN
PLANT
OCEAN
TALKING
WATER
PORCH
SAMPLE LABELS
SADDAM
MR. WILSON
MAN
FACE
COMING SOON:
MEET YOUR SOULMOUSTACHE
MATE
THROUGH THE GAME!
BUSH
PRESIDENT
DUMB
YUCK
WHY DO PEOPLE LIKE
THE ESP GAME?
ESP GAME GIVES ITS PLAYERS A
“ THE
WEIRD AND BEAUTIFUL SENSE OF
ANONYMOUS INTIMACY. ON THE ONE HAND,
YOU HAVE NO IDEA WHO YOUR PARTNER IS.
ON THE OTHER HAND, THE TWO OF YOU
ARE BRINGING YOUR MINDS TOGETHER IN
A WAY THAT LOVERS WOULD ENVY.
”
“
”
SO MUCH FUN TRYNG TO GUESS
“ IT’S
WHAT OTHERS THINK. YOU HAVE TO
STRANGELY ADDICTIVE
STEP OUTSIDE OF YOURSELF TO
MATCH
”
“ IT’S FAST-PACED ”
“ HELPS ME LEARN ENGLISH ”
BLOGS
OTHER CROWNING ACHEIVEMENTS,
“ INI HAVEN'T
PLAYED IN AT LEAST A WEEK
(PROBABLY LONGER), AND I AM STILL ON THE
TOP TEN LIST OF PLAYERS ON THE ESP GAME.
NUMBER EIGHT IF I REMEMBER CORRECTLY.
GOD, MY LIFE IS SAD.
”
“
THERE ARE MANY FASCINATING THINGS
ABOUT THE ESP GAME:
HOW IT USES A GAME STRUCTURE TO
BUILD AN INDEX OF IMAGES ON THE WEB
WHAT WORDS BECOME TABOO FROM
FREQUENT USE
HOW INTERESTING SOME OF THE IMAGES
PULLED RANDOMLY FROM THE WEB ARE
BUT BY FAR THE MOST INTRIGUING ASPECT
OF THE GAME IS HOW OFTEN YOUR RANDOM,
UNKNOWN PARTNER IS A COMPLETE IDIOT.
”
OTHER GAMES
LOCATING OBJECTS IN IMAGES
THE ESP GAME TELLS US IF AN IMAGE
CONTAINS A SPECIFIC OBJECT, BUT DOESN’T
SAY WHERE IN THE IMAGE THE OBJECT IS
SUCH INFORMATION WOULD BE EXTREMELY
USEFUL FOR COMPUTER VISION RESEARCH
PAINTBALL GAME
PLAYERS SHOOT AT OBJECTS ON THE IMAGE
SHOOT THE:
CAR
WE GIVE POINTS AND CHECK ACCURACY
BY GIVING PLAYERS IMAGES FOR WHICH
WE ALREADY KNOW WHERE THE OBJECT IS
PAINTBALL GAME
X% OF IMAGES
(100-X)% OF IMAGES
DON’T
KNOW
BALL
REVEALING IMAGES
GUESSER
REVEALER
CAR
BRUSH
CAR
GUESS
BRUSH
CAR
PARTNER’S
GUESS
MEDIUM-TERM GOAL
USE THE ESP GAME, ALONG WITH THE PAINTBALL GAME AND THE REVEALING GAME TO
CREATE THE MOST ACCURATE COMPUTER
VISION ALGORITHMS TO RECOGNIZE A WIDE
VARIETY OF OBJECT CATEGORIES
X OR NOT
CAN BE USED TO CLASSIFY ALL IMAGES ON
THE WEB AS HAVING PROPERTY X OR NOT
WE GIVE PEOPLE
40% OF IMAGES WITH PROPERTY X
40% OF IMAGES WITHOUT PROPERTY X
20% OF UNKNOWN IMAGES
IF PEOPLE LIKE IMAGES WITH PROPERTY X,
WE CAN OFFER AN INCENTIVE: THE MORE
ACCURATE A PLAYER IS, THE MORE IMAGES
WITH PROPERTY X THEY GET
THE POWER OF HUMAN CYCLES
CAPTCHA
OPEN
PROBLEM
GAME
OUR WORK ON CAPTCHA HAS
APPEARED IN OVER 100
NEWSPAPERS AROUND THE WORLD
THE ESP GAME HAS APPEARED
IN OVER 50 NEWSPAPERS
AROUND THE WORLD
Name That Image
Computers excel at sifting information, but they have trouble distinguishing a picture
of a tree from one of a turtle.
So researchers at Carnegie Mellon University knew they needed human volunteers
to successfully label millions of unmarked images on the Web. Luis von Ahn, a
graduate student in computer science, had the inspired idea of turning the laborious
process into a two-player contest called the ESP Game (espgame.org).
Each time you play, you are randomly paired with another anonymous player. You
can't communicate with your partner, although you both see the same image. The
goal is to guess the descriptive word your partner is typing for the image. Once you
both type the same word, you see a new image.
Since October the game has attracted 15,000 registered players, and they have
created more than 1.5 million labels (a label is a single descriptive word; most
photos require multiple labels). The site draws from a database of 200,000 Web
images; another 200,000 are to be added soon. Mr. von Ahn said that properly
labeling Web images would allow for more efficient image searching, improve the
screen readers used by the visually impaired and help users block inappropriate
images.
WWW.CAPTCHA.NET
WWW.ESPGAME.ORG
SLIDE
DESIGN
WARDROBE
PROVIDED
MORE
SPECIAL
SPECIAL
THANKS
THANKS
TOBY
LUIS
VON
AHN
THE
GAP
ADITYA
UDI
MANBER
AKELLA
JOSH
MONI
BENALOH
NAOR
CAPTCHA
CREW
HAIRSTYLES
LENORE
LENORE RAMM
BLUM
LUISROSENBERG
VONBLUM
AHN
ARNAUDCHUCK
SOME-FRENCH-LAST-NAME
MANUEL
MANUEL
BLUM
RONI
ANDREW
ROSENFELD
BORTZ
NICHOLAS
HOPPER
THIS
WORKSTORK
HAS
BEEN
SHUCHI
DAVID
CHAWLA
JOHN
LANGFORD
ANDLATANYA
WILL
BE
PARTIALLY
LAURA
DABBISH
SWEENEY
SUPPORTED
SHINGO
BILL UCHIHASHI
GATESBY
THE
ESP GAME
CREW
MANUEL
BLUM
BRIGHTEN
MANUELA
GODFREY
VELOSO
VON
AHN
THE LUIS
ALADDIN
CENTER
NICK
HOPPER
LAURA DABBISH
MICROSOFT
RESEARCH
SCOTT
HUDSON
MY MOM
TAKEO
KANADE
SPEAKER’S
ENTERTAINMENT
YOUR
TAX
DOLLARS
STEVEN
KATZ
PROVIDED
BY
JOHN
LANGFORD
HOMESTARRUNNER.COM
JITENDRA MALIK