USACO: Directions For The Next Decade
Rob Kolstad, Ph.D. – USACO Head Coach
ABSTRACT
The last decade of USACO and worldwide programming competitions saw huge growth in
services and participation. While USACO internet contests (via email) commenced in 1996 with
just a few countries participating, more recent widespread internet penetration has enabled online
competitions around the world including the popular USACO contests that now garner a thousand
or more participants from a total of more than 65 countries on a monthly basis.
The USACO effort appears to have reached a steady state for its current thrust toward
algorithmics contests, surprisingly with only four dozen USA participants at the highest (Gold)
level and a total of fewer than 250 USA participants across all three competition levels. While the
best students acquit themselves well at the world level (USA teams consistently place in the top
five at the IOI, most often at #2 or #3 in the unofficial rankings), it is unclear that the overall goals
of USACO are being met at proper levels since participation is so very low.
This white paper examines the state of programming competitions in general, some of the
issues that limit current participation, and suggests a host of new proposals for increasing
USACO’s ability to accomplish its mission and expand its participation.
USACO and Its Goals
The USACO1 is a volunteer organization
headed by Director Dr. Don Piele, Head Coach Dr.
Rob Kolstad, and Educational Director Dr. Brian
Dean. After successfully running the International
Problem Solving Competition (for high school students) for many years, Don Piele formed the USACO
in 1992 to select and train the USA team for the newly
formed International Olympiad in Informatics (IOI),
the ultimate international programming contest for
pre-college students. The United Nations Educational,
Scientific and Cultural Organization2 (UNESCO) created the IOI3 in 1989 to follow in the footsteps of the
successful international olympiads in mathematics,
chemistry, physics, and others.
The extremely competitive IOI pits extraordinary
programmers (and some average programmers) from
around the world against each other in two five-hour
competitions to demonstrate mastery of algorithmic
programming.
Traditionally, the USACO’s goals are to:
• Provide pre-college students around the world
with opportunities to sharpen their computer
programming skills to enable them to compete
successfully at the international level.
• Enhance the quality of pre-college computer
education by providing students and teachers
with challenging problems, training materials,
and competitions which emphasize algorithm
development and problem-solving skills.
1http://www.usaco.org
2http://www.unesco.org .
3http://ioinformatics.org/index.shtml ,
http://olympiads.win.
tue.nl/ioi/ , http://en.wikipedia.org/wiki/International_Olympiad_
in_Informatics .
Version 0.9 Preprint; Limited Distribution
• Recognize
those students with outstanding
skills in computer science and encourage them
to pursue further opportunities in the profession.
• Provide educational, motivational, and competitive materials in the form of programming
competitions and web-based training to students
in the United States and over 75 other countries
via the Internet.
USACO accomplishes these goals mostly successfully.
The background goal for all these items is the
increase of talented students entering computer science and allied fields (which covers a very large number of areas of study). This includes provision of a
peer group for talented high school computer science
estudents, and stimulating pre-college students to
study computer science along with considering a
career in that field.
Generally, the USACO goes about accomplishing the goals above with its training web site, training camp, internet-based competitions, and international travel. The training camp serves as both reward
and a final set of competitions to choose the USA representatives at the IOI, which is held in a different
country every year. A trip to the IOI is the ultimate
prize for a USACO competitor. Additionally, USACO
often takes a team to the Central European Olympiad
on Informatics (as a ‘‘warmup’’ and first international
competition experience).
While the USACO is indeed accomplishing its
goals, the scale of those accomplishments makes a too
small an impact across all the industries that need
computer scientists and engineers. Participation
remains at the ˜250 student level (out of a high school
population estimated at 16,100,0004 – a penetration
1
USACO: Directions For The Next Decade
level of only 0.00155%). USACO should probably
strive for a penetration level closer to 100,000 students (0.62%).
Current USACO Status
As an organization, the USACO currently has
three long-term contributors:
• Don Piele, Ph.D., founder and currently director. Don has retired from the University of Wisconsin-Parkside and moving toward a golden
life in Arizona.
• Brian Dean, Ph.D., Educational director. Brian
organizes the instructional and competitive aspects of training camp and additionally administers a grant for augmenting USACO instruction at the introductory level.5
4http://nces.ed.gov/fastfacts/display.asp?id=65
1
5
5
1
15
39
1
38
1
9
ARE
ARG
AUS
AUT
BGD
BGR
BIH
BLR
BOL
BRA
25
205
3
16
3
16
1
1
1
8
CAN
CHN
COL
CUB
CZE
DEU
DNK
DOM
ECU
EGY
Kolstad
• Rob Kolstad, Ph.D., head coach. Rob runs the
competition and training websites, coordinates
competitions, solicits funding, and organizes
the staffing and non-educational aspects of
training camp.
Additionally, almost two dozen associate coaches
contribute tasks and solutions for competitions,
including directing one of the three levels of the six
annual contests.
The $35-50K annual budget (principally provided by donations from The USENIX Association,
Booz Allen Hamilton, IBM, and ita Software) funds
these ongoing activities:
5NSF CAREER award CCF-0845593, for which the major thrust of the educational component is the further extension of USACO to younger students through development of effective on-line materials.
Contest Participants by Country
1 ESP
23 IDN
6 MEX
5 EST
19 IND
5 MKD
10 FRA
23 IRN
11 MNG
1 GBR
1 ISR
4 NED
46 GEO
18 KAZ
3 NZL
1 GHA
1 KHM
1 PAK
5 GRC
6 KOR
2 PER
3 HKG
7 LTU
10 POL
8 HRV
10 LVA
34 ROM
1 HUN
2 MDA
7 RUS
2
7
3
2
1
4
8
6
1
6
SCG
SGP
SVK
SVN
SWE
SYR
THA
TKM
TTO
TUR
16
17
260
2
32
17
7
1
TWN
UKR
USA
VEN
VNM
YUG
ZAF
ZWE
Table 1: 1,060 contestants from 68 countries who successfully submitted at least one solution for the December,
2009 internet-based competition.
Figure 1: Historical participation in USACO internet competitions.
2
Version 0.9 Preprint; Limited Distribution
Kolstad
USACO: Directions For The Next Decade
• Six
About Training
USACO has been training its members (and the
rest of the world) quite intensely for the IOI for several
years. Those competitors who diligently solve problems
can see and solve as many as 1,000 tasks (including
many at other sites) before the IOI. They generally
place very high in the results. In fact, they do such a
good job that the difficulty of the IOI tasks is escalating
as organizers try to differentiate the top finishers. Superior competitors from just five years ago would probably not do so well in the most recent IOI competitions;
the task difficulty is growing that quickly.
USACO thus faces a dilemma: should USACO
continue to train the world and expose all its best pedagogical technologies to students from all different
countries? Should USACO keep the best techniques
secret just for US students?
This is a very real issue because the amount of
time it takes to become a top-25 international-level
programmer now exceeds 1,000 hours of practicing.
While this is commensurate with the cost of athletic
sports excellence, it is exceedingly burdensome on
‘‘intellectual athletes’’ who have other time-consuming interests, as well (including activities that potentially yield more ‘‘rewards for the time invested’’ than
USACO). Increasing the level of competition is
liable to reduce both USA’s ability to compete at
the international level and drive potential students
away from a contest that is already very difficult.
Note that these results are antithetical to USACO’s
goals.
•
•
•
•
•
•
popular monthly internet contests each
annual season. These contests draw anywhere
from 800-1,100 competitors from over 60
countries. See Table 1 and Figure 1 to get an
idea of the growth and participation levels.
Annual bonus competitions with guest directors (e.g., China’s or USA’s top competitors)
Year-round internet-based training conducted in multiple natural languages. See
Figure 2 for the level of student visits to the
training site over time.
An annual training camp and contest (the
USA Invitational Computing Olympiad –
USAICO) for 15-30 of the nation’s best competitors (with occasional international guests. It
includes another half dozen contests (two of
which are five hours long), these days with two
divisions. Additionally, multiple teams of attendees write AI players for one or more game
competitions.
The Central European Olympiad on Informatics (CEOI) during mid-summer.
The International Olympiad on Informatics
(IOI) in late summer. The USACO generally
strives to send the ‘‘best representatives of our
country’’ in both technical and non-technical
ways vs. simply ‘‘the best programmers.’’
A new division for training college students
using extant ACM competitions from around
the world. Entrants compete virtually against
the actual contestants with scores revealed
incrementally as the contest progresses.
2002
2003
2004
2005
2006
2007
2008
2009
2010
80,000
70,000
60,000
50,000
40,000
30,000
20,000
Training Logins/Month
10,000
Contest Logins
0
Figure 2: Historic levels of student logins.
Version 0.9 Preprint; Limited Distribution
3
USACO: Directions For The Next Decade
Kolstad
Assessing USACO Participation Challenges
The USACO faces a number of challenges to
increasing its participation:
• Available training time: Serious USACO
competitors are maxed out on the time they
wish to spend training. This negatively impacts
USACO’s ability to recruit students (they don’t
wish to invest so many hours) and is easily
demonstrated simply by counting the active
contestants, a number that rarely exceeds 250
students for the entire USA (across all three
divisions). Fewer than 50 USA student qualify
for, much less compete successfully at, the
highest level (‘Gold’) division.
• Insufficient critical mass upon which to
build: In the recent January, 2010 contest, 45
USA student participated in the Gold division,
and nine of them failed to earn more than 100
points out of 1,000 possible. To keep the presentation balanced, note that five of the best
USA students placed in the top 25 internationally. Only China has more students participating in the USACO gold competition than the
USA; these participation issues are not limited
to America.
• Training and competition paradigms which
attract and reward loners rather than encouraging collaboration and socialization: Traditional algorithmics training techniques demand
hundreds of hours focused on solo programming. Coupled with geographically dispersed
online competitions, this training style increasingly seems to motivate and encourage students
who are loners, quiet students, and focused on
little other than programming. It is these students who earn berths at our training camp.
Their conversations at training camp concern
algorithms and mathematics ... and little else.
Generally (with a few notable exceptions), they
would not be viewed as well-balanced or particularly socially adept in their interactions with
adults and as observed by adults at camp and
on trips.
•
•
•
Evaluating Computing vs. Math Contests
One of my internet messaging correspondents is
a 16+ year old high school junior of high ability who
attends a public school in silicon valley, California. He
is often quite excited about math contests.
I polled him and almost a dozen other students as
to what it was that attracted folks to math contests (my
correspondent participates in roughly 35 math contests
annually – and he is not active at the national level).
I discovered several significant differentiators
between math contests and computing contests:
• Math contests foster a dramatically larger
community. This means a competitor has some-
4
•
one with whom to share their interest(s) over
lunch, someone local with whom to collaborate
and compete, and generally is not ‘‘on their
own’’ to train and compete (unlike computing
competitions).
Math competitions attract the larger community
in several ways: they often offer team competitions; math teachers (who populate every
school in numbers much greater than computer
teachers) encourage competition, often citing
the mantra that ‘‘mathematics is the foundation
of everything;’’ every student in almost every
USA school system is required to take a
decade or more of math instruction in order
to complete their high school diploma.
The reduced number of computing contest
competitors in a given community means that
they are unlikely to congregate frequently (only
a few high schools supply more than one competitor to USACO).
Math competitions offer a different experience
from computer competitions. Computer competitions (with the rare exception of ACSL) are
hardly ever shorter than three hours and thus
generally occur outside the classroom setting
(perhaps via the internet at home or off-site on
a weekend). Math competitions range from
under one to three hours (until the national
level is reached). The mathematics mentoring
community (which includes high school teachers) dwarfs the computer programming mentoring community, which is almost non-existent.
Computer contests dramatically exceed math
contests in many sorts of difficulty. First of all,
programming requires the formulation of an algorithm (a task some might argue is mathematical) and then, once the proper algorithm is
deduced, a solution requires a working program
(which some have called ‘mechanics’). Making
a small error on either one of these orthogonal
requirements can result in a task that earns no
points at all.
Secondly, the ongoing training efforts of
USACO’s site, contests, and other country’s contests has elevated the level required to differentiate among the top competitors to that of graduate
students at leading research universities.
Finally, programming contests for high school
students tend to have but a handful of tasks;
math contests have 7-40 tasks; a talented student will have total confidence in getting at
least 2/3 of them correct. Programming contestants on challenging contests often have scores
in the 30-50% range, which sooner or later
must be frustrating to students accustomed to
perfection.
My IM correspondents argue about whether
algorithmic programming contests tend to
Version 0.9 Preprint; Limited Distribution
Kolstad
USACO: Directions For The Next Decade
focus on a small number of algorithm types,
of which only the ‘ad hoc’ and ‘combination’
varieties are not relatively standardized. This
means that the variety of programming problems
is smaller when compared to the relatively larger
of math contest tasks. Simply knowing the maximum value of ‘N’ in a programming task often
reduces the number of solution types (perhaps to
a small handful), thus further shrinking the variety. Some of my correspondents (including international-level programmers and internationallevel math competitors) believe that this variance in variety is strongly evident; some argue it
is a weaker proposition.
The combination of these factors appears to combine to yield a feeling that ‘‘Programming contests
are not as much fun as other activities’’ (like
FIRST or math contests) unless you are super-talented and reaping benefits like trips. Furthermore,
the properties listed above tend to attract (and create) loners instead of more socialized, team-oriented competitors who then gravitate elsewhere since
other attractive activities are highly available.
Thus, I argue that the current emphasis on
algorithmic programming contests as a way to foster interest in computing and computing careers is
not in the best interests of either the USACO, its
fellow contest organizers, or its contestants. This
emphasis surely should not form the only basis for
accomplish USACO’s goals.
Algorithmics is not the only programming challenge and surely is not the most appealing one to our
audience; it’s also expensive to master.
History of Programming Contests
How did (algorithmic) programming contests get
to their current state?
Many modern programming competitions find
their roots in the ACM college student competitions of
the 1970s that featured a single keypunch shared by a
team of three students who submitted batch jobs
whose results were subsequently returned, printed on
fan-fold printer paper.
Modern ACM competitions generally have 9-12
tasks (vs. USACO/IOI at 3-4) in a five hour timeframe.
ACM contests return some results in real-time, a twoedged sword that raises the excitement level but puts
tremendous pressure on organizers to supply correct test
data and judges’ solutions (a quality level which too
many ACM contests fail to reach, thus betraying and
frustrating the contestants, some of whom decline to
participate in future ACM competitions).
In ACM contests, the amount of data returned in
the event of a ‘wrong answer’ is minuscule (e.g., ‘we
have some test data that makes your task fail’). While
any contest can put forth any rules it likes, this particular paradigm does not reflect real-world programming constraints and promotes perhaps excessive
Version 0.9 Preprint; Limited Distribution
stomach acid in trying to find the programming error.
USACO/IOI contests return almost no data in real
time (just a verification that your program can solve a
single trivial case). This means that tiny errors can
result in little or no credit for a task. Neither of these
paradigms yields as much fun as they might (ACM
contests give too little data to find a problem;
USACO/IOI contests don’t give results at all until
after the contest). Few sports decouple performance
and reporting of scores this much (math contests
and other written competitions being the exception).
Appendix A documents a number of modern programming and other computer-oriented contests. Such
contests are now widely available throughout the
world for pre-college competitors and others.
Recruiting New Programmers
Recruiting new interested constituents is the
lifeblood of our organization. It’s not possible to foster
long-term interest without starting at short-term interest.
Dutch IOI coach and database expert Eljakim
Schrijvers has championed the Lithuanian ‘‘Beaver6’’
competition throughout central Europe. His examination delivery system annually administers the multiple-choice aptitude test to over 80,000 students in Germany (and many students in other countries as well).
A country’s best finishers take another exam where
small prizes are awarded to the highest scorers. Of
course, the big benefit is the identification of highaptitude students to steer toward a training and competition program.
USACO Coach and Clemson professor Brian
Dean has won a grant from the National Science
Foundation that includes, as one component, the training of computing newcomers (he has his own paper on
this). Holland’s superstar Jelle van den Hooff has his
own ideas on the creation of tutorials; that effort is
ongoing in the first part of 2010. It will also encompass the majority of USACO’s effort toward teacher
training and programs for educators.
Modern trends in novice training include a move
away from Java to:
• MIT’s Scratch language – the much modernized version of Logo, at some levels: A dragand-drop procedural language that processes
interrupts just like a real graphics system. Easy
to make animations and games; kids pick it up
instantly and almost automatically.
• Python, a relatively new scripting language that
is good for teaching both object- and procedural-oriented programming; it includes some
graphics.
• GameMaker7 enables students to create games
quickly and easily. This is used in some introductory summer programs.
6http://www.bebras.org/en/welcome
7http://www.yoyogames.com/gamemaker/
5
USACO: Directions For The Next Decade
• CS Unplugged8 includes sets of classroom exer-
cises (sometimes for very large groups) that
teachers really enjoy for computer-exposure
style drills.
One of the real problems for early computer training is the lack of teachers with skills and interest in
teaching about computer programming and technology.
The set of good materials for brick-and-mortar classrooms appears to be too small. I am a big fan of coming
up with a new set of ways to teach computer science
and its allied concepts that avoid the quagmire of the AP
program with its goal of giving college credit for a
course that no two universities can agree on. Identifying
teacher needs (and teachers!) is a paramount challenge.
I am making no proposals that advocate competitive contests for novice programmers. That paradigm
is anathema outside a fun Saturday morning team
competition where most entries get a perfect score. A
better paradigm seems to be stepped completion and
achievement levels that offer encouragement and foster growth more like ‘‘merit badges’’ than rare competitive accolades that only a small fraction of students will garner.
Merit Badges
Not all rewards come from competitions. Working through a set of requirements to achieve a standard
yields a reward in the form of a certificate, patch,
book, web-recognition, etc. that many students find
rewarding. They work at their own pace and succeed
or fail as they wish. Material that grows sequentially
more difficult can lead to great self-generated rewards
as the challenges are conquered.
Participation Paradigms
While contests are not necessarily the be-all and
end-all of promoting computers and their associated
technology, they are one strong methodology with a
proven track record. Certain kinds of contests (e.g.,
projects and associated authored paper competitions)
do mimic many of the important qualities of competing in the ‘‘real world.’’
This section discusses new sorts of contests (and
other paradigms) that sport novelty and fun that differ
in small or large ways from the current algorithmic
contest format.
Proposal: Customizable Contests
The current algorithm programming system (and
presumably future systems for other styles of competitions) should have a feature for creating customizable
contests for multiple competitors and for training
exercises.
For multiple contestants, a custom event might
focus on an intraschool competition, dual meet (e.g.,
two or three schools compete against other either onsite or in a distributed manner – and in lockstep time –
8http://www.csunplugged.com
6
Kolstad
or not), a regional competition (e.g., Cleveland County
Regional Competition), or something even bigger.
For non-team events, USACO should consider
implementing one-on-one head-to-head ‘‘challenges’’ or ‘‘mini-contests’’ (probably not on-site
and either in lockstep or close to it). These could
easily lead to a challenge ladder as in tennis and
other sports.
USACO should also implement customizable
single-person practice contests with tasks chosen
from an array of yet-unseen tasks from chosen categories (perhaps with difficulties and/or ‘par’ times).
These contests offer the flavor of competition (most
notably time limits and interactive paradigms) without
the performance pressure of competing against (and
potentially losing to) other real humans.
Proposal: Increased Team Competitions
Perhaps the single most-requested feature for
USACO contests was the addition of teams (an option
which was implemented ca. 2008). When queried, students cited several reasons they like team contests:
• Collaboration (camaraderie)
• Ability to blend into a team (i.e., not stick out
with individual results)
• Reduction in the impact of one person having a
‘bad day’
• Socialization, school spirit
• Greater participation
• Ability to bring less-experienced students into
serious competitions
Thus, it’s easy to assert: successful fun contests need
to support both individual and team competitors.
Team competitions vary in their levels of collaboration. Sometimes, a team is simply the sum of a
group of individual scores (perhaps the top four, perhaps a designated team, etc.). Other times, a team is a
group actively collaborating on solving problems (and
allied programming issues). USACO should pursue
both of these as appropriate.
Programming Bee
This new style of online or on-site competition
format includes a set of contestants on the internet and
a clever server. The contest proceeds lockstep through a
series of ‘questions’ and ‘tasks’ announced in synchrony from a high-speed, well-connected site. Tasks
(e.g., what is the value of 2100? Given this set of song
times, what’s the most we can fit on a 66:00 minute
CD?) might have a time limit (perhaps two minutes for
2100) or might have an advancement limit ‘‘the first 100
solvers move to next round.’’ Problems progress in difficulty and end up leaving a single champion, the last
person standing. This competition has the advantages of
• Novelty – it feels like a spelling bee and can
provide great entertainment for observers.
• Fairness – everyone has to solve every problem
(unlike a spelling bee where students each spell
a different word)
Version 0.9 Preprint; Limited Distribution
Kolstad
• Excitement – shorter time limits and display of
those progressing make this a really high-pressure competition if one wishes, especially with
teams (particularly if three team members share
a single keyboard/screen)
Programming Bowl
This is, as it sounds, a College bowl/Science
bowl style contest for programming, implemented for
teams or individuals. Conducted on-site, it includes
verbal questions (‘‘What symbols are the block delimiters in the C programming language’’) and programming or computer/net usage questions that might
require a computer. Unlike most of the other competition styles, this one doesn’t scale past a dozen or so
participants. It has the advantage, though, of being
particularly adaptable to a stage setting with a large
audience.
Monthly Projects
Jacob Steinhardt proposes another interesting
idea for focused students. The organizers present or
reference a paper or set of papers (e.g., ‘‘Current
Directions in Machine Vision’’) and posit a project or
challenge that (teams of) students are to attack. After
some appropriate interval (probably a month or two),
students submit results (e.g., programs, presentations,
or papers) of their work for evaluation and potentially
awarding of prizes. This sort of idea also works well
in the context of conferences for pre-college students.
AI Games
In AI game competitions, teams of 1-10 players
write cooperative (e.g., MIT’s sockey9) or competitive
(e.g., Tron light-cycles10) programs that play with or
against each other in some virtual arena, usually one
that is presentable as a nice real-time graphical display.
AI game competitions can be utilized both
shorter (2-3 hours-to-build) and more intense (3-20
not necessarily contiguous hours) formats. USACO
should foster or collaborate with competitions like
Kim Schrijvers’ CodeCup (perhaps during the December holidays). For on-site games, both IBM’s Tim
deBoer and I have frameworks for building new
games.
Students generally enjoy these sorts of competitions and do just fabulous work once they commit to
winning a game.
Robotics
Robots and robot projects are extremely attractive
to students. Some believe robotics (as implemented by
battlebots and subsequently by Dean Kamen’s FIRST
organization) is driving the incredible surge of students
in the mechanical engineering sector. While the
9http://www.ai.mit.edu/lab/olympics/99/events/sockey/docs/
how-to.html
10In addition to the USACO implementation, U. Waterloo is
hosting a similar competition in early 2010 (http://csclub.uwaterloo.ca/contest/).
Version 0.9 Preprint; Limited Distribution
USACO: Directions For The Next Decade
population of CS students is just starting to increase
again after the incredible decline following the dot com
bust,11 mechanical engineering schools are full.
I interviewed talented high school academic
Jacob Hurwitz (Intel Talent Search, programming contest, Science Fair, and USACO winner) about his participation in the FIRST program. Among the distinguishing characteristics that made robots fun, he
noted:
• Robot contests create a tangible product that
you can see and touch; robots do things
• Robot contests enable contestants to make cool
results without years of training
• Robot contests feature a project that can be finished in, often, six weeks
• Robot contests almost universally feature a
team orientation with relatively large teams
• Robot contests are often mentored by skilled
engineers who bring new ideas and viewpoints
to the traditionally more staid academic setting
• Robot competitions can be made to be extremely exiting (including real-time feedback, lots of
action, and interaction among dozens of human
competitors)
• Robot competitions are extremely well-funded
with travel (a typical single high school FIRST
team will spend $50,000 on 20 students; affluent teams spend three times that much)
• Robot competitions promote the idea that it is
easy to grow skill
• Robot construction uses modular hardware (and
software) that requires problem-solving at very
different level from algorithmics
• Robot competitions often offer excellent prizes/
rewards (e.g., big scholarships)
It’s easy to see that robot competitions address many
of the shortcomings of traditional USACO contests.
Robot competitions include not only FIRST but
also the Oklahoma-based Botball. I met with Botball
founder board member David Miller12 in late December, 2009 to see if USACO’s and Botball’s interests
coincided. Not surprisingly, software is the generic
weak point of almost all robot competitions. Despite
an absolutely gorgeous embedded system box, the
Botball folks suffer some of the same challenges as
USACO. We have agreed to talk in the future to find
areas where we can cooperate.
Note that virtual robots can provide a lot of the
fun (without tangible mechanical pieces) as they simulate or model manipulations in a virtual world. The
Netherlands’ Kim Schrijvers knows of virtual competitions where winners’ programs are then run on the
real versions of very sophisticated robots that control
exotic devices.
11http://www.cra.org/govaffairs/blog/2009/03/computerscience-majors-increase-at-most-significant-rate-since-dotcom-boom/
12http://faculty-staff.ou.edu/M/David.P.Miller-1/
7
USACO: Directions For The Next Decade
‘‘Computer Clubs’’
Much like the Mu Alpha Theta folks provide a
national-level umbrella organization for pre-college
math clubs, USACO (or an allied organization) should
perform the same purpose for a kind of computer club,
information technology club, or similar association.
I propose a model that exploits the availability of
computers, data projectors, high speed bandwidth, and
preciious technology students that most high schools
possess while acknowledging that teachers often have
neither the available time or advanced knowledge to
create and present material applicable to an interested
group of technology enthusiasts.
I envision a club that meets at least every other
week (and perhaps weekly). One set of meetings (perhaps every other one) typically runs like this:
• Opening statements along with scheduling
reminders (e.g., for contests, trips, etc.)
• A 5-10 minute supplied video presentation
from academia or industry. This presentation
might show how Seagate manufactures multiterabyte disks that are never touched by human
hands from the inception of the manufacturing
process until the disk is installed in a computer
system. Perhaps the video highlights MIT’s
new research in digital mapping technologies or
CalTech’s latest results in the exploration of
using animal neurons to create better artificial
intelligence. This part of the presentation has
two purposes:
Show that the world of computer technology is broad, deep, and utterly fascinating.
Pique students’ interests in a wide spectrum of different sorts of computer- or
engineering-oriented careers.
The beautiful side-effect is the promotion of the
companies and academic institutions that provide the videos. They can penetrate to precisely
the demographic of potential employees/customers they need over the next few years.
• After the short technical news, an informative,
interesting ‘‘Powerpoint Presentation’’ is given
by an interested teacher, mentor, or student.
The presentation might cover introductions to
databases, MIT’s Scratch language, system administration, a programming language (like
LISP, which would be unusual for the high
school set), computer graphics, artificial intelligence, networking, compilers, modern commercial computer game creation, embedded systems, scripting languages, etc. The presentation
is always followed by a hands-on experience to
emphasize the concepts presented.
Clubs always need a year-end goal, so members of this
club would attend a conference (in either western, central, or eastern USA). The conference would include
inspirational speakers, technical speakers, fun competitions, and camaraderie.
8
Kolstad
On alternate weeks, club members use online
tutorials to learn programming languages, new technology paradigms, or to construct software/hardware
projects.
Contest Task Paradigms
The more popular contests these days offer either
algorithmic programming competitions or project competitions. We need a host of new styles/topics for programming competitions. A year-long casual survey of
progressional programmers reveals that a typical professional programmer will, over the course of an entire
year, implement no more than a single instance of
‘‘algorithmic’’ code similar to the sort that is the subject of algorithmic programming competitions. I assert
that we can find interesting competitions and inspirational contests by examining other areas of professional programming.
Changing the layout and/or style of a competition can yield a new contest with a different feel, even
if most elements are similar to the current algorithmic
competition model (i.e., IOI). Consider:
• Function-only submissions perhaps not too
different from Topcoder. By submitting a solution for only the heart of of the task, both the
programming and the timing for pesky things
like I/O are removed. This can enable extremely
accurate timing, huge numbers of test iterations,
and a slightly different feel for the contestants.
Of course, this idea requires a framework for
each different supported language.
• Contests with many more than three tasks.
More like ACM contests but perhaps with a set
of tasks growing incrementally in difficulty.
This might be most appropriate at lower levels
of competition.
• ‘‘No Limits’’ tasks. These problems have storage allocation specifications but no guarantees
about the maximum size of ‘N’ as far as the
algorithm goes. An O(N4) task might limit N to
4,000,000, even though no dataset that big is
ever solvable. This challenges students to figure out algorithms from task statements rather
than from the specified limits.
• Real-time scoreboard when appropriate. Just
as ACM contests show real-time ongoing
scores, our contests should do this too when
contestants are competing simultaneously. (This
functionality is in place for USACO-conducted
ACM competitions.)
• Virtual scoreboard (including contests used as
training exercises by individuals). When contestants do not compete cotemporaneously
(even when they are participating in a years-old
contest), show a virtual scoreboard whose contents are revealed in virtual lockstep with the
contestants’ clocks. (This functionality is in
place for USACO-conducted ACM competitions; watching folks compete against talented
Version 0.9 Preprint; Limited Distribution
Kolstad
USACO: Directions For The Next Decade
Russians in a contest from 2007 is quite amusing.)
• We must increase the prizes and incentives
offered to contest winners, especially when a
competition extends, e.g., throughout a school
year.
• USACO should support personal ‘‘results’’
pages that provide scores and results for a student, enable the student to showcase some of
their own work (via biographies, pictures, and
the like), and generically provide a bit of social
networking capabilities.
• USACO should create a standard recommendation program for colleges and universities so
that outstanding competitors can receive some
sort of uniform recommendation based on their
performance in elite USACO competitions.
I do not propose that USACO set up for ‘project competitions,’ which are well done by many other organizations.
New Task Topics
The refined and matured legacy of ACM contests
from the days of keypunches and line printers has
mushroomed into a large number of algorithmic-style
contests that are, in my opinion, utterly inappropriate
for, e.g., most computing professionals.
We should explore a large set of task topics – not
all in the vein of algorithmics – in order to broaden
our offerings and demonstrate a wider spectrum of
interesting programming and computer-oriented challenges. Here’s a list of all sorts of task types:
• The original: algorithms. The most popular
contest-style for almost all 1-5 hour programming contests. This includes contests that really
are about introductory ad hoc tasks.
• Simulations & modeling. The math folks (personified by MAA director Steve Dunbar of
University of Nebraska-Lincoln) are pursuing
this notion. Their prototype task: ‘‘Here’s a
model of a house; install N fire alarms in it so
that you maximize the likelihood a fire is
detected.’’ One can easily imagine modeling
contests for parking lots, baseball field mowing, embroidery layout patterns, forest maintenance, and a huge number of other real-world
systems. These sorts of tasks have a relatively
different flavor and would probably attract a
new set of aficionados.
• System programming, text, and file manipulations. Our algorithmic competitors almost uniformly have no knowledge of efficiency and
effectiveness of the algorithms, techniques, and
heuristics that buttress their user-level programs.
A contest for efficient implementations of a
‘‘word count’’ program, a ‘‘print a file backwards program,’’ file manipulation tasks, and
these sorts of things would bring a completely
Version 0.9 Preprint; Limited Distribution
•
•
•
•
•
•
new flavor to our competitions. Likewise, the
idea of writing parsers, lexers, and compilers
falls into this category.
Databases/Statistics/data mining. The storage,
retrieval, manipulation, and analysis of data is a
goldmine for tasks that probably last a few hours
and are scored subjectively. Consider:
Find the region where expanded advertising
will have its greatest impact in the next fiscal year.
Find the telephone traffic pattern that is
most likely to yield intelligence about terrorist attacks.
Supercomputing. Parallelism in all its forms is
a hot topic in computer science. The supercomputer conferences are beginning to experiment
with contests for using large supercomputers to
solve applicable tasks. Supercomputers have a
special allure – just the promise of access could
attract new talent. Imagine: ‘‘Please create a
video of these two galaxies colliding.’’
Scripting (including Python, PHP [mostly for
the web], Perl, and various shell scripting languages). Potentially in conjunction with other
areas, this area is very rich in application; I personally program 80-90% of my applications in
Perl instead of C/C++/Java. Example: ‘‘Files
1-N contain entrants in a science fair, teacher
information, school information, judge information, and project scores. Determine the
teachers with the best performance and report
the judges whose scores most nearly matched
final results. You have 45 minutes.’’
Artificial intelligence. AI contests tend to be
most attractive when designing collaborative/cooperative/competitive AI agents for (virtual) games. The previous discussion mentioned some games already used and referred to
Kim Schrijvers’ CodeCup competition.
Speculative: Tasks in a special programming
language. While this seemed a poor idea in the
context of high speed algorithmic tasks, it
seems a great idea in the context of interpreted
statements for a special purpose task or genre.
MIT’s Scratch is one such language – it’s programmed graphically with drag-and-drop. One
of the fun parts here is that the language need
not be precisely procedural; it might be LISPlike or perhaps presented simply as object code
(to be created or interpreted), or might utilize
some unusual or unorthodox representation or
programming technique (think Snobol).
Embedded applications and languages. Robots
(see below) provide one of the most popular
embedded applications, but almost any unique
peripheral can afford some exciting programming (perhaps in assembly language, C, or a
BASIC version). Plenty of robot contests exist;
9
USACO: Directions For The Next Decade
see below for comments on cooperating with
the good folks at Botball.13
Conclusion
USACO must expand its offerings if it wishes to
achieve its goals of promotion of computing careers in
the pre-college world. Until someone offers the sorts
of expanded topics listed above, students will have no
idea of the breadth of careers available in the computer technology arena (and thus will be diverted to
other popular academic majors).
The sorts of competitions discussed in this paper
can be built into various contest-administration systems and, at the end of a season, provide a truly meaningful competition (perhaps the ‘‘Programming Pentathlon’’) where teams (individuals, too?) compete in
many areas to become real programming champions
across a variety of subjects, not just in the narrow and
increasingly deep world of algorithmics.
USACO must seek funding and contributors to
bring these plans to fruition.
13http://www.botball.org
10
Kolstad
Appendix A: Overview of Some Current Computer-Oriented Contests
USACO is not the only organization promoting
computer competitions. The number (if not the styles)
of contests continues to grow as online graders (and,
more rarely, training sites) proliferate owing to spreading skills, maturing tools, and availability of technology for system protective ‘jails’ that keep submitted
code from harming a host grading server (these jails
are now available on Linux and even, apparently, Windows).
The most popular contest sites include:
• USACO – intermediate through advanced training and tasks
• Topcoder – lots of forums, divisions, and cash
prizes
• Google Code Jam – cash prizes
• Codechef – India, cash prizes
• COCI – Croatia
• ACSL – Less advanced training (includes lectures for teacher assistance)
A list of popular training sites is presented later.
These are complemented by some slightly
smaller competitions:
• Regional contests (OI’s in Central Europe,
Balkan area, Baltic area, Pacific)
• National contests in almost every nation participating in the IOI
• The IOI itself
• Smaller regional contests like the Berkeley contest, HP sponsored contests in Texas, UMd contest, and dozens more
As further evidence of contest proliferation,
2008-2009 USACO champion and IOI gold medalist
Neal Wu finds enough online opportunities that he
competes in 50-60 contests annually, generally devoting one to three hours for almost every competition
(just a few are longer).
The combination of the higher number and
increased profile of these contests (some with cash
prizes), the widespread availability of training, and the
ongoing penetration of ever-lower cost computing facilities has enabled a small number of programming prodigies to increase their skills to unprecedented levels.
Additionally, some geographical/political regions
are said to be investing heavily in academic competitions, even to the extent of hiring full-time coaches for
high school competitions. The University of Central
Florida has instituted a ‘fellowship’ program for its
undergraduates who wish to excel in the ACM ICPC
college contests. Such a program is indistinguishable
from paying students to train for competitions. CMU
offers a ‘contest programming course.’14
The concomitant blossoming of programming
proficiency has motivated elite competitions like the
IOI to increase the overall difficulty of its six primary
14http://www.andrew.cmu.edu/course/15-295/
Version 0.9 Preprint; Limited Distribution
Kolstad
USACO: Directions For The Next Decade
tasks significantly in order to differentiate among the
top competitors. The difficulty has increased enough
that easier, lower-level tasks have been added to the
competition in order to avoid the growing number of
student achieving a score of 0.
While acknowledging that correlation is not causation, USACO conducted a statistical analysis that
compared the number of training problems solved by
USAICO attendees with overall scores on USAICO
tasks. With a single outlying datapoint, the result was
that the overall performance was almost perfectly correlated with the number of training tasks solved.
While this would seem to make algorithmic programming competitions appear to be attaining a level of
strict meritocracy (which is often regarded as a positive
attribute), the resulting almost insurmountable challenge is that becoming a high-level competitor at the
high school level now requires somewhere in the range
of 1,000 to 1,500 hours of practice. While this time
commitment is similar to that of elite athletes like
swimmers and gymnasts, it is a tremendous burden to
place on students who are almost uniformly interested
in allied fields like mathematics and physics which also
require similar time commitments. I have personal
experience with teachers who have advised their students against participating in USACO because the time
requirement is simply too high.
The world of algorithmic programming competitions evolves slowly from the viewpoint of competitors: programming language C++ was added many
years ago; Java is now also used in some competitions. The administration and development systems
(used behind the scenes) are maturing more quickly,
but neither uniformly nor in public. Some formalization of these elite contests has emerged, as well: The
IOI publishes an annual journal of refereed papers on
aspects of pre-college competition.
New Contest Styles
As the technology grows and evolves, enthusiasts have created a host of new sorts of competitions
in a variety of allied fields.
X Prizes
These are serious high-powered, amazingly wellfunded competitions. From the website:15
An X PRIZE is a $10 million+ award given to
the first team to achieve a specific goal, set by
the X PRIZE Foundation, which has the potential
to benefit humanity. Rather than awarding
money to honor past achievements or directly
funding research, an X PRIZE incites innovation
by tapping into our competitive and entrepreneurial spirits.
The Ansari X prize was won by Virgin Galactic
who won the contest for a non-government created
15http://www.xprize.org
Version 0.9 Preprint; Limited Distribution
spaceship. Google has upped the ante, also from the
website:
The Google Lunar X PRIZE is a $30 million competition for the first privately funded team to send
a robot to the moon, travel 500 meters and transmit video, images and data back to the Earth.
The Progressive Automotive X prize offers $10
million to the team that makes a reasonable 100 MPG
vehicle. The Archon X Prize offers $7-10 million to
the team that can build a device to sequence 100
human genomes in 10 days at a cost of less than $10K
per genome. Other prizes are in development. The
Energy & Environment Prize Suite is to have a combined prize value of as much as $100 million.
DARPA Grand Challenge
Recent editions of the DARPA Grand
Challenge16 have encouraged college-level and professional competitors to create autonomous cars that can
negotiate urban driving courses at reasonable speeds
without accidents. First prize: $2,000,000. Second
prize: $1,000,000; third prize: $500,000.
Robot Competitions
Robot competitions are, these days, immensely
popular – and well they should be. Flash, tech, great
rewards, and just the right amount of hand-holding
combine to make an unbeatable contest combination.
The robots.net17 site lists 119 robot competitions; it’s
no wonder university mechanical engineering schools
are bursting at the seams. Mr. Wikipedia knows a large
set of competitions as well.18
Three popular competitions include:
• FIRST19 – an extensive competition where
teams of high school students construct, with a
mentor ’s guidance, a robot that will (usually)
collaborate with another robot to compete in an
arena against two other robots to see which
team can complete a task like moving beach
balls around. FIRST has a short autonomous
section of their competition, but is mostly for
remote-controlled robots.
Extremely well-funded (a team of 15-20 might
well spend $50,000 annually), the competitions
are also lavishly produced (as is the website),
complete with cheerleaders. USACO competitor Jacob Hurwitz says, ‘‘I idolize (competition
founder) Dean Kamen the way other guys idolize sports figures.’’
FIRST says they have $11,000,000 in scholarships this year. They have super participation
with programs for students from elementary
through high school:
The total FIRST Community – with over
196,000 students, 86,000 volunteers, and
16http://www.darpa.mil/grandchallenge/
17http://robots.net/rcfaq.html
18http://en.wikipedia.org/wiki/Robot_competition
19http://www.usfirst.org
11
USACO: Directions For The Next Decade
3,500 sponsors this year alone – is estimated at one million strong and growing.
• Botball20 – Botball has similar goals to FIRST
but is much smaller and hosts contests with
autonomous robots (much more technology)
and much lower budgets.
• Robocup21 – Robocup focuses mostly on the
university set (though they have other contests
for other age groups) for their theme which is
soccer-playing robots. They have used modified Aibo robot dogs as their players, TV cameras, and lots of software to play their robot
games. They are also fostering some other scenarios like building-rescue for robots.
AI Game-Playing Contests
Some competitions ask students to write one or
more programs that will play in a collaborative or
competitive setting. Few of these programs require
players to collaborate with strangers, generally nonteammates compete.
Probably the best-known AI game is computer
chess.22 These days, programming competitive computer chess programs is mostly beyond scope of amateurs.
Kim Schrijvers and a Dutch team run the Codecup competition each year. The Codecup developers
present a new game each season; developers create
programs that play in an arena and periodically in
tournaments to determine the best program.
IBM’s Tim DeBoer’s CodeRuler, CodeRally,
CodeInvaders were developed for the ACM ICPC
competition. A number of competitors (e.g., four in
Coderuler) vie to capture territory, win a race, or
accrue points in colorful arenas projected on large
screens for the enjoyment of the tournament entrants.
Perhaps one of the most interesting examples is
MIT’s sockey23 (combination of soccer and hockey)
game, where each of two competitors writes six collaborating processes to control six players on the
sockey field. Games are displayed graphically, with
the players moving all over the field, ‘kicking’ the ball
in an effort to make a goal. Configurable game boards
(e.g., faster or slower grass, different field dimensions)
along with the requirement for intra-team cooperation
(since many configurations do not have the ability to
‘kick’ the ball all the way from one end of the field to
the other) make for fascinating game play.
Project Competitions
Some competitions focus on entrants’ creations
(usually constructed over weeks or months, but occasionally in some limited time period like a weekend) of
a product or system.
20http://www.botball.org
21http://www.robocup.org/
22http://en.wikipedia.org/wiki/Computer_chess
22http://www.ibm.com/developerworks/java/newto/
23http://www.ai.mit.edu/lab/olympics/99/events/sockey/docs/
how-to.html
12
Kolstad
For high school students, the Intel International
Science and Engineering Fair is one of the most famous
and lucrative such competitions, though by no means
does it focus on computing. Huge prizes and trips motivate competitors in a truly international environment.
Large corporations like Microsoft,24 Google, and
Topcoder run project competitions for a variety of reasons. ACM’s special interest group SIGMOD conducts an annual design competition, for 2010 they are
requesting a design25 for a distributed database query
engine Design competitions are very common and
sponsored for a variety of reasons, including product
promotion.
Programming Contests
The last decade has seen an explosion of the
number of programming competitions and record
growth in established competitions at the high school,
university, and professional level.
• The IOI sees 300+ participants from 75 countries. Almost all those countries have feeder
contests to determine who represents the country at the IOI.
• The university-level ACM ICPC garners thousands of participants from around the world in
local and regional contests leading up to a
world championships.
• The good folks at dmoz.org26 keep track of
many dozens of programming competitions. It
is unknown why USACO does not appear on
this list.
• The American Computer Science League27 is a
long-running program that features teacher
preparatory materials and (relative to most)
less-challenging team-oriented competition.
Contests with both written answers and programming, multiple divisions, a registration
fee, and an annual on-site All Star Competition
are ACSL’s hallmarks.
• The ICFP functional programming contest28
focuses, of course, on functional programming.
In 2009, this involved calculating orbits and
tasks that determined how to intercept a satellite.
• The Matlab programming language puzzle
solving contest29 for 2009 involved implementing the solution to a puzzle a bit related to
drawing bit related to drawing maps.
• The IEEE extreme contest for university students30 is a 24 hour marathon with a 25 page
24http://visualbasic.about.com/b/2009/07/15/new-microsoft
-programming-contest.htm,
25http://db.csail.mit.edu/sigmod10contest/
26http://www.dmoz.org/Computers/Programming/Contests
27http://www.acsl.org
28http://icfpcontest.org/
29http://www.mathworks.com/contest/flooding/home.
html
30http://www.ieee.org/web/membership/students/xtreme/
index.html
Version 0.9 Preprint; Limited Distribution
Kolstad
instruction booklet containing ten problems,
including both algorithmic tasks and more
pragmatic tasks like a language interpreter.
• The Java 4K Programming Contest31 asks contestants to write a game whose object code is
no longer than 4,096 bytes.
• The Mario AI competition32 asks contestants to
write an AI agent to play a Mario consolebased game with up to 24 steering updates per
simulated second.
• The UMD high school competition33 is a longrunning algorithmic programming contest that
invites teams from the Washington, DC metropolitan area to compete.
• The TeraGrid supercomputer student programming contest34 (mostly for university students)
asks teams of five students to ‘‘tackle science
and parallel computation problems.’’
• The New Jersey Institute of Technology holds
an annual regional contest for high school students.35 Teams with three members; $50 registration fee; puzzle (e.g., Sudoku), simulation,
and pragmatic tasks.
• The University of Alabama/Birmingham is
another example of a regional high school competition.36 Prizes include a Dell Laptop. The
contest features pragmatic and numerical tasks.
• The Dyalog company promotes its products
with a programming competition37 that appears
to use APL.
• The MIT Battlecode AI tournament38 features a
sophisticated real-time strategy game where
software robots collaborate to take over territory. This year offers $40,000 in prizes to students from perhaps 100 different universities.
• The University of Illinois at Urbana Champaign
has conducted a long-running ‘‘strategy game39
of war and money’’ to accompany their annual
student computing conference.
• Florida International University conducts an
annual high school ICPC-like team competition40 with two divisions, 25 teams, and a $25
registration fee. Tasks are relatively simple and
occasionally touch on algorithmics.
Wikipedia41 lists a host of sites that offer online
training and contest judges:
31http://javaunlimited.net/contests/java4k.php;
http://www.java4k.com/index.php?action=home
32http://julian.togelius.com/mariocompetition2009/
33http://www.cs.umd.edu/Outreach/hsContest.shtml
34http://www.teragrid.org/tg09/index.php?option=com_
content&task=view&id=80
35http://cs.njit.edu/news/programmingcontest/
36http://www.cis.uab.edu/programs/hspc/
37http://www.dyalog.com/contest_2010/index.html
38http://battlecode.mit.edu/2010/
39http://www.acm.uiuc.edu/conference/2009/mechmania.
html
40http://users.cis.fiu.edu/˜acm/HScomp2009/index.php
41http://en.wikipedia.org/wiki/Online_judge
Version 0.9 Preprint; Limited Distribution
USACO: Directions For The Next Decade
• ACSL
• COCI
• Codechef – India, cash prizes
• Fuzhou University Online Judge,
over 800
problems. Popular among the Chinese.
http://acm.fzu.edu.cn/
• Google code jam, cash prizes
• Harbin Institute of Technology Online Judge,
over 1800 problems. Popular among Chinese
people. http://acm.hit.edu.cn/
• Lviv National University Online Judge, over
100 hard ACM-type problems. Only Ukrainian
language. http://acm.lviv.ua/
• Moscow Problem Archive with Online Judge,
supports C, C++, Java, Free Pascal, Perl,
Python, Haskell, Kylix, Lua, OCaml, Ruby, and
Scheme.
http://acm.mipt.ru/judge/login.pl?lang=en
• Peking University Online Judge, over 2800
problems including more than 200 original
problems, supports C/C++/Pascal/Java/Fortran.
http://acm.pku.edu.cn/JudgeOnline/
• Saratov State University Problem Archive with
Online Judge, over 300 problems http://acm.
sgu.ru/
• Sphere Online Judge, over 5000 problems formulated in English, Polish, Vietnamese, Portuguese; supports 39 different programming
languages, hosting own contests supported.
http://www.spoj.pl/
• Tianjin University Online Judge, over 2000
problems, supports C/C++/Pascal/Java, virtual
contests supported. http://acm.tju.edu.cn/
• Timus Online Judge, The Timus Online Judge
is the largest Russian archive of problems from
different programming contests.Accepts solutions in C/C++, Java, Pascal and C#. http://
acm.timus.ru/
• Topcoder, cash prizes
• USACO|USA Computing Olympiad Training
Site, hundreds of problems with complementary tutorials for incrementally expanding competitive programming skills; supports C, C++,
Pascal and Java programming languages; targeted to pre-college students. Full ACM contest
support, as well. http://train.usaco.org
• Valladolid Problem Archive with Online Judge,
about 2500 problems, supports C, C++, Pascal
and Java programming languages http://uva.
onlinejudge.org
• z-trening contests & training, >500 tasks at end
of 2009
• Zhejiang University Online Judge, over 2000
problems, supports C, C++, Pascal and Java.
http://acm.zju.edu.cn/
See the Wikipedia entry42 for other comments.
42http://en.wikipedia.org/wiki/Online_judge
13