Trade-off Analysis of Decision Support
Systems for Time Sensitive
Operations/Referee’s Assistant
ChristopherPaik
ThomasPhan
AdrianSolomon
AbdullahAlhauli
2
Source: http://www.nationalpost.com/sports/Germans+crush+English+defence/3208432/story.html
3
AGENDA
• REFEREE PERFORMANCE
• COMPLEXITY
• STAKEHOLDER ANALYSIS
• NEED STATEMENT
• PROBLEM STATEMENT
• MISSION REQUIREMENTS
• DESIGN OF EXPERIMENT
• VALUE HIERARCHY/
UTILITY
• CONCLUSION
4
• QUESTIONS
WHAT IS
Consist SOCCER?
of:
•Official Referee
•Runs Diagonal of field
(To maximize Line of Sight)
•Grants Final Decision(s)
•Executes Laws of the Game
•Assistant Referees
•Identifies Offside
•Aids Official to identify
infringements
•4th Official
•Soccer Players
5
CURRENT REFEREE
STATUS
Only 96.88% of Goal Decisions in 2010 World Cup were correct:
• 142 of the 145 goals awarded correctly
•13 goals were correctly disallowed for offside
• Two more goals should have been allowed
In the League Managers Association (LMA) the Additional Assistant Referee (AAR)
experiment concluded that:
• Only 78.8% for 5 types of decisions (Goal Kicks, Corner Kicks Free Kicks, Throw Ins, &
Advantages) were Correct
• 10.2% were incorrect
• 10.9% was inconclusive.
6
GAME IS
GROWING
• Players are running faster
and are covering more
distances. 9 km in 1990’s,
11 km in 2004, and 15 km in
2010.
Based on our Subject
Matter Expert, amount of
decisions per game:
627 game situation
decisions
7
POPULARITY
3500000
140000
3000000
Virginia
120000
Connecticut
2500000
Iowa
100000
Florida
2000000
Utah
80000
Total
1500000
60000
1000000
40000
500000
20000
2008
2006
2004
2002
2000
1998
1996
1994
1992
1990
1988
1986
1984
1982
1980
1978
0
1976
0
1974
Number of Registered US Youth Soccer Players
160000
Time (Years)
8
Total Number of Registered US Youth Soccer Players
Registered US Youth Soccer Players vs. Time
WHAT’S AT STAKE
• 17 teams won 1-0 out
of the 64 matches during
2010 World Cup
Average Goal Per Match in World Cups vs. Time
6
5
# of Average Goals
• The importance of a
goal being scored is
increasing due to the
decrease in average
number of goals scored
4
3
2.27
2
1
0
• Spain, who won the
2010 World Cup, won 5
out-of 7 matches 1-0
Time (Years)
Source: FIFA 2010 World Cup Technical Report
9
WHAT IS AT STAKE?
• Legitimacy of sport
• Fan base viewership (increasing popularity in other sports)
• MONEY: TV Revenue, Merchandise/Ticket Sales, TV Broadcasting rights, player trades
$3,500,000,000
900
$ on rights deals
800
$3,000,000,000
$ spent on games rights
$2,500,000,000
700
600
$2,000,000,000
500
$1,500,000,000
400
300
$1,000,000,000
200
$500,000,000
100
Time (Years)
2013
10
2012
2011
2010
2009
2008
2007
2006
2005
2004
2003
2002
2001
2000
1999
1998
1997
1996
1995
1994
$0
1993
0.000001
Overall Revenue from Broadcasting rights
1000
1992
Total $ Spent on games rights annually (Millions)
British Sky Broadcasting TV Rights Value in English
Premier League vs. Time
10
REFEREES ARE HUMAN
Referees are limited to:
• ONE line of sight
• Running Speed/ Degree of Fitness
• Limited ability to process event data (e.g. Offside: Flash-Lag effect;
Referees can not simply see two events at the same time)
11
11
REFEREE WORKLOAD
• Referee performance
shows majority of the 2nd
half of a match outputting at
85-95% Maximum Heart
Rate
• Total distance covered by
Referees must match players
increased significant
distance covered. 9 km
during the 1990s, 11 km in
2004 and 15 km in 2010
12
WHY THIS IS A PROBLEM
13
STAKEHOLDER
ANALYSIS
• IFAB (International Football
Association Board) are key
holders of the “Laws of the
Game”
• FIFA (Federation
Internationale de Football
Association) and IFAB are
governing bodies of Soccer
14
STATEMENTS
PROBLEM STATEMENT:
Currently, the game has grown more complex, but humans are limited by their
capabilities, creating a growing gap between the needed performance of the
officiating system and the complexity. IFAB/FIFA (Governing bodies of soccer)
have failed to Identify and address this growing gap.
NEED STATEMENT:
There is a need to increase the percent accuracy of the Referee in game
adjudication of a fair game.
15
MISSION REQUIREMENTS
• The system shall satisfy regulations and laws of the governing
bodies of soccer
•The system shall reduce Referee workload
• The system shall relay information to the referee within 5 seconds
of the game incident
16
UTILITY
Utility
Accuracy
(Percentage)
Usability
(Learning
Curve Rating)
Speed
(Time to Result)
Time to make
decision
(seconds)
% Accuracy will come
from model
Time to transmit
decision
(seconds)
Cost
Implementation
(Dollars)
Operation
(Dollars, Annual)
Weights will be collected from stakeholder surveys
17
METHODOLOGY
• Simulate the effects of proposed Decision Support Systems on the
Accuracy of a Soccer Official.
• Simulate the Factors that hinder a Referee’s ability to accurately
judge an event
• Simulate the probability of an accurate decision based on the
Decision Support System’s characteristics
• Perform a Trade-off Analysis comparing decision support systems
to assist the soccer official
18
CALL TABLE
Event
Affect
Freq
kicks or attempts to kick an opponent - in opponent's penalty area
Very High
10
10000
• trips or attempts to trip an opponent - in opponent's penalty area
Very High
10
10000
• jumps at an opponent - in opponent's penalty area
Very High
10
10000
• charges an opponent-in opponent's penalty area
Very High
10
10000
• pushes an opponent-in opponent's penalty area
Very High
10
10000
• tackles an opponent- in opponent's penalty area
Very High
10
10000
holds an opponent- in opponent's penalty area
Very High
10
10000
• handles the ball deliberately (except for the goalkeeper within his own
penalty area)- in opponent's penalty area
Very High
10
10000
Importance Value = Frequency * Effect on Outcome
19
Importance
CALL TABLE
Freq
%
Blind
Angle
33
54.76
Horizont
al Angle
of
Parallax
6
11.9
Visual
Field
27
47.62
Speed
18
38.1
Distance
18
38.1
26% 24%
Elsewhere
Penalty Area
52%
Offensive
3rd
20
DISTANCE
The Distance will be measured from the Event
Location to the Referee Location.
Referee Location
Event Location
Distance = |Referee Location – Event Location|
21
DURATION
The Duration is defined as the amount of time an
event takes to occur
Duration = Finish Time – Start Time
22
BLIND ANGLES
The Blind Angles are defined as the angles that
obstruct a system component view of the play
Blind Angle = |Obstructed View°Initial – Obstructed View°Final |
23
ANGLE OF PARALLAX
The Angle of Parallax is defined as the degree of separation
from the Optimal Viewing Angle and the Actual Viewing Angle.
Angle of Parallax = |Optimal Angle° – Actual Angle° |
24
VISUAL FIELD
The Visual Field is defined as the area viewable by a single
System Component.
Event 2
Event 1
Visual Field = |Visual Field°left – Visual Field°right |
25
SCOPE
The most important factors:
•Distance
•Blind Angles
• Angle of Parallax
• Duration
• Visual Field
Event’s Analyzed will be limited to the 42 calls with the most
impact on the game:
• Fouls
• Goals
• Play Stoppage
• Disciplinary
26
SIMULATION DESIGN
Event Function
Ball Position
Function
Ref Position
Function
Probability
of Accurate
Call
Function
∑
Total System
Accuracy
Foul 1 – 40%
Foul 2 – 60%
...
…
27
SIMULATION DESIGN
Ball Position
Function
Event
Function
Probability
of
Accurate
Call
Function
Ref Position
Function
0,0
0
1
2
0
0
0.1 0.0125
1
0.1
0.1 0.0125
2 0.0125 0.0125 0.0125
3
0
0
0
4
0
0
0
0,1
0
1
2
0
0.1
0.1 0.0125
1
0
0.1 0.0125
2
0.1
0.1 0.0125
3 0.0125 0.0125 0.0125
4
0
0
0
0,2
0
1
2
0 0.0125 0.0125 0.0125
1
0.1
0.1 0.0125
2
0
0.1 0.0125
3
0.1
0.1 0.0125
4 0.0125 0.0125 0.0125
3
0
0
0
0
0
3
0
0
0
0
0
3
0
0
0
0
0
4
0
0
0
0
0
4
0
0
0
0
0
4
0
0
0
0
0
∑
Total System
Accuracy
Foul 1 – 40%
Foul 2 – 60%
...
…
5 1,0
0
1
2
3
0 0
0.1
0
0.1 0.0125
0 1
0.1
0.1
0.1 0.0125
0 2 0.0125 0.0125 0.0125 0.0125
0 3
0
0
0
0
0 4
0
0
0
0
5 1,1
0
1
2
3
0 0
0.1
0.1
0.1 0.0125
0 1
0.1
0
0.1 0.0125
0 2
0.1
0.1
0.1 0.0125
0 3 0.0125 0.0125 0.0125 0.0125
0 4
0
0
0
0
5 1,2
0
1
2
3
0 0 0.0125 0.0125 0.0125 0.0125
0 1
0.1
0.1
0.1 0.0125
0 2
0.1
0
0.1 0.0125
0 3
0.1
0.1
0.1 0.0125
0 4 0.0125 0.0125 0.0125 0.0125
4
0
0
0
0
0
4
0
0
0
0
0
4
0
0
0
0
0
5
0
0
0
0
0
5
0
0
0
0
0
5
0
0
0
0
0
R = Random Number
ballPosition = bp
If (R == nextPositionCDF)
ballPosition = nextPosition
28
SIMULATION DESIGN
Ball Position
Function
Event
Function
Ref Position
Function
If (Random Number == specificEventTypeCDF)
EventType = specificEventType
EventDuration = Random Number(0,X)
If(Random Number == specificLocation)
Location = specific Location
return Event & Location
Probability
of
Accurate
Call
Function
∑
Total System
Accuracy
Foul 1 – 40%
Foul 2 – 60%
...
…
Freq
%
Tripping
23
54.76
Out of Bounds
4
9.52
Goal
5
11.9
Push
20
47.62
….
….
…..
…..
....
….
26%
Elsewhere
Event
24%
………...
Penalty Area
Event Type
Event Location
Event Duration
52%
Offensive 3rd
29
SIMULATION DESIGN
Ball Position
Function
Event
Function
Ref Position
Function
Probability
of
Accurate
Call
Function
∑
Foul 1 – 40%
Foul 2 – 60%
...
…
Total System
Accuracy
Ref Position
X,Y
RefPositionY = RefereeDiagonalPosition(BallPositionX)
Return BallPositionX, RefPositionY
*Assume Referee will always be in position according to graph
(No fatigue, No straying off path)
30
SIMULATION DESIGN
Ball Position
Function
Event
Function
Ref Position
Function
Probability
of
Accurate
Call
Function
∑
Foul 1 – 40%
Foul 2 – 60%
...
…
Total System
Accuracy
System Data
Location of Components (L)
# of LOS (S)
Processing Time (T)
Visual Field (V)
Accurate Range (R)
Accuracy Probability due to:
Input Data
Event Blind Angles (EB)
Event Type (ET)
Event Location (EL)
Event Duration (ED)
Referee Position
Distance + Blind Angles + Angle of Parallax
+ Duration + Visual Field
Probability that a single
event will be called
accurately
EX – Tripping Foul = 78% of
Accurate Call
31
SIMULATION DESIGN
Ball Position
Function
System Data
Location of Components (L)
# of LOS (S)
Processing Time (T)
Visual Field (V)
Accurate Range (R)
Event
Function
Probability
of
Accurate
Call
Function
Ref Position
Function
∑
Foul 1 – 40%
Foul 2 – 60%
...
…
Total System
Accuracy
Event Data
Event Blind Angles (EB)
Event Type (ET)
Event Location (EL)
Event Duration (ED)
Distance = |EL – L|
P(Accuracy)
DISTANCE
R
Distance
= P(Accuracy due
to Distance)
32
SIMULATION DESIGN
Ball Position
Function
System Data
Location of Components (L)
# of LOS (S)
Processing Time (T)
Visual Field (V)
Accurate Range (R)
Event
Function
Ref Position
Function
Probability
of
Accurate
Call
Function
∑
Foul 1 – 40%
Foul 2 – 60%
...
…
Total System
Accuracy
Event Data
Event Blind Angles (EB)
Event Type (ET)
Event Location (EL)
Event Duration (ED)
BLIND
ANGLES
Location of Components
P(Accuracy)
Event Blind Angles
Blind Angles
= P(Accuracy from
Blind Angles)
Angle of View
33
SIMULATION DESIGN
Ball Position
Function
System Data
Location of Components (L)
# of LOS (S)
Processing Time (T)
Visual Field (V)
Accurate Range (R)
Event
Function
Probability
of
Accurate
Call
Function
Ref Position
Function
∑
Foul 1 – 40%
Foul 2 – 60%
...
…
Total System
Accuracy
Event Data
Event Blind Angles (EB)
Event Type (ET)
Event Location (EL)
Event Duration (ED)
Optimal View Angle
Actual View Angle
P(Accuracy)
ANGLE OF
*will only apply to Off Sides, Out of Bounds, and Goal Events
PARALLAX
Optimal
Angle
= P(Accuracy from
Angle of Parallax)
Angle of View
34
SIMULATION DESIGN
Ball Position
Function
System Data
Location of Components (L)
# of LOS (S)
Processing Time (T)
Visual Field (V)
Accurate Range (R)
Event
Function
Ref Position
Function
Probability
of
Accurate
Call
Function
∑
Foul 1 – 40%
Foul 2 – 60%
...
…
Total System
Accuracy
Event Data
Event Blind Angles (EB)
Event Type (ET)
Event Location (EL)
Event Duration (ED)
Event Duration
Processing Time
P(Accuracy)
DURATION
T
= P(Accuracy from
Event Duration)
Event Duration
35
SIMULATION DESIGN
Ball Position
Function
System Data
Event
Function
Probability
of
Accurate
Call
Function
Ref Position
Function
∑
Foul 1 – 40%
Foul 2 – 60%
...
…
Total System
Accuracy
Event Data
Location of Components (L)
# of LOS (S)
Processing Time (T)
Visual Field (V)
Accurate Range (R)
Event Blind Angles (EB)
Event Type (ET)
Event Location (EL)
Event Duration (ED)
VISUAL FIELD
EL2
Θ
V1
V2
L1
Θ = acos(V1 ∙ V2)
P(Accuracy)
EL1
0
= P(Accuracy from
Visual Field)
V
60
180
36
SIMULATION DESIGN
Ball Position
Function
Event
Function
Ref Position
Function
Probability
of
Accurate
Call
Function
∑
Foul 1 – 40%
Foul 2 – 60%
...
…
Total System
Accuracy
PROBABILITY EQUATION
Probabilities due to
Distance + Blind Angles + Angle of Parallax + Duration + Visual Field
*Each will have different weights according to Type of Event
= P(Accurate Call
for Event)
37
SIMULATION DESIGN
Ball Position
Function
Event
Function
Ref Position
Function
Event 1 – 40%
Event 2 – 60%
Event 3 – 50%
…
…
…
AVG all Probabilities
Probability
of
Accurate
Call
Function
∑
Foul 1 – 40%
Foul 2 – 60%
...
…
Total System
Accuracy
Total System
Accuracy
38
VALIDATION
Must compare all Simulation Data against Actual Games
Sample Size needed to Achieve a C.I. of 10% with a 95% Level of Confidence:
Z2 * p(1-p) = 1.962 * (.5)(1-.5) = 96 games
(CI)2
.12
39
HOUSE OF QUALITY
Multiple
Lines of Sight
(Technology)
Change
Diagonal
Path
Blind Angle
X
Low
Vertical
Angle of
Parallax
X
X
X
Horizontal
Angle of
Parallax
X
X
X
Speed
X
Low
Low
Distance
X
Goal Only
Goal Only
Visual Field
X
HOW
WHAT
Low
Sensor
(microchip in
ball)
Global
Positioning
Satellite
(GPS)
X
Sensors &
GPS
X
40
PROJECT PLAN
41
BUDGET AS OF 12/1/10
Earned Value Management
25000
20000
15000
Actual Value (Cumulative)
Planned Value (Cumulative)
10000
Earned Value
5000
0
1
2
3
4
5
6
EV
AC
19148 18106
7
8
9
10
11
12
13
14
15
PV
CV%
SV%
CPI
SPI
21000 0.0544 -0.088 1.0575 0.9118
42
RISK MITIGATION
Risk:
• Data Collection
Mitigation Strategy:
• Allocate more resources to collecting Data
• Started Simulation Building a month ahead of schedule.
43
References
[1] N/A. (2007, July). 265 million playing football [Online]. Available: http://www.fifa.com/mm/document/fifafacts/bcoffsurv/emaga_9384_10704.pdf
[2] N/A. TV Data [Online]. Available: http://web.archive.org/web/20070922225713/http://fifa.com/aboutfifa/marketingtv/factsfigures/tvdata.html
[3] Stephen Master. U.S. Ratings Climb 68% for 2010 World Cup [Online]. Available:
http://blog.nielsen.com/nielsenwire/media_entertainment/usa-ratings-up-big-in-2010-world-cup/
[4] Alex Burmaster. Official FIFA Website Scores Biggest in Brazil [Online]. Available: http://blog.nielsen.com/nielsenwire/global/official-fifa-website-scoresbiggest-in-brazil/
[5] N/A. Laws of the Game 2010/2011 [Online]. Available:
http://www.fifa.com/mm/document/affederation/generic/81/42/36/lawsofthegame_2010_11_e.pdf
[6] Dan Wetzel. (2010, July 11). World Cup winner will cash in too [Online]. Available: http://g.sports.yahoo.com/soccer/world-cup/news/world-cup-winner-willcash-in-too--fbintl_dw-prizemoney071110.html
[7]N/A. (2010, June 9-10). FIFA Financial Report 2009 [Online]. Available:
http://www.fifa.com/mm/document/affederation/administration/01/18/31/86/fifa_fr09_en.pdf
[8] N/A. 2010 FIFA World Cup South Africa Technical Report and Statistics [Online]. Available:
http://www.fifa.com/mm/document/affederation/technicaldevp/01/29/30/95/reportwm2010_web.pdf
[9] Additional Referee Experiment Background Paper – FIFA 08-28-09
[10] N/A. 2010, May 18. Five-referee experiment to continue [Online].
Available:http://www.uefa.com/uefa/footballfirst/matchorganisation/refereeing/news/newsid=1489596.html#five+referee+trial+continues
[11] N/A. 2010, July 21. IFAB approves additional assistant referee experiment [Online]. Available:
http://www.fifa.com/aboutfifa/federation/releases/newsid=1276442.html
[12] N/A. 2009, September 17. FIFA AAR Program [Online]. Available: http://www.proreferee.com/article/757 written by ProReferee Staff; published 17 Sept
2009
[13] Sophie Freeman. 2010, June 29. FIFA President Sepp Blatter apologises over disallowed England goal and concedes it’s time to talk about technology
[Online]. Available: http://www.dailymail.co.uk/news/article-1290538/World-Cup-2010-Sepp-Blatter-apologises-concedes-time-talk-technology.html
[14] N/A. [Online]. Available: http://www.cairos.com/unternehmen/index.php
Figure 1.5; Courtesy of: (Jesse Chula; 27 Jun, 2010). Available: http://view.picapp.com/pictures.photo/entertainment/germany-goalkeeperneuer/image/9233659?term=People%253a0.714%253a%2522Frank%2bLampard%2522+frank+lampard&scomp=pis & (Sophie Freeman; 29 June 2010).
Available: http://www.dailymail.co.uk/news/article-1290538/World-Cup-2010-Sepp-Blatter-apologises-concedes-time-talk-technology.html
[15] Paul Mungra, Referee Role Overview. Private Conversation.
[16] N/A. 2010, August. Goal-line technology – Getting it right [Online]. Available: http://www.wipo.int/wipo_magazine/en/2010/04/article_0001.html
[17] Accurate and Reliable [Online]. Available: http://www.hawkeyeinnovations.co.uk/?page_id=1011
[18] Figure1.4 Courtesy of: Mandeep Sanghera. 2007, July 18. Football turns to technology [Online]. Available:
http://news.bbc.co.uk/sport2/hi/football/6903029.stm
Questions?
45
ALTERNATIVES
Alternative 1:
Multiple Lines of Sight (Human) + Multiple Lines of Sight
(Technology) around the Offensive 3rd of the field to monitor
infringements.
Alternative 2:
Multiple Lines of Sight (Technology) around the Offensive 3rd of
the field + Sensors (microchip in ball) to track ball (goals) and
player activity (fouls)
46
CALL TABLE
Frequency of Effect on Outcome
30
25
Frequency
20
15
10
5
0
Med
High
Very High
Effect on Outcome of Game
47
Backup Slide
Frequency of Affect on Outcome
80
70
60
Frequency
50
40
30
20
10
0
Low
Med
High
Affect on Outcome of Game
48
Very High
OPERATIONAL
CONCEPT
49