Group Consensus Support
Helping Clarify Group Opinion
by Visualizing the Matrix of
Judges, Projects, and Scores
David I. Feinstein, Ph.D.
www.AnalyticAnimations.com
dif@alumni.caltech.edu
Group consensus with partial information
has even merited the attention of evolution!
Honey bees, biologists recently figured out,
prefer quorums. In late spring, colonies divide, with
the queen and half the workers leaving the old hive.
The swarm forms a cluster outside its old home and
goes about finding new digs. But the queen doesn't
choose. Instead, the bees engage in what biologist
Thomas Seeley of Cornell University calls "a
plebiscite, where once you have a quorum in favor of
one site it wins.“
. . . "The beauty of this process is that quorum
sensing results in selection of a great site
even though no one scout knows all the
alternatives out there“
When 10,000 honeybees fly the coop to hunt for a new home, they
have a unique method of deciding which site is right: With great
efficiency they narrow down the options and minimize bad decisions.
Their technique, includes coalition building until a quorum develops.
© 2009 David I. Feinstein
From the Database of Scores to a
Graphic Display of Scores & Quartiles
• top quartile
°
second quartile
°
third quartile
94
93
90
90
ET007
ET042
ET014
ET003
ET055
ET005
ET053
ET051
96
91
82
78
75
67
61
47
ET028
ET042
ET036
ET020
ET014
ET045
92
77
76
76
72
69
68
62
ET020
ET042
ET056
ET016
ET034
ET024
ET032
ET057
93
92
87
78
78
75
72
71
ET037
ET042
ET045
ET005
ET052
ET007
ET053
ET024
ET042
ET057
ET003
ET005
ET016
99
99
98
ET042
ET034
ET055
ET003
ET014
ET053
ET051
ET007
93
90
89
88
85
48
85
82
75
73
48
• bottom
ET042quartile
J55896
J55077
ET020 88
J56064
ET005 78
J55459
ET003
70
J56182
ET014
65
J56106
ET042
57
J56169
ET057
57
J55685
ET051
ET032
48
40
ET055
ET014
ET042
ET053
ET057
ET003
ET032
ET051
93
91
82
82
72
69
61
51
95
92
92
91
90
82
82
57
Score Display with Quartile Rankings
• top quartile
° second quartile
° third quartile
• bottom quartile
ISEF 2007
category ET
sorted by
average score
Discrepant
quartiles are
important!
these mark
judges with
something
to say
© 2009 David I. Feinstein
Beyond Average Scores:
Comparing All Scores of a Project Pair
second project
How often does
a score from
the first project
beat a score
from the second
project?
first project
36.7% 100 96
93
91
81
72
62 86.0
92
0
0
0
0
1
1
1
1
91
0
0
0
0
0.5
1
1
1
90
0
0
0
0
0
1
1
1
89
0
0
0
0
0
1
1
1
88
0
0
0
0
0
1
1
1
84
0
0
0
0
0
1
1
1
79
0
0
0
0
0
0
1
1
78
0
0
0
0
0
0
1
1
86.4
© 2009 David I. Feinstein
93
How much would the “all scores”
comparison change if we reran the
science fair with a new assignment
of judges to projects?
Simulate these new assignments by
resampling the projects’ scores:
92
0
0
92
0
0
91
0
0
42.2% 96 91 93
92
0
91
0
89
91
0
88
0
0
0
88
88
0
91
91
81
72
62 85.1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
0.5
91 0.572 1
621
0.5 0.5 0.5
0.5
930 91
0
0
0
91
0
0.5 0.5 0.5
1
0
0
1
0
1
0
0
0 0 0.5
0.5
0
0
0
1
0
00
0
1
1
1
62
82.5
1
1
1
1
1
1
11
1
1
1
11
11
1
1
00
00
0 96 0 910
81
0
81
1
172 162
92 84 0 0 0 0 0 0 10
10
1
1
11
11
92 79 0 0 0 0 0 0 10
10
1
1
1
1
0
1
1
1
0
1
1
1
45.3%84100 0
92
79
0
78
90
0
90.3
96
0
0
0
0
0
0
1
0
0
0
1
1
1
1
1
1
0
0
0
0
1
1
1
1
90
0
0
0
0
1
1
1
1
79
0
0
0
0
0
0
1
1
78
0
0
0
0
0
0
1
1
78
0
0
0
0
0
0
1
1
86.4
84.4
84.9
39.1
a 
b
 
c 
a 
a 
 
b
1
19.5
25.0
46.1
70.3
21.9
a
c
 
c
a 
b
 
c 
b
b
 
b
the first project
beats the second in
2000 of 10,000
rerun science fairs
48.4
“All Scores” comparison
53.9% 100 93
list of resampled comparisons
Significance of
“All Scores”
Comparison
0.8
0.6
0.4
0.2
53.9
0
42.2
45.3
0
3
210
3
410
3
610
3
810
4
110
position in sorted list of resampled comparisons
© 2009 David I. Feinstein
2nd Display: Project
Comparison Matrix
project 007
95
93
100 97
96
95
93
98
96
96
94
90
90
100
96
96
95
95
93
0.5
1
1
1
1
1
0 0.5 0.5
1
1
1
0 0.5 0.5
1
1
1
0
0
0 0.5 0.5
1
0
0
0 0.5 0.5
1
0
0
0
0
0 0.5
0.5
0
0
0
0
0
1
1
1
1
1
1
1
1
1
0 0.5 0.5
0 0.5 0.5
0
0
0
0
0
0
0
0
0
1
1
1
1
1
0
1
1
1
1
1
1
1
1
1
1
1
1
100
97
96
95
93
87
0.5
1
1
1
1
1
0
1
1
1
1
1
0 0.5 0.5
1
1
1
0
0
0 0.5 0.5
1
0
0
0
0
0 0.5
0
0
0
0
0
0
0.5
1
1
1
1
1
0 0.5
1
1
1
1
0
0 0.5
1
1
1
0
0
0 0.5
1
1
0
0
0
0 0.5
1
0
0
0
0
0 0.5
1
1
1
0
1
1
0 0.5 0.5
0
0
0
0
0
0
0
0
0
1
1
1
1
0
0
1
1
1
1
1
0
1
1
1
1
1
0
98
96
96
94
90
90
0
1
1
0 0.5 0.5
0 0.5 0.5
0
0
0
0
0
0
0
0
0
1
1
1
0
0
0
1
1
1
0
0
0
1
1
1
1
0
0
0
0
0
0
0
0
1
1
1
1
0 0.5
1
1
0 0.5
1
1
0
0 0.5
1
0
0 0.5
1
0
0
0 0.5
87
project 007
95
project 004
96
project 060
project 060
100 96
project 004
1
1
0 0.5
0 0.5
0
0
0
0
0
0
1
1
1
0
0
0
1
1
1
1
0
0
1
1
1
1
1
1
0.5
1
1
1
1
1
0 0.5 0.5
1
1
1
0 0.5 0.5
1
1
1
0
0
0 0.5
1
1
0
0
0
0 0.5 0.5
0
0
0
0 0.5 0.5
Ordering by average score occasionally
yields a consistent comparison matrix
Numbers indicate
the percentage of
rerun science fairs
in which the project
owning the row
beats the project
owning the column
by the all–scores
comparison
measure.
© 2009 David I. Feinstein
Comparison matrix anticipates ISEF
2007 MI category award assignments
start: projects
ordered by average
rank
© 2009 David I. Feinstein
end: projects ordered
by award level
ISEF 2007 ET: comparison matrix
spots 007!
© 2009 David I. Feinstein
Group Consensus Support
Project Score Display
quartile
markers
show scores
& gauge their
significance
Project Comparison Matrix
binary comparison outcomes
of all pairs of projects
departures from triangular structure highlight
projects meriting judges’ attention
discrepant
quartiles
mark judges
with
something
to say
© 2009 David I. Feinstein