Amplifying the Wisdom of the Crowd,
Building and Measuring Expert
Consensus for Nonproliferation Issues
James Carroll
INMM Taos Technical Meeting,
May 31, 2012
LA-UR-12-21784
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Abstract
Scientific and cultural progress mostly takes place in the marketplace of
ideas. The internet has largely changed how this marketplace functions.
Although it has brought many improvements, it has also created a new set of
problems. We propose a web service that can be used to crowdsource the
process of knowledge acquisition and of knowledge summary, mediate
disagreements, and improve discourse on controversial issues, thereby
improving the efficiency of the marketplace of ideas. Our approach involves a
specific technique for integrating: a forum; a wiki organized into separate
camps, with camps organized into a hierarchical structure; a survey system
based upon camp support; a mechanism for dynamically reorganizing the
structure of the camp hierarchy while mediating user disagreements; and a
customizable mechanism for determining how votes are weighted based on
credentials or expert assignment. A version of this approach has been
implemented, released as open source, and a beta test has gone live. Initial
use seems to validate the merits of this approach. (U)
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 2
Outline
1.
Epistemology and the Marketplace of Ideas
2.
Inefficiencies in the Marketplace of Ideas
(In the internet age)
3.
A Potential Partial Solution
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 3
1. Epistemology and the
Marketplace of Ideas
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 4
Logic, the quest for truth:
Socrates
Plato
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Aristotle
Slide 5
Logic, the quest for truth:
Socrates
Plato
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Aristotle
Slide 6
The Destruction of Objectivity

David Hume (1711-1776)

Kurt Gödel (1906-1978)

The Problem of Induction


a question of how things behave
when they go “beyond the
present testimony of the senses,
and the records of our memory”
[Hume1748].

The problem of Generalization
(no-free-lunch).
for any selfconsistent recursive axiomatic
system powerful enough to
describe the arithmetic of
the natural numbers (for
example Peano arithmetic),
there are true propositions
about the naturals that cannot
be proved from the axioms.
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 7
No Free Lunch

Thomas Mitchell (The Futility of Bias Free Learning):
•

David Wolpert (The Supervised Learning No Free Lunch Theorem):
•

There are as many hypotheses consistent with past data that predict one outcome
as that predict another off training set (when generalizing).
When all hypotheses are equally likely, there is no “best” supervised learning
algorithm off training set, for the 0-1 loss function.
James Carroll (No-Free-Lunch and Bayesian Optimality):
•
When all hypotheses are equally likely, then the “best” learning algorithm will report
that all outputs are equally likely off training set.
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 8
Empirical Falsifiable

You can’t prove which theory is true, so stop
trying.

Theories should be used that are falsifiable, but
which have not yet been falsified after
concerted effort.

Such theories will be useful.
Karl Popper (1902-1994)
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 9
The Marketplace of Ideas
“Ideas that can get themselves
accepted in a competitive market of
ideas will tend to be of better quality
than other ideas. The marketplace of
ideas therefore improves the quality of
our ideas and our thinking. If different
ideas are presented in an “ideas-market”, and if that market is populated by a
maximum number of free agents expressing themselves freely, then those
competing ideas will be exposed to a maximum number of supporting and
dissenting arguments, and the balance of arguments in favor of or against an
idea will be compared to the same balance for counter-ideas. The idea with
the best balance will “survive”, because alternative ideas will be seen as
comparatively defective, given the fact that the arguments in favor of them
are weaker or the arguments against them are stronger.”
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 10
The Marketplace of Ideas


This doesn’t just lead to ideas that are true, but to hypotheses that are
useful, in the sense of making the most general predictions about the
most things, with the most accuracy.
These survive in the marketplace.
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 11
Another Rout to the Marketplace:




Rout 2… Induction…. Occam’s razor solves the problem of inference.
We compute p(h|data)….
But we didn’t personally collect all the data, ourselves
Nor can we all do the complex inference for every problem
There are more problems you could be an expert on than there are
seconds in your lifetime.

SO… we have to specialize and trade.

But some people lie, so…

We have to determine who to trust, and what data to trust, and
ultimately what to believe.

Result: The marketplace of ideas.
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 12
I, Pencil, My Family Tree as told to Leonard E. Read
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 13
Collective Intelligence
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 14
Inefficiencies in the Marketplace of Ideas:
1)
the limited access to information,
2)
the un-manageable volume of information,
3)
the reliability of information,
4)
the need to determine who to consider an expert,
5)
the need to determine what the experts believe,
6)
the difficulty in changing popular notions that are resistant to change,
7)
the incivility, repetition, and lack of focus in discourse, and
8)
the difficulty in overcoming confirmation bias.
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 15
Current Cultural and Internet Mechanisms to Improve
Marketplace Functionality:
1.
Search Engines
2.
Peer Review
3.
Wikis
4.
Forums, Bulletin Boards, and Email Lists
5.
Survey Systems
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 16
The Canonizer Solution

A wiki to leverage the wisdom of the crowd for knowledge collection and
summary
•
•
Wiki pages are organized into camps with differing opinions, instead of single
pages for each topic
Camps are organized into a hierarchical structure to encourage agreement

A mechanism for dynamically reorganizing the structure of the camp
hierarchy while mediating user disagreements to encourage agreement and
highlight areas of continuing disagreement

A survey system based upon camp support for determining consensus, and

A mechanism for customizing how votes are weighted based on credentials
or expert assignment to allow users to explore different demographic effects,
or to allow users to give extra weight to experts in different fields if they so
choose.

A forum which also functions as an email list for discussion
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 17
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 18
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 19
Weighting Votes (hierarchy of theories of
consciousness):
Property
Dualism
Theories of Mind
and
Consciousness
Representational
Qualia Theory
Representational
Functionalism
Approachable
Via Science
Ideal Monism
Mis
Mis
Mis
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 20
Weighting Votes (1 person 1 vote)
Property Dualism
(20)
Theories of Mind
and
Consciousness
(39)
Representational
Qualia Theory
(26.25)
Representational
Functionalism
(4.75)
Approachable Via
Science (36)
Ideal Monism
(2.25)
Mis
(1.5)
Mis (3)
Mis (7.5)
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 21
Weighting Votes (PhDs)
Property Dualism
(1)
Representational
Qualia Theory (2)
Representational
Functionalism
(1)
Approachable Via
Science (3)
Ideal Monism (0)
Mis
(0)
Mis (3)
Mis (1)
Theories of Mind
and
Consciousness (3)
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 22
Weighting Votes (“expert” philosophers of mind)
Property Dualism
(9.46)
Theories of Mind
and
Consciousness
(10.41)
Representational
Qualia Theory
(10.41)
Representational
Functionalism
(0.94)
Approachable Via
Science (10.41)
Ideal Monism (0)
Mis
(0.01)
Mis (0)
Mis (0)
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 23
Weighting Votes (“expert” computer scientists)
Property Dualism
(0)
Theories of Mind
and
Consciousness
(1.65)
Representational
Qualia Theory
(1.65)
Representational
Functionalism
(1.65)
Approachable Via
Science (1.65)
Ideal Monism (0)
Mis
(0.00)
Mis (0)
Mis (0)
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 24
Deliverables/Results
1.
A history of the debate and conversation
2.
A hierarchy of opinions, driven towards consensus and points of
mutual agreement
3.
A single page for each “camp” summarizing the arguments for and
against each position.
4.
A survey of participants, showing which view has the most support
5.
A mechanism for varying how the votes are weighted to determine who
believes what
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 25
References




Hume, D., 1747, “An Enquiry Concerning Human Understanding,” p. 108
Thomas Mitchell, 1980, “The Need for Biases in Learning Generalizations”,
Rutgers Computer Science.
David Wolpert, 2001, “The Supervised Learning No Free Lunch Theorem”
NASA Ames Research Center.
James Carroll, 2007, “No-Free-Lunch and Bayesian Optimality” in MetaLearning IJCNN Workshop 2007.
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 26