Relations of Ideas
are
Matters of Fact:
A Unified Theory of
Theoretical Unification
Kevin T. Kelly
Department of Philosophy
Carnegie Mellon University
kk3n@andrew.cmu.edu
Les relations d'idées
sont
des questions de fait:
Une théorie unifiée d'unification théorique
Kevin T. Kelly
Department of Philosophy
Carnegie Mellon University
kk3n@andrew.cmu.edu
I. Relations of Ideas
and
Matters of Fact
Table of Opposites
Relations of Ideas
Matters of fact
Analytic
Synthetic
Certain
Uncertain
A Priori
A Posteriori
Philosophy of Math
Philosophy of Science
Proofs and algorithms
Confirmation
Truth-finding
“Rationality”
Computability
Probability
Progress
Relations of Ideas
Matters of fact
Analytic
Synthetic
Certain
Uncertain
A Priori
A Posteriori
Philosophy of Math
Philosophy of Science
Proofs and algorithms
Confirmation
Truth-finding
“Rationality”
Computability
Probability
Progress
Certain
Uncertain
A Priori
A Posteriori
Philosophy of Math
Philosophy of Science
Proofs and algorithms
Confirmation
Truth-finding
“Rationality”
Computability
Probability
Progress
Philosophy of Math
Philosophy of Science
Certain
Uncertain
A Priori
A Posteriori
Proofs and algorithms
Confirmation
Truth-finding
“Rationality”
Computability
Probability
Old Wine in New Bottles?
Philosophy of Math
Philosophy of Science
Certain
Uncertain
A Priori
A Posteriori
Proofs and algorithms
Confirmation
Truth-finding
“Rationality”
Computability
Probability
Philosophy of Science 101
Formal Empirical
Verifiable
Unverifiable
The given
number is
even.
The sun
will never
stop rising.
Wrong Twice!
Formal Empirical
Verifiable
Unverifiable
The given
number is
even.
The current
emerald is
green.
The given
The sun
computation will never
will never
stop rising.
halt.
Aristotelian Excuse
Relations of Ideas
Completed analysis
Rational
Rational ?
Reason
Matters of fact
Unbounded
experience
Animal
??
Always black?
Observation
Leibnizian Doubts
Relations of Ideas
Unbounded
experience
Unbounded
analysis
??
Matters of fact
Rational ?
Reason
??
Always
black?
Never white?
Observation
Bounded Kantian Synthesis
Synthetic A Priori
3
3
2
4
bounded
synthesis
3+2 = 5 ?
Pure intuition
A Posteriori
Unbounded
experience
5
??
Always
black?
Never white?
Observation
Unbounded Kantian Synthesis
Synthetic A Priori
Unbounded
Synthesis?
??
Never cross?
Pure intuition
A Posteriori
Unbounded
experience
??
Always
black?
Never white?
Observation
Kantian Antinomies
Synthetic A Priori
Physical
continuum
Geometrical
continuum
??
A Posteriori
Infinite?
Dense?
Beyond all pure intuition
??
Infinite?
Never
white?
Dense?
Beyond all possible observation
Gödel and Turing
Formal Problems
Unbounded
Input
computation
Empirical Problems
Unbounded
reality
n
??
Never halts?
Computer
??
Always
black?
Never white?
Observation
Wrong First Cut
Formal Empirical
Verifiable
Unverifiable
The given
number is
even.
The current
emerald is
green.
The given
The sun
computation will never
will never
stop rising.
halt.
Right First Cut
Verifiable
Unverifiable
Formal
Empirical
The given
number is
even.
The current
emerald is
green.
The given
The sun
computation will never
will never
stop rising.
halt.
Unified Epistemology
For both formal and empirical questions:
Justification = truth-finding performance
Find the truth in the best possible sense,
 as efficiently as possible.

II. Verification
Verification as
Halting with the Truth
Yes!
Conclusion
Verification as
Halting with the Truth
Maybe you
will change your
mind.
Conclusion
Verification as
Halting with the Truth
Conclusion
Verification as
Halting with the Truth
Conclusion
Verification as
Halting with the Truth
Conclusion
Verifiability
Yes!
Conclusion true
Conclusion false
Classical Problem of Induction
I won’t present blue
until he says yes!
Empirical demon
Always green?
Empirical scientist
Classical Problem of Induction
Always green?
Classical Problem of Induction
Always green?
Classical Problem of Induction
Always green?
Classical Problem of Induction
You lose if you
never halt
with “yes”!
Always green?
Classical Problem of Induction
Always green?
Classical Problem of Induction
Yes!
Always green?
Classical Problem of Induction
Always green?
Classical Problem of Induction
Always green?
Classical Problem of Induction
You lose
anyway.
Always green?
“Problem” of Uncomputability
Never halts?
Formal demon
Formal scientist
Similar to the Problem of Induction?
Maybe I am in an infinite loop.
Or maybe I will halt later…
?
Never halts?
Apparently Not Similar
I see your program!
Never halts?
666
Apparently Not Similar
Now I can cleverly calculate a priori
what you will do!
Never halts?
666
Apparently Not Similar
I’m beaten!
666
I’m beaten!
Never halts?
shoots?
666
21456
chug
chug
chug
But Maybe Similar After All…
Aha! His program is
also showing!
Aha! His program is
also showing!
Never halts?
shoots?
666
666
23455
21456
But Maybe Similar After All…
I won’t
halt until he halts
with “yes” in my
a priori simulation of
him looking at me!
I won’t
halt until he halts
with “yes” in my
a priori simulation of
him looking at me!
I won’t
halt until he halts
with “yes” in my
a priori simulation of
him looking at me!
Never halts?
shoots?
666
666
23455
21456
666
23455
21456
But Maybe Similar After All…
I won’t
halt until he halts
with “yes” in my
a priori simulation of
him looking at me!
I won’t
halt until he halts
with “yes” in my
a priori simulation of
him looking at me!
I won’t
halt until he halts
with “yes” in my
a priori simulation of
him looking at me!
666
666
23455
21456
Self-reference by Kleene fixed
point theorem
Never halts?
shoots?
666
23455
21456
But Maybe Similar After All…
I didn’t halt…
666
I didn’t halt…
23455
21456
666
I didn’t halt…
Never halts?
shoots?
666
23455
21456
But Maybe Similar After All…
I still didn’t halt…
666
I still didn’t halt…
23455
21456
666
I still didn’t halt…
Never halts?
shoots?
666
23455
21456
But Maybe Similar After All…
666
I give up! You’ll never halt!
666
23455
21456
I give up! You’ll never halt!
Never halts?
shoots?
666
23455
21456
But Maybe Similar After All…
The
End
The
End
23455
Never halts?
shoots?
666
23455
But Maybe Similar After All…
Never halts?
shoots?
666
But Maybe Similar After All…
666
Sucker!
Never halts?
shoots?
666
But Maybe Similar After All…
The
End
Never halts?
shoots?
666
But Maybe Similar After All…
Never halts?
shoots?
But Maybe Similar After All…
Never halts?
shoots?
Rice-Shapiro Theorem
Whatever a computable cognitive scientist
could verify about an arbitrary computer’s
input-output behavior by formally analyzing
its program…
666
Q
cog-sci
Rice-Shapiro Theorem
…could have been verified by a behaviorist
computer who empirically studies only the
computer’s input-output behavior.
666
Q
cog-sci
Analogy
Universal Formal
Problem
Universal Empirical
Problem
Unverifiable
Unverifiable
Computable demon can
fool computable agent
Empirical demon can
fool ideal agent
Answer runs beyond
empirical experience
Answer runs beyond
formal experience
Apparent Disanalogy
Universal Formal
Problem
Universal Empirical
Problem
Input given
Input never fully given
Response: Digested vs. Given
Universal Formal
Problem
Universal Empirical
Problem
Input given
Input never fully given
Input never fully digested Input never fully digested
Infinitely slow-melting candy
Infinitely long noodle
Another Apparent Disanalogy
Formal Science
Empirical Science
Incompleteness
Error
Focus on Consistency
Formal Science
Empirical Science
Incompleteness
Error
Add axioms
Conjecture a theory
Who knows if they are
Who knows if it is
consistent with low-level consistent with all future
combinatorics?
empirical data?
Assume they are and keep Assume it is and keep
checking
checking
Thesis
The problem of induction and the
problem of uncomputability are
similar.
So the problems should be
understood similarly.
A House Divided
•Philosophy of Science
•“Problem” of induction demands a “solution”.
•“Confirmation” and “Bayesian rationality”
•No clear connection with truth.
•Philosophy of Math
•Uncomputability is a fact, not a “problem”.
•Logic and algorithms
•Insistence on connection with truth.
III. Beyond Verifiability
Drop the Halting Requirement
•Ought implies can.
•If you can’t halt with the truth, find the truth in the
best feasible sense.
Second amendment!
Gun control!
Q
(W. James, H. Putnam, E.M. Gold, R. Jeroslow, P. Hájek)
Focus on Questions
Answers partition the set of possible inputs
A
Finite input
B
C
D
Infinite input
Convergence Without Halting
No possible opinion change after halting
With halting
1
0
Halts
At most finitely many mind-changes
In the limit
1
0
Ever-closer to full confidence
Gradual
1
0
Success
Domain over which convergence
to the truth is required
A
B
C
D
Otherwise, non-convergence to
falsehood is required.
Success Over Binary Partitions
Verify A
A
infallibly
in the limit
gradually
Refute A
B
A
Decide A
A
infallibly
in the limit
gradually
B
infallibly
in the limit
gradually
B
Success Over Countable Partitions
Compute partial f
Formal
case
f
f=0
f=1
infallibly
in the limit
gradually
f=2
...
Dom(f)
infallibly
in the limit
gradually
Theory
selection
Empirical
case
Catch-all
T0
T1
T2
...
Retractions
•A retraction occurs when an answer is “taken back”.
•Retractions are what halting is supposed to prevent.
•When you can’t prevent all retractions, minimize them.
•Halting condition, when feasible, is special case.
2 retractions
1
0
Retraction Efficiency
Retraction-efficiency = success in the limit under the
least feasible retraction bound in each answer.
5
6
A
B
Retraction Efficiency
1
0
0
1
A
B
A
Halting-verifiable =
1-verifiable
B
Halting-refutable =
1-refutable
0
0
A
B
Halting-decidable =
0-decidable
Kantian Antinomies Revisited
Infinitely Finitely
divisible divisible
Matter
Upper Complexity Bound
Lim-ref
Infinitely Finitely
divisible divisible
Lim-ver
I say infinitely divisible
when you let me cut.
Cut allowed by demon
...
...
Answer produced by scientist
Lower Complexity Bound
Lim-ref
-Lim-ver
Infinitely Finitely
divisible divisible
I let you cut
when you say finitely divisible.
Lim-ver
-Lim-ref
Purely Formal Analogue
Lim-ref
-Lim-ver
Infinite
domain
Finite
domain
666
Lim-ver
-Lim-ref
23455
21456
Purely Formal Analogue
Lim-ref
-Lim-ver
Infinite
domain
Finite
domain
Lim-ver
-Lim-ref
I halt on another input each
time he says finitely divisible in
my a priori simulation of him
looking at my program.
666
23455
21456
Mixed Example: Penrose
Computable
Uncomputable
???
.
Undergraduate subject
Cognitive scientist
Mixed Example: Penrose
Computable
Uncomputable
You can verify human computability
in the limit…
NCNCCCCCCC…
Undergraduate subject
Cognitive scientist
Mixed Example: Penrose
Computable
Uncomputable
…but if humans are computers, then you are a computer,
in which case you can’t verify in the limit
that humans are computers.
CNCNCCNCCN…
Undergraduate subject
Cognitive scientist
Formal Complexity
n
23455
21456
...
S3
S2
S1
P3
D3
D2
D1
P2
P1
Arithmetical
Hierarchy
Formal Complexity
n
23455
21456
...
grad ref
S3
P3 grad ver
D3
S2
P2 lim ref
lim ver
grad dec D lim dec
2
halt ver
S1
P1 halt ref
D1 halt dec
Arithmetical
Hierarchy
Formal Complexity
n
grad ref
lim ver
23455
21456
S2
grad dec
3-ver
P3 grad ver
D3
halt-ver
1-ver
S3
S2
1-dec D
2
S1
0-dec
lim ref
D2 halt ref
2-dec D
3
2-ver
P2
...
Putnam’s
n-trial
predicates
S3
D1
P3
3-ref
P2
2-ref
halt-ref
P1
1-ref
halt-dec
Arithmetical
Hierarchy
Empirical Complexity
grad ref
lim ver
S3
S2
3-ver
Difference
hierarchy
halt-ver
1-ver
S2
1-dec
S1
0-dec
P2
lim ref
D2 halt ref
S3
2-dec
2-ver
D3
...
grad dec
P3 grad ver
D3
D2
D1
P3
3-ref
P2
2-ref
halt-ref
P1
1-ref
halt-dec
Borel
hierarchy
Joint Complexity
grad ref
lim ver
23455
21456
S2
halt-ver
1-ver
D3
S2
1-dec
S1
0-dec
P2
lim ref
D2 halt ref
S3
2-dec
2-ver
P3 grad ver
...
grad dec
3-ver
Effective
difference
hierarchy
S3
D3
D2
D1
P3
3-ref
P2
2-ref
halt-ref
P1
1-ref
halt-dec
Effective
Borel
Hierarchy
Wrong First Cut in Philosophy
Empirical
...
...
...
?
?
Joint
23455
21456
Formal
Right First Cut in Philosophy
...
...
...
Empirical
Joint
Formal
23455
21456
How Complexity Inverts Skepticism
•Justification is best possible truth-finding performance.
•To establish best possible performance, one must show that
better senses of success are infeasible.
•Therefore, skeptical arguments justify induction!
Impossible
performance
...
Best possible
performance
IV. Bounded Rationality
To be empirically ignorant is human.
To be logically ignorant is irrational.
?
Bayesian
Philosophy of Science 101
•“Science works by deducing theoretical predictions
and comparing them against the empirical data”.
“Scientific Method”
Empirical data
t
t
No
n
Theorem
Prover
Q
m
What if Predictions are NonComputable?
“Scientific Method”
Empirical data
t
Poof!
No
n
Q
Computable Refutation with Halting
Can Still be Possible!
“Scientific Method”
Empirical data
?
?
n
t
Q
No
Non-computable Predictions
(with Oliver Schulte)
h
Everything
but h
There exists predictive hypothesis {h} such that:
•h is infinitely non-computable;
•But a computer can refute hypothesis {h} in the
halting sense!
Inductive Complexity : Deductive Complexity
::
Implicit Definability : Explicit Definability
S2
S1
...
...
S3
P3
D3
D2
D1
S3
P2
P1 {h}
S2
Typical case
Learning complexity of {h}
= joint complexity of {h}.
h S1
P3
D3
D2
D1
P2
P1
Deductive complexity of {h}
= formal complexity of h.
Singleton Non-Basis Theorem
h
There exists h such that:
S2
S1
...
...
S3
P3
D3
D2
D1
P2
P1 {h}
{h} is P1
S3
S2
S1
P3
D3
D2
D1
P2
P1
h is non-arithmetical
h is the mu-defined Skolem function for the Delta-2 implicit definition of arithmetical truth (Hilbert and Bernays).
Cor. 1: A Foolish Consistency
Q
Computable
scientist
Inconsistency
Bayesian
Computable
demon
•Consistent computers can’t verify or refute {h} even gradually!
•So computable truth-seekers should be inconsistent.
Cor. 2: No Firewall
•No computer architecture that insulates theorem proving
from future empirical data can gradually verify or refute {h}.
“Scientific Method”
Empirical data
n
t
t
Theorem
Prover
No
Q
m
Cor. 2: No Firewall
•But a computer that has a time lag before noticing
inconsistency can refute {h} with halting.
“Improved Scientific Method”
Empirical data
Theorem
Prover
No
n
Q
m
V. Unification Unified
Which Explanation is Right?
???
Ockham Says:
Choose the
Simplest!
But Maybe…
Gotcha!
Puzzle

A reliable indicator must be sensitive to what
it indicates.
simple
Puzzle

A reliable indicator must be sensitive to what
it indicates.
complex
Puzzle

But Ockham’s razor always points at
simplicity.
simple
Puzzle

But Ockham’s razor always points at
simplicity.
complex
Puzzle

How can a broken compass help you find
something unless you already know where it
is?
complex
Bayesian Explanation



Ignorance (over simple vs. complex)
+ Ignorance (over parameter settings)
= Knowledge (that simple data can’t be produced by
the complex theory).
Simple
q
q
q
q
q
q
q
q
Complex
= The Old Paradox of Indifference



Ignorance (over blue, non-blue)
+ Ignorance (over ways of being non-blue)
= Knowledge (that the truth is blue as opposed to
any other color)
Blue
Bayesian
Not-Blue
= The Old Paradox of Indifference



Ignorance (over blue, non-blue)
+ Ignorance (over ways of being non-blue)
= Knowledge (that the truth is blue as opposed to
any other color)
Blue
Bayesian
Not-Blue
And how does he explain
formal simplicity?
Better Idea: Retraction Efficiency
Truth
Better Idea: Retraction Efficiency
Truth
Curve Fitting


Data = arbitrarily precise intervals around Y at
specified values of X.
Question: assuming that the truth is polynomial,
what is the degree?
Curve Fitting

Demon shows flat line until convergent method
concludes flat.
Zero degree curve
Curve Fitting

Demon shows flat line until convergent method
concludes flat.
Zero degree curve
Curve Fitting

Then switches to tilted line until convergent
method concludes linear.
First degree curve
Curve Fitting

Then switches to parabola until convergent
method concludes quadratic …
Second degree curve
Simplicity
The current simplicity of answer H =
the number of distinct answers the demon can force an
arbitrary convergent method to produce before
producing H.
Game-theoretic, topological invariant.
Cubic
Linear
Constant
0
Quadratic
1
2
In Step with the Demon
There yet?
Maybe.
Cubic
Linear
Constant
Quadratic
In Step with the Demon
There yet?
Maybe.
Cubic
Linear
Constant
Quadratic
In Step with the Demon
There yet?
Maybe.
Cubic
Linear
Constant
Quadratic
In Step with the Demon
There yet?
Maybe.
Cubic
Linear
Constant
Quadratic
Ahead of the Demon
There yet?
Maybe.
Cubic
Linear
Constant
Quadratic
Ahead of the Demon
I know you’re coming!
Cubic
Linear
Constant
Quadratic
Ahead of the Demon
Maybe.
Cubic
Linear
Constant
Quadratic
Ahead of the Demon
!!!
Hmm, it’s quite nice here…
Cubic
Linear
Constant
Quadratic
Ahead of the Demon
You’re back!
Learned your lesson?
Cubic
Linear
Constant
Quadratic
Ockham Violator’s Path
Cubic
Linear
Constant
Quadratic
Ockham Path
So Ockham is uniquely
retraction efficient!
Cubic
Linear
Constant
Quadratic
Same Argument Works For
•Fewer causes
•Unified causes
•More Conserved quantities
•Hidden particles
•Fewer free parameters
Formal Curve Fitting


M is given the index of a computable
polynomial function on the reals.
M must converge to that function’s polynomial
degree.
Finite stage of effective
decimal expansion
M
“Computational Experience”

M also has to refute each polynomial degree with
a special halting state for that degree.
Finite stage of effective
decimal expansion
Definitely not
quadratic!
Halting state 2
M
Formal Curve-Fitting Demon
For j = k to n-1:
act like a polynomial of degree j until:
M refutes degree k-1 and
M concludes that I am a polynomial of degree j;
set j = j+1;
act like a polynomial of degree n.
Polynomial degree?
21456
M
D(M, k, n)
Formal Ockham’s Razor
•M should not conclude that D(M, k, n) has degree
> k when M’s experience has refuted at most
degree k.
•If M is Ockham, M retracts at most n - j times on
D(M, k, n) on experience refuting degree < k.
•Else, M retracts at least n – j + 1 times.
Q
Formal Causal Inference
•Given an index of a computable joint probability
distribution whose conditional independence relations are
represented by a causal network, compute the equivalence
class of causal networks representing the distribution.
•N-choose-2 retractions required.
•Empirical complexity = # of edges in graph
•Ockham’s razor = assume no more causes than
necessary!
X
Causal network?
Y
W
Z
W
Formal Causal Inference
•Given an index of a computable joint probability
distribution whose conditional independence relations are
represented by a causal network, compute the equivalence
class of causal networks representing the distribution.
•N-choose-2 retractions required.
•Empirical complexity = # of edges in graph
•Ockham’s razor = assume no more causes than
necessary!
X
Causal network?
Y
W
Z
W
Unified Account of Unification
•When halting is dropped as a condition for success, a
penchant for elegance is truth-conducive (retractionefficient) in both formal and empirical inquiry.
Q
Discussion
1. Uncomputability is similar to the problem of
induction.
2. In both cases, halting with the truth is infeasible.
3. Without the halting requirement, skeptical arguments
justify inductive reasoning by establishing efficiency.
4. Truth-finding efficiency explains Ockham’s razor
both in empirical and formal reasoning.
5. Insistence on separation of formal and empirical
reasoning restricts the power of computable science.
Q
THE END