>> Ken McMillan: Good morning. It is our very good fortune to have with us today a pioneer in
descriptive complexity, Neil Immerman. In fact, Neil was the winner of the Gödel prize for resolving a
very important conjecture related to descriptive complexity—or at least, I understand …
>> Neil Immerman: That’s it, that’s it.
>> Ken McMillan: … the result came through descriptive complexity, yep. And so he is going to give a
survey today on descriptive complexity and some of the issues that he’s currently dealing with. So thank
you, Neil.
>> Neil Immerman: Okay. So thank you very much. It’s a real pleasure to be here, and as Ken said, I’m
gonna give a survey of things I been thinking about forever, and some new directions, and work other
people have done in this area, and what I think about it. And I really enjoy being interrupted with
questions, just to make sure you’re connecting with what I’m saying; otherwise, this can seem kind of
abstract. So all questions are appreciated. So I’ll try to give a survey about descriptive complexity and
some new trends; there’s something called dichotomy, which is a very interesting phenomena that I’ll
talk about in complexity; I’ll talk about dynamic complexity; I’ll talk—just a tiny bit—about SAT solvers;
and thoughts—which are gonna be naïve thoughts on my point—of what I think about current software
and how logic ought to—in my opinion—be able to help. And so that’s … and all of this is personal
perspective, and you can take what I say with a grain of salt, or challenge me would be more fun and …
okay. So I hope you enjoy it.
So basically, I’ve been looking at this picture of the world for quite a while. So I just want to explain this
diagram that you find on my webpage all the time. And starting with polynomial time, which in this
picture, is this horizontal line—everything down—and polynomial time is the set of all properties … in
complexity theory, as I’ll talk about a little bit more, we look at … when you think of a computational
problem, you have an input of certain number of bits, n—N and little n is always the size of my
problem—currently, n is somewhere between—you know—a thousand and a billion or some … you
know, or maybe a trillion by now, and the size of the problem that we have to deal with … and that …
and everything else is a function of that. Now those … and then we … we’re gonna computing a
sequence of bits; each particular bit is a decision problem about the input. So if I want to know how
hard this computational task is, I can reduce it to single bits. So everything is decision problem; I want to
know: given an input, is the bit one—in which case, the input has the property—or is the bit zero? Okay,
so those properties that can be checked by a computational device—a standard computer or a turn
machine—in some polynomial amount of time is called P, and it’s a very nice—one would say—
mathematical wrapper for all the truly feasible problems. So in this diagram, I have something
undefined, and it’s in dotted green—just the things that are truly feasible, by which I mean that you can
solve exactly on all the problems that you need to. So depending upon what n is for you—you know—if
n is a million, you can solve all problems to size a million in the size of computer that you can afford—
whether you’re an individual or whether you’re Microsoft, it changes a little bit, there it is—and in the
time that you have to devote to it, which might be a second; it might be your lifetime. Okay, so the … so
it’s a … it’s not a very well precise … defined thing, but the idea is that, somehow—for some reasons
having to do with dichotomies I’ve talked about before—those problems that … those natural, naturallyoccurring through—natural is not a well-defined term either, but let me just use it, anyway, the way I’m
thinking about it—those naturally-occurring problems that are in fact in polynomial time tend to be
feasible. And this is a very interesting phenomenon; that’s what makes, sort of, complexity theory and
algorithms work; if you can get a nice algorithm in polynomial time, it’s probably … some people will
improve it till it’s almost linear or quadratic, depending upon—again—the size of your n, and you can
actually get into the feasible. So a lot of the work, in my opinion, in algorithms, and complexity, and so
on, is taking a problem, defining it—maybe as you defined it, it was too abstract, and it was way up
there, undecidable, or it was here; it was NP complete—and then maybe you simplify it or say, “I’ll take
an approximate version of it,” and bring it into the feasible, where you can really solve it. And I think of
that as the process of algorithm development, and then lower bounds, I mean, I might prove something
is hard; I might prove it’s up there—r.e. complete—it’s undecidable, and yet, some version of it, I can
bring it down and actually solve it. Okay, so I’ll talk about this more as we go. Questions—even like the
first question—will help as we go. Okay, so feel free.
>> Ken McMillan: Can you push the dotted green line up a little bit for a few of us that are … [laughs]
make our livings higher up than the …?
>> Neil Immerman: Oh, go … great. So I was hoping for this question. So Ken says, “What about all
those—you know—those recursive problems that we’re trying to solve?” And my theory is: you think
you’re solving a recursive problem; actually, there’s some subset of real problems that you’re working
on where the issue is—even though we don’t know how to prove it yet—the issue is down here. So my
feeling is: if you’re really doing it in your lifetime with the amount of hardware that Microsoft can buy
for you, then it really is feasible in the way that I’m thinking of it, even though, maybe, it’s only some of
the problems that you can solve, and you haven’t identified yet which ones.
>> Ken McMillan: Of course, in my lifetime, I only solve instances, really; it’s like … I don’t solve classes.
>> Neil Immerman: Yeah, yeah, yeah, yeah. Oh, okay. No, exactly. So there’s that other issue that Ken
mentions: that we only have finite things, so there’s not really infinite. Why are we talking about infinite
problems? And this comes down to just the modelling question: why do we model our computer—
which is really a finite state machine—why do I model it as a Turing machine which is unbounded
memory? And it turns out that that turns out, modelling as an infinite machine turns out to be a better
model; you can sort of understand; you’re interested in how it’s growing with the size, which is crucial to
good algorithms. If you just thought of it as a finite state machine, you’d say, “Okay, there’s only two to
the”—you know—“to the hundred billion states; I can list them all out.” You can’t list them all out; it’s
not a good model. So even though it’s finite, I—imagining the infinite—I think it improves our focus.
Yuri?
>>: Try to be a devil’s advocate [indiscernible]
>> Neil Immerman: Please, of course, as usual. [laughs]
>>: So one can say that all complexity theory suffers from being asymptotical.
>> Neil Immerman: Yes.
>>: Now, the problem is: if you [indiscernible]
>> Neil Immerman: So Yuri says, “All complexity theory suffers from being asymptotical, or …” so yes.
>>: So if you don’t make this assumption, then it’s hard to develop any theory …
>> Neil Immerman: Correct.
>>: ... but would be interesting to see ‘proaches, because in every application, there is certain range
where instances occur.
>> Neil Immerman: Exactly, exact. That’s what my n is, is I’m trying to imagine … as I say, depending
upon the application, it might be a thousand; it might be a trillion—it’s in that range.
>>: Right, but one approach is, for example, you take log star of that range …
>> Neil Immerman: Yes.
>>: … and that number may determine this range precisely—say, log star three.
>> Neil Immerman: Oh, okay, yes, yes, yes.
>>: So maybe we can developed …
>> Neil Immerman: All this very interesting; this is—I mean—off-topic, but let me just sort of answer.
You could talk about: okay—you know—just … so Yuri says, “Let’s just look at n, and put it into six
possible categories, and do our complexity theory depending upon which one we’re in.” It’s very
different depending upon which one you’re in. That’s a great idea. I’m sort of doing it more globally
and saying—as if everything worked out nicely, which it doesn’t, so that would be great. Yeah, let me go
on from there, but that’s a nice idea. So yes—I mean—right, so—I mean—we’re doing this thing we …
proving that certain things are decidable, and we’re really please—you know—even though we don’t
have an upper bound for the time, ‘cause then we hope more that since they’re decidable, in fact, the
one we’re looking at, we’ll be able to—you know—you’ll get the answer on your machine before you go
to lunch. You know, so it’s … and it’s part of this magic, just like polynomial time; proving something is
polynomial time doesn’t do anything for me, ‘cause it might be time n to the one million—that’s not any
better than undecidable in some sense. But it’s just this phenomenon that’s not well understood yet—
this sort of dichotomy phenomenon—that says, “Once it’s in P, it’s probably … usually kind of nice.”
>>: Where you don’t even know the polynomial.
>> Neil Immerman: Or not even know the polynomial—that would be horrible proof. I prove this is in
polynomial time, but I don’t even … I can’t … just, I know … I prove the exponent exists. I don’t like that
kind of proof.
>>: There are natural results.
>> Neil Immerman: I know, I know, I know. Okay, but they’re not natural to me, okay? [laughs] Okay, so
I just want to mention nondeterministic time, ‘cause this is such an overblown concept now that I just
want to … I do this with my students: just remind them that this is not a real thing; this is just a
mathematic abstraction. So nondeterministic time, the idea is you starting in some state; at each step,
you have two choices—or it could be more than two choices, but for simplicity, two choices—at each
step, I can do one of two things: I can do the zero choice or the one choice; at the second step, I can do
the zero choice or the one choice. So at each step, there’s a zero or a one telling me which choice I
make. So if I’m running for time t of n, then by the end—you know, if I’m not worried about memory—
then there are exponentially many different choices I might have made. And now, I’m doing a
computation all along, and I’ll come to an answer which, again, is … it’s just decision problem, so it’s
either a zero or a one, and for non-touristic machine, we say the machine accepts if any of these
exponentially pa … many paths leads to a yes. Okay, then we say we accept. Fine, so you … so we’re
searching exponential space; that’s interesting, important concept; but the charging is a real gimmick.
I’m only gonna charge you for the length of a single path. Okay, so the cost is t … the nontradit cost of
this is t of n, even though the true deterministic cost, if we simulate it in a natural way, would be t of n
times two to the t of n—so it’s be exponential. So’s a big gap—or can be—between nondeterministic
time and deterministic time. Deterministic time is real; we living in a—you know—in our world, and I …
classically—you know—deterministic time is what we have, and we can talk about quantum a little bit,
but not for a while. So this is an interesting model; the reason it’s important is that a lot of problems
that we really would want to solve have this property that there’s an exponential search space; if you
find the answer, you got it; but searching the exponential search space is very hard. So it’s a very
common, very important kind of process—I think, actually, phenomenally important—but it’s still a
mathematical fiction; you can’t build a nondeterministic machine.
Okay, so that’s nondeterministic time, so then we get this class which … it’s sort of a one-sided class,
‘cause notice that yes is different from no in nondeterministic time. So if you can find the answer, then
the problem is in NP, and the complementary problem—if you can prove that the answer’s zero—then it
would be in what’s called co-NP—those properties whose complements are in NP. So SAT—Boolean
satisfiability—is NP … is in NP; the complement—the things that are not satisfiable—is in co-NP. So
there’s this two-sided nature; so the class NP is this piece and everything below; and the class co-NP is
that piece and everything below; and their intersection is called NP intersect co-NP; and it’s a very—
from a complexity theoric point of view—it’s an interesting class that we know very, very little about.
NP intersect co-NB, by the way, is where cryptography lives—so things that have unique answers that
you’d like to guess. So some people conjecture that NP intersect co-NP equals P, which would mean, at
some level, that you can’t do cryptography—so kind of unlikely, I think. Okay, so that’s our situation,
and as I mention, many, many, many problems that we’d like to solve are in NP, and there’s this really
nice phenomenon that a lot of those natural problems we’d like to solve are, in fact, NP complete—
every other problem in NP is reducible to them; they’re hardest problems in NP; they’re all equivalent—
and it’s a huge phenomenon of these kind of problems, and in this sort of overblown way, I kind of think
that everything that you … so the ans … coming back to Ken’s question again: what about way up there?
Anything that you really want to devote your life to I suggest should be in NP, ‘cause otherwise, even if
you solve it, you can’t even convince anybody that you’ve solved it. So the NP problems are the
problems that, once you’ve solved them, there’s a clear, readable proof that you’ve solved it. Okay. So
somebody who’s trying to do something bigger than NP is pretty quixotic, ‘cause even if you do it—you
know—how will you even convince anyone else? So I think—you know—so I think of NP as—you
know—what we’re all going to—the, you know, the dreamers who are trying to solve problems; they
should be in NP in my opinion. And you can … Yuda?
>> Ken McMillan: Think they should be in NP intersect cone NP, because I have to give the proof for
both answers. If I had my SAT solver, I could give a polynomial-size certificate that it’s SAT.
>> Neil Immerman: Oh.
>> Ken McMillan: I can’t give a polynomial-size certificate that it’s UNSAT.
>> Neil Immerman: Huh, okay, okay, so you want to be down … so Ken says he wants to be down in NP
intersect co-NP.
>> Ken McMillan: So SAT solver’s out. [indiscernible]
>> Neil Immerman: [laughs] I don’t agree; I think—you know—it’s just each individual question is a
question you’re searching for that you’ve … for some reason, you have some insight: I think this
formula’s satisfiable. And you do the search—you know—you come back after ten years and say,
“Haha, it is satisfiable.” If you try … if your search is in co-NP, you’re trying to prove it’s not satisfiable,
uh … [laughs]
>>: Can I interrupt you?
>> Neil Immerman: Please.
>>: There are interactive course.
>> Neil Immerman: Interactive …
>>: [indiscernible] proof interactively, which works for both NP and co-NP.
>> Neil Immerman: Okay, okay. Very interactive proofs, and there’s … I mean, there’s all this
complexity theory. So I mean, you can look at interactive proofs repeated polynomial time, and that
goes all the way up to classes we don’t have here yet, like PSPACE. Yeah, yeah, yeah, yeah. Okay, but if
you want to just publish a book—you know, no one can ask you questions, ‘cause it’s the end of your
life—you publish a book on what you’ve done, and people can read it—you know—then it should be in
NP. Okay, so that’s … this is just … I’m just specking. So I mentioned this idea before: that everything in
computation … we have some input—it’s n bits—and we have some output—it’s polynomially many
bits—and in complexity, we can just look at any particular bit and say yes or no. So we’re focusing on
decision problems; all complexity classes are decision problems; and that’s no real restriction—I’m just
focusing on each bit at a time that you’re trying to compute.
Okay, and now, I’m just … here, I’m … I want to introduce the notion of descriptive complexity which
has, for me, been a very fruitful topic, and that’s what I’m giving you this survey talk about. So we’re
just looking at decision problems; so we have some problem, S: does my input, q1 through qn, satisfy S?
And complexity says, “How hard is it to check if the input satisfies S? How much time does it take? How
much nondeterministic time does it take? How much memory space does it take? How many processes
working together does it take?” That’s traditional complexity, and instead, in descriptive complexity, we
say, “Well, how … that’s a problem; that’s a property; I can write it in some formal language if I have
some logic training, and I like to go that way. How rich a language do I need to express S?” It’s a
different kind of problem; that’s the expressing or descrip … describing that’s descriptive complexity of
the problem. And they sound completely different, right? How much time does the computer have to
take to do it? How rich a logical language do I need to express it? Kind of different …
>>: Sorry, how do you measure richness?
>> Neil Immerman: [indiscernible] great, great question. How do I measure richness? I’ll show you
examples soon; it’s just … first-order logic is very weak; second-order logic is richer; first-order logic plus
a mechanism to make inductive definitions is somewhere in the middle. So I measure in different
ways—how many variables I use—different notions of this, which I haven’t defined yet. Okay, good
question. So that’s the idea of descriptive complexity, and very surprisingly to me—and this was sort of
the basis of my PhD thesis and, which was a long, long, long time ago, and a lot of things that have come
since—is that, wow, this is just another way. You know, there are all these models of computation, like
Turing machine, and lambda calculus, and so on, that are all constructively equivalent. Well, in some
sense, this is another; they really … it’s really a way that you can view all of computation from this
descriptive point of view—rather surprising. And what I mean exactly there, unfortunately, I still don’t
exa … I don’t have that expressed well enough, so don’t challenge me too much on that, but it’s
something that I’ve found in experience; it really is constructive.
So asking this gentleman’s question—you know—looking a little bit more, I want to look at these
languages a little bit. Computational problems—of course, I can always think of an input as a binary
string, ‘cause everywhere everyone in computer science is used to that—I want to think of my input as a
logical structure. So for most typical and nice—and the best way to model any problem at all—is a
graph. So my input might be a graph like that G, which has some n vertices, and then it has a binary
edge relation—so think of this as a directed graph. So the relation E holds between this point and this
point, and we draw that as a directed edge between them. Okay, so that’s a visual and a logical
description of a graph. So that’s my input; it’s a graph; and that’s nice for me, ‘cause it’s also a logical
structure, so I have a natural first-order logic to talk about it. And binary strings, if I want to think of a
binary string—like this binary string—as a logical structure, I can; the most natural way is to say, “Okay,
the elements of my universe.” So the … so a logical structure has a universe, like up there, the graph …
the universe is the vertices, and there’s an edge predicate. And for a string, I think of the elements of
the universe as the positions of the string. So this is a string with eight bits so that there are eight
positions in the string, and I assume that I have access to the normal, usual ordering one to eight, so I
know which bits come before the other bits; otherwise, I wouldn’t have a string. So I’m modelling a …
this string is also this logical structure, where I just have a unary predicate that tells me, for each
position, whether it’s a zero or a one. So here, this predicate is true of the second position, the fifth
position, the seventh position, and the eight position. So this string is equal to this logical structure; this
drawn graph is equal to that logical structure; so my input can be a logical structure. I can now think of
my input … I don’t do anything; there’s no work involved here, just conceptual; I’m thinking my input is a
logical structure; I can write logical properties about it. That right?
In general, for those of you—I suggested there might be applications to relational databases in
descriptive complexity—so you can just think of any … your relational database in some scheme as a
universe of all the active things and then whatever relations you happen to have. That’s what a
relational database is—is just a logical structure. So when we’re doing database queries and looking at
their complexity, we can think of it from this descriptive point of view.
>>: But they ignore order in relational databases.
>> Neil Immerman: Oh, right. I haven’t tip … talked too much about … I talked about … Yuri said they
ignore order in relational databases, making for a really nice question—that’s right. Now, in the string, I
have to have the ordering; otherwise, it’s just a bag of bits; it’s not a string; but for the graph, I don’t
have to have the ordering, right? But if you feed it into a computer, those vertices have to be in some
order, so I’ll talk about that soon. And relational database theory, very nicely, they try to ignore the
order, because they want the algorithm, or the answer, not to depend upon the internal ordering of the
elements, which should be irrelevant. So they want the queries to be all order-independent. Bringing
up some really interesting questions, thank you.
So then we have the … obviously, you’re very familiar—I assume, if you came to this talk—with firstorder logic. So depending upon what your vocabulary is for graphs, for example, I can write first-order
formulas that quantify over the elements of the universe; so this formula, alpha, says, “For all vertices x
and y, there’s an edge from x to y”—so every vertex has an edge coming out of it—that’s a simple firstorder formula. This one—beta, formula about strings—says that there’s some position, x, such that, for
all positions, y, x is less than or equal to y—in other words, x is the first position in the string—and S of x
holds. So beta says exactly: “This string starts with a one.” And this formula … well, I like to have some
numeric symbols, just to make my life easier, so as Yuri was saying, I want to think of my whole universe
as ordered, and the reason I want to do that is ‘cause I want to simulate computation; you feed any
object whatsoever into a computer, you find you’ve ordered it, ‘cause you’ve given each point a binary
name—the binary numbers are ordered. So you cannot input anything into a computer unordered; so
there’s no notion of a graph in a computer; it’s just an ordered graph. So for that reason, I’ll always,
from now on, have the ordering around, and a lot of the results I mention won’t be true without the
ordering for annoying reasons. Okay, so in particular, I have numeric symbols; I always have equality; I
always have the ordering on the universe; I have the first element, which I call min; the last element,
which I call a max; I might even have … since I have ordering on the universe, my input out … my
universe is the elements one to n or zero to n minus one, depending upon … you want to look at it; we
can have arithmetic on those numbers, also; we could have those predicates; and I might want to talk
about that for really weird, weird reasons that I’ll mention later. Okay.
>>: So this is a total order?
>> Neil Immerman: Total ordering on the universe, yes.
>>: Why don’t you like lattice or a partial order instead?
>> Neil Immerman: ‘Cause literally … so Tom says, “Why a total ordering? Why not a partial ordering?”
Again, because you give your graph to your computer; each vertex has a number; they really are totally
ordered.
>>: ‘Kay.
>> Neil Immerman: So I … and what’s more, some algorithms use that. They say, “Okay, I want to
search this graph; I’ll go to the first vertex out of …” and I want to be able to simulate that in logic. So I
find if I don’t have that total ordering, I’m sort of holding both hands and both legs behind my back. And
so I’m gonna give it to you, but that makes logicians—not Yuri, but most logicians—quite annoyed,
because now, this is not classical logic anymore; it’s very different. Okay, so just by the way: so this
formula, if I have that min constant, then beta is more easily written as S of min—the first bit is a one.
Okay? So there’s three examples of first-order properties; first-order is sort of the lowest class—almost
the lowest class—I’m gonna talk about today. Okay? Okay. Great, questions are perf … coming really
well right—you know—just to just the sort of level I want to do.
So now, there’s this stronger logic that I want to mention, which is second-order logic. In second-order
logic, I can have variables not only talking about elements of the universe, but relations over elements of
the universe. So unary relations, for example, would be subsets of the universe; binary relations would
be subsets of pairs, and so on. So here’s an example; here’s a graph; and there’s a second-order formula
where I have those three new variables—second-order variable—they’re unary second-order variables
representing subsets of the vertices. So I’m saying there exists one subset of the vertices called red, one
subset called green, one subset called blue such that for all vertices x and y, each vertex is either red, or
green, or blue; and for all pairs of vertices, if there happens to be an edge between them, they’re not
both red; they’re not both green; and they’re not both blue. So this formula—this second-order
formula—says exactly that this graph can be vertex-colored with three colors. Does it make sense?
Okay, so now, I can quantify not only over elements of the universe, but relations over the universe, so
secondary’s much stronger than first-order. Okay, so there … so in pico, this formula says that I’m threecolorable, which some of you might know is an NP complete property. And something else about that
formula is that it’s second order, but all the second-order quantifiers are out front, and they’re all
existential, so that’s why we call second order existential.
Okay, so Ron Fagin noticed this back in his PhD thesis, and he proved, in fact: ha, this interesting class
NP—this important class NP—has this very, very nice, elegant, machine-independent logical
characterization; the NP properties are exactly the second-order existential properties. And this is an
example; three-colorability is in NP; therefore, it’s second-order existential. Every second-order
existential property is in NP—so an exact characterization. Now, I was looking for a thesis topic; I was a
grad student in nineteen seventy-five when I started—or seventy-six, I guess. Ron’s good friend and
roommate, Larry Carter, knew I was looking for a topic, and he sent me … he said, “Look at this paper of
Ron’s.” So I was really … I really captivated by that, ‘cause that gave an amazing insight into this class
that everyone was then trying to understand—this was the early seventies; people were trying to … they
had these questions like: is P equal to NP? Of course it isn’t, but people couldn’t prove it; no one knew
it was that hard a question back then; and this seemed like a very nice way to try to show—okay—
maybe they’re not. And that was the idea of Fagin’s thesis, basically, but to prove—say—to prove P is
not equal to NP—prove something’s not in NP—you might want to show that something is not secondorder expressible, which is very hard. So I did some work on lower languages; I never go up to second
order if I can avoid it, ‘cause it’s too hard. So I’ll talk about that in a minute.
Okay, s’everybody understand Fagin’s theorem? So that was the beginning of descriptive complexity—a
completely logical characterization of an important complexity class, okay? And so the … at the time
when I was writing my thesis, I thought first order was sort of just too weak to live. I didn’t think it had
any interesting properties. Uh-huh?
>>: Can you go back to the previous slide, please?
>> Neil Immerman: Sure.
>>: Can you say a few words about how these very different things—you know, one is complexity in
terms of time …
>> Neil Immerman: Yes.
>>: … and the other is something to do with these symbols—what is the intuition? Why are they
connected?
>> Neil Immerman: Great, great, great. Thanks for the question. Tell me your name, by the way?
>>: My name is Shaz.
>> Neil Immerman: Shaz. So Shaz says, “Huh, I just stated this fact that looks bizarre. How can these be
equal? How—you know—what’s going on? One’s about nontermistic time; one’s about logic.” Okay, so
let’s look at what a proof would look like that—of Fagin’s theorem—that NP equals second-order
extensial. First of all, let’s do the easy direction: every second-order existential property is in NP, okay?
The way to see that: there’s an existential property; there exists R1, G1, B1; and I want to build an NP
machine to check whether that property holds or not. Okay, so what am I given? I’m given an input
graph; the input graph has n vertices, okay? And I want to know whether there exists R1, G1, and B1.
Well, how many bits do I need to represent R1? For each of my n vertices, that vertex is either red or
not, so I can guess n bits that characterize R1, okay? Then I can guess the n bits that characterize G1,
and I can guess the n bits that characterize B1. Does that make sense? So I’ve just …
nondeterministically, you write down three n bits, and now I have those three relations, and now, I just
have to check a first-order property—much easier—and it’s easily done in polynomial time and less. So
therefore, any second-order existential property is checkable in NP. Does that make sense? ‘Cause
really, what this power is is guessing polynomially many bits; if it was unary, it’s n bits; if it was binary,
then for every pair of those n vertices, I have to guess a bit, so it’d be n squared bits. So this secondorder existential really just gives you … thank you for asking this question; this … I think this clarifies it …
that …
>>: Is it not known that the number of bits you need cannot be larger than the size of the formula?
When [indiscernible] problem?
>> Neil Immerman: Right. So you … so first, so why is it only polynomial? Okay, any fixed formula … so
what this means: I have a fix formula, and that represents a decision problem; some inputs—some
graphs—satisfy the formula; some do not. So every fixed formula is a decision problem; every fixed
formula has … it’s just fixed like that; it has fixed relations of fixed -arity—the maximum -arity, call it k—
so therefore, I only have to guess at most n to the k bits to guess all the second-order relations; then I
have a first-order property. Okay, so this very important … it’s very important that the formula is not
part of the input; the formula is the fixed program or statement of the property; if the formula were
part of the input, this would be much, much harder than NP. Okay, fantastic. Does it make sense? And
going the other way, well, let me just cheat and say, “Oh, look, I expressed an NP complete property;
therefore, I can get all of NP.” Okay? So I’ve already expressed an NP complete property, so secondorder existential is gonna be powerful enough. That’s just a intuit … intuitively, I can get … I got a
hardest NP property; I can probably get all of it. That’s not the way the proof went originally, but once
you introduce notions such as first-order reductions, you can translate all problems in a first-order way,
and show that machines are closed under first-order reductions, and show that, in fact, threecolorability is NP complete via first-order reductions, then that would be a proof. I got a complete
problem; I get all problems in NP. Does that answer your question now?
>>: It does.
>> Neil Immerman: Okay, so the … so thank you for asking. So the … what’s the relationship? Well,
there is that relationship sitting there, even though it wasn’t obvious at first. So these characterization
theorems are not difficult; it’s just the insight that you can understand things in this way that’s useful.
Okay, so … and then coming back, when I was originally thinking about this, I thought of first order, as I
said, as sort of too weak to live—just … I couldn’t think you could find any interesting prop …
computational problems in first order, and then later, I found out: oh, maybe you can. And one
interesting one is addition; addition is first-order. So if I … if my input is two binary strings, A and B—
okay, so I have, again, n positions—and now, instead of one unary predicate, S, I have two; I have A and
B—so I have two binary strings—and I want to know what their sum is. That might not look like—
okay—decision problem, but if I name any particular bit of the sum, I can say, “What’s that?” That’s a
decision problem.
How do I express that in first-order logic? Okay, well, how many of you know what carry-lookahead
addition is? That’s what hardware uses, right? Okay, so it’s a practical notion. So basically, what’s the
carry into the ith bit? Well, there’s a carry into the ith bit just if there’s … there’s a carry into the ith bit
just if there’s some point j to the right of i where the carry is generated—in other words, both of the bits
are one, A of j and B of j—and for all k between i and j, the carry is propagated—there’s at least one
one. That’s when there’s a carry into the ith bit.
>>: But I can do it with a Boolean circuit—and Boolean circuits are in your class—I mean, I don’t need
quantifiers; I can just write the whole thing out.
>> Neil Immerman: Okay, good, good. So Tom says, “I can do this with a Boolean circuit; what do you
need formula with quantifiers?” The answer is …
>>: [indiscernible] decision …
>> Neil Immerman: The answer is: I have one fixed formula that does this for any size input; whereas,
that formula determines, in fact … if you can write out this formula, in some sense, exists j—okay, if
you’re writing a Boolean circuit, that j depends upon the size of the input. So this is really an n-ary … it’s
an n-ary OR gate.
>>: [indiscernible] correct.
>> Neil Immerman: So I can think of this as a sequence of circuits, and I’ll tell you about that later.
>>: This is a whole class.
>> Neil Immerman: That would be a whole class; that’s right. So … but the problem of addition itself, I
can write it with a single first-order formula. So addition is first-order. Okay, so that’s … there are some
things that are first order; it’s not … it’s … it turns out, it actually is an interesting class. So then, the sum
is just, of course, A exclusive or B exclusive or C, and exclusive or I can write in terms of OR and NOT.
Okay, so addition is first-order. And I only realize this much later—maybe like five years after my
thesis—and people were then talking a lot about peril computation in this very abstract, theoretical
model—the so-called PRAM. And so I think Uzi Vishkin gave us this term: concurrent read, concurrent
write, parallel random access machine—okay—which I just call a CRAM. Okay, the idea is that you have
some number, r, or maybe even n—a polynomial number—of tiny, equivalent processors, p1 through
pr; you have some big—but polynomial-size—global memory; and this is actually a synchronous model,
so each of these little—you know—these are—you know—so they’re processors that you can put on …
sixty-four of them on a chip now, and each one has its little—you know—local registers, local memory; it
can look in global memory. They’ve synchronous; at each step, each one can look at any particular bit of
global memory; and it’s concurrent read, so they can all … a lot of them can read the same bit; then they
do a single computation; and then they can write to global memory also; and the write is also
concurrent. So several processors could write into the same bit, so we have to say how to resolve that,
and there’re various ways to resolve it that are all, more or less, equivalent. One way is the priority
write model; so if several processors write into the same bit, the lowest number processor that wrote
succeeds. Another model is the so-called common write model, which is: I will prove in my program that
if at any time, more than one processor writes into the same location at the same time, then they write
the same thing, and that’s annoying, ‘cause you have this proof obligation, but I like the common write
model, ‘cause the common write model really comps … corresponds to what a quantifier is.
So let me just show you the … a quantifier is a sort of simple parallel object. So assume that in my
memory, I’ve already computed some array, A of x, from … for x going from one to r. Okay, so I have …
that’s what I’ve computed so far, and now, I’d like to compute: is it the case that for all x, A of x? Well, I
can do that in one more step, okay? So basically, every … we’ll just write a one into a certain location,
and processor i will just check A of i; if A of i is false, then it has learned—processor i has learned—that
for all x, A of x is false, so it’ll write a zero into that location. So one step later, if there are no
counterexamples, then we know for all x, A of x; if there were any counterexamples, we know it’s not
true. So a quantifier is one step of a CRAM with polynomial-much hardware. Does that make sense?
And obviously, a existential quantifier would be the same thing; we’d start with a zero; and anyone who
finds the same answer writes it one. And you see that this turns out to be the common write model,
which is not that important, but … make sense?
So quantifiers are little parallel devices, and from this argument, you can see, in fact, first-order logic is
an interesting complexity class—way down here at the bottom—it’s the things we can do in constant
parallel time. And so this would be CRAM time one, and there’s also a model some of you may know
about—otherwise, don’t worry about it—called … a circuit model called AC0—sets of uniform,
polynomial-size, unbounded fan-in AND and OR gates; the whole circuit is polynomial-size—it’s the
same thing as constant parallel time, and that sort of relates to the—I think—the circuit that Tom was
talking about. ‘Kay, so first-order is exactly constant parallel time with … when you have polynomially
much hardware. That seemed like an kind of nice insight.
Okay, and in fact, all of these things are the same; basically, CRAM t of n—that’s what you can …
properties you can check on a CRAM, a polynomial … parallel machine with polynomial-much hardware
in t of n parallel time—that’s ex … same as what you can write in the first-order inductive definition that
closes in t of n steps; it’s the same thing you can write if you take a—basically—a block of first-order
quantifiers and write them t of n times. It’s a really weird form that I probably won’t … don’t have time
to go into, unless anyone’s really excited about it. But basically, if I take a block of quantifiers with—this
is Q1; it’s either for all or exists—x1 such that M1, where M1 is a quantifier-free formula; if I just take
this fixed block of quantifiers and literally write that same block t of n times, that’s … that this model,
first order t of n. It’s basically … so it’s more than first order, ‘cause I’m repeating this block of
quantifiers t of n times. So t of n … and that’s the same thing, really, that inductive definition does, if
you repeat it t of n times, so first-order iteration in that way—inductive definitions—is exactly parallel
time. So that was kind of interesting. S’a question about that? Okay. So okay … so this says just …
that’s what this theorem says: basically, first-order iteration is parallel time, and it works even beyond …
inductive definitions have this property that they have to close in polynomial steps, but iteration can on
beyond that. So a block of quantifiers … well, ju … this just all comes from the fact of: a quantifier is a
single peril step on a CRAM, so time on a CRAM is just how many quantifiers you’re writing again and
again.
So it’s … so that’s part of what I was saying in my abstract—that it turns out that natural properties in
complexity, like space, and time, and parallel time, have natural relations to how rich these descriptive
languages are. Okay, so where are we now? So we have few more classes now. So there’s: first order is
constant time, and if I iterate first order polynomially much, then of course, I get polynomial time—the
parallelism doesn’t matter anymore—and this is characterized … a least fixed point—probably, most of
you know—is a way—a logical way—of characterizing inductive definitions; I can define new relations by
induction, and first order plus least point characterizes that. So this tells us that polynomial time is
exactly first order plus least fixed fur … point, first order plus inductive definitions, or first order iterated
polynomially.
>>: Is that with or without ordering?
>> Neil Immerman: This is all—thank you—this is all with ordering, okay? So if you give me graph—
say—and you don’t give me ordering, and say that graph is not very helpful—maybe there no edges—so
you give me n points with no ordering; for the first-order formula, there’s no structure whatsoever, so if
you only give me a constant number of variables—‘cause in all these settings, there’s only a constant
number of variables, even if they’re iterated—then the poor first-order formula, even iterated as many
times as you want, can’t … it … say it only has three variables; it only … it can only count up to three, so if
you want to know is there an even number or an odd number of vertices there, this poor formula—even
this poor formula—can’t tell you without the ordering. So without the ordering, you’re … there’s a
certain kind of strange restriction, and so I’m just assuming under the covers that my languages always
have the ordering, ‘kay? Thanks for the question. Okay, otherwise, these characterizations would not
hold; they’d just be subsets.
Okay, and by the way, if you take a first-order formula and iterate it to death—as many times as you
want—the most times it’s useful to iterate is exponentially many, and that gives you polynomial space,
where you can think of: what’s polynomial space? It’s a … I just have polynomially many mem-mem
locations, and I can run as long as I want, which is—in the limit—the most I could ever want to run is
exponential time. So that gives you polynomial space. So I knew all this in my thesis already, and I was
just kind of thrilled by that, and I still sort of am, even though … so the idea is: you take a fixed block of
quantifiers, and you iterate it; you iterate it polynomially, you characterize polynomial time; you iterate
is exponentially, you’re all the way up to polynomial space. So question? Uh-huh?
>>: Yeah, so Datalog object, it is above PSPACE.
>> Neil Immerman: So …
>>: Datalog querying complexity.
>> Neil Immerman: So day … so well, Datalog …
>>: I’m sorry, program.
>> Neil Immerman: Okay, okay, great. So if you look at Prolog, Prolog sort of was so-called logic
programming, but then a few little things got added, such as in Prolog, you can construct new objects;
so you can increase your memory without bound, and then you become essentially Turing-complete,
okay? So if you’re thinking of Datalog—Datalog where you don’t have constructors—Datalog is
polynomial-time; Prolog is way up at the Turing machine. So do you have anoth … I can talk to you more
offline about that, but maybe …
>>: No, it just seems that this characterization’s missing something that you get in Datalog. So can … so
what [indiscernible]
>> Neil Immerman: You get … oh, what is missing—what you get with Datalog?
>>: So you get …
>> Neil Immerman: With Datalog, you can add—if you think about my data structure—Datalog can add
another element to the data structure. So I have my input graph of size n; now I can add one, so now,
it’s n plus one; I can add one, it’s n plus two; I can just keep doing that, so I can have basically
unbounded memory—unbounded size. So … not Datalog, but Prolog. Prolog lets …
>>: [indiscernible] Excuse me. Datalog intersection first-order logic is almost empty.
>> Neil Immerman: Right.
>>: So it is sort of very al … it’s an alternative way to see things.
>> Neil Immerman: Did you mean Datalog or Prolog?
>>: Datalog, not Prolog.
>> Neil Immerman: I think of Datalog is … so my understanding of Datalog is probably theoretic … too
theoretical.
>>: No, okay, so I can encode QSAT or QBF in Datalog in a simple way; I cannot encode Datalog with
QSAT.
>> Neil Immerman: Okay, so tell me your name again?
>>: Nikolai.
>> Neil Immerman: Oh, sorry, Nikola. So Nikolai’s saying—so hi; I hadn’t met you before—so is saying
that he can encode in Datalog. So it’s … so let me talk to you offline, because … so the version of
Datalog that I’m thinking about, maybe it’s a theoretical version. The version of Datalog that I’m
thinking about is just exactly first order plus least fixed point, so …
>>: But the maiden comment was that I think we’re missing something about PSPACE—between
PSPACE and primitive recursive. Or maybe that’s a future slide in your talk.
>> Neil Immerman: It’s … well, there’ll be more stuff—yeah—there’ll be more stuff within there, but my
understanding of Datalog is down here, and let’s talk offline and see what you have that I don’t have.
Okay? Uh-huh?
>>: I were … when logic was first invented by the …
>> Neil Immerman: By Aristotle, yes. Okay.
>>: Yeah. There were no computers around.
>> Neil Immerman: There were no computers around, indeed.
>>: So they were driven by a very different motivations.
>> Neil Immerman: Oh, yes, yes, yes.
>>: So what I’m wondering now is: what is it that makes a logic a logic? I mean, in our drive to
characterize various notions of computation by logics …
>> Neil Immerman: Right, uh-huh.
>>: … I’m sure that we can always make up some logic or the other that looks like it, but—you know—I
mean, it’s just—you know—you can give it whatever meaning you want; it’s a collection of symbols.
>> Neil Immerman: Okay, okay, okay, okay.
>>: I’m trying to get at here …
>> Neil Immerman: Okay, so Iaz, right?
>>: Huh?
>> Neil Immerman: Your name was Iaz?
>>: Shaz.
>> Neil Immerman: Shaz, I’m sorry. I’m terrible … really terrible with names; I apologize. So Shaz says,
“So what’s going on? What’s really a logic?” I’m sort of talking about the logics that—y’know—that
Tarski invented that—you know—basically to model the process of doing mathematics, which turns out
to be very relevant for modelling computers, so yes, you can bend the logic and change it. For me,
because these are classical things that I understand extremely well, and they exactly characterize this
things, I’m excited about it, but right, since then, there have been many, many other characterizations,
and many, many bends and twists, until it’s hard to keep track.
>>: [indiscernible]
>> Neil Immerman: So these are all sort of classical logics. The only difference is that I have ordering.
>>: So all these logics were in … they pre-date computers, essentially.
>> Neil Immerman: Of course, except for the ordering. So I’m insisting that I have an ordering on the
universe.
>>: Okay.
>> Neil Immerman: And the other thing in … standardly in logic, all interesting questions are about
infinite structures; finite structures are considered trivial; but in this setting, my input is my structure, so
it’s finite. So this is part of—also part of—something called finite model theory, where we’re asking
different questions that turn out to be more combinatorial.
>>: Bear with me.
>> Neil Immerman: Uh-huh?
>>: Objection: first, they have computer before ancient time. [laughter]
>> Neil Immerman: Okay.
>>: Ruler and compass; it was a real computer.
>> Neil Immerman: Okay, let’s …
>>: No, but wait, wait, wait, what’s to come? Logic in general, it’s a art of deduction, but class … logic
were … it was interested; it’s very objective. During the great crisis of mathematics a hundred years ago,
logic was used to solve it; in that time, certain classical logic appeared which became golden standard—
in particular, first-order logic. So it is not some kind of one of logics; it is the logic; but Datalog is always
disjoint—pure recursion—so it [indiscernible]
>> Neil Immerman: Okay, okay, so thank you. So anyway, yeah, as you said, there—I mean—there’re
now a gazillion logics lying around. And I was talking about sort of standard things, and just for me,
what was amazing is—okay—these computational problems I’m trying to understand in fact are
identical to classical, standard logical concepts—basically, parallel time is just the iteration of
quantifiers. Another rel … really interesting thing for me is that the size of the problem I’m dealing
with—the amount of hardware I have—relates to just how many variables I have in my logic. So going
out this way, the size gets bigger and bigger; I have more and more variables. Sort of one strange
thing—maybe I just … I shouldn’t go there. Let me go on a little bit. I was thinking … so I was thinking I
wanted to give an hour talk, so … ‘cause everybody has a lot to do, so I would like to sort of get to
conclusions pretty quickly. Let me go a little bit and talk about a few other things.
Two things I won’t … there’ve been a bunch of really exciting breakthroughs that I can’t necessarily tell
you about that’ve sort of complexity theoretic. One is really great; Ben Rossman has proved some really
nice bounds about how many variables you need to express properties in first-order logic plus ordering.
First order … once you put ordering on first-order logic, it’s very hard to prove lower bounds, and for a
long time, we didn’t know how to do this, and Ben showed: okay, if you … just if you want to talk about
a clique—I want to have a clique of size k—obviously, you can write that in first-order formula with k
variables—there exists x1, x2, to xk, and all edges are connected—and we want one to … without
ordering, it’s very easy to show that you need those k variables; with ordering, that was very hard; and it
hadn’t been known for twenty-five years or so that you needed more than three variables to say k-
clique. And Ben showed, basically, you do need at least k over four variables, and so this is a really
lovely breakthrough.
And there’s another kind of amazing thing that Martin Grohe has shown bay … which relates to stuff
that Yuri and I have looked at about trying to characterize properties without ordering. So if you look at
first order plus least fixed point—as I think Nikolai asked about—without ordering, I can’t even count.
So if I add counting, then I can get first order plus least fixed point plus counting—it’s a really interesting
class. And I proved with some people—Jin-yi Cai and Martin Fürer—that still, it’s not enough to get all
pony-time properties; however, what’s really interesting is: if you compare two graphs with first-order
logic, k variables and counting—so it’s a polynomial-time check—that sort of includes most interesting
graph isomorphism questions. So in fact, what Martin Grohe showed is that for many, many classes of
graphs—every class of graphs that define having excluded minors, whatever that is; let me not get into
it—k var … first order plus fixed point and counting suffices to characterize computation on that class of
graphs. So it’s … so it turns out that that language without ordering is already extremely interesting
combinatorially, and this is a … this is gorgeous work, which I don’t … I mean, maybe one question if you
want, but I don’t have too much time to do that.
Okay, that’s sort of the theoretical side. Here’s a sort of more complete picture, where beyond PSPACE,
yes, there’s exponential time and many, many other classes, but again, for me, I really am trying to get
down here; I’m thinking of: okay, NP is really what I’m aiming for, and when I can pull it down into
polynomial time, I’m very happy. Let me talk about dichotomy a little bit, because I think that’s the last
thing I have time to talk about before the end questions, and I … this is a really interesting phenomenon,
also. So basically, I was saying this before: if you take natural problems—so if you teaching an
algorithms course, you talk about the problems you run into, computational problems—they tend to be
complete for … from one of my favorite—five favorite—complexity classes almost all the time, and it’s a
really funny phenomenon. And this phenomenon has … was first pointed out by Schaefer, who looked
at a large class of satisfiability problems and showed that each one of them was either NP complete or
was in polynomial time—so that’s the dichotomy: there’s some things that are NP complete; there’s
some things that polynomial-time; and nothing in between. That’s why he called it a dichotomy.
Whereas, Ladner had proved that in … well, if P is different from NP, then there’s a dense hierarchy
between them. So all these problems exist, but in … but they don’t represent themselves in terms of
satisfiability problems.
And since then, people have found a whole lot of dichotomy problems, where certain classes of ways of
expressing properties have this dichotomy; they don’t get the weird, kinky, complicated problems; they
only get the nice complete problems, okay? So you can extend Schaefer’s dichotomy and say, “Each of
these problems is either NP complete, or it’s P complete, or it’s nondeterministic log space complete, or
it’s log space complete, or it’s complete for, basically—what’s it called? First order plus sum mod p, or
V’s base two—sum mod two. So what it means is that the problems we run into—the problems we
really need to solve—are from a much smaller class, really, than what we think. And this is incensing;
this is why algorithms works: you figure out an algorithm, like graph reachability or vertex coloring; and
you spend a lot of effort optimizing it; and then, it comes up all over the place. And the … sort of the …
so understanding why the dichotomy occurs in certain strong formalisms, like constraint satisfaction
problem, is really interesting work going on, and it seems like, in some sense, that there’s … the progress
that has come from this has been used … like a hundred years of universal algebra, results from there
have really paid off in understanding the CSP problem.
>>: Why there are few algorithms? How dichotomy related to few algorithms?
>> Neil Immerman: Well, dichotomy means that the problem you want to solve is also a problem that I
want to solve and a lot of other people want to solve that we’ve already worked on a very good
algorithm for. If every problem was completely different—at a different place in the hierarchy—then
it’d be a lot less useful to spend ten years getting the best algorithm.
>>: So [indiscernible] mature attitudes manifest itself in different ways.
>> Neil Immerman: Yes, yes. So that’s just a phenomenon that’s kind of amazing, and it comes up, I
think for me—oh, I wanted to tell you about dynamic complexity, and I just don’t have time; I’m sorry.
But there’s all the things you really wanted to hear, but let me just get to a punch line here, which was
SAT solvers—okay—which originally—you know, nineteen seventy-one—Cook proves SAT is NP
complete; that was evidence that SAT is hard; it was the first NP complete property; and then Karp
showed a lot of other problems are actually exact—that we want to solve—are exactly equivalent to
SAT. And since then, they’ve been an incredible progress in SAT solvers and SMT solvers, and so on, and
of course, every problem you want to solve—if it’s in NP, and my claim is every problem you want to
solve is in NP—is reducible to SAT via simple reduction, which means that now, these excellent SAT
solvers that people have wonderfully built are general-purpose problem solvers. So it’s a very funny
phenomenon; there’s still hard cases; there are hard instances of SAT that SAT solvers can’t solve in a
feasible amount of time; but ones that turn up in practice—for some reason that we don’t understand,
again—tend to be … and there’s a lot of work in—you know, which classes are easy, which classes are
hard—a lot of work in proof complexity. We have a general-purpose solver now, so it’s just
unbelievable, and it’s very useful, and it’s being use all over the place here in basically checking
program-synthesizing programs. I have a lot of thoughts about why that should be, and it’s just a very
funny phenomenon now. So what’s hard, what’s not hard is really subtle; it has to do with what
problems occur and understanding that better. And so I’ve just really … I’m sort of amazed that now, on
my machine, I have a general-purpose problem solver that’s much smarter than me in terms of thinking
up examples. I say, “Is there an example of this?” And just ask, and there it is or not—very, very useful;
useful for a lot of different reasons. Can I save two …?
>>: Just go; you have time.
>> Neil Immerman: Okay, so I want you to talk about software, and—you know—I—I mean—obviously,
you guys and a lot of people in many places are producing wonderful, fantastic software, doing amazing
things, such as telling me how to get between the apartment I’m—Taylor, very good to see you; Yuri,
they—my apartment in building ninety-nine here. There’s a lot of amazing software out there, and it
controls more and more of our lives, and I’m finding that—you know, just in some sense—everything
you do is computer-related. And just … I had a day where I—you know, very recently; I’m sure you’ve all
had this—where you go in … you go to … I went to the optometrist, and the optician and I waited while
the computer found something; we were just both sitting there—you know—twiddling our thumbs,
waiting for the computer. You know, twenty years ago, she could have done something
instantaneously—you know, by taking … doing it on paper—now she can’t do it anymore, and she has
waiting for this computer, which was just—you know—she just said, “I don’t know what the matter with
our computer, but …” you know, and then I went to pay my bill, and the same problem; it ju—you
know—I don’t e know if—you know—they couldn’t take my money, you know? You know, and … ‘cause
there’s something wrong with the computer, you know? It’s just sort of … so in one level, fantastic; this
really fantastic software’s getting better and better; at another level, there’s … there’re basic problems
that are—you know—they really are out there, and we should face them.
So it’s … so this is what I say: software in general, even though it’s fantastic, is also buggy, insecure,
brittle, hard-to-change; it’s not an adaptive … as adaptive as kind human beings when you go for help,
you know? It’s … if you … so very often, I’m in a situation where I say to the … my phone, “Can I please
speak with a human being?” [laughs] You know, and after a while, it lets me—you know—and then,
some of the human beings are helpful, and some of them are not. But anyway, okay, there’s this …
there is a problem; I think it’s getting worse in the sense that software is more and more important in
everybody’s lives. So … and now, we have these general-purpose problem solvers; we have logic, which
is tightly tied to computer science; so there’s … there really is this amazing progress that Ken and Mulino
did, and many, many of you and … are doing, with the help of Nikolai, to automatically check that
programs are doing what they’re supposed to do. And even—you know—we’ve looked a little bit
naïvely—and a lot of you have done more—at synthesizing correck code—you know, write the
specifications, synthesize code that’s good. So this no I dish … this idea of being robust, understanding
… I want the machine to understand, at some level, what it’s doing—understand well enough not to do
something really stupid, not to be really brittle—but to say—you know—I want the computer to say,
“No one’s ever asked me that before. I don’t know; I will ask my creators”—you know—and go and ask
that question—you know—or get better. Okay, and I think logic can help; that’s … so that’s sort of what
I wanted to say. I’m sorry if this is kind of disjointed. Okay, alright. And … uh-huh?
>>: Actually, I have a question about the previous slide.
>> Neil Immerman: Sure.
>>: You go one slide back.
>> Neil Immerman: Absolutely.
>>: So now we have this universal solver on our computer …
>> Neil Immerman: Yes.
>>: So once I’ve shown it that the problem I’m interested in is in NP …
>> Neil Immerman: Yes?
>>: … should I even bother trying to establish whether it’s in P, trying to find a provably linear-time
algorithm, or should I just use my [indiscernible] solver?
>> Neil Immerman: Okay, okay, great. So great question: once a problem is in NP, can’t Z3 just solve it?
And the answer is: only on simple instances, somehow. So it can solve it sometimes; so yeah, you know
it’s in NP, run Z3 on it, please; and if it solves it—you know—just go to the bank and be happy; if it
doesn’t solve it, then you have to look farther, ‘cause there are these sets of instances of SAT that the
SAT solvers can’t do. I have … there’s a survey paper, actually, by Jakob Nordström, who’s an expert in
proof complexity, about when SAT’s hard for SAT solvers and when it’s not. It’s really interesting survey
article that’s … which is sitting off my webpage; it’s … which I—you know—you can look at or I can send
to you. So the answer is: yeah, once you know it’s in NP, you can try the SAT solver, and you might be
lucky—fantastic—but it doesn’t solve all SAT instances.
>>: But I would not know if I’m lucky or not, because I can only …
>> Neil Immerman: Until you tried it.
>>: … try a few instances …
>> Neil Immerman: Right.
>>: … that exist in my hand, and I can try that, and I can make a guesstimate.
>> Neil Immerman: Right.
>>: And then I give my supposed solution to my user, and God knows what instance he’s gonna throw at
it.
>> Neil Immerman: That’s right.
>>: So I’m sort of [indiscernible]
>> Neil Immerman: So we need a lot more theory. So you need to look much more closely at your
problem and understand where it is; maybe it’s in a class that Z3s can do great all the time; maybe it’s in
a cla … one of the classes that Nordström identified that … forget it; and maybe it’s somewhere in the
middle, where we don’t know. So there’s this kind of question now. So there’s a wonderful quote;
Christos Papdimitriou wrote a very nice book on complexity theory, and one of the things he wrote on
the NP chapter is he said, “Proving that a problem is NP complete is the first thing we should do, not the
last thing we should do in analyzing it.” Once you know it’s NP complete, you have to look a lot more
deeply.
>>: So …
>> Neil Immerman: Uh-huh?
>>: … moving on to the next slot …
>> Neil Immerman: Absolutely.
>>: So the other big trend—if you all …
>> Neil Immerman: Uh-huh?
>>: The other big trend insert is the—I mean, maybe it’s the battle of the twenty-first century—is you
know, maybe it’s not logic and binary decisions; maybe it’s machine learning. And what is the logic
that’s in our brains, and why are we adaptable? And maybe you’re … maybe this is completely wrongheaded; I mean, maybe this is actually the worst possible way [laughter] for us to be going about—you
know—getting a good software—I mean—maybe we should be having learning algorithms; and maybe
we should be understanding our brain and what the logical hierarchy’s there. So I don’t really know; I …
>> Neil Immerman: Great, great, so …
>>: … sure it’s a … I’m not sure it’s our best hope, actually. We’re buying cars now that will stay in their
lanes, and they’re all trained by neural nets, and you know?
>>: Nobody knows what they really do. [laughter]
>>: No, the evidence is pretty much, so far, that they’re as good as humans, so …
>> Neil Immerman: Okay, so it’s … so fantastic question. So there’s … so machine …
>>: Probably better, actually. [laughter]
>> Neil Immerman: What?
>>: Probably they’re better than humans already.
>> Neil Immerman: Machine … well, they’re better than human …
>>: For staying in a lane.
>> Neil Immerman: … at some things—you know—and as I mentioned—you know—Google Maps is
better than me for finding my way here—you know—no question. And … but there’s … so machine
learning is very hot, and it’s very good at doing certain kinds of things, inc … and certain things
including—you know—some language recognition, and staying in lanes is—I mean, there’s a whole
bunch of other things that are bailable that are not ‘vailable to people. Then … so yes, I mean, maybe
there’s a way that machine learning can help us do a little better with cert—or do a lot better—with
certain pieces of this; other pieces—I think—to get the software actually right—and I would like it not
only right; I would like it secure—so then, I think I … well—I mean—machine learning can look for
patterns of attacks again. So it’s certainly a tool; so why am I concentrating on logic? In some sense, I
have …
>>: I understand why. [laughs]
>> Neil Immerman: Well, you know why I like logic, but I had this sense that we can totally characterize
what we’re supposed to do if we’re … if we were willing to be a little bit more formal and willing to use
the tools that we now have, we can—I think—we can make a huge amount of progress. But this idea, I
think it’s not only making them—you know—greater, greater, better, better, faster, faster, but also
getting them to sort of understand what they’re doing so that they can be verifiable …
>>: Right [indiscernible]
>> Neil Immerman: … and not have all these bugs that people can go in and …
>>: … the interesting intersection is—and maybe Shaz, I’m stepping on you, sorry—is that, when you say
controlling more and more of our lives, actually, a big part of that cyber-physical—so the drones flying,
the cars driving …
>> Neil Immerman: Yes, oh, yeah.
>>: … and there, you’re in continuous … a real-world, where the computers and the algorithms are
dealing with—you know—many, many more distinctions, let’s say, and it … I looked up … it’s just a …
>> Neil Immerman: Oh, wait, so that’s interesting; it’s not just Boolean anymore.
>>: So I have a comment about Tom’s observation.
>> Neil Immerman: Go.
>>: Tom, he gave the example of automatic cars, and you gave the example of maps.
>>: Right.
>>: And somehow, the implication in certain your comments is that it was machine learning that pulled
this off, but I would argue that if you look at the contribution of machine learning, as opposed to all
sorts of classical systems engineering—control theory—that goes into making a scenario like that
possible, possibly, machine learning is the icing on the cake.
>>: Okay.
>>: But … and I think the reason there is this perception that machine learning is controlling everything
is because it’s a current [laughter]—you know—phrase du jour.
>>: Fair enough, fair enough.
>>: Really, all these things require a tremendous amount of classical engineering and classical modelling
that has a cumulative [indiscernible]
>> Neil Immerman: I totally agree with that, yeah.
>>: Machine learning is also classical; it’s just optimization. That’s all machine learning is is
optimization; you turn your dials to optimize the amount of error in the [indiscernible]
>> Neil Immerman: But it’s a certain … yeah, sure, but it’s very hot technique right now, but you’re … I
absolutely agree; there’s a huge amount of …
>>: Well, there’s also some synergies there; I mean, the air carbons came to us and chatted about—you
know—I mean, people are concerned about—you know—what are these algorithms that people are
synthesizing using machine learning doing and controlling? And can you verify, or at least—you know—
monitor them in a way that …
>> Neil Immerman: Indeed, indeed.
>>: … you can guarantee that they stay within some envelope, rather than spreading [indiscernible]
>> Neil Immerman: Indeed. So … and you look cars that you can now hack into and control. If you think
about that, that’s—you know—that’s people getting the cart before the horse, if you will, and not
checking that the different modules in the compar that are talking to each other are safe. I mean
there’s … there’re conditions that could be checked—you know—that need to be checked.
>> Ken McMillan: Yeah, right, but … yeah, I mean, there are so many social things going there, you
know? I mean, you have to understand, the people who build the car, they buy, from a supplier, a radio.
>> Neil Immerman: Right.
>> Ken McMillan: They have no idea what software is in a radio.
>> Neil Immerman: Right.
>> Ken McMillan: They don’t recognize that someone can attack that radio by plugging in a CD, which
could then cause it to go on the CAN bus, and talk to the braking sub-system, and …
>> Neil Immerman: Right. Right, right, right.
>> Ken McMillan: So mean, we haven’t yet reached the point where we even know what are in our
engineered systems—you know—much less—you know—we could prove something about them at a
level that would make them secure and not [indiscernible]
>> Neil Immerman: Right.
>> Ken McMillan: … that way. But there’s sort of a long way to get just to the point where you could,
say, apply Z3 to help make that car more secure; we have a long way to go.
>> Neil Immerman: Right, right. It’s really true.
>> Ken McMillan: But it … two other comments I wanted to make: one was about—‘cause machine
learning was interesting to bring up—you know, there is this idea in machine learning, also. You know,
you have a hypothesis space which is defined by a language, right?
>> Neil Immerman: Mmhmm, mmhmm.
>>: And you have a notion of complexity; learning a concept, you put it in that hypothesis space in order
how many regar … how much data or how many examples does it take to distinguish a concept.
>> Neil Immerman: Right. Mmhmm, mmhmm, mmhmm.
>>: So is there a connection between learning theory and the descriptive complexity?
>> Neil Immerman: That’s a great question. I don’t know. I mean, there’s somebody at Berkeley now
who’s looking at, basically, first-order logic plus probability to deal with—sort of—to try to join those
two worlds.
>>: Stuart?
>> Neil Immerman: Stuart Russell, that’s what I thinking of. So he just had an article last month in
CACM about that, yeah. So the … but I don’t … it’s … I’m not an expert on that area. It’s …
>> Ken McMillan: [indiscernible] it seems that there ought to be.
>> Neil Immerman: Yeah, there ought to be for sure, for sure. And connecting logic and probability, of
course, really important.
>> Ken McMillan: Right. So my other comment had to do with—you know—this characterization of P
being the practical …
>> Neil Immerman: Thanks a lot.
>> Ken McMillan: … the practical problems. You know, the P is for practical …
>> Neil Immerman: Yes, well, mathematical wrapper for practical problems.
>> Ken McMillan: But I wouldn’t say that P stands for practical; I’d say that P stands for predictable,
right?
>> Neil Immerman: Ah.
>> Ken McMillan: In other words, if I know the problem is P, I can then calculate in advance—sorry,
sorry …
>> Neil Immerman: So the fact that P is guaranteed …
>> Ken McMillan: … how long the running time will …
>> Neil Immerman: Right, it’s guaranteed feasible, right, right.
>> Ken McMillan: Whereas—you know—for NP, I don’t have that, and so SAT solvers have this long tail.
>> Neil Immerman: Right. Right, of course, yeah, yeah.
>> Ken McMillan: … a lot of times; they may do well on average for the class of problems I want to
solve, but they’re not predictable.
>> Neil Immerman: Right. Right, right.
>> Ken McMillan: And I can very easily make a tiny change in the input that doesn’t seem to matter to
me, and the SAT solver takes much longer to …
>> Neil Immerman: Right.
>> Ken McMillan: … to solve the problem. So it has this aspect of not being predictable, which is rather
like when you went to the dentist, or the optometrist—whatever it was—how long is it going to take to
settle your bill using the computer?
>> Neil Immerman: Right.
>> Ken McMillan: On average, very short, [laughter] but the tail is long, right? But I think that when you
say brittle, brittle, actually, maybe means long-tailed, right? In other words, the amount of time that it
takes me to solve the problem goes … ranges from zero to infinity in an unpredictable way, even though
the average is small, right?
>> Neil Immerman: Oh, interesting, interesting. So for me—meanwhiles, that’s very inter—what I was
thinking about brittle is sort of—you know—software’s designed for—typically—for things that the
designers have anticipated, and when something comes along that the geners have not anticipated, it’s
usually not adaptive.
>> Ken McMillan: Yeah.
>> Neil Immerman: That’s …
>> Ken McMillan: [indiscernible] keep it simple, right?
>> Neil Immerman: Yeah.
>> Ken McMillan: I can look at my input and predict what the system’s gonna do with it.
>> Neil Immerman: Right. Right, right, right.
>> Ken McMillan: Right? If it were predictable, I would be happy, right?
>> Neil Immerman: Of course, of course.
>> Ken McMillan: It could mean runtime, or it could mean something about the output.
>> Neil Immerman: Right, right, right.
>> Ken McMillan: And the output has some nice … some pleasantness property—you know—that I
want it to have.
>> Neil Immerman: But your version of unpredictable—how long the SAT solver will take—I think is very
interesting; it’s a little different. It’s a question of how much we don’t know about characterizing this
problem. Yes, it’s NP complete; that means that there’s a certain core that’s NP complete, but where do
the problems that we really want to solve live between this? It’s hard to know.
>> Ken McMillan: My sense is that you’re never gonna have that; you’re never gonna have the ability to
say, “This is a subclass of these problems that I want to solve that I can get predictable performance out
of the SAT solver from.”
>> Neil Immerman: Right.
>> Ken McMillan: The SAT solver is a highly complex object.
>> Neil Immerman: Of course.
>> Ken McMillan: The behaviors are very complex.
>> Neil Immerman: Of course.
>> Ken McMillan: I am not able to predict, looking at the input, how it’s going to behave.
>> Neil Immerman: Right, right.
>> Ken McMillan: And that’s its … that’s the difference between the SAT solver and—you know—say,
any polynomial algorithm, right?
>> Neil Immerman: Okay, okay.
>> Ken McMillan: I could very easily look at my input—all I had to do is count the size of the input—and
I can say something very strong about how that thing is going to behave.
>> Neil Immerman: Exactly, exactly.
>> Ken McMillan: Right, which is never going to be true for SAT, but your … the reason that …
>> Neil Immerman: Well, there’s sub-classes of SAT that are guaranteed will be fast, but once you know
you’re in that subset, you could run a different, more predictable algorithm, as I said.
>> Ken McMillan: Okay, then I’d do a different problem [indiscernible] I wouldn’t be using that; I
wouldn’t be using a SAT solver. So it was too …
>> Neil Immerman: Yeah, you’re only using the SAT solver when you’re not sure.
>> Ken McMillan: They were too CNF for something like that. I wouldn’t be using it …
>> Neil Immerman: That’s right; that’s right.
>> Ken McMillan: … using a SAT solver. So my point is that the reason that some of those … some of the
practical problems live outside of P—outside of predictable, right?
>> Neil Immerman: Mmhmm, mmhmm. Mmhmm.
>> Ken McMillan: Is that we’re willing to live with unpredictability.
>> Neil Immerman: Mmhmm, mmhmm.
>> Ken McMillan: We’re willing to live with the fact that sometimes, we go to the optometrist, and it
takes twenty minutes to pay the bill, because there’s some kind of software problem, because there’s a
very complex system whose behavior I’m not able to predict.
>> Neil Immerman: Correct, correct, yeah.
>> Ken McMillan: Alright, and so that … and so to the extent that we can live with that, then those
problems are prac—that we can live with the long tail—then those problems are practical, alright? And
that’s really a fun … that’s function of our tolerance—you know—that’s our function of our tolerance for
uncertainty and … as opposed anything particular about the problem space.
>> Neil Immerman: Okay, okay, interesting. See, I had this feeling—maybe it’s just naïve—that there
are design methods for software including—you know—specifying what your module’s gonna do, and
checking that that fits in correctly with the other module that would make things so more resilient.
>> Ken McMillan: So I can—right—so doing this, I can buy some predictability in the behavior of that
module by proving a property of it, like that it doesn’t crash, let’s say.
>> Neil Immerman: Right.
>> Ken McMillan: Okay, so I can buy some predictability—right—in order to do that, I have to pay
somewhat. I have to pay Nikolai—or somebody …
>> Neil Immerman: Yeah. Right. Right, right, right, right, right.
>> Ken McMillan: … to use the … to use this very flakey, unpredictable tool [laughter] in order to prove
that. So I’m really pushing the brittleness around by doing that.
>> Neil Immerman: Uh-huh, uh-huh. Interesting, interesting.
>> Ken McMillan: You see what I mean? So I’m paying so that you don’t have to pay; I’m paying in
advance.
>> Neil Immerman: Right, right, right, right, right, right. Bath … yeah—I mean—to me—you know—
again, he’s gone, but … I mean, there could be—you know—people have talked and people are talking
more about how we’re—you know—can’t we build engineering standards that require software to be
secure? And the people who build software say, “Of course not. There’s no way to do that.” But I think
as these … some of the … our tools get more … more whatever—you know—more ready for the real
world, they—you know—could have to be used. And so Microsoft requires certain tools to be used in
building software, yes?
>>: Mmhmm, not.
>> Neil Immerman: And—you know—these tools can be improved to a certain point that things would
be better, I think.
>>: Mmhmm.
>>: Yeah, and I mean, the big problem we have generally is that people don’t do formal designs, and
they jump into a Turing-complete, theory-complex programming language …
>> Neil Immerman: Right.
>>: … where you get many low-level details. And what’s been happening more recently—like in the,
maybe, last ten years—is, like, people using higher-level, domain-specific languages are using …
>> Neil Immerman: Right, right.
>>: … are doing some modelling with TLA, and at least they’re carving out smaller algorithms—they’re
still complex—but as opposed to what we did with—you know—sign-on and sent it to our verifier was:
you just give us all your C code …
>> Neil Immerman: Right.
>>: … and we’ll try to extract some sense from it.
>> Neil Immerman: As best we can’t, yeah, yeah.
>>: You know, and so I think part of it is really also saying, “Well, what is software? And how do we
get—you know—how do we get to the software from some higher level, where we have more hope for
these things?” Because we’ve done some careful design, and in a way that makes more … again, it’s
hard to say: is it tractable or not? We just know it’s simpler in some sense; there’s less detail, so it’s
easier, but I think it’s in—you know—again, it’s about having human supportive of the machine at the
right level of abstraction.
>> Neil Immerman: Mmhmm, right.
>>: There’s some logic there for that.
>> Neil Immerman: There is; there is.
>>: I mean, I think the classic example is sort of SQL with the relational algebra.
>> Neil Immerman: Yeah, yeah, yeah.
>>: Right? Which gives you all this beautiful algebraic properties, and query optimization, and …
>> Neil Immerman: Right, right, right, exactly.
>>: … we don’t have a lot of other examples like that, unfortunately. [laughs]
>> Neil Immerman: Right, right, so that’s a …
>>: I mean, we have communicating finite state machines, and Shaz has worked on a language for
having them communicate, but concurrent state machines are already … like, if you just have more than
a handful—or even a handful—you’re already in a very bad spot, so you have to—you know—carve out
your state machines and then do modelling of the environment.
>> Neil Immerman: Right. Right, right, right.
>>: So I think, also, we have to sort of think about what we mean by software, and a lot of the progress,
maybe, I think we’ll make is maybe with some higher-level signs or things like Ivy, where you have a
description—higher-level description.
>> Neil Immerman: Right, so there’s some hope—right—that says—you know—if you can force yourself
and to squeeze yourself into a certain box—right—say, “I’m going to use languages of very restricted
power.”
>>: Yeah.
>> Neil Immerman: Alright, so that all my questions become easier …
>>: Yes.
>> Neil Immerman: You know, and then the question becomes: how hard is it to get the problem I want
to solve—you know—into that restrictive language?
>>: Right, and …
>> Neil Immerman: From which point, I can get the machine to do something for me reliably.
>>: Exactly, so …
>> Neil Immerman: I think that’s sort of … that’s a hope [indiscernible] right?
>>: That’s a hope, but the … but you’re immediately going to cast out many people who will not … who
want to write some English and then say, “Ah, yes, now I know how to take my [indiscernible]”
>> Neil Immerman: Right, right.
>>: And so there’s … it’s going to be more specialized.
>> Neil Immerman: Right, so this is a cultural thing, too, because as part of—again—the descriptive
complexity, you can see the complexity on its face, so you could say, “Okay, I’m only gonna write in this
language,” and the question is: how annoying does that have to be?
>> Ken McMillan: How annoying is it to get into that language? Or y’know?
>> Neil Immerman: Right. Right, right, right. And it … again, it’s a question, Bill—I think—maybe,
perhaps, building flexible tools … I mean, so first of all is: your standard programmer’s not gonna want
to have to deal with that; you’re gonna have to make it nice for them.
>> Ken McMillan: But this is—right—this is what I was saying about—you know—paying now or paying
later.
>> Neil Immerman: Yeah. Yeah, yeah.
>> Ken McMillan: People always want to pay later.
>> Neil Immerman: Course.
>> Ken McMillan: Because, especially it … ‘cause we’re … you live in a marketplace, right?
>> Neil Immerman: Yeah.
>> Ken McMillan: And for the thing to be successful, it has to happen at exactly the right time. Too
early doesn’t work; too late doesn’t work; your timing has to be perfect.
>> Neil Immerman: Yeah.
>> Ken McMillan: Right? So you can’t pay in advance for this brittleness.
>> Neil Immerman: Right, right.
>> Ken McMillan: You can’t say, “I’m gonna subject this thing to formal analysis that’s going to have a
very long tail”—it’s going to be very long-tailed distribution.
>> Neil Immerman: Of course not, yeah.
>> Ken McMillan: Right? Because …
>> Neil Immerman: You have to get it to market.
>> Ken McMillan: You have to have … the thing you most need the predictability for is not the
correctness or the runtime; it’s time to market.
>> Neil Immerman: Right. Right, right, right, right.
>> Ken McMillan: And so that means that you’re going to pay the brittleness problem later rather than
earlier, and it explains the architecture of the world where I live in many other things …
>> Neil Immerman: Right.
>> Ken McMillan: … that you might look at and say, “My God, how could anyone think and design
something like that?”
>>: Neil Immerman: Right. Right, right.
>> Ken McMillan: But—you know—the answer is those people were working under constraints …
>> Neil Immerman: Of course.
>> Ken McMillan: … in which—you know—questions about security and correctness were sort of
secondary, tertiary …
>> Ken McMillan: Way down the line. Yeah, yeah, yeah, yeah. Right.
>> Ken McMillan: You know, and so if the market’s going to decide that what the most important thing
is is predictable time to market …
>> Neil Immerman: Right.
>> Ken McMillan: Right, then maybe some of these issues about complexity are in … that we never got
secure systems.
>> Neil Immerman: Right, right. You have to change—right—you have to change that economic
insertion structure …
>> Ken McMillan: Mmhmm.
>> Neil Immerman: … in order to get the incentive to … people actually have to prove that their
programs are safe in certain ways.
>> Ken McMillan: Mmhmm.
>> Neil Immerman: You know, you could change the liability laws, so they’d have to …
>> Ken McMillan: And that’s what I was saying about it being a first a social problem …
>> Neil Immerman: Yeah, yeah, yeah.
>> Ken McMillan: … and second a technical problem.
>> Neil Immerman: Right, right. Right, right, right. But you can’t make the laws until the technical ability
is there to—you know—that what you’re requiring is possible.
>> Ken McMillan: Mmhmm.
>> Neil Immerman: Okay, okay.
>> Ken McMillan: Don’t know. Anyway …
>> Neil Immerman: So thanks for your … thank you so much.
>>: Thanks a lot. [applause]