>> Jay Lorch: [laughs] Okay, welcome, everyone, to Colin Scott. He’s no stranger to the area; he went to the University of Washington for his undergrad; and although he’s a PhD student now at Berkeley, he spends most of his time here, collaborating with people at UW and living in our glorious area. He does research on networks and distributed systems, and ‘cause distributed systems are notoriously buggy, he spends a lot of research on ways to find bugs, and understand them, and ultimately fix them. And he’ll tell us about some of his research in this area now. >> Colin Scott: Thanks, Jay. Yeah, so like Jay mentioned, I work on a systen—sorry—systems and networking. I also pursue research that’s at the intersection of software engineering, and programming languages, and formal methods with systems—distributed systems. So part of the reason that I’m excited about Microsoft Research is that there’s an opportunity to collaborate with all four of those groups, and you have phenomenal groups … people working on that. Okay, so what I’ll be discussing today is my dissertation work; I also have some other work that I’ll briefly touch on at the very end, but this is basically my dissertation. Okay, so I’ll just go ahead and jump in. When a software developer receives a bug report for a distributed system, they typically start their debugging process by looking at the console output—or the log files—for each machine in the cluster, and their intent here is to try and understand what events in the execution caused the bug—triggered the bug. Now, if the system was running for a long period of time, these log files could be quite large; if it’s running for several hours, they could be multiple gigabytes. So on a lucky day, the developer might immediately find some obvious piece of diagnostic information—maybe like a stack trace—that immediately tells them what is the root cause. What is the line of code that is causing my problem? Or maybe they can just look at the last few events in these log files and then use their intuition to try and understand what was actually causing the bug. But often, developers are not so lucky; in these cases, they need to find the events throughout the execution—not just at the end—that were responsible for getting the system into that unsafe, buggy state. Okay, so effectively, what they need to do is filter out the events in this execution that were not relevant—there might be many thousands of extraneous events—so that they’re left with the handful of events that are responsible for triggering the bug, that help them understand what actually went wrong. So you can imagine that this whole process could be very time-consuming. So more generally, software developers spend a significant portion of their time on debugging—some form of debugging—at work. According to one study—this was actually a … I think this was a study of Microsoft product groups, just a random sampling of product groups—they found that forty-nine percent of time at work was spent on doing some form of debugging, okay? Now, some portion of that time is spent understanding: what were the environmental conditions that led the system to trigger the bug in the first place? And then, some other portion of that time is spent coming up with a fix to the code so that future executions don’t exhibit the same problem, ‘kay? So our goal is to allow developers to focus most of that effort—most of their effort—on that latter part—coming up with a fix to the code—instead of first having to understand what were the conditions that trigger the bug in the first place, okay? The way that we achieve our goal is by identifying—thanks—what we call a minimal causal sequence of events that triggers the same bug as before. And what I mean here is just that if we were to execute just this minimal sequence of events—instead of the entire execution—we would end up with the system exhibiting the same bug as before, okay? Now, the key idea here is that the smaller the execution, the easier it should be for a human to understand—the less time it should take for a human to understand what was going wrong. And in general, human time is much, much more expensive than machine time, so the value that we’re adding here is that we’re saving human time. Now, this is certainly true of my experience in debugging—I mean, it’s also corroborated by research in the field of psychology—that smaller event traces should be easier to understand. So for the rest of this talk, I’ll start by defining a simple model of distributed systems—a computational model—to give us intuition for what exactly we’re trying to achieve; then, we’ll look at how we obtain faulty executions using our tool, DEMi—the distributed execution minimizer. You could imagine— sorry—we happen to use fuzz testing—a randomized concurrency testing—to find our bugs, but you could imagine using production executions and then feeding them into our tool, and then also minimizing them. For the core of the talk, I’ll discuss how we perform minimization once we’re given one of these faulty executions; and then lastly, I’ll look at how well this works in practice before I conclude. Okay, so you can think of a distributed system as just a collection of N single-threaded processes, and each process is a little automata; so it has unbounded memory—it’s a … it’s essentially a little Turing machine—it starts in a known initial state; and then it changes states deterministically according to some transition function, and I’ll describe what that is next. Yeah? >>: Is there anything special about single-threaded? You mention it’s a single-thread process. >> Colin Scott: Well, conceptually, if you have a multi-th … if you have a machine that’s running multiple threads, you can treat that as a separate thread, as long as there’s no shared state between the threads. At the end of the talk, I’ll discuss how you could adapt this to a multi-threaded system, where … but essentially, the key idea—the key insight—is that message passing—this is a message-passing model— message passing is equivalent to shared memory. If you interposed on reads and writes to shared memory, you could treat it in exactly the same way as a message-passing system, yeah. Okay, right. So you can think of the network as just a giant buffer; it’s a buffer that keeps all messages that have been sent from one process in the cluster but not yet delivered to the … to its recipient, okay? Now, the execution takes place one step at a time. Conceptually, we’re going to linearize all of the events in this execution. In reality, these things—these processes—may be running concurrently, but there’s always some partial order that we can define, so conceptually, it’s easiest to think of it as a linear order. So one at a time, we’re going to—each step in the execution—we’re going to pick one of these pending buffers to deliver—in this ca … sorry—pending messages in the buffer to deliver—in this case, there’s only one—and then based on the contents of that message and the current state of the recipient, it will … the recipient will then go through some transition—some straight tan … sate transit … state transition—based on its transition function, okay? So it goes through a state transition, and in the process, it will send zero or more messages to other machines in the cluster; does this make sense? And note here that we can treat timers as a special kind of message; this is just a special kind of message that we deliver back to the sender at some known, later point in the execution. So in this model, we assume that the … none of these processes have access to a local clock—or at least, we interpose on that local clock, okay? Now, as an … a step in this execution could also be external, and by external, I just mean that it’s outside of the control of the processes themselves. So a couple examples might be: we … if we assume that there is some process outside of the system that we’re not modelling, it might … we might—for example, client—we might have a client send a message and put that into the message buffer; or we might have a new process be sort of non-interministically created; or we might force one of these processes through a crash recovery event—so we force it back to its initial state. Now, note here that I didn’t explicitly model fail-stop crashes where a single machine crashes, because that’s equivalent to a schedule where we just partition that process from all of the other processes in the system. So this is sort of … that … fail-stop crashes are sort of implicit in our model. >>: How ‘bout message loss or duplication? >> Colin Scott: So a message loss is equivalent to having the message just stay in the buffer indefinitely—this is an infinite buffer. Duplication, we could model that; we don’t happen to model that; but it would be a simple change to the model. [coughs] Okay, sorry. So a schedule—which we denote with this symbol here, tau—is just a sequence of these events, either external—which I’m showing here in dark green—or internal—which I’m showing here in blue. And a valid schedule is just a schedule such that, at each step—if we were execute that schedule—at each step of the schedule—the execution—the next event to play would be currently pending in the buffer. So it’s valid … it’s a valid schedule for us to execute. Now, recall that we’re trying to find bugs here, so we assume that we have access to some invariant, and this is a safety condition—some predicate p—that’s defined over the local variables of all of the processes, okay? And this predicate will tell us whether or not the system is currently in a safe or unsafe state. So an example of a invariant violation might be if you violate linearizability in the … if you’re trying to do raft consensus or some sort of consensus protocol. Okay, now ultimately, what we’re looking for is some faulty execution; this is just a particular schedule such that, if we were execute each events in a schedule, we would end up with the system violating that invariant. Does this make sense? Okay, right, so now that we have that sort of formalism in hand, we can now get an intuition for exactly what we’re trying to achieve. We assume that we’re given one of these faulty executions—some schedule, tau—such that we … if we execute it, we end up with an invariant violation. And now, what we’re looking for, informally, is a locally minimal reproducing sequence, tau prime. And just to unpack what I mean by this, by reproducing, I just mean that if we were to execute tau prime, we would also get the same invariant violation; tau prime should be at least as short as the original and should not contain any external events that were not also in the original. And then, by locally minimal, I just mean: if we were to remove any single external event, e, from tau prime, there would not exist some other schedule, tau double prime, such that if we were to—containing just the external events … same external events minus e—such that tau double prime also violates p. Yeah? >>: [indiscernible] all the states of the process are known, or …? >> Colin Scott: Sorry, what … we assume that we can always take a snapshot of the … at any point in the execution, we can just halt the state of the system—we can halt the execution—take a snapshot, and then the user is going to define—the application developer, who has some domain-specific knowledge about what kinds of bugs they’re looking for—they define this predicate over the local variables contained in that snapshot, yeah. So there is some work that the developer has to do for us. >>: I have a question. >> Colin Scott: Yeah? [coughs] >>: When the … when your message handler sends a whole bunch of messages out into the network … >> Colin Scott: Right. >>: … is that treated as an atomic step in your computation model? >> Colin Scott: Yes, and that’s actually crucial. So I’ll discuss a little bit more about the implementation details, but the systems that we interpose on, we need a well-defined point where the syst … where the process blocks—like you said, an atomic start and an atomic end. This way, we can ensure that every step is sort of … this way, we can ensure linearizability when we do our minimization. >>: But what is worrying me is that that doesn’t really match what happens in reality, right? I mean, in reality, you could have … you could do a send … >> Colin Scott: Right. >>: … and that can cause a de-queue immediately—right—before the other sends have happened, and that can cause that message handler to start executing, if you know what I mean? >> Colin Scott: Yeah, you’re absolutely right. So I think for now—for this part of the talk, where I’m just kind of defining it clearly—let’s assume that we’re alway … we’ll always be able to interpose on any point. Like you said, maybe there would be some de-queue that we don’t know about. Let’s assume for now that we’re going to interpose on it; we sort of play God; and we get to interpose on any nondeterministic event. >>: I see. >> Colin Scott: And there’s a lot of engineering effort that I had to do to make sure that, in the systems we were testing, that didn’t happen. >>: It’s just an engineering question; it’s not something foundational to your model here. I see. >> Colin Scott: Exactly, and later in the talk, I fully admit that it may be a lot of engineering effort to do … to get this to work in practice. So later in the talk, I’ll sort of touch on the engineering aspects of this, yeah. Okay, right, so the reason that we focus … we focus on removing external events in this formulation, because external events are the first layer of abstraction that developers think about when they’re reasoning about the behavior of their system. After we’ve minimized external events, we can go ahead and trying to minimize the internal events remaining in our schedule, tau prime. And conceptually, what this just means is that we’re gonna try keeping some of those messages in the buffer, and not delivering them, and seeing if we can still trigger the invariant violation. Does this make sense? >>: How will you define it in terms of removing exactly one? You can imagine removing more than one. >> Colin Scott: Yep, you’re exactly right. Yeah, so that’s … it’s an excellent point. Unfortunately, if you try to remove more than one, that’s equivalent to enumerating the power set … >>: Yes. >> Colin Scott: … and that’s a problem. So if we had infinite time … actually, well, one nice thing about our algorithm that we use is that if we had infinite time, we would give you a provably minimal result, but of course, we don’t have an infinite time, so this is why we’re looking for a local minima here. Okay, and in general—by the way—in general, if you get stuck in some local minima, you could run it again and induce some sort of nondeterminism, and we might find some other local minima that might be smaller. >>: [indiscernible] local minima. >> Colin Scott: Exactly, yep. Okay, right, so now onto how we obtain these faulty executions. [coughs] You can think of a distributed system just as an application process that runs on each machine in the cluster; I’m showing three here. That application process calls into some messaging library—some RPC library—which, in turn, calls into the operating system to have the bytes sent across the wire. So what we do is we interpose on this messaging library; in our case, we interpose on a particular library called akka, but you could imagine applying this same sort of ideas to a different messaging library. We interpose on this library such that if … whenever—sorry—whenever an application sends a message, we intercept that message before it’s sent to the operating system; so we place it into a buffer that we control. And now, at this point, we essentially get to play God, so we get to choose arbitrarily which of these pending messages we want to deliver next. So we’re going to essentially linearize the events in this execution. In this case, there’s only one choice to be made, but there might be other choices. So we record our choice to disk, including the contents of that message, and then we go ahead and allow this message to be delivered to the recipient. Now, based on the content of the message and the recipient’s current state, it’ll go through some state transition and then send some zero or more messages to other machines in the cluster. Now again, we could make some choice, or based on some probability, we might also decide to inject some external event, and let’s suppose that we decide to have this process go through a cache … crash recovery. Now, be—again—because we precisely control when each ex … when each process gets to execute or not, we can—at any point in time—we can just stop the state of the world—stop the execution—and then take a snapshot of the local state of each process, and then run our invariant over it, okay? And we’ll just keep doing that until we find our bug— some invariant violation we care about. Okay, so now, once we’re given one of these sequences recorded with our tool, now we’re gonna go ahead and try and minimize it. So to make the discussion more concrete, I’m gonna briefly … I assume most people probably know what consensus is trying to achieve, but I’m gonna briefly go over the mechanics of how raft consensus works; raft is a particular kind of consensus protocol. So in raft, we have some processy … some number of processes in the cluster, and they vote; the first thing that they do is that they vote in order to decide on some leader. So in this case we have Putin over here is our leader. >>: Why debug it? >> Colin Scott: What’s that? [laughter] Exa … that’s the joke, exactly. Putin doesn’t really need consensus; he doesn’t need a quorum, but in raft, he would need it; he would need a quorum. Okay, so let’s suppose that he received a quorum of votes; now client requests will be directed to the leader; the leader will mark in his log that he might commit this entry; he hasn’t yet committed this entry. Now he tries to replicate that entry to the rest of his peers; they also log that they might commit that entry; and then, they acknowledge back to the leader that they have received the entry. Now, at this point, the leader knows that a quorum of the processes in the cluster have received that entry in this particular slot, so it goes ahead and marks that it has committed this entry, and then it tells the rest of its peers that they can go ahead and commit that entry as well. Now of course, the hard part about consensus is that there might be failures or arbitrary message delays. So what I’m showing here is a particular bug that we found and minimized using our tool. So this is a bug in an implementation of that raft consensus protocol. So the bug here—the invariant violation— was that at the end of this execution—at this point here—we had two processes in this cluster who both believe that they were leader in the same election term. So in raft, that’s a very, very bad state to be in, because those two leaders might overwrite each other’s log entries; they might violate the linearizability … the main linearizability constraint that raft is trying to achieve, okay? So the two leader … the two processes here, by the way, were the green process and the orange process, and if we look at this minimized execution, you can almost, sort of, immediately tell what was going wrong. So the green node here requests a vote from the red node, and then the red node grants that vote, and then later on, the green node requests a vote from the very same red node and then grants … it again grants the vote. So the bug here—and by the way, the orange … the same thing happens for the orange and blue node— the bug here was that we were accepting duplicate votes from the same … the very same peer. In other words, we—the actual implementation of this—we had an integer that we incremented once where … every time we received a vote granted message, instead of—what you really wanted was—sort of a hash map that points from which peer you got it from to which … how many votes you’ve received, okay? And by the way, I gave a talk at Salesforce with … and my host was Diego Ongaro, the guy who invented raft, and he said that in his implementation of raft, they had the very same bug. So apparently, this is not … even Diego himself is having this kinds of implementation issues. Now, in retrospect, this bug is easy to figure out, but the original execution for this—this initial fuzz test that we generated—was something like fifteen hundred events. Okay, so what we’re looking for is this minimal output—something that’s ver … pretty easy for a developer to understand—and … but we’re not initially given that minimized output, right? So the initial execution we’re given might be thousand of events; now we’d like to find that minimal output. So how do we do this? One straightforward approach might be to just simply enumerate all possible schedules—all possible interleavings of external events and internal events—that are at least as short as the original. So the original is some point in this schedule space, and we’re going to enumerate all of them, and we’re going to execute every single event, and then at each step, we’re going to check the invariant violation. >>: Yeah … that’s O of n factorial many loads. >> Colin Scott: Yeah, oh, very good. [laughter] Yeah, sorry. >>: I’m wondering: why did M get to be so big in the first place? See, you said you got a fifteen hundred length trace for this thing; why didn’t you just restart and only try traces of length two, three, four, and five, and so on? >> Colin Scott: Yeah, that’s a great question. So the point is that if you do breadth-first search when you’re finding bugs, then your bugs are minimal … your executions are minimal by construction, and that is a good idea; we actually tried that; it turns out that it takes a very long time to get to interesting lengths. So most bugs happen at at least length, like, thirty, say, and even getting to executions of length thirty with breadth-first search takes a very, very long per … amount of time—like we ran it for twenty-four hours, and it still hadn’t got past, like, length five. So we just found that—I mean, in general—that this space of possible interleavings is so large that we—in our experience—we found that randomized testing was much more effective for finding bugs. Incidentally, the MoDist—I don’t know if you guys know MoDist; it was done by MSR here—they also did an evaluation; they’re doing something very similar to us in terms of finding bugs; what they found was that systematic enumeration often gets stuck in these … that only exploring some small part of the schedule space. So they—when they combine systematic exploration with randomness—they found that they found bugs much more effectively. Now of course, the problem is: when you’re using randomness, now you have to minimize, which is where our technique comes in. Yeah? >>: I have another quick question. It’s about the engineering of your system, really. >> Colin Scott: Right. >>: So you mention that there is this invariant that can refer to states of all the processes. So how did you actually implement that? I mean, the … you … the state of a process—I mean—there could be stag, and it’s just all sorts of heap and stuff. >> Colin Scott: Right. >>: How does somebody even write that, and how do you know that this thing that you wrote in the invariant refers to this bit of memory? >> Colin Scott: Yeah, good question. So in our case—in the case of raft—our job was made much, much easier, because Leslie Lamport did that work for us; he said, “Here are the well-defined safety conditions that we care about.” >>: Okay, so you can just bake it in. >> Colin Scott: Yeah, so … and then each process has the local variables that are defined by Leslie Lamport or Diego, and then we just examine the values of those local variables and see if they match what the safety conditions prescribe. Now, I totally agree that they’re some … and so most systems— unlike Leslie Lamport’s work—in most systems, we don’t write our specs first; we actually just write a bunch of code, and then maybe, we realize that we want something else, so we change our code, and it’s just really hard to actually understand what it is that we actually really want. So I’ll get to Ratul in a second; I have just one idea there. Michael Ernst at UW did his PhD dissertation on this thing called Daikon; it’s inferring likely invariants. So this is one way that you could help developers sort of … >>: No, I think that you read too much into my question. I’m just talking … >> Colin Scott: [laughs] Oh, okay, great. >>: … basics. Well, I’m making sure … >> Colin Scott: Yeah. >>: How do you write code that simultaneously looks over the state of multiple processes? >> Colin Scott: Well, a distributed snapshot is a well-known … in our case, we … I don’t actually need Chandy-Lamport, but a distributed snapshot, conceptually, just takes a consistent snapshot of all the local variables. So you … what … if you can—sorry—as long as you can look at the local variables of one process—you say, “I know the locations in memory that I want to examine”—then you can just use that same code to run it across multiple processes. So there are separate questions: how do you obtain the snapshot, and then how do you run the invariant? I think Ratul was first, and then I’ll get … >>: You saw one thing that’s a … invariants that we’re running here, like some invariants may get violated, like, transiently. >> Colin Scott: Yeah, good … yeah. >>: And so does your code know when to actually value its own invariants? >> Colin Scott: So if it—sorry—if your safety condition is violated transiently … so a safety condit … the definition of safety condition is that it should never be violated. So if it’s violated transiently, that means that maybe you haven’t … one answer—one snarky answer—might be that you just haven’t written your invariant violation … or invariant very well. That you sh … >>: I think we have maybe converse ... so I was just thinking, like, take distributed-out computation; let’s say a set of processes are trying to compute shortest paths. >> Colin Scott: Right. >>: Right, so I think: before the whole thing converges, like, paths are not short … >> Colin Scott: Right. >>: … and what your invariant is correctness, that—you know—after things converge, you should basically get shortest paths. >> Colin Scott: The … so what you’re actually describing is a Leibnitz condition. I think that’s the challenge. >>: So it’s a safety condition, but it still the … you need to be defined as after the [indiscernible] that is your safety … it’s part of your safety property. After you converge, then paths are short, right? So before you converge, you just [indiscernible] >>: But how does … I’m just wondering, like, how does this engine know that things have converged, and now’s the right time to check it, versus you shouldn’t have checked it before this has happened? Or these types of invariants are not checkable? I’m trying to … >> Colin Scott: No, no, I think they are checkable. I think in practice, like, it … people do—developers— do exactly what you said; they define some sort of threshold. They say, “If five seconds have passed, then the assertion should hold.” And the problem with that—like you said—is that it can be transiently violated. If the execution times are slightly different, then all of a sudden, we have flakiness. I don’t know. I don’t have a great answer. I mean, one thing you could do is empirically build a distribution of how long the convergent times take and then pick some time in that convergence ti … in that distribution that would be reasonable. I mean, it’s a hard question. What you really want … I think what you really want is a Leibnitz condition, but it’s … unfortunately, Leibnitz involves infinity, and you can’t execute systems infinitely long, so … there’s a—sorry—there is one answer. There’s a nice paper called MaceMC. I don’t know if you’ve heard of it. The basic idea is that you can convert Leibnitz conditions into safety conditions using some heuristics. So they do random walks of the search space, and they say, “If we’ve done n random walks, and we haven’t … and the Leibnitz condition still doesn’t hold, then we’ll … well, we’ll consider that a Leibnitz condition … a Leibnitz violation.” And that’s … that might be one way of getting rid of this transient issue you mentioned. I don’t know if that … are you satisfied? >>: We can continue to talk offline later, perhaps. >> Colin Scott: Yeah. Sure. >>: [indiscernible] You said you could do the minimization in factorial time. Before, you said that you— pity for you—if you were given infinite time, you could do it, but actually, you … >> Colin Scott: Yeah, you’re right. I meant factorial time. That’s right. Yeah, you’re absolutely right. Which is still … could be potentially years. Right, okay, so one observation that others have made is that many of the schedules in this schedule space are commutative. And I’ll explain what this means, but incidentally, I don’t know if … many of you probably know Patrice Godefroid. This was actually his dissertation work on this topic. So the basic idea here is that: suppose we have two pending events, i2 and i3, or they’re both in the message buffer at the same time, and let’s further assume that they have different destinations—so they’re destined for different processes—and they are concurrent with each other according to happens-before relation. So they’re … essentially, they’re happening on different machines. Now, if you consider the state of the overall system at some step, n, before we execute either of those, and then we decide to deliver i2 and then deliver i3, we’ll actually end up at exactly the same state if we had instead … as if we had instead executed—sorry—delivered i3 before i2. So we know that any schedules that only differ in the order of i2 and i3 are … will end up … end us up in the exact same state. Now, the algorithm that we use to reason about this kind of commutativity is called a Dynamic Partial Order Reduction—again, this was Patrice. Conceptually … sorry, I guess I’ll explain how this works in a few slides, but essentially, what it does for us is it allows us to only explore one of those commutative schedules, and the rest of the equivalent schedules we get to ignore. Okay, so this is great. We’ve reduced the size of the schedule space by some factor k, where k … again, I’m being somewhat sort of loose in my notation, but k is like the size of each equivalence class. But in general, this is still intractable. So our approach is to prioritize the order in which we explore schedules within this schedule space, and the idea here is that we’re given … the user is going to give us some fixed time budget—they’re only willing to wait so long before they hit control C—and then we’re going to try and make as much minimization progress as we can within that time budget. Okay now, some of you may observe—may have already observed—that for any prioritization function that we choose, an adversary could compute … or—sorry—could construct the program under test, or even just the initial execution, such that whatever prioritization function we chose makes very, very little progress within our time budget. And the reasoning behind this objection is actually absolutely sound, but our conjecture is that the systems that we care about in practice are not adversarial—in particular, they exhibit some set of program properties, or they adhere to some constrained computational models that make them amenable to the kinds of prioritization functions that we’re gonna define. Does this make sense? So the research agenda is really trying to define what are these program properties that we care about in practice and then defining prioritization functions for them. So I’ll briefly go over the main program property that we assume. Consider a single process. You can think of the single process as a state machine—some potentially-infinite I/O automata—and each transition here is the event of receiving a message. Now, at each state in this state machine, we have some set of local variables. I’m only showing two here—x and y—but you could imagine it might have many more local variables. Now consider the cross product of all of the processes in the cluster. This is, again, some state machine. This defines, sort of, the behavior of the overall system. Now, let’s assume that two properties hold. One is that any given invariant—we might care about lots of different invariants—but any single invariant is defined over a small subset of the process’s local variables. So we only look at, maybe, x and not x and y. In this case … again, imagine that there are many, many variables here. Another property would be that each event—each transition in this diagram here—only affects a small subset of the receiver’s variables’ values. So in other words, when we receive a message, the receiver is not going to flip all of its local variables, it’s only gonna flip some local variables that are relevant to that kind of message, okay? Now, if those two facts hold, then it seems highly likely that the initial execution—the initial execution is some path through this overall state machine here—that initial execution will define a path that contains loops. And essentially, what we’re going to do is remove those loops. So if we’re—sorry—if we were to reduce this state machine to only transitions that only affect the invariant’s local variables, then that state machine would contain loops, and we’re going to reduce those loops. Does make sense? Okay, now the challenge, of course, is that we don’t know … we treat the … we’re treating the system as a black box, and because we’re treating it as a black box, we don’t know which local variables are actually relevant or not, or which events are actually relevant or not. So our approach is going to be … is going to experiment … we’re going to experiment with different executions in order to infer what those local variables or what those events are. Does this make sense? Okay, and the key insight—sorry—the key insight here is as follows: we know one very important piece of information, which is that the original execution, when we execute it, caused the system to violate the invariant. So you can think of this original execution as sort of a guide for how we can get the system to progress through its state machine so that it ends up in this buggy state here. But again, like I said, there are some … there are gonna be some loops in that st … in the path through that state machine. So what we’re going to do is we’re going to selectively mask off some of these original events and see if we can still trigger the invariant violation so that we can find a shorter execution. Okay, so the way we do that is somewhat detailed; I’ll walk through it slowly. Recall that we’re trying to minimize external events first, so just consider the external events from that original execution. Now let’s suppose that we only consider the right half of these external events. We’re going to ignore the left half. Now, the algorithm that we use to do this splitting between right halfed and left half is called Delta Debugging. Conceptually, you can just think of it as just a modified version of binary search. Okay, so now, [coughs] we’re just gonna consider these last three external events, and now, we’d like to know: is there some schedule—some execution—containing just these three external events that would trigger the same invariant violation? So the way that we find that schedule was as follows. We walk through each of the original ext … internal events, and we check: is that message that we originally delivered, at this point in the execution, is it currently pending in the buffer at … in our current execution? And if it is, let’s go ahead and deliver it. If we ever get to a message that is not currently pending, we’ll just skip it over. And we’ll keep doing this for all the original messages that we delivered in the original execution, okay? And now at the end, we just check: is the invariant still there? Now, let’s suppose in this case that it was; it was still there. What that means is that we can now ignore the first three external events. Those are not necessary. >>: Does this require re-execution? >> Colin Scott: Yes. Yeah, exactly. We’re going to re-execute, starting from a known initial state, for each of these schedules, yep. Okay, now we’re gonna proceed with the rest of our minimization. So now we’re gonna consider just the rest … right half of the remaining external events, and again, we’re gonna try and find some schedule. Now, let’ suppose, in this case, that we didn’t find the invariant violation at the end. That doesn’t necessarily mean that we’re done. There could be some other schedule containing just these last two external events that also triggers the same invariant violation. So remember that algorithm I told you about earlier, DPOR. D … what DPOR will do will … it will set backtrack points at each step in this execution where there was some alternate, noncommutative choice it might have made about which message to deliver next, okay? Now, what we’re going to do is continue exploring these backtrack points until either we find some schedule that triggers the invariant violation, or until we run out of our time budget for that subsequence, okay? Then we’ll continue this until we finally produce our minimal output. Now, there was one major detail which I swept under the rug here, which is that it’s not always straightforward for us to compare messages in the current execution with messages from the original execution, because we’re essentially modifying history. So as an example here, let’s look at one … we have some message that we originally delivered on the left here. And now, in our modified execution, we’ve got some pending message, which looks very similar except for one field, which is our sequence number here. So let’s suppose, in this system, that the processes keep a sequence number; this is just an integer that they increment by one for every message they send or receive. Now, we removed some of the events, so now the sequence number has lower value. But we, as humans, know that—at least for most bugs invo … for any bugs that don’t involve the sequence number, which should be most bugs—we know that we should be able to mask over that message field. We know that that … the value of that local variable should not affect whether or not we trigger the bug. So we know that we can mask over that when we’re comparing the equivalency of these two messages. And by the way, the intuition for this observation is exactly the same as the one I showed you earlier. This sequence number in the message is reflected in some local variable at each process, and that local variable does not happen to affect our vi … invariant. Yeah? >>: Is it possible to infer this by analyzing the program … >> Colin Scott: Yes. >>: … to see some local var … the relation between local variables? >> Colin Scott: Yes, that’s exactly right. Another … an even another thing you might do—other than applying a program analysis—you might also try and experimentally try and infer which of these fields are nondeterministic. We don’t do that. For our prototype, we just assume that the user is going to give us some fingerprint function which tells us which fields to ignore. But I think you’re absolutely right; that’s a … that would be an interesting avenue for future work. In general, we want to decrease the amount of engineering work it would … effort it takes to get our system to work. Okay, so in the first phase, we’re gonna use this user-defined fingerprint function to choose the initial schedule that DPOR executes. And then, in the second phase, what we’ll do is we’ll only match messages according to their type. So by type here, I mean a class tag or an object type—at a lang … a language-level class tag. And the intuition here is that messages that have the same type should be semantically similar to the original, except they’re gonna differ in some field … some of the messa … the values of the message fields. So again, we’re trying to stay as close as possible to the original execution, except now, we’re going to explore backtrack points that only differ in the type in small numb … small values of the fields. This make sense? Yeah, Ratul? >>: So tell me something. So what exactly happens … so suppose you don’t … you want to drop message with sequence number three and instead, go directly to one with five; your matching will say they’re equivalent, but when you inject it into your system, is the sequence number three or five? >> Colin Scott: In the example I gave, we’re trying to play this message on the left here, but it doesn’t exist in our pending buffer. There’s only one message that matches; it just happens to have a sequence number of three. So our user-defined … your … the user-defined function tells us that these two are equivalent. Now, suppose that there were … there was some ambiguity—there was some other message that had the same type—in that case, we’re going to explore a backtrack point. Sorry, I have too many animations. So if there’s ambiguity in which message we might choose—there are multiple pending messages that match—we can … we’ll try one of ‘em now, and then backtrack, and try the other one in its place later, if that makes sense. Okay, so the last observation we make is that often, the contents of external messages affect the behavior of the system. And if we try and minimize the contents of those external messages, we can get better minimization. So I’ll go over a brief example here. Let’s suppose that our system assumes that we have some bootstrap message. This bootstrap message tells each process what are their peers. So in the beginning of time, they don’t know which … who their peers are, and we send them … at the … in our execution, we happen to send them a bootstrap message with a list—a, b, c, d, e—which tells them the cluster. Now, let’s suppose that we’re going to mask off some of these values of that list. We’ll mask off one, and then we’ll mask off two. At some point, the quorum size will actually change; it changes from three to two, and now, the remaining processes in this cluster have to wait for fewer messages before they can achieve quorum. So if we minimize these external contents—which we control—we can in fact get better internal minimization by doing this. Does this make sense? Okay. >>: Is it the saving [indiscernible] I mean, by changing the quorum size, you might miss a bug, or— right—maybe the bug was only happening—you know—with f equals two, because—you know—you sort of hard-coded a quarine of looking for two messages only or something. >> Colin Scott: Yeah. >>: And they won’t find the schedule. >> Colin Scott: Yeah, so we’re always doing this experimentally. So in that case—in the case that you described—if we tried … we had the original execution, which triggered the bug, and which had five entries; we try masking off one of the contents; and everything we try doesn’t trigger the bug. We try masking off two; everything we tried doesn’t trigger the bug. So we say … so we put our hands up, and we say, “Okay, fine, we’ll go back to five.” Yeah. Okay, so just to … yeah, go ahead. >>: Yeah, so back to that example, so I think just follow-up to Christoph original question. So this … there’s the process of exploring this state space; you can do that in a very smart way; and you argue that—you know—combining that with randomness actually helps sporting a bigger space. >> Colin Scott: Right. >>: But for this particular one, I think it probably be better off having—you know—three … >> Colin Scott: From the be … very beginning. >>: Yeah, we will probably find the same bug … >> Coin Scott: Yeah. >>: … if it actually shows up. >> Colin Scott: I totally agree. So if you’re just doing randomized testing, you should just start with a small cluster size, and then increment it. However, if you’re trying to minimize production executions, you don’t necessarily have control over the size of the initial cluster. So this technique might be more useful for a production run, yeah. >>: But hang on one second. I mean, your technique fundamentally requires this ability to replay executions. >> Colin Scott: Yeah. >>: If you have generated a fault from a production execution, how are you going to do replay? >> Colin Scott: That’s a … it’s a great question. We don’t … we have a section … I have a dissection in my dissertation which discusses how you might do that. So you need a couple things; you need a partial order—so you need … you might need a Lamport clock on all of your messages, so you can always partially order the execution. >>: Uh-huh. >> Colin Scott: You would also need to make sure that—like you said—that each external event, like a failure, is non-redundant—so you have exactly one of those, a unique event for each external event. So that might … I totally admit that that might be hard to obtain in practice. We also have some thoughts about how you might reduce that overhead. So if you’re looking at the kinds of systems we’re looking at, which are actor systems or message-passing systems, it gets a lot easier; you don’t actually need— because there’s no shared state—you don’t actually need Lamport clocks. So we have some thoughts about that, but we haven’t pursued it yet. Yeah? >>: Did you let these … skipping something like [indiscernible] is impossible. >> Colin Scott: Sorry, say that again? So … >>: Do you assume that you are just skipping over some events—that’s what you’re doing as in this binary search. >> Colin Scott: Yeah. >>: This message being [indiscernible] You assume that your doing that will not crash [indiscernible] >> Colin Scott: Ah, you’re absolutely right that—sorry—that when we do this splitting between left half and right half, you’re absolutely right that we might have an invalid or semantically invalid split, which might cause a crash or just doesn’t … it’s nonsensical set of external events. >>: So that’s the invariant. >> Colin Scott: Yeah, so in that case, we assume that the invariant that the user defines correctly disambiguates those cases. So if there’s a crash that we don’t … that was not our original bug, we’ll save that for later, like, “By the way, we’ve figured out a way to trigger a new bug,” but we’re not gonna use that for a minimization, yeah. Okay, almost done here, so I’m just summarizing all of our techniques in one slide here—I guess you can read yourself. One thing I will say is that once we’ve minimized external events, we’ll go ahead and use the same techniques that we use to minimize external events to then minimize internal events. So like I said, that just means we’re gonna keep them in the buffer, and not deliver them, and see if we can still trigger the bug. Okay, so before I end, I’ll quickly look at how well this works in practice. We’ve applied our tool to two different system—distributed systems—so far; one is an implementation of the raft consensus protocol; and the other is the Spark data analytics engine. So we should note … you should note that these are obviously two very different systems; they’re trying to achieve very different things; they have very different behaviors. And also note that the raft implement we looked at was a somewhat early-stage development project, whereas Spark is obviously very, very mature. So what I’m showing here is the size of the executions on the y-axis, and then on the x-axis is each of our case studies—so each of this is a distinct bug. And then the blue bars here are showing the size of the initial execution that we found with randomized concurrency testing; and then the green bars here show the size of the minimal out … the minimized output that our tool produces. So a few takeaways from this graph: I don’t actually show it here, but it turn … we found that randomized concurrency testing—or fuzz testing—turns out to be very useful for uncovering bugs that developers didn’t anticipate. So that … so this raft implementation had existing unit tests and integration tests, but each unit test only told explores one particular ordering, and there’s an exponential space of possible orderings, so there are a lot of things that they didn’t anticipate. Yeah, go ahead. >>: Question about the use of the phrase “fuzz testing.” >> Colin Scott: Yeah. >>: I think by that phrase, you mean that you have control over all scheduling choices. >> Colin Scott: That’s right. >>: And you randomly pick one choice. >> Colin Scott: Yep. >>: Right? So I just wanted to point out that that’s what you’re assuming. A lot of people in the industry … >> Colin Scott: Yep? >>: … they just think that: “Oh—you know—fuzz testing is when I artificially inject some delays here and there; I create big workloads to indirectly—not directly—influence schedules.” >> Colin Scott: Right, right. >>: Okay. >> Colin Scott: And I agree that in production, that’s actually might … that might be what you want to do. So in production, it might be very … it might be too much engineering effort to actually interpose on all the sources of nondeterminism, so one way to deal with that nondeterminism would be to just replay multiple times—flip the coin multiple times—and then hope that one of those coin flips is gonna trigger the bug. So that’s one way. We had a prior … the first chapter of my dissertation, we had a prior SIGCOMM paper on how you would do this for systems where you don’t have as much control. One messy part is: now you come dependent on wall clock time, which is not … it’s not a clean way of thinking about the problem, but it’s more practical, yeah. But you could, in practice, do that without enough interposition. Yeah? >>: So raft and Spark here are re-implements on that [indiscernible] where you took some implementation, and you’re looking at bugs that already existed. What are these bugs here that you’re talking about? >> Colin Scott: So these are … so—sorry—Spark and raft were both implemented in Scala; well, Spark is actually multiple languages, but the play … the parts of the process that we interposed on were Scala. Raft is completely written in Scala. >>: So you used implementations that already there? >> Colin Scott: That’s right. >>: I see. >> Colin Scott: So Spark obviously is on Apache, and then this one we found on GitHub. These bugs … all of the bugs in raft, in fact, were un … previously unknown. So all we did was we sat down and re … we wrote down all the five safety conditions that Leslie Lamport prescribed, and then we just random … we did randomized testing, and then for each … and then we checked those five invariants at each step, and then this is what we uncovered. Yeah? >>: Were these fixed bugs at the end? >> Colin Scott: Yes. Yeah. >>: Then if it’s a fixed bug, is it possible to come up with the optimal number of messages? >> Colin Scott: Yes. I’ll get to that in one slide. >>: Okay. >> Colin Scott: Okay, so what we’re showing here is that across all of our case studies, we get pretty good reduction. We’re improving the state of the art. So they would have started with this, and we give them this. So we’re … it seems like we’re helping developers, but you might also ask, “How much room is there for improvement? How far off are we from optimal?” And so what I did here was, again, I’m showing here the green bars are the size of our … the output of our tool, and then the orange bars are the minimal trace—so this is the smallest manual trace that I could produce by hand. So it took me … it was actually fairly painstaking; it took me about a month to do this, but I … what it shows here is how much room there is for improvement. So across all of our case studies, we’re within a factor of five—of four point six, actually—from that smallest manual trace. In these two cases here—raft-58a and raft42—part of the reason that we were so inflated that we were far away from that optimal was that we ran out of our time budget. So what I’m showing here is the minimization runtime on the y-axis, and we had some maximum time budget. But in general, across all of our case studies, we’re doing pretty well—most of these finish within ten minutes. In a few cases, if we had a better or a more smart prioritization function, we could have found our minimal output in much less time. So there is definitely room for improvement, but in general, we’re doing pretty well. Yeah? >>: How many Berthas are you running, and how does that play into the size of the trace? >> Colin Scott: Yeah, good question. We were … in this case, we were only running four processes in— sorry—for raft, we were running four processes; for Spark, we were running—I mean, they have a master and a … they have a whole bunch of processes—like, roughly a dozen. If we had more processes, I think the executions … so in practice, the executions could be much, much larger, so … >>: Yeah, I’m impressed that there are bugs that require even, like, fifty-something messages to reproduce in four processes. So that’s impressive. >> Colin Scott: Right. Yeah, that’s right. It’s essentially be … that’s … if you think about it, this is only a couple of round-trips. >>: Yeah. >> Colin Scott: If you have four processes, and there’s some bootstrapping … >>: Oh, each RPC is its two, and then … >> Colin Scott: Yeah. >>: Yeah, alright. >> Colin Scott: And there’s always gonna be some bootstrapping messages that are always gonna be present in the execution, so … okay. Alright, so there are many more details that I didn’t have a chance to go over here. I’d encourage you to check out our NSDI paper for a much more lucid explanation of those details. So I probably don’t have time to do a demo. I have a pretty cool demo, it’s like GDB for your … it’s essentially a GDB breakpoint for your network. So you get to choose which pending message to deliver next. So I think it’s pretty neat. >>: Yeah. >>: Go ahead. Do it. >>: Let’s do it. >>: Do it. >>: Let’s do it. >> Colin Scott: Okay, sweet. Okay, so I’ve got a Java program here. This is a … I com … what I did is I compiled the akka … the—sorry—the raft implementation, which is a Scala or Java program, and then I interpose on … so I used AspectJ to interpose on the RPC library that it uses. So I’ve got this dash-dashinteractive; that’s just telling our tool how it’s gonna behave. So what I’m showing here: I’ve got four raft processes in this cluster, and then, I’m just printing out the external events. So we have … in this case, we … the processes assume they have some bootstrap message that tells them their peers, and then, we also have two client commands: in this case, append word—please append a word to the raft lock—and then this here is just our prompt for the console. So a little bit further up, I’m just showing you: this is just the console; this is—here at the top—is the console output printed out by each process. So it’s just saying, “Oh, I’m getting ready to be … to run my raft implementation.” And then, now, what I’m showing here is the set of pending events. So these are the messages that have been sent, but not yet delivered. So we interposed on them, and then they … and then we get to choose, now, which of these we’re gonna deliver next. So now I’ve got a little—essentially—like a GDB prompt. I just type “help,” and it tells us … tells me all the commands I can run. And then I … then … so now, what I might do is deliver some message, or check the invariant, or cause one of the processes to fail, et cetera. So let’s suppose that we’re gonna … so the way … >>: Scaling all that on … processes on your machine, right? >> Colin Scott: Yeah, so … yeah, that’s right. So each of these processes is running locally on my machine, but they are not aware of colocation. So as far as they know … >>: Oh, okay. >> Colin Scott: … as far as they know, they’re running a distributed system across the network. >>: Yeah. >> Colin Scott: Okay, so now, let’s suppose that we allow the first process to have its bootstrap message. Now it’s gonna set a timer. It set a timer, and it says, “I’m gonna try—when this timer goes off—I’m gonna try and elect myself as leader.” So let’s go ahead and allow that timer message to go through. Now, it’s going through a state transition where it wants to begin an election. Again, I’ll let that message go through, and now, it’s sent request-vote messages to all of the other processes in the cluster; so it’s gonna go and try and get itself elected. Now, if I had some bug in mind about—some particular interleaving in mind—about how to trigger some bug, I could use this system to try and trigger it. And then, at any point, I can check the invariant, and in this case, we haven’t done much, so there’s no var … there’s no violation yet, but we could just keep doing this until we find the bug. And now when I exit out of this prompt, we saved a recording of the … of all the events that we played. So I could now, if I wanted to, replay the execution that I had generated here. I could also do this programatically instead of doing it interactively; I could also just make random choices programmatically. And one thing to note here is that because we’re interposing on timers, we’re able to execute these … run these executions much, much faster than you would be able to in practice, and because we don’t have to wait for the wall clock time of each timer to go off. So it’s actually pretty amazing how many executions we can go through in a minute. If we get lucky, we might trigger the bug. I know there’s a known bug here, but maybe I won’t test my luck. So I actually have a … I also saved a recording of some fuzz test that ended up with triggering the invariant violation, so now, what I’m a do is try and minimize the recording that I put to disk. So now, what it’s doing is it’s—like I said—it’s … what I describe in the talk is just experimentally trying smaller subsequences of events, and eventually—after about ten minutes—it’ll stop and tell us that it has some minimal output. Now, one cool piece of this—sorry—I think one cool aspect of this tool is that we’re now allowing developers to generate regression tests without having to write any—well, they have to write a little bit of code—but without having to write much code. They just told our tool, “Hey, please fuzz,” and given … using the invariant that we gave the tool, it’ll just run fuzz tests until it finds some bug, and then it’ll minimize them, and then it’ll save them to disk. And now, we can actually go change the system. We can go add print statements to help us debug, or we might even change how the system behaves. And then we can just—say, a month later—we can just rerun that execution that we saved to disk—that minimal execution we saved to disk—and unless the protocol has changed … if the protocol radically changes then that save … that recording will no longer be valid, but as long as the protocol doesn’t change too much, we can rerun that test to see if the bug is … it came up again. So I think this is pretty neat. >>: So what is the complexity of this minimizing tool that you are running right now? >>: Colin Scott: Well, the complexity in the worst case is n factorial, like what we said. >>: [indiscernible] factorial. >> Colin Scott: Yeah, but we … our … I actually have system configured to exit fairly soon. So I give it a small time budget; I give it a time budget of … well, this—in this case—it’ll actually finish in about ten minutes. Yeah, I don’t know if we want to wait ten minutes, but … [laughter] yeah? >>: How large is the raft program you are testing with? >> Colin Scott: Raft is relatively small; it’s fifteen hundred lines of Scala. Spark is much, much larger; Spark is something like forty thousand lines of code. >>: Yeah, but you only interpose certain messages—right—you are not trying all the possibilities, right? For instance, I start with a fifteen hundred lines, but I assume the code that are touched will execute probably a small fraction. >> Colin Scott: That’s right. So in … so one major advantage of doing black-box testing is … yes? >>: [indiscernible] sorry. >> Colin Scott: Oh, sorry. One major advantage of doing black-bock testing is that the system might be written in multiple languages—like you said, there might be lots and lots of code that’s actually been used—and we’re agnostic to that. >>: I understand. Just walk through together at … but did you know how big of the raft program that is actually involved in this? >> Colin Scott: There was a little bit of work I needed to do to deal with nondeterminism, so there are some sources of nondeterminism outside of the … conceptually, all we do is we interpose on the RP slayer, and that’s it. So we actually interpose below the application; we shouldn’t have to touch it; but in some cases, they also depend on nondeterminism that’s outside of the control of the RPC library. So an example would be like a hash map; suppose you keep values in a hash map, and then you iterate through all the values; the order in which the JVM chooses to put values in the hash map depends on the memory address, which is nondeterministic. So in some cases, we had to sort the values of the hash map to get better determinism. So there was some changes we needed to make the application, but for the most part, we’re basically agnostic to it—conceptually, we’re agnostic to it. >>: Mmhmm. >> Colin Scott: Yeah? >>: You talk about fixating this to multi-threaded apps … >> Colin Scott: Yeah. >>: [indiscernible] Yeah. >> Colin Scott: Yeah, good question. I’ll just go right to a slide here. So like I said earlier, shared memory is functionally equivalent to message passing, so there’re actually some existing … there are a bunch of other papers that look at—essentially what we’re trying to do—systematic exploration of schedules for multi-core systems instead of distributed systems. And the basic idea here is that you just interpose on the language runtime, so you detect whenever the pro … whenever a thread writes or reads to shared memory, and then you trap, and you block that thread, and then you treat it as if we had just sent a message—it’s conceptually the same. It would be if some amount of engineering effort to interpose on that runtime—which we haven’t done yet—but you could do it. >>: And the invariant is [indiscernible] shared. >> Colin Scott: Yeah, the invariant would be defined over the shared memory, yeah. >>: Better not be using too much shared memory over net scale. >> Colin Scott: That’s right, or the schedule space would be too huge. And then—I don’t know—Shaz Qadeer has a bunch of techniques for helping with that problem, yeah. [laughs] >>: I’m a huge fan of binary search, so I’m glad you’re using it productively. I have comment about the problem formulation. >> Colin Scott: Yeah. >>: You start out by saying you want to minimize an existing trace … >> Colin Scott: Right. >>: I’m wondering if you had set the problem in the following alternative way … >> Colin Scott: Right. >>: … and you just say that I have a faulty execution, and I have a generic tool for doing prioritized search available to me, except that the tool needs a particular prioritization function. >> Colin Scott: Yeah. >>: And all I’m going to do is—I’m not gonna bother trying to minimize this execution—I will just try to learn from that execution a prioritization function, or maybe a family of prioritization functions, that’s likely to meet to a violation of the same invariant. >> Colin Scott: Yep. >>: Forget about trying to minimize that particular execution itself. >> Colin Scott: Right. >>: So how would you compare that problem formulation to what you already have? >> Colin Scott: I actually think your intuition is right that it’s very, very similar—it’s almost the same. >>: Okay. >> Colin Scott: So in future work … we don’t currently learn. So I have some master’s students who’re— who I’m working with—who are looking at using this tool called Synoptic; Synoptic looks at log files and then tries to infer it … learn a state machine from those log files. So what they’re looking at is try—like you said—trying to learn some prioritization function based on the behavior of the system—the observed behavior of the system. So another answer to your question is: conceptually, what we’re doing is essentially explicit state model checking. >>: Okay, prioritized search. >> Colin Scott: Prioritized search. >>: Yeah. >> Colin Scott: And … except that we have one additional piece of information, which is we know that the original execution triggered the bug. >>: Right. >> Colin Scott: But that’s essentially what we’re doing. >>: Okay. >> Colin Scott. Yep. Okay, I had a couple slides here before I ended. So this is the technical conclusion—y’know—the results we found for those two systems were pretty … leave us pretty optimistic that these kinds of techniques can be applied more broadly. Our tool is open source. You can check it up on GitHub. And of course, I’d encourage you to check out our paper. I had a … I wanted to go over a couple slides about where I want to go—where my research trajectory is currently headed. I have done work on a lot of prior topics, not just minimization. The common thread through all of these is troubleshooting and reliability. My area … my background is in networking distributed systems. I think this is an increasingly-important area, because everything is becoming distributed. We all have smartphones in our pockets. But unfortunately, the kinds of tools that we use to develop concurrent and distributed systems lag significantly behind with the kinds of tools we have for sequential code. So actually, this is a quote from Parkinson; Parkinson is an MSR researcher. I never met this person, but they have a conjecture that—or a claim—that the kinds of tools that we use to develop concurrent or distributed code lags behind sequential tools by about a decade. So I see … >>: More like twenty years. >> Colin Scott: Yeah, maybe more, that’s right. [laughter] Maybe even more. So I see a great need to kind of bridge this gap. Now, there are great … there’s a lot of great research from the software engineering, and the programming languages, 401 methods communities on how to debug, verify, et cetera—lots and lots of cool tools. There’s amazing things you can do with program analysis. >>: [indiscernible] I thought it was sequential. Sequential [indiscernible] what does sequential mean? >> Colin Scott: Well, a conc … a distributed system is just a concurrent system that happens to also have partial failure in asynchrony. So what I really just mean: I’m just trying to distinguish between concurrency and sequential code. Most of … when you … lots of verification tools, for example, assume a single-threaded sequential computational model. >>: There’s also huge gap between, like, concurrently working and also [indiscernible] >> Colin Scott: That’s right. Yeah, that’s right. >>: Numbers together one place. >> Colin Scott: That’s right. I think parsh … you’re exactly right. Partial failure makes things even worse—even worse. So anyway, the point is that I see a great need to bridge this gap, and there’s a lot of great research from these two communities that we could use to help us deal with these problems. So I have a bunch of projects that I would like to pursue. These are just some ideas that I’ve had in the last five years for ideas that we could … where we could take ideas from software engineering and programming languages to help us adapt them—it’s not just straightforward adaptation, we just have … we have to do some research here—but we could take these ideas and apply them to problems in concurrency and distributed systems. So this is basically the direction that I’m planning to move in. So thanks a lot. If you have any questions you can e-mail me. My e-mail’s very easy to remember, [email protected] [laughter] Thank you. [applause] >> Jay Lorch: We peppered him with many questions throughout the talk. Are there any remaining? Okay, great. Then we’re done. >>: Whoo-hoo.