>> Eric Horvitz: Okay, we'll get started. It's an honor today to have Marty Tenenbaum join us.
Marty is currently, he founded and runs Cancer Commons. It's a nonprofit, open science initiative where cancer patients are treated aimed at getting cancer patients the best treatments in accord with the latest knowledge and with that knowledge, for the vision, is continually updated on each patient's response; and you can find out more about that or we’ll be hearing about it of course at cancercommons.org. That's one word. Marty is a well-known
AI researcher, Internet entrepreneur. Beyond Cancer Commons he founded CollabRX based on getting the best treatments to cancer patients. He's a well-known Internet commerce pioneer.
He cofounded Enterprise Integration Technologies which was acquired by VeriFone,
CommerceNet, Veo Systems which was acquired by Commerce One, actually even before the
Internet craze that took over a couple years later, Webify Solutions, acquired by IBM. He’s served as an officer and Director of Medstory, which our company acquired a few years ago.
Medstory, we had our Healthcare Solutions Group, and he's a fellow and former board member of the AAAI. He was a consulting professor at Stanford in Pure Science for a number of years and currently on the boards of Efficient Finance, Patients Like Me, and the Public Library of
Science, PLoS, which we all know. He did his bachelors, his masters, and his PhD, let me clarify,
Bachelors and Masters at MIT, PhD at Stanford. I didn’t want to get that wrong for the alumni in the audience here. Thanks a lot Marty for being here.
>> Marty Tenenbaum: All right. It's a great pleasure to be at Microsoft Research, and I have many, many friends here. I have some colleagues here. I'm here to try to enlist your help in the battle against this horrific disease, and what I mean by Beating Your Cancer is we're really on the cusp of a new era where we can to try to match precision targeted therapies to the precise targets of a particular cancer patient based on information about what works best for people with of those combination of targets. And to do that kind of, to get that kind of information we are going to have to learn in a much more efficient way than has been traditionally possible through clinical trials or the clinical practice of medicine. And so informatics plays a large part,
[inaudible] revolution plays a large part, and it’s the combination of those two together that we'll be talking about today.
The paper is, the talk is in a real sense an update of a talk in a paper which I gave the angle more, Memorial Lecture at AAAI in 2010 which became an article in AI Magazine in 2011; and two things have happened since the time this was written both of which reflect growing humility when dealing with cancer, and everyone who deals with cancer goes through this transformation. First of all, notice the title, A Computational Disease that AI Can Cure, and what we are now hoping to do is be able to at least help individuals manage this disease through an evolving cocktail of drugs much in the same way that AIDS patients manage their disease but in a very information rich way. And the second is that I no longer believe that this can be done nomadically through AI. We're going to add collective intelligence as well. And so

it's that combination that in fact makes the problem a lot more interesting actually to work on and certainly a lot more realistically that we can get to places that can actually help patients.
Let's begin at the beginning as to what this means to a patient who has just been diagnosed with cancer because literally every day I get calls from a desperate panicked people. My own sister, on Monday, called me up and said, I was just diagnosed with breast cancer. And the words she used are just, I can't do better than she said, I'm numb, I'm paralyzed, I'm frightened.
And this is, you know what happens in a cancer diagnosis is like you're fine, you walk into the doctor's office, he walks in and in my case he said well, there’s no normal node the pathology report shows. And all of a sudden you're not hearing anything anymore, you're thinking I have cancer. In my case, the doctor never used the term. He never said I had cancer. It wasn’t a normal node. But suddenly your life will never be the same. And at the beginning when you're facing this horrendous learning curve and you're just trying to educate yourself and your doctor’s telling you to do one thing and you say, well don't I get a, I need a second opinion?
And you go get a second opinion, they tell you something different. Or a third opinion, fourth opinion, and my case it was like a half-dozen opinions. And they were all different. And there was no data on which to help make that decision in a rational way and no way to recognize that you've at least considered all the possibilities. Such is the state of medical knowledge today.
So patience is lost in this forest, and it's quite understandable if you think about how much information there is. There’s a couple of million cancer papers, at least 100,000 published each year, tens of thousands of ongoing clinical trials. At ASCO, which is the American Society of
Clinical Oncology, every year they have a national convention. It's held in Chicago, McCormick
Place; it’s one of the two venues in the world that can fit 30 or 40,000 people. They come in for a long weekend and they get a report on maybe 5,000 clinical trials, what the results were. And this goes on in like 20 parallel sessions all weekend long, and if you're not in the room at the moment that that particular thing was announced that was maybe going to be helpful for your patient it might as well not have ever existed. You'll hear about it through the grapevine five years later. So the entire way that medical knowledge is communicated and disseminated it can take literally up to a decade to diffuse through from the major cancer centers out through the rural parts of the US and another decade to get to the rest of the world.
And this matters. What this slide is showing is that the five-year outcome for major cancers, five-year survival rate, can vary by up to 50 percent or more depending on where you are treated and who treats you. Now this is outrageous in the Internet age. I don't know any other word for it because it's largely about information arbitrage. It's like some people know things and other people don't know, and this is something that we really can and need to do something about. Yes, question.

>>: How do you know the difference in [inaudible] years [inaudible] due to the diffusion information count? Do you do, let's say, high variance on the expertise of these different hospitals [inaudible]?
>> Marty Tenenbaum: There's certainly some procedures that the call for, that have expertise being, playing a big role. Surgery, radiation, oncology, but these numbers actually are largely about medical oncology where we are talking about just administering drugs through infusion centers and the difference comes from there plus mixing it up with these other things. So yes, it's confounded. But the interesting thing is the more difficult cases go to the comprehensive cancer centers and they have the higher results. So, you know, there's one simple lesson you could say. If you've got cancer don't go to the Seattle Cancer Care Alliance or the Hutch as opposed to some, not universally true, and I'll give you some counterexamples to it, but the big lesson is to take away from this slide it matters and be, we can make a really big difference. We can save a lot of lives with no new cancer science, just through better information. That I think is the real important message that I would take away from this slide.
So I started Cancer Commons to do something about this, basically, to provide cancer patients and their physicians and their families and caregivers with the information that they need to make better decisions and to get better outcomes and to rest assured that they left no stone unturned. That really is the thing that really causes angst in a family. Was there something else
I could've considered that I just didn't know about? And we set it up as a nonprofit. I actually tried to do this as a for-profit first and no one was willing to share information. It's hard enough as a nonprofit.
This was borne of personal necessity. I mentioned kind of in passing 15 years ago I was diagnosed with a metastatic melanoma, metastatic to liver. Fifteen years ago there were no treatments that were effective, and the odds of my being here are literally one in 1000. I should not be here. And I went to half a dozen doctors, local doctors, and everyone told me my odds were get your affairs in order. That was my odds. But they all told me something different that I might try if I wanted to try something. So what's the data? There was no data.
And I was fortunate that at the time I was consulting for Rick Lausner[phonetic], who was head of the National Cancer Institute, and he put me in touch with some of the amazing clinical researchers that NCI had both internally and they were sponsoring externally, and I talked to another half-dozen or so people. If I had followed any of the advice of the local oncologists I would be dead. At the next level up the odds, the options were a lot more interesting, the renovative immunotherapies, the beginnings of targeted therapies, and just the level of practitioners, surgeons who would be willing to be, undertake intrepid surgeries to remove cancers when everyone else would say is too late, it’s metastasized. But I can tell you one

thing, formaldehyde kills cancer. You can get it out, take it out, then worry about what else you can do.
In any case, this was me, 12 months expectancy and no drugs available and looking at a 15 year pipeline to get a new drug. And looking at all these different things I said if I get better I can do something about that. So this was going through all my medical guesswork. In the end I just made on a hunch, bet my life on a clinical trial and that clinical trial failed, but the trial helped a few people and I was fortunate enough to be among them and the question is, why me? I still don't know the answer to that, and 15 years ago it wasn’t impossible to determine. But it is possible to determine now, not that it would make any difference because they took the vaccine that failed and they destroyed it. The lawyers made the company do it for getting rid of contingent liabilities; but the lesson was certainly ingrained on me that this was one more reason that once I got my health back I had a mission, and the reason I would get my health back was to fulfill this mission, and that was Cancer Commons.
If we fast-forward 15 years later, and indeed just on the timeline that I showed you on schedule almost in 2011, melanoma, simply because it's a relatively easy cancer to understand molecularly, it’s a wicked cancer but it's relatively straightforward molecularly, and it didn't have any good alternatives, was a place of choice for researchers who are trying to develop these targeted therapies that would hit particular targets not try to hit rapidly dividing cells like chemotherapies but be able to bind particular signaling pathways and shut them down. And so two amazing drugs were licensed in this one year, and there’s been two or three more since then. So melanoma remains a hotbed that now it’s not the case that there's nothing can be done.
But that said, these drugs are hardly panaceas. They each took about 1 billion dollars to develop. They about 150,000 dollars a year, they're not curative, they’re maintenance drugs, and almost all patients eventually recur because what you're doing is the same thing you do in a laboratory if you're trying to sensitize a cell line to be resistant to a drug. You keep feeding it that drug and make cells that survive, you feed it more of the drug and then the cells that survive that become ultimately very resistant just through a process of evolution. So this is really leads to the conclusion that the future, at least of this line of cancer attack, is going to be cocktails of drugs in which you try to anticipate the escape routes and head them off ahead of time and basically evolving, managing, this evolving disease in the same way that one would manage an evolving disease like AIDS through an evolving cocktail, try to stay one step ahead of the disease.
Now, this of course puts the need for information higher than it's ever been, information on what drugs should be applied to what patient. Clinical trials, which were long the gold standard for getting this information, are quickly becoming impractical in this world; and the reason is

simply that we now realize that at the molecular level cancer is thousands of distinct molecular subtypes. A single patient may have multiple subtypes going on even within the same the tumor much less at multiple metastatic sites; and if these need to be treated with combinations of targeted therapies, and the good news here is that there are hundreds of these targeted therapies coming down the pipe now, there's probably a couple dozen that have been approved, maybe 100 more than are in trials of various stages and a few hundred beyond that that are close to being ready to go in the trials, but if these drugs need to be used in combination then just to think about the combinatorics, thousands of rare diseases, hundreds of drugs that need to be used in combination to treat each of those diseases in a unique way.
There's no way there’s enough patients to populate those trials. We're going to have to find a much more efficient way to make use of what we learn from each and every patient in order to be able to populate this.
And for that I turn to what happens every day in community oncology where millions of patients get seen by thousands of oncologists and these guys experiment off label all the time with drugs and cocktails. They don't work in universities, they are not running formal trials, they're not spending their time writing papers or IRB approvals or FDA whatever, they're just doing what doctors do which once the drug is approved they can use it off label, or if the drug’s not approved they can apply for a compassionate use to use the drug on a single patient which is often much easier and faster to get than trying to get a trial approved which puts thousands of people at risk.
But essentially, nothing is learned from these exercises, most of the time it doesn't work so no one writes a paper, and even when it does work the journals nowadays you look down at case studies which used to be a mainstay of the medical literature. Nowadays it's all about trials.
We've been ingrained to do that. So, come back when you have 100 more patients and can demonstrate a statistical result. But that kind of statistical learning isn't that helpful. If this trial says that the drug is good for 50 percent of the patients and this other trial says the drug is good for 20 percent, this drug is good for 20 percent which drug should you take? Depends on whether you're more similar to the responders to this drug or to the responders to that drug.
The statistics actually don't matter much at the level of an individual once we have molecular information to guide us.
So, inspired by the experiments that go on every day with the desire to capture it, the vision for
Cancer Commons is to build a model of molecular subtypes of cancer, this model, and how to treat them; and that model can be derived in lots of ways, whether it's lots in reading the literature or consensus opinion of experts, people talk about evidence-based or imminencebased. But regardless, you build this model up and you select the treatment from this model based on the data for treating the patient in front of you, analyze what happens, and if the

patient responds that's great, notch up a success, if the patient doesn't respond try to go in deeply. This patient now becomes a study a subject of research. Try to understand what's happening and what we can learn from it, update the model, and then go around again. And whatever we do learn as much as possible and disseminated rapidly. Don't disseminate it through the literature and through conferences which adds months to years, not to mention the diffusion after the fact. The key is to disseminate this rapidly using Internet methods and to make sure that the people who need that knowledge find out about it when they need it as opposed to having to go look through the Library of Medicine for it, PubMed and so forth.
All right. Diving down on one level and looking at what these models look like, this is a model, a simple model of molecular subtypes of melanoma. There's about 10 of them here and I could, if there was interest, talk about how these the subtypes corresponded to mutations in one or more of the signaling pathways that keep these cells in equilibrium with each other. This pathway controls growth, this one controls proliferation and arresting of proliferation, MAPK is
PI3K pathway. These are color-coded, so mutations with yellow on them are mutations in this pathway and blue and so forth. And the more interesting ones have mutations in multiple pathways. And for each of these there are treatments and recommendations which either come from data or from data mining, data analysis, or from experts.
Regardless, this is a baseline model and what happens is that patients come in, they get tested, they get assigned to one or more of these subtypes, let's say to one of these subtypes, and some of those patients will respond to the treatment and some won’t. In that case the subtype has to get split into responders and non-responders. And so, over time these 10 has actually proliferated and now have 10 to 20, a couple dozen I would say, 20, 24, something like that; but it's not likely that this is going to keep exploding forever because some of the newer subtypes are subtypes we have seen before in other cancers. This sub subtype of melanoma looks a lot more like a subtype of breast cancer than it does other types of melanoma and there's a drug for it. And so the logical way to be able to do rapid learning is to be able to say hey, this drug worked on this combination of mutations in this other cancer let's try it here because fundamentally cancer’s a molecular disease. So that's the direction that we're trying to head in.
Yes.
>>: So for those cures [inaudible] those combination subtypes, are they simply just take the combination of the [inaudible] for individual pathways or?
>> Marty Tenenbaum: That would be the simplest thing you would try, and often that's the case where there's two, when you have more you try to do deeper network analysis to try to understand where these networks come back together and if there's a place where you can, if I can, if there’s a mutation here and here, for example, there are two drugs for doing that or I

might be able to block something down here that would be able, so this is kind of experiments that get done.
When I talk in my abstract about using drugs as probes to understand what the drugs do and how the biology works, that's the sense in which we talking about it. There's a lot of experimentation that can be done, and you can do this experimentation in animals, and animals are just not very good models. If you look at something that works in an animal the odds of it working in a human being or actually getting a drug out is pretty close to one in 1000, the same odds I had. It's a very long, there's a lot of things that can go wrong. But if you have something that you can test rationally on a person who is out of options, so there's no ethics involved, you’re in a human model, and you're in the human model years, perhaps a decade earlier than you would otherwise get. So not only is this a way to be able to inform treatment in the short term but more fundamentally it's a way to transform the way drugs are developed and clinical research is done by tightly integrating research and care around individual patients; and that's one of the things I'm hoping to use Cancer Commons as a platform for driving this kind of systemic change in the medical community. Yes, sir.
>>: So today, what percentage of melanoma patients you had diagnosed even get this subtyping done?
>> Marty Tenenbaum: Today I would say a very large percentage.
>>: [inaudible].
>> Marty Tenenbaum: Three years ago, none. It's happened in a very rapid way, and I have to say that rather than the medical literature driving that it's been the patients who find out about these things on patient blogs and drive the dissemination like wildfire. The urgency of patients just so far exceeds what happens in, once the academics get their paper published they’re done, they get their credit, but the patients when they see something that works they can upend research programs by saying to their doc, you know I want to be tested for this thing because I just saw this paper on this drug that if I have it I may only have a two percent chance of having that mutation but if I do I'm cured so I want to be tested for that. The researcher who published the paper had in mind another 10 years of NIH funding without having to deal with patients, but that's the power of patients.
So what we did is create what I call a rapid learning platform which is a way for all of the stakeholders to interact with the knowledge, with the data, and with the resources that are needed to both get the data and act on the knowledge. And so the way this works at a 50,000 foot level is the patients and the doctors understand what the options are and they can come in with another option too, that’s how it learned, just what we asked them to do is do whatever you’re going to do and tell us how it works so that the community as a whole can discuss and

debate and keep updating this model. And the model also serves to organize the services and information of all the different partners. So we're going to talk about this little bit later in the talk, but I want you to think about running this loop 30,000 times a day and learning and doing it not in just one cancer, but across all the cancers, so we can get this cross learning.
So now I'm going to, that was like a 30,000 foot level, let's go down and ask what happens when we are dealing with an individual patient who has not responded to standard therapy and what is it that genomics and personalized medicine, as you heard it, can actually do for that patient today? I think at this point I'm going to spend just a minute talking about three generations of precision oncology. The first drugs I showed you, Vemurafenib and Ipilimumab for the melanoma, these drugs came out with companion diagnostics. If you had a certain type of BRAF mutation you were eligible for the BRAF inhibitor. If you had a certain type of thing on your T cells you were eligible for the immune blocker. And so these companion diagnostics I would call precision oncology 1.0. This is precision oncology 2.0 and it’s just beginning to be rolled out at the leading cancer centers and a few independent labs like Foundation Medicine that are set up to do this.
So what happens is the patient who is, let's say out of options, gets biopsied and they basically take a specimen from the tumor and the specimen of normal tissue in the case is nearby. So the case of skin cancer it would be a, maybe a sample of skin, and then what you do is you run both of these specimens, you extract the DNA and you run them through next generation sequencing machines which give you a read of, if not all of the sequencing at least of the exomes which code for the proteins. Then what you do is you compare these two sequences to see where they differ and those are the characteristics that are distinguishing the cancer cell from the normal cell. Some of those may be significant in driving the cancer, those are called drivers, others are known as passenger mutations which are just manifest as a result of errors that the cancer causes but they're not driving the cancer per se. So the goal is to identify the driver mutations and then do some research, and typically this is literature research in PubMed, to find out which of these differences, potential drivers, are actionable in the sense that there are drugs that are available to target them, either approved drugs or investigational drugs available in a trial.
And then if there's time, one can attest these hypotheses in either cell models, you take cells from that patient or from a similar patient and you culture the cells, make a cell line, and you test whether or not these predicted drugs are knocking them out, or you can implant those the cells in an immune deficient mouse; and this is perhaps a more accurate model, and in the future people are developing analytic models of how drugs interact with signaling pathways in order to be able to predict what might happen in that patient. It’s a little bit early because

these models, while they exist, haven't been thoroughly validated and there are commercial services that will provide in vitro or in vivo testing.
Once you decide which is the drug you want to do you treat the patient and then you monitor, not whether the patient lives or dies in three years, but whether that's tumor is responding to therapy on like a six week or two months basis using a combination of biomarkers or imaging.
Did the tumor physically shrink? Did it physically become less hot on a PET scan or did the numbers go down if you are actually tracking markers like that are correlated with the cancer levels of certain antibodies that are associated with the cancer? And if the patient starts to recur then you have the opportunity to do some more science. What you do is you do another biopsy comparing the treated cancer cell with the original cancer cell. What changed since the patient was responding? And now you go sequence those and you find the salvaged drug.
So this is precision oncology 2.0. And there's been some remarkable success stories in 2.0. This one is one of the most famous because it was written up in the New York Times. Lukas
Wartman was a leukemia researcher. He looks older, he was 28 years old a few years ago when all this came down and he contracted the disease he was researching which made it a good subject for a newspaper article, and he was on his deathbed. Nothing was working. So people in his lab decided to do the kind of precision oncology 1.0, 2.0 that I just described and discovered that that the guy’s cancer overexpressed FLT3 which is a particular kind of protein which is relatively rare in ALL but relatively common in kidney cancer, and there's a drug that actually is active against that called Sutent. And so they put this guy on Sutent and literally within weeks he was back in business. And I believe he remains in remission and on Sutent to this day.
You don't always have such happy results. A very sad case involved a personal friend of mine who was one of the world's leading Glioblastoma, brain cancer researchers, who contracted pancreatic cancer. He worked in a Swedish Hospital and last summer, those of us who were working with him in his own lab, his own research fellows were told that there was a VIP patient, not that it was the master, they were doing a deep analysis, they discovered a potentially actionable mutation, they actually tested since they had a lab, they built avatars and tested the drugs; they weren't wiping out the tumor but they were controlling it, slowing its growth, getting stable disease. So there was this drug they needed, you can get these drugs for research purposes, getting them in approved form to put in patients is harder. You have to get them from the drug companies who make them. There were four companies that made
PIK3/mTOR combination inhibitors which three of them turned us down. One of them, Pfizer, came through, gave us the drug. We did get the FDA approval to this drug in this patient within
48 hours and it did stabilize the disease briefly but he was already infunctionably to weak and he passed away.

But this is the front lines of this kind of research. The goal is for every patient to learn something about the drug and about the molecular models so that we can make this better and better for each new patient. And the knowledge that we accumulate, Gleevec is kind of the poster child of targeted therapies in the sense that it was probably the first one that really had a widespread major impact. It was developed by Brian Druker at Oregon Health Sciences
University. It attacked chronic myelogenous leukemia and basically it took patients from their death beds again to dancing in the streets within a month, and someone discovered that this drug also bound to a different target for gastrointestinal stromal tumors, a target called C-Kit which was a surface signaling protein, and you had the same effect there. And this is a place, another place where patients spread the word by like wildfire completely upended the plans of these guys who were planning to slowly roll the stuff out.
And I notice here that in 2009 this drug was approved for one or two percent of melanoma patients who have a Kit mutation. So these people have a fundamentally different disease than
I had. I didn’t have a Kit mutation. Someone actually asked the obvious question like, tell me about the melanomas these guys had. It turned out they were all patients who had acral melanomas which are melanomas that form on the soles of the feet or under the nails, other places where the sun don't shine. So it's a fundamentally different disease. It involves melanocytes but a different disease in the fact that the same melanoma doctors were trying to treat this using melanoma treatments was like someone said why don't you try Gleevec on melanoma and someone said I already did and it failed. It only helped two patients out of 50.
So someone raised their hand and said, tell me about those patients. And everything started.
But the point I want to make is that the connection between Kit mutations driving a small subset of melanomas was published in a research paper in 1991. There was nothing you could do about it. It was just forgotten.
>>: [inaudible] I believe that for the acral melanomas that sun exposure is a risk factor and is mysterious connection [inaudible]. It comes on places on the body that doesn't see the sun.
>> Marty Tenenbaum: There's all kinds of things like that. The big point to make here is that people knew in 1991 that Kit mutations were a factor in these two percent of melanomas. They knew, there was no drug. CML came, was developed in the late 90s, approved in 2001, in record time, it’s the fastest drug ever approved through its process like two months just, they discovered that it worked on C-Kit, but now it took another most of a decade to apply it to melanoma, right? Machine inference could have connected those dots and should have connected those dots. And we should have, in fact even back here before this when we saw what this drug was and if someone had computational chemistry expertise and could look at and predict that the receptors, that this drug could bind to the receptors of these other acral

genes and we could hit this off; anyway, subsequently, this was approved in the one trial for six different types of sarcomas. It’s just a very, very interesting drug. In the back.
>>: So has this led to a new definition of success for human trials? That if one person out of
1000 is cured that rather than it being declared a failure there's further investigation into what that mutation or that was and is there new success because then there's some population of people-
>> Marty Tenenbaum: The answer is no, it hasn't changed. That clinical trial would be considered a failure. However, bright researchers when they see things like that, will say this is an extraordinary opportunity to do research to find out what disease that person had that we cured and how many other people have that disease, and then a platform like Cancer Commons with its ability to have lots of patients who have been profiled and we can connect up those patients with this researcher to test the hypotheses not in a clinical trial but the next day, and sometimes the clinical, those hypotheses can be tested even retrospectively on existing data.
And I'll give you examples of both of those before this talk is over.
So you're absolutely right. That's the way trials need to be done because they're not really trials that all. Maybe a better way to say it is every patient is involved in a continuing trial. This is an example of negative learning. So this is the drug Vemurafenib, it was the melanoma drug for BRAF approved in 2011, and it attacked a very specific mutation, v600e, in this BRAF gene, very specific. And then someone found that that very specific mutation existed in some small subset of colorectal cancer. And so everyone got excited. Do we have suddenly a solution that can stop for a while that kind of colon cancer? This drug didn't work at all there in very few patients. And so clearly there’s other things going on. BRAF 600e is not a driver there. It's being preempted by something else downstream or cross a stream from it. But even that negative learning is very valuable. Why? Because, for one thing, it will save thousands of other patients from, this looks like a good hypothesis, let's try it. People will try it over and over again, keep failing over and over again, they're just chewing up patients as if there's so many data points. That's what happens in clinical trials, just use up patients without using them wisely to learn as much as you can and doing as much as you can for that patient.
And the second thing is the company that developed this will now go off instead of spending a lot of money trying to set up clinical trials in colorectal cancer they'll go put their ready resources into more productive things. So whether that knowledge is positive or negative it's just very, very important to capture and spread.
I want to take a small look into the future, into the near future, which is, let's call this precision oncology 3.0. This is going much deeper into understanding the biology than really trying to identify mutations and comparing them with normal tissue. And I will explain what this is and

that 2020 is not far off now. It's six years off. But this is going to be transformative, and I'll show you what the difference is. So instead of biopsing have few cells from the original cancer, with this we recognize now that heterogeneity, the full heterogeneity of cancer. So we want to take a sampling of cells from all the different, from one tumor and all the other tumors. So we understand what we are working with and then we do very high coverage sequencing so we can pull those apart and understand how many different mutations that we have to deal with all at once, understand which of the most dangerous ones, prioritize those. Sequencing will be replaced by panomics, so not just mutations which are one the start of the information that you can get but gene expression, metabolomics, epigenomics, all of the different things, proteomics, all of the different things that can be measured should be measured. At this point you're probably up to at least a terabyte of data per patient. So we can now do big data on one patient. Yes.
>>:. How long does the heterogeneity occur?
>> Marty Tenenbaum: It’s strictly Darwinian evolution. The tumors are cells that are quickly replicating, there's 3 billion base pairs every time that happens, and mistakes happen. And as cancer cells get more and more advanced their ability to correct mistakes also gets less. So these mistakes accumulate, and so is there is typically a tree in which mutations don't go away once they occur. They stay on the tree and then new ones that show up. What’s your follow up?
>>: But within a single tumor-
>> Marty Tenenbaum: Within a single tumor.
>>: So is the thought that as the treatment or even the body starts to perhaps defend against the cancer then there's a response-
>> Marty Tenenbaum: That's half of it, and the other half of it is it's strictly mutations as DNA gets copied over and over and over again 3 billion times.
>>: [inaudible] 5400 dollars today [inaudible] generating. Your estimate, what would it cost in
2020?
>> Marty Tenenbaum: They been estimating, I think the price of the testing will worth the cost of the administration of getting the specimen and signing the paperwork and all that other stuff, and I think that getting this data will be not the problem. There will be a problem in getting people to share the data because people think it has proprietary value and the big cancer centers are very reluctant to share. We'll talk about that in a second as well.

So we do panomics. The comparison is no longer a simple comparison but a deep network analysis. So this is a chance for the Bayesians in the audience to really say I want to understand what's going on at all of the different levels of knowledge. If I have gene mutations that’s going to predict possibly some differences in expression I need to see that they’re there, I need to see that the proteins are there that are causing it, I’ve got to see that these things are lined up, and that's the depth of analysis that is done at a handful of places today. One of the leaders is a fellow who developed his skills at Network Analysis in Seattle which is where our chat and went off and is now running a center at Mount Sinai Hospital in New York.
So as a result of this analysis you're going to find the need to be able to find multiple mutations that you’re going to try to simultaneously target. You can think about this as playing chess against the cancer. I move here and the cancer’s going to come back here so I'm going to block it there and then it will come back here and I'll try to preemptively block it there. And the one thing I can say is that computers are better than most people with playing chess, certainly better than most doctors. So this is where, a place where computers are going to have a major impact. Testing is, I predict, increasingly going to be done in silico because you can run all kinds of experiments in the course of an afternoon verses the really problematic things of getting cell lines to grow and getting them to grow in time to be able to test things for a patient. So you
[inaudible] that test, treatment is going to involve cocktails and then monitoring increasingly is going to involve biomarkers as we get smarter about seeing what these proteins, these tumors themselves are throwing off into the bloodstream to do their nefarious work elsewhere in the body and we’ll be able to monitor progression and then of course combine.
So one comment I want to make about this slide is that everything on this slide is available today. That's why I can make these predictions very accurately. They're just not available at one place. And so one of the things that a platform like Cancer Commons can do is connect the places which have the best laboratory assays with the places they can do the best computation with the places that can do the best analytic avatars, silico avatars to analyze and so forth and to be able to take the learning from each of these patients and be able to combine them so that we can immediately learn from a handful of patients what might be good for the next patient.
And that allows things to, if they work, to be quickly replicated; and if they don't work to be quickly killed off and thereby to do literally 10 years of research in human models rather than mice and ethically, because in cancer you’re typically dealing with people who are beyond standard of care. That's the way to make a real difference for cancer patients.
So I'm about to start the second half of my talk which is going to talk about informatics. This was your kind of introduction to cancer and personalized oncology. So before I go to the second part let's take more questions on this one. Yes.

>>: So one of the questions that I had was, so instead of the actual DNA level cancers cells actually develop, there's mechanisms for becoming resistant to drugs.
>> Marty Tenenbaum: Yes.
>>: [inaudible] is one example of it. Another one is they actually created channels where the drug gets pumped into them and gets pumped out. So where does that fit in this-
>> Marty Tenenbaum: It's a complication, and the way it fits is there are drugs that one has that try to shut down those channels. A good example is that cancers tried to hide themselves.
They become stealth from the immune system and they do it by sending out proteins that try to tell the immune system to shut yourself down, and why there are receptors on the immune system for that is otherwise if you have an auto immune disease you get or an infection you'll eat up the infection and then you’ll keep going and eat up normal tissue. So at some point the signals have to come out and say stop and cancers have figured out how to take over those kinds of signals. It’s just another example of what you're talking about. There's a class of drugs that people are developing, immune modulators, to be able to preclude those sorts of things.
So it just, it adds to the complication but it’s something that absolutely needs to get factored in.
And not all drugs, etofogy[phonetic] where the cancer cell eats itself in order to be able to get energy without having, once you kill off its blood supply or something like that. And, you know, not all cells are capable of that and the mechanisms you use may be in a different channel that will shut it down from wanting to replicate in the first place. It doesn't become a question of energy anymore. Question in the back.
>>: Kind of a social rather than scientific question. As an engineer, I look at this and say great, and then I think about is [inaudible] in my insurance company and wonder how all this custom work is going to get funded.
>> Marty Tenenbaum: One of the things that we're trying to do in Cancer Commons is bring all of the different players to the table and get them to look at this, what I call systemic problems in terms of it's not just how you get paid, but it's the regulatory approval mechanisms, it's how you get access to the drug because if the, I said there's only one company out of four that was willing to give us a drug and we were lucky there, there are nice solutions which align everyone's interest with the patient, which is the key thing.
And I'll just summarize what I think that nicest solution is and we are working with all of the different players to try to move them slowly in that direction. The nicest solution is if there’s strong scientific evidence that some drug works in a patient and you even have proof in the laboratory model or a computer model the drug the company provides the drug for free and if it works the insurance company agrees to pay for it because it works so it should be reimbursed. And if it doesn't work they don't pay for it because some implicit guarantee here.

The good news is that the drug company doesn't lose very much. The value is in their IP not in the powder. So if it doesn't work they write off the cost of the powder and it actually saves them a lot of money in terms of wasting more money on 10 more years of mouse studies to try this good idea which seems so promising. You try it, you get past it, and you move on. And I think the payers themselves are going to find the data from precision oncology very valuable because in reality, 70 percent or more cancer drugs just don't work for the patients that they are prescribed for, and we can do a much better job just at the pharmacogenomics level of being able to see how people metabolize drugs much less the matching to targets. Yes.
>>: It seems like of these low level, you still have to motivate them, the producers to attempt to research in all these different areas. My personal background here is that I had a rare sort of brain tumor where there are only 10 people a year in the continent who get this. And so what that means is in the 20 years that I've survived no one has come up with any new treatments for that because there's no market.
>> Marty Tenenbaum: This kind of a platform will, it's not like, the cost comes in inventing new drugs; but the good news is, at least for cancers, that the cancers in all different parts of the body, including the brain, keep it reinventing the same pathways; so over time there will be drugs that may be, that will likely be applicable to any particular rare tumor. And the question is to have a systematic way to be able to identify what those are for a particular patient in need.
So you ever, God forbid, to have a reoccurrence you could use these techniques very likely to find existing drugs that might be helpful; and then it would be my goal to try to help you get, obtain access to those drugs for compassionate use which would be very helpful not just to you but to all patients who had similar disease as well as to the drug companies who suddenly would get a new market for rare diseases. They charge through the nose for this and insurance companies pay it. So if it works the insurance company should pay it. Question?
>>: You used two terms, compassionate use and standard of care.
>> Marty Tenenbaum: Yes.
>>: Are you saying that, does a person have to run out of normal treatments before you can go-
>> Marty Tenenbaum: Usually that's the way it's done and that's where the lawyers get into the act because if you don't do standard of care the doctor can be accused of not giving you the proper treatment. But one of the problems is that standard of care typically doesn't work very well and it can wind up giving you, making you much weaker and are having, giving your, taking your cancer to a more advanced stage where investigational drugs really should be used earlier.
Once we build up our confidence showing and demonstrating, and we do have to demonstrate that this stuff really does work more often than it doesn't work in terms of giving patients better outcomes, that we can get our confidence up and start moving earlier. But I think, in my

case in melanoma, I ignored standard of care completely because I realized one in 1000 if I tried everything standard I’d get standard results and I'd be dead. So I didn't even considered standard of care. I just jumped, and I think you'll see more and more patients who want to aggressively fight the disease take that same outlook. You and you. I have till three total?
>>: On record, yeah.
>> Marty Tenenbaum: Because I want to make sure we talk about informatics. So two more questions and then we'll move on.
>>: So why haven't the drug producers and insurance companies have come to this agreement that you just described by now?
>> Marty Tenenbaum: On their own?
>>: Yeah. On their own.
>> Marty Tenenbaum: Because the whole system is set up to approve blockbuster drugs that hit a very large number of diseases one at a time. And the FDA won't even allow these drugs to be used in combination until they get approved as independent agents because the thing is too tricky. We’ve got to make sure they're safe and that they work and then we’ll combine them.
The real problem is with age drugs is that the individual drugs don't often, they may be safe and they may be biologically active but they’re not efficacious by themselves. The early protease inhibitors didn't cure AIDS and didn't control it. They slowed it down a little bit and the AIDS activists got the FDA to license these things in this kind of premature conditions, as least far as the FDA was concerned, and that changed the game because suddenly the physicians and the patients had this small pharmacopeia, this toolkit of molecules that they could start mixing and matching and out of those can the cocktails which were effective and which are now the standard of care. And by the way, virtually all of these cocktails are designed for a particular
HIV patient by computer. It's all computerized now because no human physician can keep up with this stuff. One more question and then I’m going to get to the second part.
>>: So imagine why this clinical trial failed but they contain sort of a small function of success
[inaudible] case and those if you can go in and compare the molecular profile and whatever then you might be able to figure out what A, are the trial markers distinguished on them and B, what would be the targeted promising. But how, as of today, do we have access to this kind of failed clinical trial data?
>> Marty Tenenbaum: We're beginning to get access to failed clinical trial data. Not successful ones, but most of them fail. So there's a lot of data. I think, let me, in the second part of the

talk I'm going to talk about what we're doing which is really a big part of what I'm doing to try to figure out how to pull the data out of the institutions, kicking and screaming in many cases.
So we're going to move to the second part of the talk which is the informatics and how this is going to make a big difference in terms of applying these ideas at scale to be able to really help millions of patients. And in this paper I’m going to summarize the conclusions of this paper with a set of ground challenges which, where at the end of the paper it said, the goal is to organize the world's knowledge of cancer and use that knowledge to basically design treatment plans for individual patients and then to coordinate those plans so that in an ethical way, this is the exploitation, exploration and trade off, in an ethical way to be able to do good for the patient but also good for science, right, and then to be able to integrate all of the resulting data back across all patients in order to be able to very rapidly learn as a community.
So think about what we're doing as organizing a vast adaptive search over the space of targets and trials and over the space of targets and drugs and then using all kinds of knowledge and heuristics and pragmatics and ethics to be able to guide that search so that as a whole we can do the most good for the most patients and always putting the patient first, by the way, which is not what happens and clinical trials. Clinical trials put the drug first and then try to learn as much as we can to help other patients. So that's the goal, what we're trying to do, and I'm going to show you examples of how we’re going about doing that starting with organizing with the world's knowledge.
So we first discovered that’s a lot of knowledge to organize. One approach to this has gotten a lot of publicity which is IBM’s Watson who said, I'm simply going to go read it all. And they talk about reading 2 million articles of cancer, and there's some very good things about that, namely, it's all automated and that knowledge is ostensibly encyclopedic. If there was a some rare event that if some, some rare finding that happened to be relevant to some particular patient that was reported at the San Antonio Breast Conference that a few hundred people went to and most people didn't, Watson would have come up with it. The learning is continuous in the sense that they try to update the findings after by reviewing the recommendations for each patient and how they worked; and based on I should say one person, Mark Chris, and that's one of the foibles right there which is we want the collective understanding of everyone and not just one genius that runs lung cancer at Memorial.
And you can do inferences of the kind that I mentioned. So Watson, in principle, if it got its act together could infer that Gleevec early on might have worked against the acral melanomas with
[inaudible] mutations, the two percent of them. On the other hand, it might've even been able to do earlier if it could reason not just in terms of connecting the dots at the knowledge level but also being able to go back and say I understand computational chemistry and I’m looking at the shape of the receptors and I'm predicting this drug, you could imagine looking at all drugs.

There's 5000 drugs and they've been approved, and there’s something like 20,000 molecules that have been approved safe for use in humans that are not drugs. They failed for whatever reason orbusiness reasons to not progress to be drugs, but they are still available. You can mix and match. So that's an opportunity.
Okay. But there's also a downside, and I'll summarize the downside which is most of the literature is either irrelevant, out of date, or wrong. We just don't know it’s wrong yet. Next year we'll know it's wrong. And the fact is that the reason you don't see Watson applied yet in practice is that it's still learning. The results that it comes up with are like a first-year medical student and Mark Chris have to keep correcting them. It will get better. Watson's the way to go, right way to go. But that's the challenge.
Not having the resources of IBM, my colleague Jeff Schrager[phonetic] started working with a group called the Melanoma Molecular Map Project which was an international consortium of about 100 melanoma researchers. And they did a kind of hand curated version of this. Now each of the lines in this spreadsheet represents a coding of the major result from a paper. And these papers were carefully selected, curated by this group to say this was an important finding. And if you read this table in detail it would say that the presence of some of protein either enhanced or decreased the sensitivity to some drug or increase the sensitivity to a second drug in conjunction with the first drug. This is kind of the level of conclusions of these papers. And so it went through and there’s like 2000 papers that they covered and they included strength of that evidence which was very straightforwardly coded, namely if the evidence came from a petri dish that would be one, if it came from a mouse model, two, if it came from a case report, three, if it came from a small trial, four, a large trial, five, a metaanalysis of many trials, six. You get the idea here.
And then the way they did this was if you had a patient and you studied what markers they had you could decide what papers, excuse me, what drugs you were thinking you could be interested in and based on that profile you would then select the weighted evidence from the rows in this table and you’d get a personalized drug rack. It was just that simple. A patient has certain, I have a drug, the drug patient has certain things, I’m going to look at all rows that talk about the patient's mutations and that drug and I'm going to add up the evidence up, the evidence down based on how strong the evidence was and I will get a result. And this worked fairly well as an approximation to what Watson was trying to do. It's hand curated, it’s not automated.
We went further in Cancer Commons and then this company CollabRX that I did. We didn't need to take 100 people. We took one editor and a couple and a scientific researcher and a small editorial board of three or four and we basically hand curated, we went to them and I said how many subtypes of melanoma do think there are? And how should they be treated? And

then I went to a handful of other experts, and that consensus gave you a baseline model but this was all this was intended to be. And then from this we could start to do learning, right?
We can capture data as to whether it worked or didn't work and start to refine these, split them. So there's various ways you can organize the world's knowledge starting from encyclopedic to this, and obviously it be good to have some combination of them because at the moment a handful of good doctors will have read all the literature and will have precompiled and digested that and make it available in this nice digested form.
The one problem with this again, and it’s what I talked [inaudible] about is that these are per cancer type, and I really want to do a pan-cancer model by molecular subtype because that's the way we have to start thinking about cancer, not by organ. Talk about head and neck cancer. I don't know what the organ is. The reason it’s a medical specialty is because the muscles and so forth require a certain kind of surgery when you are operating on it. So it's a specialty, but it’s nonetheless treated as an oncology specialty as well. So you have to think about cancer at a molecular level.
Okay. So that's knowledge. What we do with this knowledge? On this platform we've turned this into basically an apps platform, and we've developed a small set of apps to be able to do rapid learning; and by that I mean for patients and doctors the ability to look up what treatments and trials are recommended, number one, number two to be able to discuss those treatments and trials or treatment plans with peers, and number three is to report outcomes, and then for researchers to be able to capture those outcomes, analyze them, discus those with your peers, discuss updates to the consensus model, and then to be able to update the model and rapidly recruit patients to validate results or to continue testing. And so this became a platform for being able to do this kind of basic rapid learning for precision oncology 1.0 and maybe 2.0. This is not the deeper stuff yet. We'll get to that in a second. But it’s the thing that we need right now to be able to provide information to patients.
So I'll give you an example of some of these apps. This is our first app which was to capture information from patients. And this is just a Wizard, basically, that captures information about this is demographics, talks about date of birth, postal code, and blood type and things like that, the markers that had you’ve had, the treatment that you've had, and the responses. And this is a timeline that shows you're getting better as you go higher if it's turning green, you're getting a record of things. And as an incentive for patients to provide their data what we had initially was an application which gave them targeted news and alerts to new studies that affected them.
And in fact this could be precisely targeted, and this is something I'd love to work with researchers at Microsoft who actually care about wanting to do precisely targeted news. At the moment we're matching keywords and UCI and marker and patients are in a particular

molecular subtype or a phenotype that's based not just on what their subtype is but how they’ve been treated, how they respond to the drugs, patients with the same molecular markers may respond differently to drugs if they're in a different phenotype. So if we could take the news and other information and classify it in the right way and then hang it there so when a patient is asking for what's new for me, these are personalized dashboards. They get precisely the information that's right for them for people who are in their particular state of the cancer journey. That would be very, very helpful.
>>: [inaudible] think about even on these kinds of slides have some of these experts, [inaudible] the one that you went to that aren't available in the community come on and provide some information and some expertise.
>> Marty Tenenbaum: Absolutely. To be able to put a social collective intelligence around this is absolutely the idea, and the reason I don't talk all that much about it is that we are committed to channel partners who do that for a living. That's all these advocacy groups, and my challenge is to be able to mine the patient blogs and discussion forums and QA's and so forth, go on all these sites to pick the one percent of postings that are actually scientifically important and then organize those so that they show up so people can find them. The problem is that no one can find this stuff, and so if we can organize the world’s social media, and this is something that I don't see myself having expertise in doing, I'd love to do it with you guys, that would be extremely helpful.
This is concept pictures of where we'd like to go. As people report data we have, in principle, the ability to be able to show people information they've never had before, very straightforward information what treatments are they taking, and what the outcomes were; so this is a simple depiction of I'm a patient and here's all the patients that are similar to me in subspace and instead of having a subtype this is using nearest neighbor type analysis and some of these people, you might not see it, but some of them are not smiling. These guys are smiling so I think let's see what this guy did, and let's see what, for the patients like me, what treatments did they use, what the outcomes were. You can started displaying it in ways that really make clear for the first time for a patient who is just like you, has the markers, has been treated in the past with other drugs, at this moment in time these are your options and more people, bigger circle, more people are using A then B and D is the best because it's both the most efficacy and the highest safety and so I want to go try that one. Similar, one can crowd source other things. So you can imagine various kinds of apps like this. It's not my goal to write them all. It's my goal to create the platform and that people will use this patient data, send them to the apps, and the apps will come back with all kinds of interesting things.
So now let's talk about where the data comes from. The patients will give us the data if they have it and they'll know typically, a breast cancer patient will know if she's triple negative or

estrogen positive or whatever it is, but they won't know the detailed that they can't know the detailed sequence information that comes back. So to do that we've got to get the information out of the institutions that have that and we're doing it in two ways. One is we are actually trying to sign up institutions to become network partners, and by that there's a quid-pro-quo; we will send patients to partners that are willing to share data and they'll send back the data for the patients who we send them or that are part of this program. And what they get in return is something that's pretty important to them, de-identified data from all the patients in the program, identified data from their own patients; and why that's important is Foundation
Medicine, for example, is a lab that does molecular testing. When they give you recommendations they have no relationship with the patients, so they have no idea whether patients have followed the recommendations or if they did what the results are. So that's what we're doing there and of course they get referrals. So, this is high-value and it's of high value especially to the smaller institutions. M.D. Anderson may have enough data on its own to be able to do its own thing, although even they are beginning to recognize that on a global scale they have to join forces with other people.
Now for some institutions, maybe because they're smaller and we don't have relationships with them, we have another vehicle which is we use the patients as leverage points to be able to pry data out of these institutions whether or not they explicitly want to cooperate, and we have two things going for us which we never have before, and for people involved in HealthVault this is actually very important because there's two things that are happening this month that never happened before. One is that the HIPAA regulations have expanded to guarantee that patients can get access to their medical records, which they always could, but now they can get it in digital form within 96 hours. When I used HIPAA to get my medical records some years ago they let me go on the records room with my folder, pick out a few things and then pay 25 cents per to get a Xeroxed on a piece of paper.
>>: Is digital [inaudible] recommendations?
>> Marty Tenenbaum: Pardon me?
>>: Is the word digital-
>> Marty Tenenbaum: Yeah, it's in there. The problem is that it’s likely to be ASCII organized in some random way so there's still good work to be done for people who want to take information and the structure it for sure. But the other thing is that on the flip side of this the institutions have a real incentive to give them the data because meaningful use too, which is part of the Affordable Care Act, actually has positive incentives to a provider to have a thirdparty, not themselves, but a third party make meaningful use of their data and this qualifies. So the patients have a legal right to the data and these guys have an incentive to give it to them

because they get better Medicare and Medicaid reimbursement. So that's what we are counting on in order to be able to get data. And so once we have that data then these same institutions, many of them are like commercial labs that might want to provide a second opinion app, for example, because they're M.D. Anderson, they want to recruit patients and show how great they are, get some patients out of that, or clinical trial finders because you're a drug company and you have some studies that you want to do. I'm hoping that this will grow to be a very vibrant ecosystem of apps that are coming from all kinds of third parties. And my goal, again, is not to turn Cancer Commons into a site, I'm perfectly happy to white label all of this behind HealthVault or Bing or advocacy groups or what have you, the goal is to help cancer patients, not to bill an institution, compete with them.
This as an example of a third-party app that actually exists that came out of the company I started which provides a little expert system that you can actually run it today and try it. It runs off of the lung cancer model that's similar to the melanoma model. You can try the melanoma
1, 2. There's about three or four of these CollabRX.com.
>>: [inaudible] two advisors [inaudible]?
>> Marty Tenenbaum: No, no. These are the co-, Fisher and Flaherty are the co-editors of the melanoma model that I showed you. They have a small editorial board and they built a consensus model and that's the way that came to happen.
>>: In that case, why the hetero-targeted therapy finder advisors?
>> Marty Tenenbaum: This is called target therapy finder and [inaudible] advisors.
>>: Okay.
>> Marty Tenenbaum: We are trying to get a lung cancer pilot going, rapid learning involving a number of institutions that are directly or indirectly associated with the University of Chicago at
Art and Health Sciences, which are the two charter institutions, and they have colleagues and we are trying to just build a rapid learning pilot in lung cancer; but again, I want to generalize that quickly to all cancers and so we are also trying to add institutions that by leveraging the
Stand Up To Cancer, Prostate Cancer Dream Team and the Melanoma Dream Team that I'm associated with to try to build up the Cancer Commons network and build a sustainable funding model for Cancer Commons by getting these people to all contribute a little bit to sustaining of the network and then use this network to run their studies in a much faster way than they been able to do before and capturing the data from every study in order to make it available to patients.

Okay. At this point all of these things, planning the experiments, integrating the data, we are now moving beyond 1.0, 2.2 to really precision oncology 3.0 where we are trying to do deep reverse engineering of these tumors and integrating the results, and I just want to show you a few examples of how this works. Again, the trade-off between exploration and exploitation, you're always trying to do the best for the patient while at the same time maximizing others.
You saw a picture of melanoma pathways that was a severe simplification, and this too is a simplification because this is just like a pool of thousands of proteins all interacting with each other in random ways, and I don't know how to depict it other than a cartoon, but the point is it's too complicated for people to really anticipate all of the horizontal and vertical feedback among these pathways. So it really is something that machines should do. And this is what's involved typically when you’re reverse engineering a tumor. You start with a genetic profile which can include expression and other things. On the basis of that make hypotheses of what protein ought to be there, you try to line those up with pathways that are there, select targets and select drugs, and you are trying to come up with, if you like, a Bayesian inference across all these different levels of knowledge that are going to identify based on the evidence that you have, what are the best targets, what are the best drugs I have for those based on the evidence. And there's various ways you can try to improve your ability to deconvolve[phonetic] all this information.
So basically you have multiple specimens which can help you put noise in its place. You're dealing with panomics which allow you to do this multiple levels of hypothesis validation. The fact that I, probably the most useful level to start with is not mutations but expression, and so based on expression you look for proteins that you expect to be present or possibly phosphorylated and then you can go back and say are there mutations that maybe account for that? But in any case, you're looking for the master regulators and the interesting thing when I talk about, people talk about the big data and I'm talking about either small data or deep data, data from one patient, the point is that you have 20,000 genes, right, that you’re getting expression data from in a single patient that's likely to be very noisy, any of those. But if those of you who are familiar with concepts like Hough transforms any one of these genes can be a noisy but if its expression is over or under-regulated it will implicate some number of problems here. The next one that's like that will indicate some number of problems but they'll be scattered and the real master regulator that's responsible will keep getting votes from all of them. That's the way this works. This is very straightforward.
>>: So if you're getting false sequences [inaudible] aren’t we resolving lots of variables into
[inaudible] little bit of noise on them or is it-
>> Marty Tenenbaum: No. The point is you would get a very strong signal. If you want to put a
P value on it where the doctors in the clinical trials think of P value of .05, one in 20 is fine, the

P values on dealing with 20,000 mutations to try to isolate where you think the action is is like
.0001.
>>: So it’s hoping that the sequence information will give you some current knowledge.
>> Marty Tenenbaum: Yeah. Well, you’ll see it’s also the pathway information. You're trying to-
>>: You have regulatory stuff going on.
>> Marty Tenenbaum: You're trying to put it back in place in a way that makes a story that compares what you're seeing with what the public models are. So you may not from a single patient have the ability to be able to determine what is the de novo model of cancer. But you can look at all the public models and you can say this is most similar to that model with the following deviation, and that's where big data comes in as a kind of baseline that you use as part of this interpretation. Similarly, sequential biopsies, so every time a drug interacts you want to figure out did the drug, why did this tumor not respond? There's lots of potential reasons. Either the drug didn't reach the tumor, or if it did reach the tumor and it didn't bind to the receptor, if it did bind to the receptor the receptor didn't turn off the pathway as we thought it would, and so on and so forth. So this is, and response to therapy. So this is using drugs as probes as we talked about in order to be able to infer, in a deep way, and I'm not going to tell you guys how to do this kind of work, I’d just love your help in trying to work with individual patients with data to try to help those patients and in the process to learn. This is not something that should be done, in my opinion, as disembodied research because the doctors are there working and they can use your help.
>>: Has there been some [inaudible] models of work of this kind of-
>> Marty Tenenbaum: Absolutely. And [inaudible] is-
>>: [inaudible] levels. For example, [inaudible]?
>> Marty Tenenbaum: Yes. I can help you. I've been involved in this personally. I can absolutely help you, but it's not something I have the time or even the expertise to be able to do on my own. But we can certainly provide the expert help with directly and through our own collaborators for people who want to work in this. And the second big one is these are all the people doing big data like the Cancer Genome Atlas and this is ASCO trying to capture data from all of their oncologists. But this is this cumulative trial that I talked about where we're trying to analyze and learn from every patient.
This is my colleague Jeff Schrager's diagram and I'll cut it down to size for you. This is what is happening when a patient is, a standard patient comes in, he gets tested for –omics and

standard treatment is available, they get treated and monitor response and if not you come back, you have more information and you retest the tumor and you go around this loop. But there are no options in the standard thing that actually integrate research. This is clinical.
Now if you take the first step of what we're doing, if in fact there are no standard treatments available, then you come down and you look at what are all the research options that are available? So in those subtype models, if there's 20 of them, if I sort them by evidence, the top
10 may be standard of care, the next 10 are like hypotheses that someone wants to try. So are there some good things to try? And if there are then you say is this a best choice for the patient? And if it's a good choice for the patient then you go and treat; if there's no options at all then you consider that a dead end, and if is not a best choice, that is the patient doesn't want it, then what we can do is go and say, I'm sorry the patient doesn't want it, it’s not clearly one best choice it’s like two equal choices, then we can choose the best one for science. So this is the priority that I was telling you about in exploration, exploitation; namely if it's a clear best choice for the patient then you do it always. If it's not a clear choice, it’s like there's two things and we don't have enough evidence to know which one's best, everyone is doing this one let’s try the other one because we want to explore the rest of the field.
>>: Suppose you have an impatient cancer patient who, given the set of targeted therapies each with a few side effects, just want the cocktail right away we’ll see what happens?
>> Marty Tenenbaum: You know what happens then? It's this guy. No options, right? You've seen this, right? This is where we give them. And now suddenly we have a new whole new set of choices. And the goal with this analysis is to be able to integrate the evidence from every patient in an optimal way that gives each patient the best possible outcome for that patient based on science and drives the entire learning forward for the field. So I would love to build a global cumulative treatment analysis process on top of the Cancer Commons platform. We're not there yet. That's where I want to go.
I'm going to skip this and come back to it if we have time after my three o'clock cut off time which is talking about what we can do beyond trials to change the way medicine is practiced. I want to spend the last bit of time talking about what we and Microsoft might to do together.
So Cancer Commons is collecting, building an infrastructure that will have data and knowledge and various network partners from pharmas to payers to providers, patients and whatever, we are bringing all these people together and what we need are partners of many kinds, and I look at Microsoft as this extraordinary organization with, most importantly, 1 billion users some of whom are going to be concerned about cancer at any moment in time. And you have some interesting properties, Bing and HealthVault, which are intriguing to me because HealthVault is something that is, can be turned into a registry for the kinds of data that we are collecting with

our Donate Your Data; and Bing is, to the extent that it can also become a decision-support engine can take that information, that profile, and be able to do true decision-support using the kinds of apps that I was showing you. Those could be integrated. Similarly, if to the extent that
Bing is actually going out and proactively looking for news for someone, this all fits. I want to build apps that can be used with HealthVault and Bing; and so the specific ask is I would like to do a collaboration with HealthVault and I would like to get some Bing cycles or add words so I can start playing with that.
With Microsoft, and this is for a company that on your health page explicitly says that the goal is better outcomes, we share that, that's what this is fundamentally about. As far as research is concerned there are many, many things going on, and I've only listed a few of them in
Microsoft Research, that are absolutely essential in order to be able to fulfill this vision. And
Haiphong[phonetic] and I are working on, have been talking about building a pan cancer knowledge-base, largely mine from the literature but also been corrected by human experts and ultimately by the data, reverse engineering with tumors be talked about, the global cumulative trial analysis was that diagram that I just got through showing you, on the natural language front there are two things that I've identified, one is patients have questions. They want questions answered about the information that we are giving them, and just putting up any kind of a display is not going to be enough; and some of these questions may need to be answered by machine and some of them need to be passed off to experts at Sloan-Kettering or
M.D. Anderson or the Hutch, and we want to be able to make that seamless. Applications that do crowd sourcing of second opinions, this is very interesting you know. What do other patients who have been in your boat recommended doing? There's all kinds of, or physician referrals not based on what your friend knows. I need a good doctor for this. But we have the numbers now. We will have the numbers that say for patients who are like you, right, who have been with this molecular phenol, molecular subtype and these treatments. These are doctors who have had the best outcomes and these are the methods that they've used to it.
We'll have those numbers and I want to be able to get them to patients.
So with that, let me close on a personal note and say for just one moment forget that we’re here at Microsoft Research or researchers at all, we are people, we are all going to be patients someday, we are all going to be taking care of patients someday, it’s just a question of getting old enough; and this is the scary thing about getting older, suddenly you hit a certain age and all of your friends are dealing with this stuff. And when this happens what you're going to want is to know that the collective knowledge of everyone in the world is available so that you will make the best possible decisions, get the best possible outcomes, and know that you've left no stone unturned. And so, please, please, please join me in this urgent question to kick cancer’s ass. Thank you. Thank you very much.

>>: Other questions?
>> Marty Tenenbaum: I would like, I know, anyone who wants to leave, officially over, but I want to go ahead and show the one more slide, if I can.
>>: I’ll ask later.
>> Marty Tenenbaum: It's just one slide. This is, if we can get to a point where we can get
100,000 patients on this then we have patient power. We have an army, right? And what can do that with that army? Someone asked before what happens if you have one responder in a trial? Does the trial succeed or fail? And the answer is today it fails, but this is an example of someone who did it right. There was a trial of a drug, Everolimus, for I believe it was kidney cancer that was ostensibly targeting mTOR which is a well-known cancer target. And there's only one lady who responded, and they did this complete sequence of her tumor to find out what it was that this drug was hitting, and they discovered it was something downstream of mTOR call the TSC-1 that she had and few of the other patients had; and so they then went back and looked at the rest of the data and they saw that this TSC-1 was present in all of the other patients who had at least a small partial response to it but a lesser extent that she had.
And it turns out there's lots of other cancers in which TSC-1 is quite prevalent. So, suddenly there's an opportunity to take a drug which otherwise might've been considered a failure and to go look at all the other few percentage of every cancer and suddenly this becomes a major success. So this is one example, exceptionable responders.
End of one study’s, this is a true story, there was an investigator at Columbia University who was presenting at ASCO last summer in front of 2000 people in a big hall, and he said based on molecular results and reverse engineering and finding the master regulators I knew exactly where there was an opportunity to interdict this tumor, and I got a hold of the drugs and I combined them and just knocking it out of the park, killing the tumors, mice are doing great, and now I'm going to start a clinical trial. My plan is in 2014 to do the paperwork, get to the IRB approvals, the FDA approvals, the drug company approvals so I can get access to the drug, apply for funding, start, at some point start accruing patients, eventually you have enough patients that you can begin this study and then you run the study for a few years and then you write it up and then you submitted it to ASCO and then ASCO escrows the results for 3 to 6 months because God forbid they should be presented before the conference, you're talking about a decade.
I went up after the talk and I went to the microphones and asked him, I said, so you're doing good with these tumors? Where'd you get them from? He said, well, I get them from the tumor bank. I said whose tumors were they? He said, I don't know. They're de-identified. And right at that point, right, I raised the obvious suggestion that said, well these tumors came from

someone. If we could find out who they came from you could, tomorrow, these guys are dying.
So it’s not a lot, you can do one of these compassionate use things and we can actually test your theory and decide if it's good or no good; and if it’s good we can replicate it on four more patients and then apply for fast track FDA approval, and if it's no good then you can go off and do something else and quit wasting the next five years of your life doing that. That's the kind of revolution that we can do with a platform like this.
Final example, which is not unrelated, is that this is a drug, Enzalutamide, which is used in prostate cancer to, recurrent prostate cancer, to block androgens; and it stops the growth for a while, but these patients all recur because the cancer will find other hormones to drive it. And typically, often, it uses metabolic, hormones from the metabolic pathways such as insulin, and so you find that Metformin, which is a commonly taken drug, they’ve done some experiments to suggest in vitro, in vivo in vitro, that this will in fact, by blocking the insulin, make some of the tumors responsive again to Enzalutamide. Now, of course the answer is we’ve got to start a clinical trial to test that. And in this case, I had a similar response. I went up and I said, certainly there are large numbers of patients who are older men who are on this drug for their prostate cancer, some have recurred, some haven’t, some are on Metformin because they're also battling diabetes or not, why can't we then in an evening go back and crunch the data get the answer to your question?
So another example of retrospective studies that could be done very, very quickly on many promising things, the answer is, the question is there's so many hypotheses that need to be tested and so few patients especially as you get these sub, sub, sub types of cancer, we have to find a better way to exploit the patients for the benefit of the patients and the benefit of science; and tightly integrating research and care is kind of what the endgame is with the aligned incentives that we talked about so that the insurance company pays when it works and doesn't pay when it doesn't work. That's where we're headed. And the final little build here is that there are obvious examples for informatics to drive this, namely what we need to do certainly is to be able to look at all the results to identify exceptional responders from all the individual patients who are reporting one thing or another. It's not that they're all equally interesting, I want the ones who are going to teach us something and then we'll really drill down on them. Here, there's an opportunity to be able to directly match patients and the researchers who are interested in them, and here we have an opportunity to build an e-Trials database by integrating information from all of the partners that Microsoft can get a lot easier than I can so that we can do these retrospective studies overnight. So with that, I will again say thank you very, very much and I hope we can find some very productive ways to work together.
Thank you.

>>: I was hoping to hear punch lines [inaudible]. I mean even though it's two months out from your ACSO meeting. That’s what we get [inaudible] figured out a way to go to the company.
>> Marty Tenenbaum: No, I will go ask him if I hadn't intimidated him enough; but there is an interesting punch line on this one which is interesting because there's a different person who was, came up with the same result in a different institution, and they didn't know about the fact that this, I couldn't talk about it. It was a perfect example, but I couldn't talk about it because they hadn't published this finding yet, so it was like another six months til they published it before they then start thinking about their trial. And then like the next week I was at the ASCO meeting, I literally went on an airplane from this meeting the ASCO meeting and I hear about another group that actually had a trial in process or in the process of starting up and
I didn't want either of them to do a trial, but nonetheless this one group was like, you can see how all of the replication and redundancy of medical research, the same experiments get done over and over again and the people who lose are the patients every time because they, you only have one life to live. Why would you want to be in something that's not going to work?
>>: So if today we want to do this kind of trial what’s [inaudible]?
>> Marty Tenenbaum: This one?
>>: Yeah.
>> Marty Tenenbaum: You know, I don't think, there's certainly no technical bottleneck.
Determining trials in precisely in this way was actually coined by Andy Grove who wrote a wonderful editorial in Science. And you can just Google Science and Andy Grove. And he caught so much flack for this. His article was saying the drug industry needs to be more like the semiconductor industry. We need more knowledge churns. Instead of if something fails you throw it out, if something fails you try to see what part of it might've worked and learn from that and build. And his best idea was to say I want to do retrials. And so in order to do this you really need to get access to a lot of data. But you might be able to do it by partnering with one organization. And a big pharmacy benefits manager like CVS Caremark will have data on 50 million patients. That's more than enough to answer a lot of these questions. And so if together we can go and propose a study like that, there used to be-
>>: [inaudible].
>> Marty Tenenbaum: Yeah, well sure. There was a company, Allscripts, any of those. There was a company called Medco which was in that business and they got sold, but before they got sold they had a very visionary management who was willing to run experiments for people, if you went and proposed them they were willing to run those experiments for you, but you had to apply each time, and I wanted to tell them, I said I can make it easier for you. We could

create some kind of an API or whatever and people could submit and run their own experiments. But he said that the reason they wanted to put their professional health services researchers in the middle of this is it's so easy for people who aren't skilled at this to kind of design what they are looking for anyway that will come up with wrong conclusions. So we tried to act as a little bit of a break to, or a little bit of advice to make sure that the studies were well conceived to answer those questions right. And I think that's probably fair enough. But we need to make some deals like that. And those are the kinds of things that together we could have much more of an ability to be able to get impact. And doing this not just as a business but as a data philanthropy project, right, this is something that if we could get all of corporate
America to do that, and I left one thing off of my thing, my natural language question answering remember at lunch? For those of you interested in machine translation out there in Microsoft
Land-
>>: [inaudible].
>> Marty Tenenbaum: I want the opportunities, when I say it takes 20 years to get to the hustings[phonetic], it takes a century to get to the third world, we need to be able to translate these findings and this information to every language because it's a different planet out there.
They are like 50 years behind and in an era where we are in the Internet, right? The information can flow instantly, you can ship specimens with FedEx, you can get drugs back by
FedEx, there's no reason we can’t deliver state-of-the-art if we get state-of-the-art information to the people who need it. So I would love to work with the group that would be like to take our content and figure out a way to internationalize it. Thank you.