Transcript

Transcript for VeHU 2007 Session #227

VistARad - From 0 To 65

Good morning everyone. I think it's time to get underway. Just so that we're all on the same page, this is course #227, it's in the technology and IT track, VistARad for

Coordinators. It's going to be a discussion divided into three parts.

I'm Dr. Peter Rosenthal, I'll be doing the initial portion which is thinking about PACS, how to set up workstations, the general principles, general issues. Then my colleague Dr.

Kim Wilson will take over with a discussion of the practical implementation of VistARad

65 in the work setting. And then I will finish off with a few issues relating to experience from the field, efficiency, cost, and other issues. Just in case there are any errors in the program I just want to clarify. This particular track starts with this session and then there are two hands-on sessions that are part of this track. The first of those hands-on sessions is on Wednesday, and please consult your program for the time, I'm talking now about course 222. Course 222 is going to be the basic approach to using the VistARad system, the manager window, and an initial approach to setting up hanging protocols. And then the second hands-on session will be consolidating, doing customization, and the high end stuff that you need to learn to implement this, and that is on Thursday. Please check the times and if what I've now said about the content overrides any errors that might be in the program. So with that I'll get underway.

What I hope that you will take away from this meeting, from this particular session, will be principles for planning and designing protocols to read all basic diagnostic imaging modalities in radiology. You'll have some idea of the kinds of monitors that you'll need, how to set them up. An arrangement of the reading room environment, display functions and how to divide up the monitors. And in these discussions the key parameters will cover exam resolution, the number of images, the number of series, how many comparison exams you want to be seeing, and the use of other IT resources.

Can you see those X-rays there? So when thinking about PACS the issue that we all faced in making the transition was what should we do with it, should we just replicate

1

Transcript for VeHU 2007 Session #227 what we used to do with the old auto-alternator where we put a bank of four images in front of us, the current images in the middle and the comparisons to either side. And as you can see from the crossing out there, that that's really not the way to go.

What I want to present to you is what I believe is the consensus model of at least the minimum that you would need at a workstation, and this is its shape here. You have the dedicated high res diagnostic monitors, which you see two of there, and then you have a third monitor that you should use for control functions, and you can use that also for images that contain color. You can see the diagram indicating the color Doppler there.

And then off to the side, or you can have it incorporated into the system, a voice recognition system and the ideal would obviously be to integrate that. So if this is the consensus model, how did we get here?

Well I flew in from Charleston, South Carolina, and maybe we could have VeHU there, although I'm not sure we have too many facilities that are quite the right size for this. But be that as it may, this is the Charleston skyline, there's our new bridge which adds to the numerous historic landmarks in the city and it's a lovely place to visit. Come and see us.

I've been at the Ralph H. Johnson VA medical center for approximately ten years now, and I've seen the transition from film to filmless, and now almost paperless. We have four to five attending radiologists, we do all modalities with the exclusion of mammography, PET, and high-end interventional. We've had VistARad since 2003 and

VistARad 65 and Voice since approximately 2005, although the official release came a bit later.

The question is what have I brought to VeHU to discuss in terms of the consensus model and the experience? Already when I was working with VistARad 16 and 32, although we started with four monitors I was already looking for ways to kind of narrow down where I was working because we have to do the dictation stuff and look at stuff on CPRS and so forth. So I was already looking to narrow that down. I then became an Alpha user for version 18 through 65. I also then went on some site visits nearby our hospital to see

2

Transcript for VeHU 2007 Session #227 how other people were using their PACS systems, and I've also looked in on the material that Siegel and his group from the Baltimore VA have published online. From all of this not only do we have the consensus model but we also have the notion that the whole thing is fully customizable.

So there's a question that arises from that I'll come to in a moment. This is just to show you a number of the online resources, and when this is posted online you'll be able to use these links to go to those particular resources. That's from Elliot Siegel and the

Baltimore group.

So this is the consensus model. An MR station is a little different from the basic model in that it has three high resolution monitors because MR tends to have a lot more sequences and you need a little more real estate. Here at VeHU we're going to be working with two monitors and a control monitor, so we kind of have to simulate what we do in regard to MRI with a little bit of space limitation.

In regard to thinking about PACS, why do we need to think about it? If it's fully customizable why don't we just say okay, we're going to make it do what the radiologists have always been doing. So the question also arises, why optimize the radiologists use of the system's capabilities?

So to explain this I'm going to do a short aside about the lessons from the Civil War and from Gettysburg.

And you're all familiar with Marie Antoinette saying, "Let them eat cake." Here's a quote from a German militarist, General Helmuth von Moltke. He said, "The Civil War was nothing than two armed mobs chasing each other around the country, from which nothing can be learned."

Richard Moorehead, military scholar, disagrees with this, and from the Military Review has an article dating to 2004, and he says there that there were three technological issues

3

Transcript for VeHU 2007 Session #227 that affected the outcome of the Civil War. One was the rifled musket, the other was the electric telegraph, and the third was the railroad.

We'll just focus on the rifled musket. If you think back to your European history you'll recall that Napoleon came to rise after the French Revolution of 1789, he had his ascendency in that period and so his war tactics date to the early 1800's. They had smoothbore rifles at that time and they were accurate to about 100 yards, and if you wanted to inflict damage on an opposing army, in addition to cavalry attacks and artillery you would bring your infantry men up in a group, they'd fire in unison to try and inflict sudden impact on the enemy. Range of 100 yards for the smoothbore muskets, canister artillery 400 yards.

At the time of the Civil War, 1860's, a development that became very important was the development of what is called the Minie ball or bullet, and you can see a draft design for it there. This was shaped more like we would expect a bullet to be shaped and it had a cone shaped defect at the back. This still had to be pushed down from the front end of the musket, so it had to be smaller than the barrel. But the barrel now had grooves in it, so that when this was shot out of the barrel it would begin to spin, and that spin gave it greater stability and it became accurate to 500 yards. The other thing is, for those of you who are into self cleaning modalities, this bullet actually expanded in the barrel, so it took any garbage, any residue that was left over from the previous shot out with it, so it was self cleaning.

So the implications again from our military review here, the Civil War infantryman could now target the artilleryman at 500 yards, before the artilleryman could target him. With the rifled musket they could attack the cavalry from further away. And basically the shift in firepower meant that the defensive armies were in a much stronger position than those who were trying to attack, whereas before attack was very much to the fore. Rapid cavalry charges, move your infantry up, fire.

4


So the way to accommodate that, you can see here General Lee's plan for day two at

Gettysburg and this comes from the Wikipedia site. The best answer, or one of the ways to answer this change in the equation was to attack the flanks. Unfortunately those attacks remained frontal attacks, even though they are to the side of the enemy's position, and so they drew a heavy cost.

Come to day three, Lee's initial plan was to repeat the day two plan, but then an initial engagement up on the right flank of the Northern Armies offset them a little bit and he decided to initiate the central charge, Pickett's charge, which was a regular frontal assault across an open plain with the resulting significant loss of life and the defeat of the

Confederate Army.

The lesson from the Civil War then is technology can impact very significantly what you do at any given moment, and the smart thing to do is make the optimum use of the technology and to function accordingly.

So we don't just replicate the auto-alternator, we go for the new consensus approach. The consensus approach in the context of

VistARad is based on the following principles. First off all, scrolling through images beats scanning over them with your eyes. So in the old days we used to hang lots of films and we used to have to scan over those with our eyes, it's much quicker to just use the mouse wheel and run through them in scroll fashion. And then in VistARad, if you plan the ports appropriately you can change the size of your image very rapidly, and there are two ways you do it. You can either do a double click, and I'll show you how that works, or if you create variable size of ports to begin with you can move one image set to another location and it will enlarge or get smaller as the case may be.

In regard to scrolling beats scanning, it's basically a reinvention of the wheel, and in this case it's the wheel on the mouse, the conventional image you have there in front of you. I just had an aside on this, I thought well what about the mouse on the wheel. So you're

5

Transcript for VeHU 2007 Session #227 familiar with the standard version, here's the cartoon version, and you know, we're not the only ones who are thinking of other ways of using other things or people.

Here is a mouse that reengineered the frog to get out of the water. So riding beats swimming.

Alright, so looking now back to Patch 32 of the VistARad program, I call it the ye olde auto-changer because we kind of came at it with that mindset and we put everything up in layouts. You see a CT layout on the right and you see a chest layout in the middle.

But then we also had the component of stack mode, so you can see now the chest X-ray is in stack mode, the images are behind each other, and the layout of the chest now it's preferable to go through that scroll on the right through the lung window slices that you see illustrated there. So again, scrolling through images beats scanning.

Now here are some of the techniques that you can use when working with VistARad.

One is to zoom in, let's say you've got two monitors in front of you, you've divided them into six view ports each. There's your lung stack, your lung window's at the bottom there.

When you double click with the left mouse anywhere on that image what happens is that. So that's a very quick changeover, you've now got a big image that you can see in more detail.

When you're done with that you double click on it and it goes back.

Another way to change the size of the images, here is a diagram showing an oval which could be like the brain in CT head. If you bring them up four on one and you want to move to another size, you can click that image and drag it to the monitor that has the full screen and you get a larger image. You want to move it back or move around, you can exchange other images.

6


So with those sort of concepts in mind it's now a question of thinking how are you going to use the space on the monitor, and a full screen obviously is critical for general radiology.

A segmented view such as here, horizontal, is useful for body imaging such as CT and

MRI.

Here's a more rectangular vertical orientation, that's good for the oval shape of the head or the longitudinal images of the spine.

And you've got quite a bit of flexibility, so you can have sagittal images, axial images, all on one thing, and some free spaces as well.

And you can do current and comparison, and compare imaging that way. And at any time you want to change where you are or how big something is, you click and drag or double click and move along.

So when you get started and you're working with VistARad, one of the key issues I think is to start off with a smaller set of variables and try and design your protocols that will meet the needs of 80% of the work you're going to encounter so that you've got very little adaptation to do, but you're ready for the cases where the patient moved during a sequence or whatever, to accommodate to a change in the protocol. Stuff happens. And if you avoid over choice initially things will be simpler for you and for the computer.

Now let's move onto the issue of control functions and heads-up display. This is a concept that was initially developed for the Air Force so the pilot could watch where he's flying and have data presented to him and not have to look down at various gauges and panels and so forth. The same principle applies here. You've got some control functions, you put them on the control monitor, and you're going to peg them there. There's a little thumbtack device built into the system that will help you keep control screens up where

7

Transcript for VeHU 2007 Session #227 you can find them very rapidly. And the other thing is you do want to try and bring your workspace into one area. If you're going to spend your whole day turning your neck like that, you're not only going to get carpal tunnel, you're going to get stress headaches and stuff like that.

So I have shown it here as three monitors, and in the teaching situation three monitors is still probably reasonable. The control functions are over on the left hand side. Manager, preview, scouts, and you'll get more of that later, and you'll select how many monitors you're going to work with by filling in the little checkbox.

So what space do you need? We've discussed control. The question is what is your current exam, how many comparisons do you need to see, is it a modality where you need to see scouts, how much resolution do you need? Ultrasound you don't need quite as much resolution as you need for plain films. All those factors go into choosing a basic imaging size, some for previewing, some for optimum viewing, and so forth.

So the monitor becomes the unit of space, you can divide it up all sorts of ways. Here are just some examples, and you can also turn the monitor horizontally and work on some of the divisions there.

So let's move on.

In the system there are pre-built hanging protocols, so when you first start up you can go in and just double click on the exam that you want to look at and see how the system brings it up, explore that a little bit and see if it suits your needs. If it suits your needs you don't really have to do much else with it. But I can tell you radiologists are quite stiff-necked, they don't like to be told how to do things, they want to tell you how they want it done for them. And so you're going to need to be ready to put things together for the radiologists and ease them into the future.

8


So you've got to be ready to move things around a little bit, and here we go.

So here is my basic set-up for a general radiology protocol. You've got two monitors there, the current exam, and then you can see I've divided the right hand screen into four separate viewing ports, I've got the first comparison top left of that, the second comparison top right, and then if you want to bring up a third, I continued the arc, the conceptual arc of going back in time, three goes back to the bottom, or alternatively you can leave the bottom two free so that if you decide gee, I want to look at another chest without closing anything, or I want to bring up a CT and see what that showed, you've got some spare windows down there to bring stuff in without having to go and reconfiguring a whole lot.

Remember all you have to do once you've got that up is a double click, and now you've double clicked on the comparison, and now you've got two full screen views to look at say a chest X-ray, the current and the comparison that you think is really appropriate for that exam. You can read the exam.

If you're reading with residents it's useful to have the extra screen, and you can see here the same basic concept, current exam on the left, one comparison, two, three, going backwards in time. And there's some free space there.

Alright, this is my layout for ultrasounds. This is a mix of layout and scrolling, and the way the mix works is because ultrasound is a little different, you're not going sequentially unless you're doing one of those sequential acquisitions in sort of volume mode, but generally speaking our images are not done like that. You're looking at a lot of different images from different places. I think it's reasonable then to put six on a screen, Dr.

Wilson likes to put twelve, and if you just click on one of those images, just a single click, you can then use the page up and down button and you'll move through a full volume of images, a full page of images, without having things click through one at a time. If you just use the scroll button it goes one at a time, snaking through the image.

9


That will become clear to you when we do the hands-on. This is how I arrange things for

CT. You can see on the right hand side are what I've called my primary viewing port.

On the vertical monitors, and this diagram is a bit squat, but on the vertical monitor that horizontal arrangement is optimum for body imaging. It's a nice size, you get to use the volume of the screen well, so the primary case, the current case, the image would be read at the top. Any comparison images I would place below, and you can link them and scroll through them. What I do is I create another batch of sort of preview windows which are about the same size as our hanging films used to be, so they're still reasonably useful images. So you can see that actually the current exam loads top left hand corner, mediastinal windows and lung windows, and then one free. And then the first comparison and the second comparison going down.

For MRI it's a similar configuration, but I tend at this point to open only one MRI exam at a time, and then use something called the browser which you'll hear about to bring up the comparisons. But we're going to work on some of those things further, and Dr.

Wilson will also show you how she works with two monitors to get the whole of the MRI up for you to read.

So we have a consensus, not just from our experience but from other people's experience of the kinds of ways to lay things out and how to work. Two high resolution monitors for most purposes, three monitors for MRI, and those you can divide up as four on four since you're not quite as constricted by space. If you use horizontal monitors you can go six by six for MRI.

At this point I'm going to introduce Dr. Kim Wilson, who has worked as I have as an

Alpha and Beta user with the system, and she's really taken the protocols to the nth degree.

Dr. Wilson: I'm just going to talk a little bit about how I have it set up at my house, but it can apply at work. I read from my home three days a week, and I go into work two days

10

Transcript for VeHU 2007 Session #227 a week, and when we first got the PACS system we put up four monitors everywhere because we were sort of kind of going by the old way that we were used to reading, which was usually four images wide. But we found, especially with the newer version that is currently out with Patch 65, it's so easy to scroll through images, it's much more comfortable to look at an image scrolling through. People have been used to that with

CT, scrolling through the images in CT, people have been doing that for years because actually you get a higher accuracy picking up nodules and lesions, but for plain films I wasn't so sure that's going to work out, but it actually works really well. At my house I have just two high resolution monitors, and then the third monitor on the right is for my voice recognition software, we use Dictaphone or PowerScribe, and we did try one time to combine the PowerScribe with the PACS system, but at that time the PowerScribe version was 3.0 that we were using, it had a high error rate and I end up using the keyboard too often and it didn't work out. But we're going to try it again with

PowerScribe 4.7, it's much better and I rarely even touch the keyboard for the

PowerScribe anymore. And so it may be that I can just combine the keyboard and mouse for that version. And right now the total space that I use at my house is 5 by 7 feet including the chair. At work you could probably if you used that kind of space you could fit all of the reading stations in one of our reading rooms, but the noise would be terrible, everybody dictating at the same time would be very distracting. But I do think that you can go to smaller size reading rooms. You don't want to have too many people in there at once though, I get distracted, I keep listening in on everybody else's dictations, it's terrible.

And this is a view of my keyboards.

With this VistARad you have multiple windows, and some of them it's preferable to have them up while you're reading your films, and others you don't need up. So the windows that usually I like to see displayed while I'm actually reading an exam is the preview window. Preview box pops up and displays all little thumbnail images of each of the images or series that I'm looking at, and also it would be nice to have the window that shows the report or requisition, so you can refer to the clinical history or you can look at

11

Transcript for VeHU 2007 Session #227 the prior report for the study that you're comparing to. Some windows that come up that

I sometimes want to have displayed is the browser. The browser is a window that you can use to display any exam from the preview window, including a prior exam that you might ad hoc just decide to open another exam. There's also a scrapbook that only comes up when you decide to mark an image and save it. And then the scout images for CT and

MRI, you do want them displayed at least initially so you can review it, because sometimes there's information on a scout image that's not on the CT itself, and you don't want to miss that, could be a lung mass or something. Then the manager itself is the window that you use to find and organize exams, you use it to find additional prior exams, but really you don't need the manager displaying while you're looking at a study.

So going from the four monitors to the two, I did make adjustments to the way that the hang protocols were set up. First of all, I do think it's much more comfortable to sit and just look at two monitors straight ahead than it was looking back and forth to four monitors. The middle mouse you can use to scroll through all the images very rapidly.

And also if you look here on the right side on the top, that's how it initially displays but all I have to do is double click on any of these chest films and they become full size. So you can kind of sort of look at all the patient's recent ICU films and then you decide which one you want to compare to.

I also made changes in the way I viewed certain studies, for example, cervical spine where you want to see all the images together at once, or you may want to see several of the images together. Foot films and lumbar spine, this is important especially for the older radiologists that are used to looking at all the images on a big monitor that's four wide because they're probably going to want to see all the images at the same time, or have the option to do that. And also I have a second stage in the hanging protocol, pretty much all of the hanging protocols, where both of the monitors will be used to display the current exam if there is no prior study for comparison.

And I like to set it up so that all of the pop-up boxes and anything that I'm going to be using my mouse on is on the right monitor just to reduce my mouse riding around all

12

Transcript for VeHU 2007 Session #227 over. I also recommend putting the smart move mouse setting in the windows setting, or it's also snap to, depending on your software, that causes the mouse to jump to any defaults pop-up box. I also think it's preferable to put the requisition on the left monitor, or if you have a third monitor put it on there so you can keep it up while you're looking at the studies, and the same thing with the preview. And then the browser window I would open it up and fill up the right screen. When you first get VistARad and you open it, it all comes clustered together, so you have to make a decision where to put all those windows and over the years I've come up with I think a way to do it that works well. So the browser fills the right screen and then you can still look at the current exam on the left while you're looking at some old study on the right.

And here's just an example of how the case at my house would initially open.

And I've gotten rid of the requisition on the left. I can look at the current study, I can still look at the manager and see if there's an older film I want to compare it to.

This is an example of the browser expanded, so I'm reading an MRI and the preview window is over here on the left, so if I double click on one of the old CTs it will pop up in the browser window which fills the right side. Now I can scroll through the current

MRI on the left while referring to the old CTs on the right.

And the third monitor at work we either have four monitors set up, and then in that case most people just use the two middle monitors for actually reading, and then they put their separate windows on the flanking monitors. A few people use all four monitors. Usually neuro, MR uses all four monitors because they have multiple sequences they want to see at the same time. Otherwise we do have two monitors set up with a third low resolution color monitor that we use to put the preview window, the requisition report window, the scouts, and what's nice about that is well it's cheaper because it's a low resolution monitor but also you can use the browser window has that capability of looking at color images so if you have a color Doppler you want to look at in color, you just drag the browser over to the color monitor and look at it there.

13


And as I mentioned, the default hanging protocols, most of them anyway, come with a second stage that uses both the screens for the current exam if there is no prior.

So here's an example of the cervical spine hanging protocol, it initially would open, if there is a prior, would open like this so you'd have six images on the left and six on the right, these are the current images and these are the prior. That gives you a chance to compare if there is any gross overall change, and also maybe for the older radiologists they can then at this point pick any of these images and double click on it. It will become full screen and then they can just use a scroll bar to scroll to see all the other images.

I also have a stage 2 where it comes up two on one, and the reason is I like to see the obliques at the same time so I can discuss the neuroforaminal narrowing and things, and also you can compare. So that is useful.

And then stage 3 is the current exam going across all, so I can still see the obliques at the same time, but

I'm not wasting that right monitor because it would be blank if I didn't have a prior. And then again double clicking on any of these, they'll become full screen.

For a foot, similar thing. We have podiatrists who do a lot of surgery at our place. We get a lot of bilateral weight bearing foot films. Six images, they have priors, they've had surgeries, so they have more than one prior, maybe a pre-op and a post-op.

So for that reason I have this set-up. So to get a feel for the alignment basically it's nice to see all the views, and again just double clicking comes full screen for a real detail, and

14

Transcript for VeHU 2007 Session #227 the comparison they'll both be on the right screen but in this case there was only one prior so there's nothing in the bottom.

And this was just by double clicking on the images on each side they became full screen.

In stage 2 is the current, will just fill up both screens.

And lumbar spine a similar thing. I have a couple of stages so I can look at them like that.

And then stage 3 they come up with just the current.

Ultrasound, this is basically the way it comes as a default hanging protocol, except I think

I have slightly more boxes and Peter likes to look at 6 images on his screen and I like to look at 12, some people would like 9. It's very easy to change that.

And again, if there's no prior the current will come up on both screens.

And this is how, there's a button at the top left, and the radiologist could just pull the pull down box here and adjust how many tiles he wants on each monitor and quickly goes from 12 to 6 very fast.

You can also go into the hanging protocol, the one that comes with the system, they have this at the end, SYS_internal INT, and those ones can be adjusted and then resaved, so you could change the columns from 4 over 3 to 2 over 3 or 3 over 3. And then it would permanently change the way they come up.

So CT chest is another one. Because we have two CT scanners and it was sent different numbers of series and different types of series, I couldn't come up with one single hanging protocol that really worked, so I have multiple stages and for stage 1 I have it explicitly mapped. You could take a characteristic of a series of images and select that

15

Transcript for VeHU 2007 Session #227 characteristic and then use that characteristic to apply it so that it goes to a certain location. So this current mediastinal window would go here and the prior mediastinal window CT would go below it. And then the lung window current would be on the top right and the prior would be on the bottom right.

This is pretty much the way I think most radiologists like to see about this size.

This is also fine. 4 on 1 is also usually good enough, but it depends on some of our residents make it full screen and then sit back about 3 feet away and read like that, so everybody has a different way of doing it. We had to have another series in case there were multiple additional series other than just the chest. We do a lot of times the chest, abdomen and pelvis are all combined, so this way you would see, if there was a third series over here it would show up right there.

Then I also have a stage that compares to three prior CTs, so a lot of times we follow lung nodules for a total of two years and there's usually often at least three priors.

Therefore they'll automatically display, you don't have to go searching through the history to find the old priors, they will automatically be pulled and displayed.

And then on the stage 4 chest X-ray, the current and prior. Again, that saves you the trouble of finding the prior chest X-ray when you open it on the manager, it will automatically open, which is real convenient.

MRI abdomen, this is very similar also to the default, basically the series will display 9 on one on the left, and if at your institution the MR abdomen actually usually has 12 series, then you could have this come up with 12 view ports on the monitor.

16


Generally if you want additional priors there's not extra real estate sitting on the monitors, so that's a good time to use the browser. The browser is like a whole other window to show whatever cases you want, works very well.

For studies that are done in a routine way, the same way every time, you can have really specific hanging protocols. For MRI knee what I do is have the axial images, which are taken in the same alignment, linked so that when I scroll through one, like this as I go up and down this will automatically will move with it.

And the same thing with the sagittal, you can have those linked. And also automatically pull up the prior X-ray so it's displayed down here in the bottom, double clicking on it becomes full screen, double clicking again it goes back small again. And basically I created this for the department and everybody uses it and loves it. No complaints. So straightforward hanging protocols like that you can create and share with everybody.

Another thing that's very important I think as far as efficiency and management of the department and reading is to create custom lists. So if you can work with the radiologists or managers to come up with how they want to divvy up the workload. You can create custom lists based up on the modality, CT, MRI, plain films, or the location of the patient. Or the location where the images were taken, so it could be based upon the location that the patient was when the exam was ordered, or it could actually be the inpatient radiology area or the outpatient radiology area. And then you can also make the custom list based upon specific exams. We have a musculoskeletal list and we have a chest list. And they even can make a custom list based upon the last four of the social security number because that was a request from some places that divvy up the workload among the radiologists based upon the last four of the social security number.

17


When you're using a custom list it allows you to use something called a readlist, which is really efficient, where you just click on a button and it starts going down the list of cases one by one. As you read them, the next case automatically preloads and opens. The thing that's really nice about that readlist is that STAT examinations pop to the top of the list and will get read next, no matter where you are in the list. And I was talking to another person who uses AGFA and he's having a problem with that, with the STAT not getting over. So they're having trouble knowing when to read the next STAT case, it's not getting through, they're not going to the top of their list and they have to go and find them. But that's not a problem with VistARad, and it's really good, we never have a problem keeping up with the STAT and urgent exams, they're always getting read usually within minutes if not within an hour. And then the routine exam will be listed in order from the oldest to the newest, basically that's the order you want to read them. You can also have the custom lists ordered in any way that you want, and you can also reorder them at any given moment if you decide today I want to read them alphabetically, you can do that. As you read down the list, as I said, if a STAT exam is done it pops up to the top of the list and it will be loaded next.

This is just explaining about how to use the readlist. Basically at the end of reading a case you just click a button, it closes the case, and as the next one opens, there's just a few seconds of lag time, and I correct my dictation and send it through.

The readlist doesn't work quite as well for cases that have multiple accession numbers because what it does is uses the first accession number that's at the top of the list and it might not be the one that you really wanted for your hanging protocol. So for CT, MRI and angio we tend not to use the readlist or make what's called a selection readlist which you can just choose the cases that you want by holding the control button down and selecting the individual cases. It will create like a little temporary custom list and then you can read down that list.

18


Dictation integration has been really helpful. For one thing, you don't have to type in any numbers, scan a barcode, we were doing that for years. That was subject to error, especially typing in the numbers, and we've been unable to identify any mistakes in this.

So far it has always sent the correct information.

PowerScribe 4.7 we only got this year, and it's got much improved voice recognition, which also speeds up efficiency. They also have templates now which allow you to have default phrases that you can just replace or leave in by toggling through them. If you want to replace it you just dictate over it. Because of that I now use a template for about

90% of dictations, and it saves a tremendous amount of talking.

For example, I use the old version of the microphone because it's simpler and I like it, but there's a tab button here, and as I hit that it tabs to each of these bracketed areas. So if this person did not have a prior chest film I would just tab right beyond that, and if they did I would dictate the date in right there, it would replace the word none with the date.

And I go through and tab through all of these areas and usually at least half of these remain and I only dictate over part of them. Even on abnormal cases I leave part of it, and that saves a lot of correcting as well.

This is a little picture from the PowerScribe monitor itself, and on the right hand side there's a list of the templates. We have many different ones that are organized by modality and each person can have their own list. So some people have hundreds, and some people have none. They just use the ones that I made for the department.

Like for musculoskeletal we have probably 60 or so.

And at the end of the report in PowerScribe we get a pop up box that asks us if we want to send a view alert. This I think is really important for the radiologists to understand what the view alert does. It sends an e-mail alert to the ordering physician, but depending on how you have it set up with the CPRS, it can send it to other people. For example, in the emergency department when I first started working there we found out that the view

19

Transcript for VeHU 2007 Session #227 alert for abnormal studies was only going to the ordering physician, who could be a person that only worked one night in the emergency department, and then nobody was getting follow-up on those patients. So we changed it so now the view alert goes also to the primary caretaker, so they're alerted that their patient has an abnormality. The other thing is the retaining of the view alert records in the computer we discovered was being purged after 30 days. And for the radiologist protection it would be preferable to keep that information in the computer for maximum time because otherwise you have no documentation that any kind of alert was sent to a provider that there was an abnormality.

Diagnostic codes may be underutilized in some places, I think they're supposed to be the same at all the VAs, but we at one point went in and made some minor changes and eventually got in trouble for that. Code 4 I believe is the same everywhere, and that's abnormality, attention needed. And it not only sends a view alert to the ordering provider and/or primary care person, but also it will mark the study as abnormal in CPRS so they look down the list of studies it will say complete, abnormal, and bring the attention to anybody opening up the CPRS. Also there's a code 5, major abnormality, physician aware. We use that if we call the provider because of something very important that's going on, like a collapsed lung, and we use this when JCAHO came by and said we want you to show us that you document properly, then we just ran this code 5 and could give them a list of those cases that had been called, because in the report we also say discussed with Dr. so-and-so at such-and-such time. Code 7 is unsatisfactory/incomplete exam.

We can use that also for tracking for QAQC for the technologists. And code 9 is a code 4 possible malignancy, usually when you initially discover it.

So the tumor registry has used code 9 to identify new patients. We've also added additional codes for mammography tracking, so the BiRads codes were on there. And then we could track mammo patients that required follow-up and make sure that they're getting a follow-up.

Voice recognition, if you're not using it and you introduce it, will probably be problematic. It's got advantages, the main advantage is your report gets uploaded

20

Transcript for VeHU 2007 Session #227 immediately into the medical record, and that is huge because it stops all the phone calls from clinicians, basically went down to almost nothing. They realized that if the report wasn't in there, nobody had looked at it yet, whereas before we would be reading films all the time and there would be no report yet because the transcriptionist hadn't gotten to it.

Also because you correct the report while you're still looking at the study you're going to have greater accuracy, right, left, that kind of thing. It can be faster than the old system of free dictating for normal exams because you use the templates. And also when you're reading with a resident, before the resident would just review and tell you what they said, and then later you'd get a report and you'd have to remember is that what we discussed?

But now the residents will actually open up their report while you're reviewing the case, and then the radiologist can have the resident correct the problems while they're sitting there. So that results in better accuracy. The disadvantage that it takes longer to correct these reports than with a real transcriptionist because the radiologist has to correct it instead of the transcriptionist correcting it. It's also slower for long and abnormal exams because a certain percentage of words that you dictate are going to come out incorrect.

So the longer your report is, the more corrections you're going to be making. And some people cannot adapt to the multitasking, keyboard, the mouse, we've had a couple of people come in and just could not do it. And you need to modify the old way of dictating and adjust and dictate differently. I tell people if you talk like a computer it works really well, and it really does, because it's the slurring that makes the words get all muddled up.

Which brings up the point that when you're rushing, if you're behind, then it goes really badly.

Now the health summary is really useful, and that's something that the IT people have to work with the radiologists to come up with what they want in their health summary. So in the VistARad there's a button that they can click on and pull a health summary from

CPRS or VistA, and it's very useful. Most radiologists I've talked to really want that, and they use it a lot because what it does is it gives them access to like the emergency department note, and they can see if the ER doc saw a fracture or not, and helps to determine when the caretaker needs a phone call if they missed the finding. So the way that it's created is certain components of medical record are chosen and then they're

21

Transcript for VeHU 2007 Session #227 imported into VistARad at the click of a button. I created one for our department called diagnostic radiology and it lists the allergies, I mean obviously that's because of the iodine allergy for a CT, and then problem list, the last five progress notes which usually includes emergency department note from the night before, and then specific laboratory results, BUN/creatinine, which is pertinent to us. And the CBC and lactate, you could put also like alpha-fetoprotein or other things that might be pertinent to a radiologist.

Also list all surgery, pathology, and discharge summaries.

And then this just shows that there's a button here in the middle of the manager, just a click on that and it will display.

The first time you do it you have to choose which health summary. The health summaries are in the whole hospital, everybody can have the health summary, so you have to pick yours.

And then here's an example where I pulled up and the emergency department note is there.

So as a summary, when I sit down to read, first log into the voice recognition and then lost into VistARad second. It will then recognize that voice recognition is on. Then I will just pick a custom list that I want to read for that session. Click on the readlist, send the information to PowerScribe. After reading the first case I close it, and then after that a single click will close a case and open the next. While the case is closing and opening the next one, then I would just correct the transcribed report.

If somebody comes by and wants to review a case, or calls me and wants to review a case while I'm in the middle of a readlist, I don't have to stop the readlist, you can just keep it as it is, review the case, and close it.

It works quite well. And at the end there's a stop readlist button near the middle of the manager, which I use. Then I collapse in my chair.

22


The efficiency of this system is in the older days the time was spent looking at the film and checking the patient name and the date, am I looking at the right film and is this the right prior. Don't have to do any of that anymore. Now the time is spent looking actually at the image itself. Getting the clinical information is really quick, just clicking that health summary I can get all this additional clinical history, and then I do spend more time dictating and correcting the report than with the old transcriptionist. But I do believe that the result is a better quality report, it's more accurate. And it also gets uploaded much faster. I do not spend time arranging the images, hunting for priors, or figure out what case to read next.

So on average if I'm just sitting down without interruptions, I can read 30 plain films an hour no problem, paying full attention to the clinical history, putting a full dictated report in the medical record, not rushing, and also calling the occasional result to the ordering clinician.

And now back to Peter.

Dr. Rosenthal: Part 3 of the talk, Kim thank you very much for that section, I'm going to discuss a number of elements that I think are important, they may not pertain to what you do on an everyday basis but you are the advocates for the system and I think there are things about it that you need to understand so when you're having a discussion or your department is talking about doing something new, that you have a perspective to put this all into. So I'm going to provide you with some experience from the field, look at the issues of efficiency and VistARad's impact on productivity, try and make a comparison to other systems, although I've not worked with any other systems, present you some facts and figures, and then discuss how these fit into the big picture.

I've been at my position now for about ten years, we have four radiologists, five sometimes. We have had almost level technology staff for that duration, very minimal

23

Transcript for VeHU 2007 Session #227 increase, and our workload has gone up as I'll show you, and we've had residents ranging from 3 to 5 at any given time.

So what I did to try to bring you some information to assess the impact of VistARad is I went into the VistA section radiology management, and I pulled up some data just on the month of March from each year starting in '96. That material you can see here, so in

March of '96 we read a little over 3000 exams that month. There was little drop the next year for reasons that aren't clear. Then you can see a steady increase to over 5000 a month. If you annualize those figures we're going from around 35,000 a year to 60,000 a year. So that's a significant increase in workload, and I haven't mentioned the technologists in this equation because they barely changed at all, but you can see there are 3, 4 and 5 across the bottom of that graph, those are the number of attending radiologists that we had at that time. So in '96 we had only 3, '97 had 4, and only between 2005 and 2006 did we get the additional fulltime radiologist.

What I then did with the data is I averaged out the first three years and the last three years in order to kind of get an idea of what was happening over more than just one month of a year. So averaging out the numbers for '97, the March exams were just shy of 3000. For

2006 just over 5000. Annualized again 35 and 60, and if you look at that as a percentage change, from the baseline of '97 72% increase in workload. Radiologists went from approximately 3.7 to 4.7, that's almost a 27% increase, so we went up about a third of our capacity there, and in terms of technologists we went up 1 or 2, safely below 10%.

And a lot of the stability to absorb that extra work I believe can be traced to having a

PACS. Now the question is when did we get the various forms of the PACS. VistARad

16 came in at 2003, that was our first PACS system. You can see it corresponds with an acceleration in the workload, that's not because the PACS brought in more patients, it's just we had more to cope with, but we coped with it without much change in staff. 16, that's where we got 32, this is where we got the Alpha version of 18, and then we got 65 somewhat ahead of everyone else around that time there.

24


So again those are the numbers in summary that I discussed earlier, and I think the takehome message there is steady increase in overall workload, but the same number of people could handle it because of the PACS system.

Voice recognition, just a few comments here. It slows down the reading of studies, it increases the number of errors, the more you say the more you have to correct. If you are not as diligent as Dr. Wilson is about correcting your reports, then more reports go out with garbage, and periodically you get these e-mails back saying what did you mean about a rectum showing up in a foot film? Don't ask me. But anyway. So you make some corrections. But it does improve report turnaround. We have a VA mandate, we have to have 90% of our studies read and finalized within 48 hours. I know that we're certainly meeting that target in Charleston. My real question was how well are we doing in 24 hours, which is kind of a real world number, and actually we're at 89%. So we're practically meeting the mandate on day one. Good going. And that again has to do with

PACS and the talk technology. But voice recognition is the most important rate-limiting factor to radiologist productivity.

Here's where we got it in our analysis, and you notice we got a radiologist just after we got that. I don't know if that was the impetus or not, there are a lot of other variables that would go into this analysis.

When you're thinking about PACS systems and their efficiencies there are a lot of different things to take into account. There's the basic speed of the program, how quickly does it load. Flexibility, does it allow you to chop and change. You're busy with a readlist, someone wants you to help them on another case, can you step out of that, do that, and then come back to your work without losing your orientation and having to reboot things or whatever. What's your overall efficiency, how many cases can you read?

Is it easy to use, can you sit down at the system and get on with your work? And once you've learnt the system is it really efficient at the point where you're really familiar with it? Is it easy to customize? These are all issues to look at. And then finally, does it interface well with the other systems? Human and informational.

25


So in evaluating VistARad and trying to compare it to other systems out there, I think our experiences I've shown you from those figures has been favorable. But there have to be other ways to evaluate VistARad, and if you haven't worked with it you have to look somewhere else. And then those are the issues that I'm going to look at, the impact on productivity, cost issues, and system integration.

So I ran a time test myself to see if I was reading plain films, and I don't read plain films all that often, but if I sat down to read plain films how quickly could I go through them, and I could achieve at my highest speed about the same as Dr. Wilson mentioned, about

30 cases in an hour including some phone calls and stuff. I couldn't maintain that for 8 hours a day, but certainly the system runs fast enough when you're using the readlist and you've got your protocols set up, that you can do that. I'm going to do a little system process analysis to discuss those cases that take a bit longer to load, but before I do that just mention some other resources I went to to evaluate how other systems compare with

VistARad. I asked our residents who work with a commercial system at the main hospital that they're based at, we have a couple of new staff members part-time who've worked other systems, I asked them now that they've worked with ours can they give me some sort of appraisal. And I've done a couple of site visits as well, teaching hospital and community hospital.

I've discussed the time sample test, and the only thing to take away from this slide now is general radiology is 63% of your cases. So if you've got a system that's working very well for general radiology, what you've got left to fix is on the downside, 40% and less if you're having difficulties with the efficiency of the system there.

There are 24 residents in the program. At any one time we would have about 4 at our hospital, which means we've got less than 20%, which means that they're only working on VistARad about 20% of their time. And they say that the commercial system is more user-friendly, it loads large volumes faster, it handles larger volumes for check out, for

26


MR/CT, and for very high caseloads in general radiology, it handles them better in the teaching situation.

Having said that, in terms of user friendliness one of the issues that impacts that is how often you use it. If you don't use something very often, when you come to use it you're not going to necessarily find it all that user-friendly anyway. There are going to be differences between solo work and working with residents anyway. I'm going to do an analysis that discusses the issue of the load rate relative to the read time shortly, and the residents are often not so happy to be at the VA and they may be a little bit negative in their approach to how well the system works.

So we have to kind of take it as a measure. This is not cycling quite as I want it to, but I will show you anyway what I'm trying to tell you now. I want to discuss the issue of load time, and load time is defined as how long does it take from the time that you select a case that you want to read, to the time that the images are in front of you and you can begin to read. And obviously we want that period of time to be as short as possible. So the load time is signified in red here, and then the read time is signified in blue, for exams generally if the load time is long you'll have an element of inefficiency. What VistARad has in it is a process through the readlist where the next case is pre-loaded in the background, so that load time occurs underneath or simultaneously with your reading, your primary reading over there, so when you're finished reading you don't have to wait a significant load time, you go to the next case and it's there for you to start reading. Now it's not quite instantaneous as this graph shows, but it's not a long factor of time. So that's the read time. Short load times are obviously optimal when we're evaluating PACS systems. For studies with long read times the load time factor as a percentage of your working time is reduced. It's frustrating to wait 30, 40 seconds for an MR to come up, but if you're going to spend 20 minutes reading it, as a factor of your time you haven't lost all that much time. So workstation preloading is helpful, and if you have a very limited range of things that you read, and you just have a limited number of protocols, then you can use the readlist to facilitate that and make it more efficient.

27


Okay, some cost issues. VistARad versus commercial PACS. Questions, questions, questions. To put this together was quite a task because so much is changing. You know just from your own experience that computers themselves, the cost change is dramatic.

You buy a laptop today, two months later for the same money you could have bought something with way more memory, a readable and writable CD and DVD component.

Everything is just changing very, very quickly. So how quickly do prices change for the systems that we're using? They're probably going to change as quickly. Technology is changing incredibly quickly as well. If you think about it, CPRS was only put into gear ten years ago, only ten years ago and we've got a complete Electronic Patient Record and it's impacting the healthcare already in less than a decade. That's unbelievable. And when you're costing something and you're speaking to a vendor, where are they in their business cycle? Are they just trying to get out there, they want their presence a little further in the market so they're willing to give you some deep discounts, or are they well established and they're now in that cycle where they want to make more money? So those issues make it very difficult to do the kind of analysis that I've done, and you need to understand that in terms of looking at these numbers. The final question I'm driving at is what are the right choices to make in our government system when we're already spending large amounts of money?

So to try and attempt to answer this question, that's one of the reasons that I wanted to get the consensus model of what a workstation is so that we can start costing its different elements. At our VA we do 60,000 exams a year, again doing all modalities except mammograph, and the following parameters go into this analysis. VistAImaging is a VA mandate, everybody's got VistAImaging, and you'll see how that works because that's an underlying structure where we have hardware and software in place, we have to have that. VistARad dovetails with VistAImaging. Commercial PACS is an add-on.

So a model department, we have one MR workstation which has at least three monitors, and then we have approximately five standard stations at our hospital. That's what we run with. You could fine tune the resolution of your monitors and save a few dollars here and there, my attitude is we want the high resolution everywhere because if I'm looking

28

Transcript for VeHU 2007 Session #227 at an MR and I want to see what the chest X-ray showed, I don't want to say well gee I can't look at the chest X-ray here. So all of the monitors, the costing here is the high res, as if you were reading plain films.

Again to remind you, two high screen, one low screen PC with color for control and that sort of thing, MR having three.

Let's see what it would cost to just pay for a VistARad workstation set-up at a hospital our size. Standard workstation unit price $20,000. We would need five of those, the subtotal is $100,000. MR workstation, extra monitor $37,000. We need one of those, we get to a total of $137,000 for our size hospital.

If you look at a commercial model, and I went to a hospital that had acquired one within the last three years and looked at their numbers, and I scaled it down because they're doing nearly four times what we're doing. I scaled it down to try and get it to the same sort of size. So their unit price for a workstation was $55,000. We need five of those, that's $275,000. MR workstation with the extra screen, $70,000. $345,000. Alright?

And if you look over there I've put the VistARad number in, so you can see there's a difference there of around $200,000 already. So now let's look at the systems diagram, you've got one modality represented there, CT scanner. There are other modalities for which the same would apply. The data from the CT scanner, the imaging goes to an imaging gateway. From there to a background processor. And from there it goes to two storage systems, the juke box and the server. The juke box is long-term storage, it's slower. We haven't broken into that yet. We're still pretty much on our 8.8 terabyte

RAID system and the 2 server which is quicker, and you would use that for your current imaging and recent comparisons. From there we have two end users, we have those using PACS, top right, and then bottom right, anyone with a PC can use the same data for

VistA imaging. And long-term storage actually ties into, this is the correct tie-in there, the long-term imaging will also go through those pathways. So the question is what happens when you get a commercial PACS? You have to replace the workstation. That

29

Transcript for VeHU 2007 Session #227 we know. It turns out you also have to duplicate the server because when you go with a specific provider they've got their particular way of doing things. Some of them, the software is in the server. For others, the software is on the workstations. So these systems are somewhat variable, but they all have to use their own RAID system and server, and Atlanta in our VISN is one of those that has gone with commercial PACS and

I've confirmed they have a duplicate server and RAID system set up there. So the question is how does that impact the cost? Everything else that you have here stays because that's what VistAImaging runs off, and that's a mandate.

So looking at the server issue, 5/8 of their 14 terabyte system, the unit price for each of those eight units was 50, we only need five of them for the same memory. 250K. They have some intelligent database functions on one of their servers, that one costs 140. That includes 3D stuff. Okay I'm going to factor out the 3D because we're not talking about that now, so I came up with an estimate of $70,000.

Now you can put the whole thing together. The PACS workstation is $345,000. The basic server is 250. The server with database, 70K. Your total hardware cost, $665,000.

There are installation and other start-up services. Modality hook-up, for every CT scanner that you hook up, every ultrasound and various other workstations that you hook up, the vendor charges you a fee. I factored that down to the number of scanners that we have with their pricing, came up with a modality hook-up fee of $276,000. Interfacing and training, you have to get various information systems to talk to each other, they had a charge for what they were going to do to make sure that all their components spoke to hospital components within that system, and they were going to train everyone to use it.

$160,000. Total for the system just over a million.

If you factor that VistARad costs you about $140,000, your net increased expense is just under a million on that particular model.

Now not included in this analysis is that you need special tech stations because some of the stuff that we do in VistA, that the technologists do in VistA to get the data to

30

Transcript for VeHU 2007 Session #227 coordinate and to finalize exams and so forth, case them in and out, that gets done in

VistA. If you buy a commercial system you may need special stations that do that, and monitors in additional places, and some of those you can make hybrids so for example the station that the tech would use in the ICU can then also be used by the ICU staff to read their stuff. So I put that together at about another 125 K. There are maintenance costs, in this particular system around $250,000 a year. That's just an ongoing maintenance cost. Now I think that what Atlanta has done is they've hired a person to do the maintenance in-house, but it's still an extra cost, it's an extra person that has to be paid, you might save some there if you do it in-house. Upgrade and frequency, when they purchased this particular model at the hospital I went to, they weren't thinking about upgrades, they didn't know that there was one just around the corner. They bought in, within two to three years the company is offering them an upgrade to another system, another $200 or $300,000. I don't know what the life expectancy of these systems is going to be, but there are going to be constant upgrades anyway as the technology moves forward, as you go from a four slide scanner to an eight slide scanner to a 64 and so forth.

Anyway, how does this analysis compare with other analyses going on in the field? I went to the chief of our imaging acquisition and I said to him this is my analysis, I've come to a figure of around a million for a commercial PACS, what do you think? He said if we were putting in what we have now and doing the equivalent, I think we'd spend

2.5 million. Then he said there is another one of our sister hospitals, they've made a request for a commercial PACS, and their quote is somewhere between $700 and

$900,000. And he said he doesn't know that that's necessarily a complete system. I've got some additional information from him in regard to Atlanta. Atlanta spent, I believe, if I remember the figures correctly, they paid about $365,000 for the initial package, and they are going to pay by the case. So a lot of the commercial PACS guys will say to you here we'll give you the system and then after that you'll pay by the case. And so if your system grows, they want more money because you're doing more work on their system.

They are budgeting at this point for $40,000 a month. First year then becomes $480,000.

In two years they'll have spent just under a million just on the budget, not on that initial buy-in. And if you play around with those numbers, the relative cost to the 95,000 cases that they are doing per year is almost identical to my analysis at 60,000 cases a year. So I

31

Transcript for VeHU 2007 Session #227 think this gives you a perspective of where the costs are if one goes with a commercial system. There are other issues, lease versus purchase, the volume paradigm which I mentioned, data migration is a big one, vendor changes and data migration go together.

Interfacing with other systems, I haven't even discussed with you what other systems will charge you when you bring in your commercial PACS, but it doesn't work with their particular computer system that you're using in a certain section of the hospital where you need it. And then there's the whole issue of web access and routing. Web access is a little bit better with commercial systems for remote reading, but routing is certainly a very viable way to go. Just in terms of data migration, it's a very important point. If you decide to go with a commercial system and you've already been working with VistARad, or you've been working with another commercial system and you've got all your comparisons in this database, they might say to you well you know, our system is not going to be able to access that data in the form that it's in. What you're going to have to do now is you're going to have to migrate that data which involves time and money and hardware and software to get it over to the new system, and that affects the costs there as well and that's a big one to bear in mind.

Okay, in the big picture change is incredibly rapid in everything that we're dealing with.

As you heard this morning, the VA has got a tremendous record of cost-effective care and we're getting results through the preventive, risk reduction, patient-focused care, and looking at the veterans health rather than just stepping in when they need a procedure performed. CPRS, the Electronic Patient Record, is an industry leader, and it saves a lot of money just by reducing the need for replication of lab tests and imaging. The quote this morning I think was 20%. I've heard a quote of 30% that are the extra costs of repeated lab tests and imaging that you save if you've got an integrated system and everything's there where you need it. So VistARad 65 offers us functionality that is comparable to what is out there on commercial systems. It does so at a fraction of the cost, and there are very important key advantages to this. One is system integration, everything is already working together. There's continuity, there's a team working on future developments and upgrades as we go forward. And the end user, you and the

32

Transcript for VeHU 2007 Session #227 radiologists, are involved in that feedback so the system gets refined and evolves to meet our needs.

So with that I'd like to just briefly say that I've enjoyed getting ready for VeHU and it's been fun to work with people at all sorts of levels. Kim and I have spent time on the phone with the team in all sorts of places across the country and that's been entertaining and rewarding. I think we owe a tremendous debt of gratitude to all the people who've developed VistARad over the years and fought hard to keep it going and keep it there when the reception wasn't always that bright, and everybody out there in the workforce, I see Christine Haas here from Charleston, I won't tell her to stand up but I just want to say that I recognize that one of the reasons I'm able to stand here is from the back-up I've had from her and I know Dr. Wilson has had from John, and so I just want to say thank you all for coming and let's keep going forward.

33

Transcript

Related documents

Products

Support

Transcript

Related documents

Add this document to collection(s)

Add this document to saved

Suggest us how to improve StudyLib