Session 209
Squeezing All You Can Out of Reminder Extracts
Section 1: Overview
Good afternoon everyone. So we’re almost done, we’re in the home stretch. So I know some of
you guys are ready to go home tonight, some of you guys are leaving tomorrow, and I promise I
don’t want to bore you guys or I always want to make this as painless as possible for everyone.
Actually going back to this morning’s session with Dr. Jesse on his talk on health informatics, he
mentioned about having a registry to establish a cohort of patients that’s manageable. He used
an example of a physician, or any physician, who wants to know who were his diabetic patients.
And I think that’s an idea that leadership in general thinks about all the time, we want our
physicians to know who their patients are, a manageable list, a reporting structure that is not
only robust but also transparent to management level and physicians, and basically the idea of
an extract came to be, and so that’s why I want to introduce to you guys our session 209,
Squeezing All You Can Out of Reminder Extracts.
I’m Raymond Tavera, I’m from James J. Peters VA Medical Center in the Bronx, New York City. I
am a former CAC and as a CAC I was involved with the extract project. My co-presenter is Philip
VanCamp CLIN 2 Team National VistA Support.
So the idea of extracts wasn’t something that we were like set on from the beginning. We didn’t
know what we were going to do, leadership wanted a robust report that captured a cohort of
patients that would kind of mimic what EPRP was looking at, which were our composite
workload. During that time going back a year, I think summer of last year, VISN 3 funded for
Alan Montgomery to come to our VISN and to teach the CACs clinical reminders, and during that
time we had an influx of new CACs, we also had seasoned CACs who knew reminders like myself.
I’ve been involved with clinical reminders for a few years and still I’m confused with clinical
reminders, it’s not easy at all. So it was really helpful that Alan came to us and actually showed
us the ropes of clinical reminders. One of the topics that was discussed was extracts and he kind
of left it for the last day because I had a feeling he didn’t want to overwhelm everyone with
extracts, but we got the gist of it and it was something that we wanted to know more about, you
know, the robust structure of extracts, how we could use it, and can we incorporate it in our
VISN. And he actually went about and discussed the advantages of using extracts versus clinical
1
Session 209
reminder reports, which I’m going to discuss in a little bit. And what we did is as a VISN we got
together all the folks who were involved with clinical reminders, both new and seasoned CACs,
and we ended up having weekly meetings, well conference calls with Alan involved, and we
decided to form a game plan.
Section 2: Extracts versus Reminder Reports
So extracts versus reminder reports. Extracts can be customized, you create a desired output,
extracts, and I’m not sure, how many of you guys are familiar with extracts? So you guys use
extracts one way or another. Great. So you guys know that with extracts you kind of can
manipulate to get the output you want, versus a reminder report which is kind of a linear way of
pulling in data. So with an extract we kind of manipulate what EPRP is looking at, and one of the
components of EPRP is pulling in the Nexus visits. Nexus visits are patients that were seen
within a primary care stop code, some of the sub specialty care stop code, and also mental
health stop codes. We’re also looking at patients who had an anchor visits, not only were you
seen in a Nexus clinic within a reporting period, but you had an anchor visit maybe 24 months
prior to that Nexus visit. And also to include veterans, and also to remove deceased patients.
That’s some of the stuff that’s available in reminder reports but is not as flexible as using
extract. And yes, reminder reports are linear and that way and I’ll also show you what are the
constraints using reminder reports.
Reminder reports are useful but can be very limited. When you’re running by location you’re
pulling in patients that were seen within a location, so if you choose location and you choose a
couple of clinics, let’s say a couple of primary care clinics, you’re running it by date range. Now
you’re running every patient that was seen in that location within that date range. So you have
your established patients, which is fine, but then you’re also pulling in new patients. That might
not fall on the EPRP’s pull list, you might pull in veterans, well you are pulling in veterans, but
you have a chance of pulling in patients who are not veterans. So for example if a primary care
doctor ended up seeing a patient that was a spouse of a veteran or child of a veteran, chances
that you’ll the meet the core criteria of a clinical reminder, chances are that patient is going to
show up in your report. And the problem of the deceased patient function, so if you go towards
the bottom of running your reminder report you will have the option of excluding deceased
patients, which is great, but the problem is if those patients were alive during the reporting
period and are currently deceased, it’s going to exclude those patients from the list. So if you’re
2
Session 209
running a report going back let’s say two or three months in the past, and the patient has been
deceased for a while but was alive during that time frame, you’re not going to get that patient in
your report, and chances are if that patient was alive during the reporting period EPRP might
pull that patient in their pull list.
So here’s a screen shot, and this is a reminder report option. Everyone is familiar with this, and
like I mentioned before, by location, which is fine, I can live with that, I’m running a simple
reminder just to get a list of patients that were seen somewhere else. But this is not exactly the
best tool to use when you’re trying to create something that you want upper level management
to look at or providers to look at.
Now jumping ahead, this is a view of an extract. Since most of you guys are familiar with
extracts you know what the screen is. It’s an inquiry of an extract and it’s pulling up the rules
that are in the extract, sequence five, sequence ten, and what it’s showing you is a customized
source of patients. That is a lot better because number one I’m seeing well sequence five is
pulling my Nexus, and it’s adding the patients, so I’m pulling those patients that were seen in a
Nexus clinic. And then sequence ten is telling me well the anchor visits, which is a select option,
so now I’m having not only my patients that were seen in Nexus clinics, but now I’m selecting
those that had an anchor visit, so now my list is truer to what EPRP is focusing on when they’re
pulling in their patients.
So there is a huge learning curve behind this, so going back to what I said before about
conducting weekly meetings, and like I said there were CACs who were knowledgeable in
reminders, there were CACs who were new to reminders. We had to get everyone up to speed,
and so we had to familiarize everyone with building reminders. Not all CACs had the same level
of knowledge, some CACs are used to building reminders left and right every day, some CACs
know of reminders but weren’t too familiar, kind of like iffy about creating reminders, so we had
to at least get everyone up to speed with that. One of the things we did was standardize clinical
reminders. If we’re creating reminders as a group for the VISN there are certain elements we
need to standardize, which I’m going to get into in a little while. Just because you build clinical
reminders doesn’t mean you know what performance measures are. That’s one big thing,
because you can build a simple clinical reminder that a clinician can tell you I want a reminder
that does this and that, but then you might not be familiar with what the performance measures
3
Session 209
are. A lot the clinical reminders, believe it or not, they’re available in CPRS cover sheet are
based off performance measure guidelines. So another thing we had to go into was going over
what was inside the EPRP technical manual. And this was brand new training on extracts so we
had to make sure everyone’s familiar with what were extracts, what was an extract, and what
compromised an extract. Basically the list rules, what was a patient list, and building the extract
itself.
Section 3: Extracts 101
So a little bit of extracts 101, this isn’t to teach you guys what an extract is, this is just an
overview of like what are some of the rules that compromise an extract. So the list rules, it’s
kind of a concept, it creates a list or cohort of patients that are used in reminder extracts, and
there are four kinds of rules. We have a finding rule, patient list rule, reminder rule, and rule
sets. And I’m going to go into details of what each of those rules are.
A finding rule. A finding rule is based off of a reminder term. You can only include a term in the
finding rule. And then a term can include anything, it could include like a set of different finding
items or like or similar finding items in a term, it could just have one finding item, so my finding
item I’m using this example of hemoglobin A1c greater than 9, it could just be a simple lab test
of hemoglobin A1c greater than 9. You can’t put a finding item in a finding rule. You still have
to create the reminder term. So once you have a reminder term, when you run that finding rule
or when you establish a finding rule in a rule set, it’s going to use what’s inside that term as its
criteria. Kind of like in a clinical reminder when you have a finding item and it’s either true or
false. So if it’s true and it matches, in the rule set if it’s using a finding rule it’s going to agree
with that finding rule and then whatever you use that finding rule in for the rule set, it’s going to
pull in those patients. A rule like this, hemoglobin A1c greater than 9, is something that you
might want to use a select not to get a source of patients but select those patients that might
have a hemoglobin A1c greater than 9. One of the things we did with the finding rule is going
with the cohort we created our Nexus list from a finding rule, so we had a finding rule called a
Nexus cohort, we had what we created was a term called Nexus cohort, inside that term we had
location list. So location list had those stop codes. So when I added that finding rule into my
4
Session 209
rule set in the extract, that was my add patient. I’m adding patients from that finding rule
because we need to get that base list of patients.
Now the patient list rule uses an existing patient list. A patient list could come from more than
one source. I could run a clinical reminder, save it as a patient list, and then pull that patient list
into my patient list rule. Or if I run an extract I could program my extract that every time you
run a specific rule set to save that output in the patient list, and then use that patient list in the
patient list rule. And what we did with our extract in our rule set is we have one base rule set
which was create our Nexus cohort, that Nexus cohort was saved as a patient list. When you go
to your subsequent rules in your extract, for example your composite rules, and we did create
composite rules, for example our diabetic composite rule, it needs a list of patients to run the
reminders from. And so what we did was we used that patient list that was generated from the
first rule set because if you kept on rerunning that finding rule over and over and over again it
becomes labor intensive, it’s going to take forever to run your extract, so why not use a list
that’s already created by your Nexus and your rule sets. So that’s what we did and then that
became our patient list.
The next rule is reminder rule. We didn’t use reminder rules but because it’s in it and it is a rule
I feel obligated to talk about it. A reminder rule uses, when you create a reminder rule it uses a
field name of L and the usage field must be L. A reminder rule is you create a clinical reminder
and it has a cohort, it has a kind of cohort that you might not be able to get using a finding rule
or a patient list rule. And it could be, for example, on the PowerPoint there’s a reminder
definition that has a rule of age or finding item one and finding item two, which is kind of
something you can’t really create using other rules. So reminder rule uses that clinical reminder
to create its cohort of patients.
Section 4: Rule Sets
So rule sets. The extract creates a patient list by using a rule set. Each rule created will contain
one of the three rules, a finding rule, a reminder rule, or a patient list rule. As discussed before
each of the above rule that is used will contain a certain type of patient cohort. The rule
functionality will allow once the rule is added to behave by using the following operations,
5
Session 209
which is listed, which is add patients, which acts like a Boolean or, it will add your patient to the
rule set as a list. You have another function in it called select patient, which is a Boolean and,
because now that you have your patients I need to get a certain list of criteria of patients as the
select patients, so that will be your and functionality, or remove patients, for example if you
want to remove deceased patients from a extract it’s an and remove and it’s also going to work
as an add not because we don’t want to include those patients in our extract. And finally an
insert finding, we don’t use inserts in an extract, but insert is useful for other reasons. For
example, if you’re going to use a patient list and going back on hemoglobin A1c example, and
you want to insert certain findings in your demographic report, you could insert certain values
from that rule that comes out of your demographic reports, so if you want to see a list of
patients and you want to see their SSN, their hemoglobin A1c value, their last A1c and the date,
the insert finding actually helps you out and adds that information to your demographic report.
So I find that useful, and I used that a couple of times when I run different reports.
So here’s a screen shot and it’s kind of small, hard to see, but it shows basically what the
sequence is in the extract, and it shows the different sequences and it shows that I’m adding
patients from my Nexus, it shows that I’m selecting patients who are veterans, selecting patients
who had an anchor visit, and it shows I’m removing those patients that are considered
deceased.
So the next step is since there was a whole huge learning curve on extract itself was reminder
building, and we had to build reminders like a team, like I mentioned before, and the reminders
had to be standard, so the elements that we needed to standardize were reminder name, the
print name, we needed to have a standard reminder name and a standard print name. In the
long run the information gets extracted from VistA and goes to a SQL database. In order for it to
have that link from VistA into a SQL database it needs to have a common print name that helps
that each reminder that each facility builds has a common print name. Another thing is baseline
frequency. Every reminder that’s created, if it’s the same reminder, should act the same way.
So we needed to have that common baseline frequency in each of the reminders and also
reminder terms. Because of the whole issue of having different names for different labs,
hemoglobin A1c could be the name for one facility but could be called something else at a
different facility for the same lab test. Health factors is a big problem. One facility could create
6
Session 209
a health factor called life expectancy less than six months all spelled out, another facility might
call it life expectancy less than sign number six months. So not everyone had the same factors,
the same labs, and so forth, so the best way to standardize those finding items was to create a
reminder term, keep the reminder term name standard, and then use that in our clinical
reminders for extracts.
The next step was to understand the performance measures. We need to understand what was
EPRP pull list, and like I mentioned before in the EPRP pull list it’s looking at patients that were
seen in a Nexus clinic that had an anchor visit that were are VA veterans and who were alive
during the reporting period. We also had to understand what were the quality measures, so the
next step was to pull up the technical manual, understand what were the composite quality
measures, read through the performance measures and go through each of the definitions of
each measure. Next one is the technical manual, which I just mentioned, which was what we
did, we needed to familiarize ourselves with what was in the technical manual.
Section 5: EPRP
So give a little bit of history of EPRP. EPRP was established around 1992, which was a
consultation service, coming from WVMI, which is West Virginia Medical Institution for VHA
since ’92. Now EPRP has consultants that may come physically to a medical center, or they can
work remotely from home. And what they do, they do an audit of medical records. Most of you
guys are probably familiar with EPRP reviewers, and so they’ll either work from home or onsite,
but then they’ll look at our records, it’s a random pull of patients, and with guidelines that they
have they’ll go in and then at the same day they’ll have the results available, they’ll do an exit
report with the facility, they’ll meet with the facility medical center director or they might meet
with the director and the chief of quality assurance, and give an exit report of their findings. So
it was like a near real-time interval of data and then that data gets rolled up to office of quality
and performance shortly after.
Screen shot of the technical manual, this is what we used, kind of like our clinical reminders
bible, to get information on what were the performance measures.
7
Session 209
This is a screen shot of the table of contents, or part of the table of contents of the technical
manual, and it breaks it down by composite. So we see here in the screen shot we have our
behavioral health composite, or our mental health composite, that compromises our depression
screen, PTSD screen, alcohol screening. Below that we have our diabetic composite which has
our blood pressure measure for diabetes, hemoglobin A1c, LDLs, retinal imaging, and renal
testing. And below we have our cardiovascular and our prevention composite.
And I mentioned before what are they looking for? They’re looking patients that had their
Nexus and anchor visit, and also VA veterans only. And that’s actually established in one of the
CAC technical manuals. The technical manual is like two sections, there’s one section dedicated
to performance measure, there’s one section that’s dedicated to the catnum and the cohort of
what they’re looking for. So for example they have the overall for the CGPI, which is your
outpatient measures. They’re looking for the Nexus visit and the anchor visits. And then there’s
also other catnums, like your diabetic patients. Not only are we looking for patients that had a
Nexus but also looking for patients that had a code of diabetes within their reporting period,
and also a previous code of diabetes. So they really want to make sure they’re picking patients
that are established, that are being followed within the VA, and that they have more than one
coding of that comorbidity.
So this is a screen shot from the catnum manual what compromises Nexus clinics. So we’ll see
here, like I mentioned before, we have our 323 primary stop code, they’re looking at certain
sub-specialty stop codes, and they’re also looking at multiple mental health 500 series stop
codes as well.
Now we need to establish a cohort. Now the reason why I’m talking about establishing a cohort
is going back to what I said about the Nexus, a lot of stuff could be established using a rule set.
A rule set is a great way to create a cohort. We could do the same thing when we work with our
composite measures. A lot of the composite measures, the core could be established within the
rule set, which leaves little system time dedicated to establishing a cohort in each individual
reminder that you add to the extract. So when you’re creating a rule set one thing you have to
think about is what can I establish as coming across the board within that composite? So using a
8
Session 209
diabetic composite as an example, what can we use to create something that’s common, what
can we do to speed up running the extract?
Well one thing not all cohorts are the same. Some composites might have some similar cohorts.
For example, diabetes, patients 18-75, who have a code of diabetes in the medical record and
also have certain exclusions like patients in hospice care or patients with life expectancy less
than six months. And if you scroll down to each of the individual diabetic measures you kind of
notice it has the same cohort in each single one of them. And cardiovascular it has something
similar to that, and mental health also has a similar base cohort. Other composites like your
prevention measures have different sets of measures within that composite. We have our
cancer screening measures, we have our immunization measures, and we also have the obesity
screening. All different kinds of measures, all different age range, all different criteria, even
some of them are sex specific. So for those we had no choice but to leave less in the cohort, I
mean in the rule set, and establish more in each individual clinical reminders.
So here’s a screen shot of a piece of the technical manual in reference to hemoglobin A1c
annually. So we see here what arrows are pointing to our numerator and denominator. The
denominator is telling us exactly what they’re looking for, which is what I mentioned, diabetic
patients 18-75. And if you scroll down technical manual you see the same thing over and over
again for each of the diabetic measures. The numerator will tell us not only the cohort but what
the measure is specifically looking for, which we could call a resolution. And what are you
looking for, patients that meet a cohort criteria but had an annual hemoglobin A1c performed.
Each measure has their own specific resolution, which makes each measure unique. And they’re
set up in each individual measure, which means each of our individual clinical reminders is going
to have that resolution in it. So basically the concept is leave less of a cohort criteria in the
reminder establish as much as you can your rule set but leave the resolution within the
reminder.
Section 6: Performance Measure
9
Session 209
So going back to our hemoglobin A1c snapshot, looking at the denominator you could say you
know what, I could use this in my rule set since it’s common across each of the measures, the
numerator will be which will compromise the resolution for the clinical reminder.
So let’s just keep working with the hemoglobin A1c performance measure. So the first step was
identify the core resolution. Patients with diagnosis of diabetes 18-75, and we did this for every
single clinical reminder that we used in our extract, and the resolution was an annual
hemoglobin A1c. Can the cohort be established in rule set? Yes, because it’s the same cohort
across the composite.
Next step is create the composite rule set using those elements. So next step was to create a
rule set, now going by what you guys know about extracts we ended up creating finding rules
established those criteria of age range to have those patients within the age range to also make
sure that we removed those patients that had those exclusions that were set up in each of the
performance measures, and of course also we wanted to add the diabetic taxonomy as part of
the finding rule. Now what I was mentioning earlier about using that patient list of Nexus, it
needs to have an add patient option, or we need to get a list of patients that diabetic rule set is
going to run, so that patient list came from that Nexus rule set that’s the first rule set being run
in your extract. Diabetic composite for our extract was the second sequence in our extract. The
first sequence was that Nexus, that Nexus created our patient list, so had that patient list, it’s
available in the patient list menu. When the diabetic rule set kicks off it’s going to grab that
Nexus patient list as an add patient and then uses the following sequences in the rule set, those
are the rules that I mentioned, to select and remove those patients.
Next step was to create the reminder, or reminders. And to create a reminder we identify the
finding items. Finding items were what’s the resolution for the reminder. For example, the
hemoglobin A1c was the lab test hemoglobin A1c. Now since, and it was kind of like a lesson
learned, since we already established an age range in the rule set it’s best to omit it from the
clinical reminder because that’s considered double work, and then you’re actually making the
extract go over the same thing it already established as a rule set, so we omitted the age range
from the clinical reminder, so we kept the cohort nice and simple and we just had the resolution
added and some additional finding items in it as well.
10
Session 209
And we did the same with each individual clinical reminder, so we completed diabetic cohort,
we did a hemoglobin A1c annual, hemoglobin A1c less than 9, the blood pressure. Same thing
for our cardiovascular, we established a common rule set, we created everything we could in
that rule set and then we created our individual reminders and so forth. The one that was kind
of tricky was our prevention one. Like I mentioned earlier, since that composite is diverse we
had to create as little as possible in the rule set. Something that was common across each one
of them. And then leave everything as much as we could in the reminder itself that was specific
to each of the performance measures. Once we created each of the rule sets that mirrored each
of the composites what we did was move next was to create our extract, and the rule sets were
used to establish a set cohort, the first rule set in our extract was our Nexus rule set. And then
the subsequent rules contained individual composites for those rule sets, diabetes,
hypertension prevention, and then those reminders were added to those rule sets. So for
example, in our diabetic rule set we had our diabetic clinical reminders added to them. One
neat thing that we did add in each of the rule sets was the counting rule, if you further drill
down to stop code and location level, because this is facility run and you’re running your Nexus,
it’s based off your facility, you’re going to get facility results. But to make the data clearer, to
get more of a drill down by not only facility but stop code and also by location level, what we did
was we created a counting rule, and a counting rule breaks down the results further by drilling
down what specifics you added to that counting rule. So for example, we create a counting rule
that included stop codes. We also created a counting rule that had included location in it. So
you have your facility breakdown and a counting rule breaks the results down even further. I
have screen shots I’ll show you later on what I mean. Breaks it down even further that says well
this is your facility raw numbers but now we have your counting rule showing how your stop
codes look like, with the results, and your providers’ locations with the results.
So here is a screen shot of the name of the extract, which is Bronx facility, or the name of the
facility, monthly performance measure, each site had their name on it. Compliance and finding
totals, that’s what we used. It’s a monthly run and then the time period was within the month,
and we tried to mimic basically the month that was a reporting period for EPRP, so we basically
ran the report month by month.
11
Session 209
And then that’s our first rule set, which is the Nexus cohort, and since we started working on
this last year it’s called FY09, and then like you’ve seen in previous screen shots those are the
rule sets that were established, or the first rule set that was established in our extract.
And then going further within our rule set we have our VA finding rule date of death in it and
our finding rule that pulling patients that were veterans.
Now this pulls up one of the rule sets that we used after running our Nexus, our cardiovascular
rule set. What you see the arrow pointing to is actually the name of that patient list that’s going
to create Mnn, YYY, is actually code that you establish when you create your rule set, and what
it does is it automatically stuffs in the name of the month and a year, numeric value of the
month and the year that was used to run that extract, and what it does it dumps all the patient
lists into that, the names of those patients in that patient list, so for example for the Nexus we
created a Nexus cohort Mnn, YYY when the Nexus is done running it has the name of those
patients in that list so when this next rule set starts running and it starts grabbing from that
patient list it will have those names available to use.
And then this is the counting rule I was talking about. It has a counting group, and the group
goes into the rule, and the name of the group is Bronx primary care clinic stops, and it has the
list of the stop codes so you see that these are names of stop codes. These are actually names
of reminder terms because the location lists are added in the terms because like finding rules we
had to enter terms into our counting group. And that’s identified in each individual rule set.
So the process basically just to branch it out is the name of the extract, the Nexus cohort rule
set, and then finally after the Nexus cohort rule set the next subsequent rule sets are the
composite rule sets, the name of the clinical reminders that fall within each of the composite
rule sets, and then the counting rules that go with the rule sets.
Section 7: Results
So what were the results? Well data is viewable by three different levels. So you have your
facility breakdown, you have your stop code level breakdown; you have your location level
12
Session 209
breakdown. This information gets translated over to a SQL database, so a programmer goes in,
pulls the information out, puts it in SQL database, and I have screen shots later on to show you
that you could choose by facility level data, you can choose by stop code level data, you can also
choose by location level data, so you can just play around, do the selection, and pull up the data
you want to look at. And like Excel in the screen shot I’m going to show you, there’s conditional
formatting so depending on what were the results if it meets the benchmark or not those
conditional formats will come up red if it’s below benchmark, those conditional formats will
come up green if it hits or exceeds the benchmark.
So this is the output. So the extract is run, we’re done, we ran it for the month, we have an
output, what I show you right now is the cardiovascular output and on top where you see the
4,519, that’s our facility number. This is the main number you get by running it off the Nexus.
So we have total number, we have the applicable, we have not applicable, due and not due,
then because we have that counting group established, as you can see here it breaks it down
even further. So out of the 4,519 how many of the patients end up being primary care patients,
and we get that number. How many of them fall in cardiology, diabetes, so we have that
breakdown. This translates into the SQL database where you get those same numbers and
results, but in a GUI end.
So what was the aftermath? Alright, so just because you run it, it doesn’t mean it’s right. I
mean you have data, but you always need to validate data. So one way we kind of thought of,
which was a very creative process to validate it was to use the patient list that was available for
each of the rule sets and run them via reminder reports.
So now this is the patient list option that I have in the Bronx. So the arrow is pointing to our
cancer screening measure patient list, so going back to that old screen shot I was seeing about
the cardiovascular list where it has the name of the rule set with Mnn, YYY, once that’s done
running this is what it looks like. It’s the name that was established in the extract, M1, month
one, 2010. If you create an additional patient list using the same reporting period it will have a
slash and it will give it a digit after that. So if I ran like five lists I have the main list and then the
extra lists will have like a one, two, three, four, to differentiate it from the main list. So I do
have a patient list, this patient list shows me all the patients that met that rule set cohort I
13
Session 209
established from my cancer screening rule set. Same thing for my diabetics. What’s in the
patient list are all the patients that met the diabetic cohort of Nexus with the diagnosis diabetes
with the age range and with the exclusions.
Next step is what we did was go to reminder reports and we’re selecting that we’re running off
a patient list, and when we get to the option of choosing the patient list we just added the name
of that patient list that we’re going to validate the reminders from. So for example I’m calling
forth the cancer screening patient list.
When it asks for a clinical reminder we’re going to add that clinical reminder we’re going to
validate, and also we have to make sure the report is detailed because we’re going to use the
report, we need to see the name of those patients. So what we did is we always made sure
there’s a detailed report and we pull in the name of that clinical reminder that we’re going to
validate.
And the output is those lists of patient names and what we’ve been doing as CACs and also we
have a collaborating physician on board, see where each site has a collaborating physician
where we get the output we get the names and then we go and do a reminder test on those
patients, find out whatever triggered them as due now, if it matches what’s in the medical
record, if it’s something that is beyond us we give it to our collaborating physician, they’ll look at
it, some sites opted to run reports off specific clinics with those collaborating physicians, the
clinics they have, and then get those patients that belong to those clinics because they know
their patients and say these are the patients that came up due for let’s say colorectal cancer
screen, can you check on those patients, check the medical record to make sure it’s true.
Sometimes, and we did find this, where a clinical reminder became due now in our extract but
then we checked the medical record and it said resolved, and it maybe didn’t recognize FOBT
orders or it didn’t recognize a code for a colonoscopy. So then we had to go back and then
change the reminder and add the code, and so it was kind of like a little back and forth work, but
it helped out because we find out what was the mistake, we stopped it there, we fixed it, so
next time it ran it actually pulled in valuable data. And I think I remember, I think it was one
site, I think it was for a diabetic retinal one where the numbers were way off, I think the site
came up like 45% compliant or something and it wasn’t because the site was doing bad or
patients were falling through the cracks or whatever the case may be, it was because there was
14
Session 209
maybe an exam that was missing from the clinical reminder, maybe a set of CPT codes that were
missing or ICD-9 codes that were missing, so by going through the charts we were able to focus
what was the problem, hone in on it, add it to the reminder, rerun the report, and then when
the site got its results we went from 50% to maybe like 85%, 90% compliant. So validation does
help.
Section 8: Automatic Runs
Automatic runs. In the beginning we were running them monthly and we had to remember
when to run them and it was dependent on us running each extract because the next step was
having a programmer go in, pull the data out, and put it in SQL database, if someone forgot to
run it we’ll be missing data. So there was a bright idea about having it set up automatically
where IRM would go in through I think Fileman and set up an automatic run and so it will be
queued to run on a certain date every month. So this is just a quick view of a screen shot of the
field that a programmer will go in and it’s a three-step process for the set-up, and they had to fill
in what was the next reporting period in a designated field, and it’s shown below in the screen
shot.
And then had to create an option to have the extract end it and then through Taskman it will run
every month, so it kind of was like a queued report, it was queued to run every month and then
every time it was run it will have the results available so when a programmer went in it grabbed
the results and pulled them into a SQL database.
So what was the aftermath continued was that those results were put into a SQL database.
This is kind of a screen shot of what it looks like and as you can see here you could hit by facility,
by stop code, or by location level. So basically what we did in the extract and what we defined
in the extract using rules in the counting groups, we were able to have those options available in
the GUI.
And here’s a snapshot and it kind of reminds you of an Excel spreadsheet but it’s really not an
Excel spreadsheet I don’t think, and it breaks it down by your composites are what you see
bolded here is our composites, below each of the name of the composite is the individual
15
Session 209
measures, and next to the measures a breakdown of the month and their compliance or their
performance measure rate for those clinical reminders. Like I mentioned before, what we have
in the red are those that didn’t hit benchmark and what we have in the green either met or
exceeded benchmark. So for if I was the manager, if I was the chief of primary care, if I wanted
to find out how my practice was doing I would look at this information because this information
is transparent, it tells me what I want to look for, if I want to drill it down by provider and as a
chief I have the option to drill it down by provider and I find out well Dr. So-and-so is not
performing up to speed with the rest of his clinicians, let me contact someone from informatics
office to run a detailed report for me. And so it’s great to use in a staff meeting and that’s the
idea is to have that level of information available to management and to practice chiefs.
So the outcome is to raise provider awareness. I mean this is something that they’ve been
asking for a long time, that they’ve been wanting to use. This is something that pay for
performance, if you want something, you have an element for paid for performance, this is a
great and accurate tool, more accurate than using a reminder report. It’s close to
representation in EPRP, I mean in the Bronx we’ve been using clinical reminder reports and we
actually have a mechanism where the clinical reminder reports gets posted on the Intranet and
then the physician has access or the chief of the clinic has access to view that information, but
like I mentioned before, it is not 100% accurate because we’re pulling not only established
patients, but new patients, non-vets, we’re missing those patients that might be deceased at the
current time but were alive during reporting period. So this is more of a pure way of looking at
things when we’re trying to compare our data with EPRP. It’s going to be really robust because
we’re going to break down the data by facility, stop code provider level, hospital locations. Now
I’m stressing hospital locations because as of yet we cannot pull in PCMM panel data. So to
come close to PCMM panel data we’re pulling in by location, and by doing so we actually had
the CACs find out how many clinics does a provider use, they might have a clinic in the main
hospital, they might have clinics in a satellite hospitals, they might have resident clinics that they
see residents, and what we do is we created individual location lists that house those individual
locations, mix them all together, and so when we have our output and using our counting rule
we could find out well Dr. Smith performance in so-and-so, but then that’s because we have all
this data for his main hospital clinic, his satellite clinic, and his resident supervision clinic. In the
future, well I hope in the future we’ll have the option to run it by panel, and I think this is really
16
Session 209
important because we are going towards medical home. Medical home is panel and teamdriven and I mean this is a great marriage with medical home and one of the focus is to have this
information available by panel. We’re close but we’re not there yet, but hopefully in some
future patch or release we’ll be able to see this.
Section 9: Monitoring CHF
The next segment is monitoring CHF using extract and once I got my feet wet with extracts and I
kind of got familiar with how to use an extract, I was like wow, this is pretty cool, I can use it for
other things, I can create patient lists and monitor different measures. One of the measures
that, and probably not just my facility but other facilities we’re struggling with was compliance
with CHF discharge instructions. It’s like the discharge instructions are available in the discharge
summary, it’s there, physicians can click on it or whatever, but we’re just missing cases. We
were getting hits on it, not every patient with CHF were receiving discharge instructions, so I
kind of thought of a clever way to actually monitoring and using patient lists and extract.
So the idea was to create a new process that monitors patients that are admitted with CHF and
then make sure that they receive the appropriate CHF discharge instructions. And the goal was
to not to just improve the performance measure but to also enhance patient care, to make sure
those patients that get admitted with CHF get their appropriate discharge instructions.
So here’s a screen shot of what the CHF performance measure looks like and we’re looking at
patients who are 18 years of age and older, who of course are inpatients, and then patients are
discharged and one of the biggest thing is we’re looking at patients that are discharged with
their CHF discharge instructions.
Here’s your exclusions. There’s a lot of exclusions, there’s a lot of things I needed to look at. Of
course patients that didn’t meet the age range were automatically excluded from getting
discharge instructions. Patients that were not discharged at home, maybe patients are
transferred to nursing home, were not included in the mix. Patients who are deceased
obviously will not get discharge instructions. And so forth. So those are different things I
needed to hone in on when I was creating my report.
17
Session 209
So the idea was to use the reminder extract functionality to create my cohort list, and so I had to
tailor the reminder to meet, to kind of like give me the desired results that I wanted, which were
patients who received CHF discharge instructions. And with my reminder the cohort was
patients with CHF. Now I mentioned before in the rule set well why can’t I just add that to the
rule set, well I kind of did something cool with the rule set, I didn’t kind of want to add too much
to it, and since this is simple reminder it wasn’t too overwhelming like the other extract, I left it
alone, I left my CHF diagnosis as part of cohort in the clinical reminder. The resolution I created
was patients with CHF discharge instructions. That could be a note title, if you have a note title
that’s specific to CHF discharge instructions you could use that note title, which is I think the VA
progress note computer finding where you’re able to add that note title in it. We have, for
example, a health factor that if you give the patient discharge instructions it adds a health factor
to the medical record, so the thing I used was the health factor. Frequency, I chose once in a
lifetime because it’s actually dependent on the admission itself and if the patient have that
diagnoses. So I didn’t want to have it due now every year, every so often, since maybe it’s a
one-time admission I didn’t want it to come up again in the medical record if patient wasn’t
admitted again.
So here is a snapshot of the clinical reminder. I’m looking for patients that meet the criteria of
CHF and the fact that they received CHF discharge instructions, that’s my resolution.
So the next step was to create a finding rule for each inpatient unit that contained a reminder
term which I called discharge for a date range and then gave it the name of the unit . The reason
why I did one for each unit is because after extract report is run the result will be broken down
by each separate unit versus one combined result. So we have I think three med/surg locations,
one ICU like main area, and then the inpatient mental health ward. So if I run a report and I go
to one of the med/surg units I say well this is your result, I need a result specific to that unit.
The only way to do that was to create a output that was related to each specific med/surg areas
or different areas in the hospital. So the finding rule utilize reminder term that will look at
specific unit and capture specific patient movement. So each of the finding rule, and this is
where I want to talk about which is something cool that I did, was utilize a computer finding
called VA discharge for a date range. It worked hand in hand when you run the extract because
18
Session 209
if I run the extract for a month it’s going to look at all the discharges that happened for that
month. And I ask myself well why am I focusing on discharges? Because it’s kind of like I want
to have a report that if I give it to the unit and say well these are the patients that you had
admitted, these are patients that you missed giving the CHF discharge instructions. Kind of like
trying to mimic what EPRP does. EPRP is looking at medical record, they’re looking at the guy
that was discharged and wasn’t given the discharge instructions. So that’s the approach I took
with this.
So this is actually a snapshot of one of the reminder terms, and the finding item is the VA
discharge for a date range, and I added there multiple times because you had to specify what
kind of discharge you want when you use that computed finding. And if you go to condition we
see regular discharge, OPT-NSC, and OPT-SC, and they all kind of mean regular type of discharge.
This is the type of discharge that EPRP is looking at. One is transfers and other types of
discharges so we make sure we’re only honing in on one specific discharge. And it needs to
know what location I’m using, so when you look at a computed finding parameter you see 7B
med/surg, 7B is the name of one of our med/surg locations because it needs to know that it’s
looking at discharges in the 7B location.
So the VA discharge for date range computer finding contains two useful parameters. So not all
movements are required for CHF instructions, transfer to nursing home is not considered a
discharge that needs a CHF discharge instructions, that’s not included, so what it does is that if it
finds a patient let’s say that was transferred to nursing home it’s not going to pull that in as a
discharge that needed CHF discharge instructions. And then what happens when you run the
list in the extract, when it goes to that finding rule my cohort has been compromised those
patients that met that discharge criteria that I set up in that term.
So next step was to create a rule set including the finding rule that was created. Since each
finding rule had a specific location and since I wanted the output to be specific to those
locations, I had to create a rule set also for each of the units. And then each individual rule sets
were added to the extract, so I had like my 7B rule set, I had my 7C rule set, which is my other
med/surg area, my ICU, and also the mental health ward added to it.
19
Session 209
So here’s just a snapshot of what it looks like and this is a rule set discharge instruction for 7B
and this first sequence is to add the patients, so I’m adding patients using the finding rule of 7B
med/surg discharges. That finding rule has that reminder term that has that has that computed
finding that VA discharge for date range, so when that rule set starts processing for its finding
rule it’s going to pull in only those patients that met that criteria and that VA discharge for data
range computed finding, which is those regular discharges only because that’s what EPRP is
focusing on.
So after adding each individual rule set the next step is now that I have my rule set for each of
the areas I need to create my extract. Once the extract is created then I add my rule set in
them, each rule set will have that CHF clinical reminder added to it. So basically step by step is I
have my list of patients for that unit, and then it’s going to run it by the reminder and find out
well out of all those patients that had regular discharges how many of them have CHF discharge
instructions? And then run it for a reporting period which is most likely the prior month.
So here’s a screen shot of the output. So what we see here for item one we have discharge for
date range 7B, we have total 20 patients that had regular discharges, so we could have had
more patients but those that didn’t have regular discharges were not pulled in in part of total.
And remember how I established a reminder as a cohort of patient CCHF, well we have six
patients with CHF, so out of 20 patients that had regular discharges, 6 of them had discharges
from the unit with CHF.
So provider feedback was that how far do I look back for CHF? Well since I’m not a clinician and
you know I just create a reminder, and the only thing I did was add the taxonomy of CHF, when I
ran the report what happened was that I pulled in patients that had CHF that I noticed going
back to like 1999. And then it wasn’t recoded again after that. If the patient happens to be in
the unit it means that a patient was seen after 1999 again either as an outpatient or a previous
admission. Chances are the patient wasn’t coded with CHF again since for over ten years he or
she might not have CHF. When I gave the information to the clinicians at first they did a spot
check, found this patient and said well you know what, I looked up the echo, I looked at the
problem list, I don’t see CHF, I read the old progress note, it must have been something that was
erroneously entered, I think this report would be more useful if you restricted your view to
20
Session 209
maybe look back one or two years. So that was one of the feedback I got was maybe to just look
back two years for the diagnosis. So when I ran the report again I had it look back for two years
for CHF. It also raised staff awareness. A lot of staff in the unit probably thought well if it’s not
the reason why the patient was admitted he might not need CHF discharge instructions. The
patient might have a history of CHF but was admitted for something else. CHF, everyone knows
CHF doesn’t go away, if it’s not the primary reason for admission it’s definitely a secondary
diagnosis somewhere, and you still need to give the patients discharge instructions. So it raised
awareness so when I gave out the list of names to the patient, to the unit, it showed that well
okay, I didn’t give them the discharge instructions but I also gave them information about where
that diagnosis came from. It came from some encounter somewhere, it was maybe during echo
report when they signed off the report and they had the code in it, it was probably somewhere
in a problem list because it probably could be in a problem list but it wasn’t picked up by the
staff nurse or the clinician. So it did raise awareness. Like the charge nurse in the unit, some of
them actually like spoke with the staff and said we need to be more cognizant and aware that
we need to look at not only the problem list but echo reports and certain encounters to find out
even if it’s not a primary diagnosis we still need to give those patients CHF. And actually it does
help improve performance measures because those records do get pulled by EPRP and they’re
looking at that stuff.
And I’m going to hand it off to Philip.
Section 10: Reminder Extract Outputs
Okay. I get tired of talking after talking all morning and talking in the afternoon. So pardon me
if my throat gets drier than the desert.
Okay, reminder extract outputs. I have the data. What now? We’re done, right? No.
Leadership wants nice and pretty outputs so they can click on a button, pull up right on the
desktop – what I’m going to do is take the opportunity of going over some ways that you can go
and get that information and present it in various fashions. But in order to do that let me first
kind of review for you where is this information stored.
Well when you execute the extracts the data points that are extracted are filed in file 810.3,
which is titled reminder extract summary. And within that reminder extract summary there are
21
Session 209
several fields, but the three key fields that I primarily use, and most sites would probably
primarily use, is the extract name, what was the name of the full extract, not the reminder, the
beginning date, and the ending date. So what time frame that that extract ran. Within 810.3,
the reminder extract summary, there is a multiple called compliance totals.
As Ray mentioned that when you build your extract you can set it up to extract out the
compliance totals and this will get filed into the multiple, and these are some key fields that
you’ll most likely want to utilize. Obviously the reminder field, which is the name, and the
specific clinical reminder that the extract was run on. What patient list that extract created
from that reminder. Total patients evaluated, applicable, not applicable, due, not due. Similar
fields that we’re used to in regular reminder reports. Now, to further complicate things, there is
a multiple of the compliance totals multiple of the reminder extract field called finding totals,
so we’re now getting deeper, deeper, deeper. Within the finding totals those are going to be
the specific data elements that we had defined within the reminder extract for the compliance
totals of the reminder extract. So again we’re going to get to the term level. Within the term
it’s going to show us the total count of patients, applicable count, not applicable, due, not due.
Seems similar in nature to the compliance totals, they are two totally distinct and different sets
of data points.
Now extracting the extracted data. Many of us have many ways of going about utilizing this
information. In my previous role, I just joined Clin 2 back in May, but at my previous site in
Maryland besides doing all the reminders and the extracts and pretty much everything else, I
managed the SQL database so I was very fortunate to have my own SQL database to run. I know
every site is not that lucky, but what I’m hearing and knowing is that a lot of sites and a lot of
VISNs are moving towards their own personal data warehouses. How many people have access
to a local data warehouse or VISN data warehouse? Not many, okay. How about the ones that
don’t, do you have local personal SQL servers at your sites? A couple. Well for those that don’t
have the data warehouses I’m sure it’s going to be coming to a site soon. My presentation is
going to show a couple of aspects for the data warehouse and SQL server. How many people
have Access and Excel at their desktop? Every hand, every hand’s got to go up. Okay. That’s
how I can do this presentation because I know there’s a way that every site can somehow
22
Session 209
emulate a piece of what I’m going to do today. Yes, Access and Excel is a very valuable tool.
Now how do you get that information out? There are numerous ways. I was very fortunate I
was able to use a class, a third party software called MDE, which is a Mumps data extractor.
There are other Mumps data extractor tools out there. A lot of sites have the Cache,
intersystem Cache, and there’s a process within there that you can extract the information.
Some sites have WinSQL and of course the rusty, trusty Fileman.
I want to take a step back and revisit a comment that Ray had made that when you create the
patient list within an extract there is the patient list management menu option that you can go
out and do an output and from within that patient list option you have several fields that you
can include into that output. So I wanted to throw up here a listing of all the various data points
that you can include in your patient list output. Basic demographic information, appointment
information, eligibility information, if the patient was an inpatient obviously that information is
there, and then of course if you had some finding item included insert into the extract those
would also be extracted as well. And one of the great features with any VistA output is that you
can have it delimited,
so once you have it delimited get into a text file, well that just has opened up several other
doors for several other packages that we can now try to utilize this information and manipulate
it into a useful form.
Taking Microsoft Access for an example, Microsoft Access allows you a couple of different ways
of getting to this information. Say I have that text file. Microsoft Access is very good at
importing a text file into a table. There’s a very good wizard that will move that in for you, it will
help you clean it up a little bit. So having that Fileman output either from the patient list or just
from direct Fileman based what I talked earlier, we can create tables within Access that are
standalone. For those sites that have data warehouses or on SQL servers that you can actually
point to the VistA data or have automatic downloads of the VistA data onto your SQL tables, you
can do the link table manager, link the tables to those data points same way. You’ll be creating
the same tables to be able to create various types of queries upon.
The previous screen I was just showing you is an example of a single table, this is a parent level
table of the actual extract summary, so this is the first level where I was talking about which lists
23
Session 209
the extract name, it has the month and the year as Ray had mentioned, that is put into the
extract, the beginning date and time for that extract period.
And then going down a level to the compliance totals, this is where I create separate table that
will list the individual clinical reminder and then of course similar data points that we are used
to in the reminder reports, applicable and not applicable, due and not due.
A word of note from populating tables from Fileman. It’s pretty simple to create a print
template to extract the information from Fileman for this purpose from the extract table.
However, I do want to make known that there is some clean-up that has to be done. As I
mentioned, yes, once you have a text file, moving it over to Access is pretty simple. Fileman on
multiple fields has a little quirk in that, for example, on the screen you can see that for the
reminder extract summary it has an internal number, so for example, the top level, I think it’s
465 I think is the one, when I go to do my Fileman output for the compliance totals Fileman just
does one 465 and then does all the other line items. So I have to actually clean up and just, and
I usually take an Excel spreadsheet and just click on the 465, drag it down so it populates 465 all
the way down. The reason why I say that is that when you put into Access obviously you need
cross points to be able to link the two different tables together so I’m going to look at which
extract I’m going to look at, I’m going to look at which reminder I’m going to look at, so to have
those fields you have to – so there is some clean-up that you have to do in Fileman. I don’t want
to lead you down the wrong path to say that Fileman is going to be an easy process but it can be
done.
Here I want to demonstrate or just display for you what a SQL table looks like. I have an
automatic download each night from the reminder extract field that populates separate tables,
one for the extract summary, one for the compliance totals, one for the finding totals, and what
I did here was just create a very quick view of all three of those separate tables into one, so it’s
going to have the summary, it’s going to have the reminder, it’s going to have the various counts
for both compliance finding and the extract summary.
Section 11: Microsoft Excel
24
Session 209
Microsoft Excel. Again, Excel is very good at importing text files or having them linked to tables.
So in either case you can either create your own, import your own text file, or use a linked table.
What’s nice about the linked tables obviously is as the data points get updated in your parent
table in your data warehouse, Excel is automatically updated every time, refreshed when you
open up. But what I like about Excel is it really gives the users, the end users who are going to
be reviewing these reports, the ability to do some simple filters and queries.
Within Excel obviously they can go in real quickly and click on which date range that they want.
I’m going to give them all the data points. They can have all the data they want, I don’t care
what date range that they want, I’ll let them mess around with that. So with a quick filter they
can click on the column header, there’s drop-down arrows for each of the column headers, and
they can pick and choose. They can pick it by month, they can pick it by year, however they
want to combine it. Likewise for the clinic locations. Some of the extracts I had included the
clinic area so that they could drill down a little bit further so that they could filter it down to
their own personal clinic or the clinic team manager could do it for their particular team group,
or the managers for the whole primary care could look as a whole. So again, any of these. Even
the clinical reminder, they could filter it down to a specific reminder if they wanted to.
However, over time I was kind of asked to make it a little bit more simpler. Again, leadership
wanted to be able to just click a single button. Well obviously we all know there’s no way to
click a single button to get everything that you want, so the closest thing that I did, I’m going to
jump back to Microsoft Access.
What I did was within Access you can create forms, so with the forms then create little buttons
that execute queries in the background that will provide them additional information. Some of
these buttons will actually ask them for additional prompts, like what date range do they want,
what specific reminder do they want. Some of these will go to a raw table that they can get the
raw information that they can capture and move over to Excel if they want to do something
else, or some of it is just a plain old, it’s a form fitting report and I’ll show you.
This is a little bit more of a close-up of some of the various options that we gave some of the
providers is that we can have them select by clinic, by date, clinic and date, we can look at
25
Session 209
overall PCC points, we can look at our all outpatient, our Nexus clinics for specific clinicians,
specific reminders, we did monthly totals and we also did daily totals.
So for example if one of the management or leadership chose the overall PCC monthly review
that would link directly to just a form fit report, so the query wound run in the background, it
gives them a list for that timeframe, listing of all the reminders that they specifically wanted to
address, and it just showed them for all primary care, and we have eight facilities across the
whole state of Maryland, so this would have been for the whole state of Maryland and not just
one location, but however we built the query to do it by location, again because we had our
extract set up by the findings by location as well. But it gives them a real good snapshot of just
seeing where their compliance is, where their faults and failings are.
This is an example of a selected reminder where when they chose I want output just by a
selected reminder it gives them a drop-down list, they can pick and choose which one, give
them the outputs
so if they chose the diabetic here’s the monthly reminder report by reminder four, the diabetic
eye exam, listed by clinic, so now management could see specifically which clinic location,
where the patients were seen for that period of time, who was due, not due.
Once you have that data outside of VistA, the tools, the options, just your capabilities are just
numerous. There’s a ton of different things. There’s web services. I’m not a web person but I
know the VISN I came from is doing a lot with web services to do a lot of data points. Ray’s
example was a web-based one so I’m sure that they’re using that as well. Again, expend the
time to get that information out of VistA and people will come knocking at your door if you like
that kind of attention.
Alright, so in summary what we tried to do was to cover that reminder extracts has a lot of
flexibility. To be able to create customized outputs, really drill down to some data points that
we’re looking for. We tried to provide a couple of site’s examples of how we utilize the
reminder extracts, and again I can’t reiterate, once you have that data extracted that there are
many tools and options available for you to use that data.
26