Center for Militar y Health Policy Research
A J O I N T E N D E AV O R O F R A N D H E A LTH A ND TH E
R A N D N AT I O N A L D E F E N S E R E S E A R C H INS TITU TE
THE ARTS
This PDF document was made available
CHILD POLICY
from www.rand.org as a public service of
CIVIL JUSTICE
EDUCATION
ENERGY AND ENVIRONMENT
HEALTH AND HEALTH CARE
INTERNATIONAL AFFAIRS
NATIONAL SECURITY
POPULATION AND AGING
PUBLIC SAFETY
SCIENCE AND TECHNOLOGY
SUBSTANCE ABUSE
TERRORISM AND
HOMELAND SECURITY
TRANSPORTATION AND
INFRASTRUCTURE
WORKFORCE AND WORKPLACE
the RAND Corporation.
Jump down to document6
The RAND Corporation is a nonprofit
research organization providing
objective analysis and effective
solutions that address the challenges
facing the public and private sectors
around the world.
Support RAND
Purchase this document
Browse Books & Publications
Make a charitable contribution
For More Information
Visit RAND at www.rand.org
Explore theRAND Center for Military
Health Policy Research
View document details
Limited Electronic Distribution Rights
This document and trademark(s) contained herein are protected by law as indicated
in a notice appearing later in this work. This electronic representation of RAND
intellectual property is provided for non-commercial use only. Unauthorized
posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are
protected under copyright law. Permission is required from RAND to reproduce,
or reuse in another form, any of our research documents for commercial use. For
information on reprint and linking permissions, please see RAND Permissions.
This product is part of the RAND Corporation monograph series.
RAND monographs present major research findings that address the
challenges facing the public and private sectors. All RAND monographs undergo rigorous peer review to ensure high standards for
research quality and objectivity.
Measuring the
Strategic Value of
the Armed Forces
Health Longitudinal
Technology
Application (AHLTA)
James H. Bigelow, Katherine M. Harris, Richard Hillestad
Prepared for the Office of the Secretary of Defense
Approved for public release; distribution unlimited
Center for Militar y Health Policy Research
A JOINT ENDEAVOR OF RAND HEALTH AND THE
RAND NATIONAL DEFENSE RESEARCH INSTITUTE
The research reported here was sponsored by the Office of the Secretary
of Defense (OSD). The research was conducted jointly by the Center
for Military Health Policy Research, a RAND Health program, and
the Forces and Resources Policy Center, a RAND National Defense
Research Institute (NDRI) program. NDRI is a federally funded
research and development center sponsored by the OSD, the Joint Staff,
the Unified Combatant Commands, the Department of the Navy,
the Marine Corps, the defense agencies, and the defense Intelligence
Community under Contract W74V8H-06-C-0002.
Library of Congress Cataloging-in-Publication Data is available for this publication.
ISBN 978-0-8330-4314-6
The RAND Corporation is a nonprofit research organization providing
objective analysis and effective solutions that address the challenges
facing the public and private sectors around the world. R AND’s
publications do not necessarily reflect the opinions of its research clients
and sponsors.
R® is a registered trademark.
© Copyright 2008 RAND Corporation
All rights reserved. No part of this book may be reproduced in any
form by any electronic or mechanical means (including photocopying,
recording, or information storage and retrieval) without permission in
writing from RAND.
Published 2008 by the RAND Corporation
1776 Main Street, P.O. Box 2138, Santa Monica, CA 90407-2138
1200 South Hayes Street, Arlington, VA 22202-5050
4570 Fifth Avenue, Suite 600, Pittsburgh, PA 15213-2665
RAND URL: http://www.rand.org
To order RAND documents or to obtain additional information, contact
Distribution Services: Telephone: (310) 451-7002;
Fax: (310) 451-6915; Email: order@rand.org
Summary
Background
The purpose of this project, which was undertaken between June 2006
and October 2007, was to help the Military Health System develop an
analytic framework and define specific outcome measures for assessing and reporting the efficiency, safety, and health benefits of AHLTA,
DoD’s electronic medical record system, as it becomes fully deployed.
Deployment of AHLTA is occurring in several planned phases or
“blocks.” Block 1 deployment to military treatment facilities (MTFs)
throughout the world began in January 2004. The timing of Block
4 deployment (the last planned block) has yet to be determined. As
of the end of 2006, AHLTA Block 1 had been installed at 138 fixed
military installations worldwide, and virtually all outpatient care they
delivered was being documented in AHLTA’s clinical data repository
(CDR). The CDR can be accessed (with appropriate permissions) from
any installation, so if a beneficiary is treated at one installation and
later seeks care at another installation, his record is available.
The CDR cannot now be accessed electronically from a theater
of operations, such as Iraq or Afghanistan.1 The MHS is addressing
this problem through the Theater Medical Information Program-Joint
(TMIP-J).
AHLTA will gain functionality as blocks 2–4 are rolled out. By
the end of 2007, Block 2 will add ordering and management of eye1
AHLTA has been deployed at a number of permanent overseas installations in Germany,
South Korea, and Japan. The CDR can also be accessed from these locations.
xi
xii Measuring the Strategic Value of AHLTA
glasses and dental charting and documentation. By the end of 2011,
Block 3 is scheduled to replace legacy pharmacy, laboratory, anatomical pathology, radiology, occupational health, and surveillance capabilities. Also by the end of 2011, Block 4 will extend AHLTA to inpatient
encounters. By this time, essentially all aspects of health care provided
by the MHS (but not care purchased from civilian providers) will be
documented in AHLTA.
The MHS is simultaneously deploying a clinical data mart (CDM),
which imports data from AHLTA’s CDR and facilitates enterprise-level
analysis and decision support. The CDM was in operation as of the end
of 2006, but its analysis capability was immature. There are also plans
to implement a clinical data warehouse (CDW). This capability would
access and link data from sources beyond the CDR—such as eligibility
and enrollment data, accounting data, and surveys—and would have
the potential to support more complex analyses than does the CDM.
Views of Senior MHS Leadership
Senior leaders of the MHS will be the primary customers for measures
of the performance of the MHS and of the strategic value of AHLTA.
Therefore, we conducted a series of 14 interviews with senior MHS
leaders. Through these interviews we discovered the dimensions of performance that the leadership deemed important, and how the leadership anticipated that AHLTA would affect those dimensions. They
were unanimous that the correct dimensions of performance to use
were the strategic objectives contained in their strategic plan (OASD,
2007a). The strategic objectives go beyond efficiency, safety, and health
to include the medical readiness of service members to deploy and the
ability of the military medical system to provide outstanding health
care in support of military operations.2
The leadership anticipated that AHLTA will have effects on performance similar to those reflected in the published literature on electronic medical records (EMRs). These effects include improved qual2
Specific strategic objectives are depicted in Figure 2.1.
Summary
xiii
ity and safety of care through point-of-care alerts and reminders and
reduced cost through avoidance of duplicate tests. AHLTA-generated
reminders, they expected, will improve individual medical readiness.
Improved continuity of care will benefit service members evacuated
from theater with wounds, injuries, or disease. A few participants mentioned AHLTA’s potential as a tool to support process improvement
efforts through measurement and feedback.
The Framework
To assess the strategic value of AHLTA, we recommend the standard
treatment-outcomes3 methodology described in textbooks on program
evaluation (Mohr, 1988; Rossi, Freeman, and Lipsey, 1999). The methodology has four elements:
1. Outcome measures capturing valued domains of system performance plausibly influenced by the presence of AHLTA
2. A treatment describing a way of using AHLTA that is expected
to influence one or more outcome measures
3. A logic model describing the mechanisms or processes by which
uses of AHLTA (i.e., treatments) influence outcome measures
4. An evaluation design for estimating AHLTA’s effect on outcome
measures in quantitative terms.
Given AHLTA’s complexity and evolving nature, we do not consider AHLTA, as a whole, to be a “treatment.” Instead, we consider a
treatment to be a specific use of AHLTA under defined circumstances.
An example of a treatment is the following: “When an active duty service member arrives for an appointment, determine whether the patient
meets all the criteria to be medically ready to deploy; if not, remind the
physician to inform the patient of his readiness status.”
3
Throughout this monograph we use the word treatment as it is used in the program evaluation literature: the action or intervention whose effects on outcomes is to be estimated. This
is more general than the use made of the word in healthcare, where a treatment is something
done to a patient to affect his health status.
xiv Measuring the Strategic Value of AHLTA
Logic models trace the cause-and-effect chains through which
treatments influence outcome measures. Each treatment requires its
own logic model, although many treatments should have very similar
models. A logic model describes cause-and-effect relationships qualitatively, and data are needed to turn the qualitative arguments into
quantitative estimates.
An evaluation design is the strategy for separating a treatment’s
quantitative effect on outcome measures from the confounding effects
of other factors. The evaluation design will determine what data
are needed and how those data could be used to generate the estimates. It is standard practice to measure this effect by comparing outcomes observed in the presence of the treatment to estimates of the
outcomes that would have occurred in the absence of the treatment.
If the data are insufficient to estimate AHLTA’s effects quantitatively, it is possible to use methods that combine logic models and a
mixture of qualitative and quantitative evidence to support the existence of a cause-and-effect relationship between AHLTA and relevant
outcomes. However, such methods rely heavily on quantitative measures of AHLTA-relevant outcome measures.
Specific Measures
The MHS strategic objectives are too abstract to serve as outcome measures. We have suggested detailed measures (i.e., measures specific to
care setting, condition, procedure, and patient characteristics) for a
wide range of strategic objectives that AHLTA could plausibly influence. Where possible, we suggest using detailed measures of healthcare
quality that are in the public domain, which can be obtained from
the National Quality Measures Clearinghouse (NQMC). While the
NQMC measures represent the practical universe of existing quality measures, they cover only a minority of MHS strategic objectives.
Measures for other strategic objectives must be developed from scratch.
Where possible, we have suggested what those measures might be.
Summary
xv
Before they are used, however, measures should be assessed for reliability (the degree to which the measure is free from random variation)
and validity (the degree to which the measure is associated with what
it purports to measure).
In order for high level managers to avoid information overload, we
discuss methods for creating higher-level measures by selecting, aggregating, and summarizing the detailed measures. With this approach, it
remains possible to drill back down to more-detailed levels to pinpoint
a problem when a high-level measure reveals a problem.
Implementation Issues
Calculating a detailed outcome measure requires specific data elements
and a precisely defined algorithm for combining them. To implement
a given measure, therefore, one must identify sources for each data element, and those sources must reliably contain correct values for the
data elements. Moreover, the data should not be buried in free text. For
the measure to be practical, it must be possible to retrieve the needed
data automatically.
Many measures of potential interest cannot be implemented
because the necessary data are not collected. If the MHS deems a measure sufficiently important, MHS leadership may choose to change
their information systems and policies to collect the necessary data.
AHLTA enhancements are fielded regularly, and the need to collect
additional data may be the purpose of future enhancements. Changes
for this purpose could be made to other information systems; they
need not be confined to AHLTA alone.
Finally, to implement our proposed framework, there must be
an organizational home for a measurement system. The organization
chosen will need ready access to data from many other systems as well.
In addition, it will need considerable analytic capability. We anticipate
that many of the treatments to be analyzed will originate as process
improvement exercises at individual installations. That organization
xvi Measuring the Strategic Value of AHLTA
will need to provide analytic support for these efforts, and it should act
as a clearinghouse for information on all installations’ improvement
efforts.