LRI Validation Suite Meeting
October 11th, 2011
Agenda (Red=topics today)
• Action Item List
• Test data update
–
–
–
–
Selection of core message set
Review of lab results test category spreadsheet
First set of test messages
NIST data sets and management of test data
• Review policy proposal for validating receiver processing of terminology
• Identify/Create/Verify Value Sets
– HL7 Tables, LOINC, UCUM, SNOMED CT
•
•
•
•
Review ELINCS Test Plan and Test Tool (09/29/2011)
Update on LIS Test Plan Template
Update on EHR Test Plan Template
Spreadsheet Analysis/Juror Document
–
–
–
–
Overview and purpose of spreadsheet
Mapping CLIA requirements to HL7 Elements
Identifying Reportable Conditions to PH Lab Results
IG Example Messages (IG Analysis Call)
• Tooling Update
• Planning
Draft Test Messages
• Completed/Located Draft Messages
–
–
–
–
–
–
•
Hematology
Chemistry
Microbiology (has been reviewed by Riki)
Special Chemistry (Lipid Panel)
Susceptibility
Other (Surgical Pathology )
Work in Progress
– Routine Urinalysis
– Complete Urinalysis
• Provides at least one message for each of the categories we
identified
• We will use these to finish test plan skeletons, to develop data
management spreadsheet, create test case variations, etc.
• Disclaimer: These messages are not verified to be correct and do
not yet meet the technical/structural requirements specified in the
IG.
Draft Test Messages
• Hematology
– Hemoglobin 718-7 – use as example for minimally populated
message
• Started mapping to IG
–
–
–
–
•
•
•
•
•
•
Potentially maximally populated message (with no repeats)
Unknown patient
OID version
Non-OID version
Chemistry
Microbiology (has been reviewed by Riki)
Special Chemistry (Lipid Panel)
Susceptibility
Other (Surgical Pathology)
Work in Progress
– Routine Urinalysis
– Complete Urinalysis
ELINCS I
• Met with Walter Sujansky and team on 09/27/2011
• Generally speaking, in-line with validation suite approach
• Automated LIS Validation and Inspection Testing for EHR (EHR Display and
mostly CLIA)
– Other data items not specified as CLIA requirements were not examined
– I think we can relax our requirement that all required elements need to be
inspected; some will be implied (a lot of MSH, set id, etc.—it is just part of the
message processing)
– Other items will need to be inspected (access to DB)
– Did not test “RE” usage elements; did test for conditionals when feasible
– LIS testing was context free testing
• About 20 test messages covering the various scenarios (e.g., P, F, C).
– Messages posted on Wiki
– We can adopt these messages with modifications
– Limited in scope (in terms of the breath of lab results)
• Agreed that an “Expert Reviewer” is needed to access terminology
equivalence
– ELINCS was self testing tool (CCHIT did the certification)
– We’ll follow same model (we’ll provide recommendations for “Expert
Reviewers)
ELINCS II
• Since Labs and trading partners will have worked out
mappings we can’t necessarily expect a particular
mapping for our test cases (test data sheets)
– This is OK; we can validate to make sure it is a valid LOINC
and also provide a set of acceptable LOINCs
– The tool can provide an “ERROR with exceptions”;
meaning that the validation flagged it but it may be OK
based on the inspection of an “Expert Reviewer”
• Code is .net/C# (Desktop application)
– We’ll leverage ideas, not code base
• Configuration for certain data elements
– Addressing, Patient IDs, local codes?, etc.
– Dynamically create messages from message templates
– Inline with the validation suite WG design
LIS Validation Tool Test Flow
Use Case
Test Case
Data Sheet
Specific Lab Test
• Provide LOINC Code
• Allow for local code (from
Order)
Vendor LIS
System
Message
LIS Test Tool
select test case
Validation
Report
LIS Test Harness Test Flow
Testing the EHR – Model 1
Use Case
1.
2.
3.
4.
LIS Test Harness
Test Cases
Specific Lab Test
• Provide LOINC Code
• Allow for local code (from
Order) via configuration?
Juror
Document
Test Data
LN-1
Test
Message
Validation
Lab Message
Comm
LN-1
On-site inspection
Over-webex technology
Printed Report (?)
Screen-scraper – screen-capture
(include clock)
What are the requirements on the
EHR with regards to LONIC?
• Persist original LN code?
• Display Lab Results (options)
EHR
ACK
LN-1
Optional Techniques
Tech 1  Database access
Tech 2  View configuration files
Tech 3  CDA/CDD
Tech 4  HL7 message to public health
Report
*In all cases, some level of inspection test should be performed
LIS Test Harness Test Flow
Testing the EHR – Technical Option #2
Use Case
LIS Test Harness
Test Cases
LN-1
Test Data
Test
Message
Validation
Report
Should we expect the
same LONIC code?
Juror
Document
Lab Message
Comm
LN-1
EHR
ACK
LN-1
HL7 V2
HL7 V2
Validator
Mapped
LN-1 Text
on display
Report
Comments from Vocabulary WG (C. Johns) I
• LOINC codes often indicate a specific processing method for a particular
lab test; the NLM has recommended use of LOINC codes that do not
specify a method (ie, are “methodless”)
– Read: Our test cases/messages should focus on LOINC codes that are
methodless; however a test case that specifies a LOINC code (that specifies a
method) is OK
• The Vocabulary Standards workgroup’s understanding is that EHR
conformance testing is only to determine if the system can accommodate
LOINC codes, not to determine that the system correctly applies LOINC
codes to the lab tests
– This was not our assumption; The MU term is “incorporate”. If the system
doesn’t have to apply/incorporate the standard codes, what is the purpose of
requiring standard codes?
• Conformance testing should allow for both a LOINC code and a “local”
(proprietary) code for a lab test
– Agreed. Test tool will allow for this in configuration. Lab and EHR may have
agreed upon codes that they use. If both LOINC and local codes are sent then
mapping is straight-forward (bound in the message).
– OK for inaction with real labs and pilot testing, but what about certification
where the Lab is the test tool (no mapping agreements made). Does EHR have
to do mappings from just a LOINC code?
Comments from Vocabulary WG (C. Johns) II
• Conformance testing should allow any LOINC code to be used for any lab
test – ie, if a result message sent to an EHR uses an ESR LOINC code for a
Retic Count lab test, the system is still in conformance because it was able
to accommodate a LOINC code at all
– Not sure what this means; but if a certain type of lab result is ordered don’t
you expect to see a LOINC code that represents that lab results?
• The LRI Validation Tool should be designed to allow any kind of character
(alpha, numeric, or alpha-numeric) for the LOINC fields of the HL7
segments (e.g., OBX-3)
– Does LOINC have a certain format? If so, then this format is the only code that
should be accepted (For the OBX.3 triplet that contains the LOINC code).
• If the LRI Validation Tool is designed to test for a valid LOINC code in the
LOINC fields of the HL7 segments, the Tool should be able to check the
entire LOINC database to determine if the LOINC code used is in the
database, but the Tool should not determine if the correct LOINC code was
applied for a specific test
– Even if given a specific order/test case data sheet? Correct 
Reasonable/equivalent?
– We did not plan on checking the LOINC database for context-provided test
cases (for context-free testing this may be appropriate)
Challenge of Testing Terminology I
(Receiver System)
• Problem Statement:
– A specific terminology is specified for use in the IG (e.g., LOINC;
OBX.3.1 (code), OBX.3.2 (text), and OBX.3.3 (LN-coding system)
– The LIS Test Harness sends the EHR a test message containing a
LOINC code
• How do we test?
• How do we test the universe of possible codes?
• Considerations:
– Meaningful Use Requirements (See next Slide)
– EHR likely translate/map code to internal code and display local
text representation on display
– EHR could use standardized codes internally
– Inspection Testing
• Automated testing is limited
Meaningful Use Requirements
•
•
LOINC is a named standard for lab test results in the §170.306.g Reportable Lab
Results Stage 1 ONC certification criterion.
In the §170.302 (h) Incorporate laboratory test results criterion it states:
– Receive results. Electronically receive clinical laboratory test results in a structured format and
display such results in human readable format.
– Display test report information. Electronically display all the information for a test report
specified at 42 CFR 493.1291(c)(1) through (7).
– Incorporate results. Electronically attribute, associate, or link a laboratory test result to a
laboratory order or patient record.
•
The criteria use the following verbiage when referring to LOINC:
РҤ170.306 (g) Reportable lab results. Electronically record, modify, retrieve, and submit
reportable clinical lab results in accordance with the standard (and applicable implementation
specifications) specified in §170.205(c) and, at a minimum, the version of the standard
specified in §170.207(c).”
– §170.207(c) states: “Standard. Logical Observation Identifiers Names and Codes (LOINC®)
version 2.27, when such codes were received within an electronic transaction from a
laboratory (incorporated by reference in §170.299).”
– In the ONC Final Rule Preamble for 170.304.i Exchange Clinical Information and Patient
Summary Record ONC stated the following:
•
“For the purposes of electronically exchanging a patient summary record, we expect the patient
summary record to include health information that is coded, where applicable, in accordance with
adopted vocabulary standards. Therefore, unless otherwise required in the context of a meaningful
use objective and measure, an eligible professional (or eligible hospital) would be permitted to map or
crosswalk local/proprietary codes to the adopted vocabulary standards prior to transmitting a patient
summary record.”
– MU Stage-2 requirement not finalized
Processing of Terminology
• Receiver requirements for processing
terminology:
– The receiver shall persist (store) the original
standardized code and the original standardized
code text as received in exact representation.
– The receiver may perform a translation/mapping
to locally defined representations.
Assessment of Terminology
• Proposed procedure for assessing the receiver for incorporation of
terminology: (for Lab Results name not results—results need to be exact)
– Where applicable to meet CLIA requirements the receiver shall display on the
EHR GUI the equivalent representation of the received coded lab results. The
receiver shall display at least one of the following:
• Original standardized code text
• Original standardized code (will the actual code ever be displayed; is it adequate or good
practice to display just the code?)
• Local code text
– The receiver shall be capable of demonstrating the persistent of the original
standardized code and the original standardized code text. Acceptable
methods for attestation:
• Administrative access to database
• Inspector approved method
– If applicable the receiver shall be capable of demonstrating the linkage of the
original standardized code the locally translated/mapped code. Acceptable
methods for attestation:
• Administrative access to database
• Browse capabilities of configuration files
• Inspector approved method
Equivalent Representation
• The exact original code
• The exact original code text
• For translated/mapped local code text an equivalent representation
as determined by clinical terminology expert. The following rules
and guidance are given to promote consistency in assessment:
– Rule: A terminology shall never be made more specific in the
translation/mapping.
– Rule: A terminology shall never be made more specific in the display
of standardized terminology.
– Guidance: A limited number of synonyms are provided to assist the
clinical terminology expert. Note: a predefined definitive set is not
possible—there are too many possible equivalent local
representations.
– Guidance: The inspector must consider the context in which the code
is used as this may impact the translation/mapping.
Questions
• What should be the assessment if a more detailed
terminology term is received that is mapped to a less
specific term?
• This mapping will lead to a loss of information if the original
code is not persisted.
• Is it valid to display a representation that has less
specificity?
• Is it valid to display a representation that has less specificity
as long as the original data is persisted in the system?
• Is it acceptable for a loss of information to occur when data
is rendered as a report or forwarded on to another system
(e.g., public health)? That is, we sent a specific code and
then a general code is forwarded on to public health. Is it
acceptable for either the general or specific LOINC code to
be forwarded?
LOINC-RELMA List for Hemoglobin
LOINC #
11559-2
19949-7
19951-3
19953-9
19955-4
2714-4
2715-1
2716-9
2717-7
30369-3
30370-1
34969-6
Component
Ex. UCUM Units Ex. Units
Order/Obs
Oxyhemoglobin/Hemoglobin.total
%
Observation
1
Oxyhemoglobin/Hemoglobin.total
%
Class
MFr
%
MFr
%
2
Oxyhemoglobin/Hemoglobin.total MFr
%
%
2
Oxyhemoglobin/Hemoglobin.total MFr
%
PULM
2
Oxyhemoglobin/Hemoglobin.total MFr
%
PULM
2
Oxyhemoglobin/Hemoglobin.total MFr
%
%
Observation
1
Oxyhemoglobin/Hemoglobin.total MFr
%
%
Observation
1
Oxyhemoglobin/Hemoglobin.total MFr
%
%
Observation
1
Oxyhemoglobin/Hemoglobin.total MFr
%
%
Observation
1
Oxyhemoglobin/Hemoglobin.total MFr
%
% of tot
Observation
1
Oxyhemoglobin/Hemoglobin.total MFr
%
% of tot
Observation
1
Oxyhemoglobin/Hemoglobin.total MFr
%
% of tot
Observation
1
No Method listed in RELMA for any of these tests
Property Time Aspect System
Scale
Long Common Name
Type
Pt
Bld
Qn
CHEM
Fractional oxyhemoglobin in Blood
8H^max
PULM
BldA
Qn
Fractional oxyhemoglobin in 8 hour maximum Arterial blood
8H^min
PULM
BldA
Qn
Fractional oxyhemoglobin in 8 hour minimum Arterial blood
Pt
Bld.preductal
Qn
Fractional oxyhemoglobin in Blood Preductal
%
Pt
Bld.postductal Qn
Fractional oxyhemoglobin in Blood Postductal
%
Pt
CHEM
BldA
Qn
Fractional oxyhemoglobin in Arterial blood
Pt
CHEM
BldC
Qn
Fractional oxyhemoglobin in Capillary blood
Pt
CHEM
BldV
Qn
Fractional oxyhemoglobin in Venous blood
Pt
CHEM
Plas
Qn
Fractional oxyhemoglobin in Plasma
Pt
CHEM
BldCoV
Qn
Fractional oxyhemoglobin in Venous cord blood
Pt
CHEM
BldCoA
Qn
Fractional oxyhemoglobin in Arterial cord blood
Pt
CHEM
BldMV
Qn
Fractional oxyhemoglobin in Mixed venous blood
Testing Options/Policy-OLD
LOINC # Component
Ex. Units
Class
Type
11559-2Oxyhemoglobin/Hemoglobin.total MFr
%
1
19949-7Oxyhemoglobin/Hemoglobin.total MFr
%
2
19951-3Oxyhemoglobin/Hemoglobin.total MFr
%
2
Property Time Aspect System
Long Common Name
Scale
Ex. UCUM Units
Order/Obs
Pt
CHEM
Bld
Fractional oxyhemoglobin in Blood
Qn
%
Observation
8H^max
PULM
BldA
Qn
Fractional oxyhemoglobin in 8 hour maximum Arterial blood
%
8H^min
PULM
BldA
Qn
Fractional oxyhemoglobin in 8 hour minimum Arterial blood
%
OBR|1|111325^EHR^2.16.840.1.113883.19.3.2.3^ISO|1132896^Lab^2.16.840.1.113883.19.3.1.6^ISO|7187^Hemoglobin^LN|||20070701152505|||||||||100^Hippocrates^Harold||||||20070701162505||CH|
F|NA&Not Applicable&LB
OBX|1|NM|11559-2^Hemoglobin.Total^LN||^12.4|g/dL^grams per deciliter^UCUM|12.0 to
16.0||||F|||20070701152505|||||||||Effective Labs,
Inc^^^^^DRSD&2.16.840.1.113883.19.1.11&ISO^XX^^^6543|3434 Test Loop^^Ann Arbor^MI^48103^^B
•
Depending on what the order is, what are possible test results? Are there more than one valid test result base on local
conventions, etc.?
•
Potential Test Case Policy: In the data sheet (test case) we can allow any of these to be selected. This assumes that nothing
else in the message needs to be changed. It would be an easy way to provide the capability to tests all of these LOINC codes
with minimal effort. The Long Common Name is the differentiator.
•
One of the determining factors likely will be whether or not the same result value can be used for all of the various versions
of Oxyhemoglobin/Hemoglobin.total.
Issues/Comments:
•
Other data elements in the HL7 message might also need to be test-specific. For example, if data for the SPM-4: Specimen
Type and SPM-8: Specimen Source are included in the message, then the same message probably could not be used for all
of the versions of Oxyhemoglobin/Hemoglobin.total. OBX-7: Reference Range also might be a limiting factor if venous blood
and arterial blood result values have normal ranges that don’t overlap.
•
The LOINC code is the anchor that discretely and accurately identifies the lab test that was ordered/performed. No matter
what a hospital or physician chooses to call the test, the LOINC code is the one constant that tells everyone which it test
was. For our purposes, we’ll need to be sure that all of the data for the data elements in the HL7 message are appropriate
for the test indicated by the LOINC code.
Note on the use of LOINC
• The list of tests has been organized by order panel or,
for micro related tests, by target organism for easier
readability.
• The LOINC terms included here are considered
examples only – though they may be the more
common LOINC terms, each laboratory needs to be
sure to map their own tests to the most appropriate
LOINC, even if it is NOT on this list.
• This is NOT an exclusive list of LOINC terms – ANY valid
LOINC should be accepted in data exchange projects
based on this specification.
Other Issues I
• EHR – Implementation Choices for Terminology
– Use standardize coding internally
– Map to local codes
– Need to account for options in test procedure
• How to we test for complete coverage of recommended terminology?
– Create all messages for all possible (recommended) terms (for important code
system, e.g., LOINC)—This is the preferred approach
– Alternative approach: Create a subset and inspect/verify tables for coverage
and accuracy
– The LAB is not restricted to sending the recommended list of LOINC codes and
the EHR should not fail when it does not recognize a LOINC code.
• How can this be tested; should it be tested?
– We should test that the EHR recognizes an invalid LONIC code and rejects the
message.
• Given that LONIC changes often what mechanisms will a system use to identify invalid
codes (based on a certain format?)
• We could send messages where code and text don’t match as another negative test
– Selection of approach may be made on a case-by-case analysis
– It may be necessary to provide coverage for all recommended LOINC codes.
However, for other terminology it may not be a priority (e.g., state) or may not
be feasible (SNOWMED).
Other Issues II
• Use of UCUM. Can/should UCUM be mapped
locally and displayed as local representation?
• What terminology can be mapped?
• Is it in scope to test the actual results of given
in the lab results (abnormal values was
mentioned)
CLIA Requirements
42 CFR 493.1291(c) The test report must indicate the following:
1.
2.
3.
4.
5.
6.
7.
For positive patient identification, either the patient's name and
identification number, or a unique patient identifier and identification
number
The name and address of the laboratory location where the test was
performed
The test report date
The test performed
Specimen source, when appropriate
The test result and, if applicable, the units of measurement or
interpretation, or both
Any information regarding the condition and disposition of specimens
that do not meet the laboratory's criteria for acceptability
CLIA Requirements Mapped to Data Elements
42 CFR 493.1291(c) The test report must indicate the following:
1.
For positive patient identification, either the patient's name and identification number, or a
unique patient identifier and identification number
–
–
2.
PID-3 : Unique patient identification number
PID-5 : Patient Name
The name and address of the laboratory location where the test was performed
- OBX-23/24/25: Lab Identification Fields
3.
The test report date
- OBX-19: Date/Time Analysis
4.
The test performed
- OBX-3: LOINC codes for Observation Identifier
5.
Specimen source, when appropriate
–
SPM-4: Specimen Type
6.
The test result and, if applicable, the units of measurement or interpretation, or both
–
–
–
–
–
7.
OBX-5: Observation Value
OBX-6: Units
OBX-7: Reference Range
OBX-8: Abnormal Flag
OBX-11: Observation Result Status
Any information regarding the condition and disposition of specimens that do not meet the
laboratory's criteria for acceptability
–
–
SPM-21: Specimen Reject Reason
SPM-22: Specimen Quality
Required CLIA Report Elements
–
–
–
–
–
–
–
–
–
–
–
–
–
PID-3 : Unique patient identification number
PID-5 : Patient Name
OBX-3: LOINC codes for Observation Identifier
OBX-5: Observation Value
OBX-6: Units
OBX-7: Reference Range
OBX-8: Abnormal Flag
OBX-11: Observation Result Status
OBX-19: Date/Time Analysis
OBX-23/24/25: Lab Identification Fields
SPM-4: Specimen Type
SPM-21: Specimen Reject Reason
SPM-22: Specimen Quality
Are all elements required to be displayed on screen (including the
components and subcomponents of these fields)?
Action Item List I
• Select message to handle core lab results
– Identify 20 or so common lab results (In progress)
– Obtain/Adapt/Create test messages to cover the core set of lab
results (In progress)
• Identify/List all pertinent data elements (In progress)
– Create spreadsheet of all data elements with usage of R, RE, and
C (rows)
– Columns will identify:
• Juror Document (How to assess the element)
• Identify the elements required for CLIA testing
• Identify static, configurable, or indifference data elements
• Identify/create/verify value sets (In progress)
– Create Spreadsheet; convert to NIST XML tool format
– Incorporate the value sets in PHINVADS
– Develop download mechanisms and transformation of values to
support the NIST tooling format
Action Item List II
• Review LRI implementation Guide and create a list of
all conformance requirements (Not Started)
– Create matrix based on data elements
– Link all conformance requirements to data elements when
possible
– Create “higher” level list of conformance requirements
• Determine the policy for assessing receiver side
terminology (Done: need to write policy statement)
– Inspection test requirements and procedure
– Automated test requirements and procedure
• Complete development of LIS Test Plan Skeleton
• Complete development of EHR Test Plan Skeleton
Action Item List III
• Identify and document the test dimensions (Not Started)
–
–
–
–
–
Coverage of Lab Results
Scenarios (e.g., Preliminary, Final, Corrected)
Reporting formats
Negative testing
Minimally and maximally populated
• Contact CLIA and CAP inspectors to get their lab inspection
process (Need contacts)
• Determine a process for verifying test cases (Not Started)
• Implement process for verifying test cases (Not Started)
• Research ELINCS Test Tool (DONE)
– Determine what we can leverage
– Process flow, source code, test messages
Action Item List IV
• Identify all the public health reportable lab results (In progress)
• Identify the data elements that differ from the public health IG and
the S & I LRI IG (Not Started)
• Determine a policy for validating LRI messages using EHR PH lab
results messages (Not Started)
• Develop spreadsheets for managing test cases/data (In progress)
– Adapt tooling to process and incorporate data
– Phase 1 nearly complete
– Phase 2 will include the multiple dimensions (Data, Profile, Juror)
• Create the HL7 standard message profiles (Starting soon)
– MWB (then produce XML message template)
– Need to make updates to the message profile based on changes made
in version 2.7 and 2.7.1
– Write XSLT to modify XML message profile
Action Item List V
• Identify the CLIA conformance requirements and
compare to the requirements in the implementation
guide (In progress)
– Mark in spreadsheet – (DONE)
– Make sure conformance requirements and IG match
– Write conformance requirements in IG where necessary to
match CLIA requirements
• Prototype tool (In progress)
–
–
–
–
Requirements and design (In progress)
Development (In progress)
Incorporate test cases (Not Started)
Testing (Not Started)
Validation Methodology
HL7 V2.5.1 Message
Profile (LRI IG
assertions)
Test Case Specific
Assertions (Validation
Context Files)
Vocabulary (Stored in
PHINVADS than
translated to V2.8
Table Library Format)
Validation Engine
and Juror
Document
Generator
Tooling Update
• LRI Implementation Guide is in ballot process
– Open for comments until October 17th, 2011
• We will begin developing the HL7 standard conformance profile
– MWB
– Need to make updates to the conformance profile based on changes
made in version 2.7 and 2.7.1
• Data Management of test cases/data will be with spreadsheet
– Spreadsheet is processed to build messages and to create validation
context files
– Validation context files encapsulates test case related assessment
– Leverage/adapt existing NIST Test messages to S&I Framework LRI IG
• Early version of prototype tool developed
– Limited functionality
– Handles message validation based on message profiles and validation
context files
– Message Editing