Introduction to
Encounter Data Validation
Presenter:
Thomas Miller, MA
Executive Director, Research and Analysis Team
1
Welcome
 About me
 Rules for engagement
 Presentation overview
•
•
•
•
The importance of encounter data
Trends in Federal policy
CMS protocols
Florida EDV study
2
Objectives
1. Learn why Encounter Data Validation
studies are important.
2. Identify the core evaluation components
outlined in CMS’ protocols for validating the
quality of encounter data.
3. Understand the proposed scope of work
for Florida Medicaid’s SFY 2013-2014
encounter data validation study.
3
4
Importance of Encounter Data
 Accurate and complete data are critical
to success of managed care programs
 Essential for overall management and oversight of
Florida’s Medicaid program
– Ability to monitor and improve quality
of care
– Establish performance measures
– Generate accurate and reliable reports
– Obtain utilization and cost information
5
Importance of Encounter Data
6
Importance of Encounter Data
 Used by MCOs and the State for many purposes
–
–
–
–
–
–
Performance measure development and calculation
Performance improvement measurement
Focused studies/quality activities
Rate-setting
Compliance monitoring
Provider practice patterns
7
Key Trends
 Importance of Federal and State monitoring
– Development of core measurement sets
• Medicare versus Medicaid
• Health care reform
• Holding health care accountable
 Data, not anecdotes
8
Key Trends in the News
9
Key Trends
 Findings from a recent article in Medicare and
Medicaid Research Review, Assessing the
Usability of MAX 2008 Encounter Data for
Comprehensive Managed Care
– Objective: Assess availability, completeness,
quality, and usability of encounter data
– Results: High rates for reporting by key encounter
data types
– Conclusions: Completeness and quality of
encounter data were high
10
Objectives
1. Learn why Encounter Data Validation
studies are important.
2. Identify the core evaluation components
outlined in CMS’ protocols for validating the
quality of encounter data.
3. Understand the proposed scope of work
for Florida Medicaid’s SFY 2013-2014
encounter data validation study.
111
12
Objectives
1. Learn why Encounter Data Validation
studies are important.
2. Identify the core evaluation components
outlined in CMS’ protocols for validating
the quality of encounter data.
3. Understand the proposed scope of work
for Florida Medicaid’s SFY 2013-2014
encounter data validation study.
13
EQR Protocol
 Developed and refined with the maturation of the
External Quality Review program
14
EQR Protocol
 Specific guideline for
External Quality Review
Organizations (EQRO) to
use when assessing
completeness and accuracy
of encounter data.
 Data submitted by Managed
Care Organizations (MCO)
to the State
15
EQR Protocol
 State establishes standards for
encounter data
 State must establish the
following standards:
– Definition of “encounter”
– Types of encounters
– Data accuracy and
completeness
– Objective standards for data
comparison
16
EQR Protocol
 Five core activities
1. Review state
requirements
2. Review MCO’s
capability
3. Analyze electronic
encounter data
4. Review of medical
records
5. Submission of findings
and recommendations
17
EQR Protocol
 Attachment A: Encounter Data Tables
Table 2: Data Element Validity Requirements
18
EQR Protocol
 Five core activities
1. Review state requirements
• Develop understanding of State-specific policies
and procedures for collecting and submitting
encounter data
• Identify data exchange protocols and layouts
• Evaluate encounter data system interchange
flows, including system edits and submission
timelines
• Review existing encounter data quality activities,
requirements, and performance standards
19
EQR Protocol
 Five key activities, continued
2. Review MCO’s capability
• Develop, conduct, and review MCO’s
Information System Capabilities Assessment
–
–
Identification of IS vulnerabilities
Key findings address:
»
»
»
•
Data processing and procedures
Claims/encounter processing and system
demonstration
Enrollment
Key informant interviews
20
EQR Protocol
 Five key activities, continued
3. Analyze electronic encounter data
• STEP 1 - Develop data quality
test plan to determine:
– Magnitude and type of
missing encounter data
– Overall data quality issues
– MCO data submission issues
21
EQR Protocol
 Five key activities, continued
3. Analyze electronic encounter data
• STEP 2 - Verify integrity of encounter data
– Macro-level analysis
– Encounter file completeness and
reasonableness
»
Volume and utilization by encounter type and
service setting
» Internal field consistency
» General field completeness and validity
22
EQR Protocol
 Five key activities, continued
3. Analyze electronic encounter data
• STEP 3 – Generate and Review Analytic
Reports
– Micro-level analysis
– Encounter record completeness and
reasonableness
»
»
Follows similar analysis as outlined
in Step 2
Analyzing volume/consistency by
time, provider, service type
23
EQR Protocol
 Five key activities, continued
3. Analyze electronic encounter data
• STEP 4 – Compare findings to state-identified
standards
– Identification of appropriate benchmark
population
24
EQR Protocol
 Five key activities, continued
4. Review of medical records
• Verification of the accuracy of coding
• Protocol assumptions
• STEP 1 – Determine sampling for medical record
review
– Identify valid sample size
– Encounter- vs. recipient-based samples
25
EQR Protocol
 Five key activities, continued
4. Review of medical records
• STEP 2 – Obtain and review medical records and
document findings
– Procurement efficiencies
– Abstraction staff and training
– Categorization of errors by level, type, and
source
– Procurement tracking and abstraction tools
26
EQR Protocol
 Five key activities, continued
5. Submission of findings
• Narrative report summarizing findings from
Activities 1-4
• Actionable recommendations for overall encounter
data quality improvement
27
Questions?
Proto what?
28
Objectives
1. Learn why Encounter Data Validation
studies are important.
2. Identify the core evaluation components
outlined in CMS’ protocols for validating
the quality of encounter data.
3. Understand the proposed scope of work
for Florida Medicaid’s SFY 2013-2014
encounter data validation study.
29
30
Objectives
1. Learn why Encounter Data Validation
studies are important.
2. Identify the core evaluation components
outlined in CMS’ protocols for validating the
quality of encounter data.
3. Understand the proposed scope of work
for Florida Medicaid’s SFY 2013-2014
encounter data validation study.
31
SFY 2013-2014 Encounter Data
Validation (EDV) Study
Agency for Health Care Administration
VALIDATION OF ENCOUNTER
DATA
32
SFY 2013-2014 Encounter Data
Validation (EDV) Study
 Review proposed encounter data validation process
– Submitted as part of EQR RFP response
– Will be conducted in alignment with CMS’ EQR
Protocol 4
– Evaluates the accuracy and completeness of encounter
data submitted to AHCA by capitated health plans
33
SFY 2013-2014 Encounter Data
Validation (EDV) Study
 Background
– Experience
– Core competency evaluating data
• Information system reviews
• Comparative analyses of MCO and State Medicaid data
• Medical/clinical record review
– Methodology is constructed to provide an effective
validation of the quality of data maintained by State
agencies within resource requirements
34
SFY 2013-2014 Encounter Data
Validation (EDV) Study
35
SFY 2013-2014 Encounter Data
Validation (EDV) Study
 Four key steps for conducting successful evaluations
– Project implementation
– Study design
– Data collection &
analysis
– Reporting &
recommendations
36
SFY 2013-2014 Encounter Data
Validation (EDV) Study
 Project Implementation
– Kick-off meeting with AHCA
•
•
•
•
Initiated during contract implementation period
Review and define overall scope of project
Discuss anticipated timelines
Define evaluation parameters
– Number of MCOs included
– Data requirements and limitations
– Implementation procedures to validate AHCA’s encounter data
– Kick-off meeting with participating MCOs
• Description of project and finalized study methodology
• Expectations for MCO involvement
37
SFY 2013-2014 Encounter Data
Validation (EDV) Study
 Study design
– Prepare draft methodology including:
•
•
•
•
•
Study objectives and research questions
Data source and collection procedures
Measurement methodology
Analytic methods
Timeline
– Review and approval of methodology by AHCA
– Develop of detailed analysis plan or technical companion
document methodology
38
SFY 2013-2014 Encounter Data
Validation (EDV) Study
 Data collection and analysis
– Information systems review
• Scope to be defined in collaboration with
AHCA
• Identify key encounter data policies and
procedures
– Selection of key evaluation fields, service groups,
and encounter types
– Identification of existing/proposed standards
– Review of processes affecting data quality
• Expected to be limited in scope
– Focused on building contextual knowledge of
systems to facilitate development of effective and
actionable recommendations
39
SFY 2013-2014 Encounter Data
Validation (EDV) Study
 Data collection and analysis
– Information systems review, continued
• Request for supplemental documents
– Encounter data submission process
– Previous studies conducted by AHCA
• Documentation will be used to assess encounter data quality
• Used of NCQA® Roadmap where appropriate
40
SFY 2013-2014 Encounter Data
Validation (EDV) Study
 Data collection and analysis, continued
– Encounter data source files
• Review of State encounter data file layouts
• Prepare data requirements documents
• Receive, process, and load encounter data
– Final status encounters from the Florida Medicaid Management
Information System and Decision Support System (FMMIS/DSS)
– Final status claims/encounters from MCO adjudication systems
– Includes all claim/service types—i.e., inpatient/outpatient, physician
visits, dental, and pharmaceutical
41
SFY 2013-2014 Encounter Data
Validation (EDV) Study
 Data collection and analysis
– Comparative data analysis of State and MCO
encounter data
• Evaluates the extent to which encounters submitted by MCOs to
AHCA are accurate, complete, and reasonable
• Preliminary file review
– Ensures files are sufficient for processing
– Involves the basic checks
» Percentage present
» Percentage valid
» Percentage valid values
42
SFY 2013-2014 Encounter Data
Validation (EDV) Study
 Data collection and analysis, continued
– Comparison: State data to MCO data
• Indicators to measure degree of completeness and accuracy for
each encounter type
– Overall record matching—percentage of state encounters present in
MCO files
– Field-level matching—percentage of state encounters with exact
value match in MCO file for each select data element
» Standard fields include: date of service, recipient ID, provider
ID, primary diagnosis, procedure code(s), and payment fields
43
SFY 2013-2014 Encounter Data
Validation (EDV) Study
Table X—Diagnosis Code Matching Rates for Institutional Claims
Encounter-Level
Matching
Plan
Statewide
Total Number
of Matched
Claims
Field-Level Matching: % Correctly Matched
% With All Diagnoses
Correctly Matched in In First Diagnosis
In Second
In Third
In Fourth
Both Files
Field
Diagnosis Field Diagnosis Field Diagnosis Field
In Fifth
Diagnosis
Field
4,655,817
92.1%
99.1%
82.1%
88.2%
93.0%
94.9%
Plan A
144,090
96.3%
97.8%
99.0%
99.6%
99.8%
99.9%
Plan B
500,980
99.5%
>99.9%
99.9%
99.8%
99.8%
99.8%
Plan C
2,429,624
89.1%
100.0%
75.4%
85.0%
91.5%
94.9%
Plan D
737,587
92.3%
>99.9%
68.2%
75.3%
84.0%
89.8%
Plan E
224,193
>99.9%
>99.9%
>99.9%
>99.9%
>99.9%
>99.9%
Plan F
367,800
89.8%
89.8%
>99.9%
>99.9%
>99.9%
89.9%
Plan G
251,543
>99.9%
>99.9%
>99.9%
>99.9%
>99.9%
>99.9%
44
44
SFY 2013-2014 Encounter Data
Validation (EDV) Study
Table Y—Second Diagnosis Field Code Matching Rates for Institutional Claims
% Mismatch Due to:
MCP
Statewide
Total Number of
Matched Claims
% Correctly Matched Diagnosis Omitted in Diagnosis Omitted in True Diagnosis
in Both Files
State File
Plan File
Mismatch
4,655,817
82.1%
0.9%
12.1%
4.9%
Plan A
144,090
99.0%
<0.1%
0.0%
1.0%
Plan B
500,980
99.9%
0.1%
0.0%
0.1%
Plan C
2,429,624
75.4%
0.0%
23.2%
1.4%
Plan D
737,587
68.2%
5.7%
<0.1%
26.1%
Plan E
224,193
>99.9%
<0.10%
<0.1%
<0.1%
Plan F
367,800
>99.9%
0.0%
<0.1%
0.0%
Plan G
251,543
>99.9%
0.0%
<0.1%
<0.1%
45
45
Phew… Questions?
46
SFY 2013-2014 Encounter Data
Validation (EDV) Study
 Data collection and analysis, continued
– Medical record review
• Represents the “gold standard”
• Evaluation of service level accuracy
and completeness
• Proposed methodology
– Only include MCOs operational as of January 2013
– EQRO Contract Years 1, 2, and 3 (7/1/2013-6/30/2016): review onethird of selected plans each year
– EQRO Contract Years 4 and 5 (7/1/2016-6/30/2018): review one-half of
selected plans each year
– Procure and abstraction 25 percent of all sampled records each quarter
– Minimum 50 cases reviewed per plan
– Target professional, dental, and pharmacy encounters
47
SFY 2013-2014 Encounter Data
Validation (EDV) Study
 Data collection and analysis,
continued
– Medical record review
• Sample selection methodology
1.
2.
To generate list of randomly
selected encounters for medical
review, HSAG proposes using data
files from comparative analyses
Two-stage stratified sampling
design used to ensure:
» Member’s record is selected only
once
» Number of encounters included in
final sample covers all encounter
types and proportional to total
distribution of encounters
48
SFY 2013-2014 Encounter Data
Validation (EDV) Study
 Data collection and analysis, continued
– Medical record review
• Sample selection methodology
– Identify all users by encounter type per MCO
– Determine required sample size of each encounter type based on total
distribution of users
– Randomly select users form each encounter type based on required
sample size
– Identify all encounters associated with applicable encounter types for the
selected users
– Final sample will consist of 50 cases randomly selected from applicable
encounter types per MCO per year, OR
1,200 cases for 1/3 of all MCOs being reviewed per year
– For each encounter type, HSAG will define specific data elements for
validation
49
SFY 2013-2014 Encounter Data
Validation (EDV) Study
 Data collection and analysis, continued
– Medical record review
• Procurement of selected sample records
– General Process
» Once sample is selected, each MCO to receive list of its study
cases
» HSAG will match selected date of service for each sampled
member with rendering provider
» MCOs will procure and submit identified medical records to
HSAG for review
50
SFY 2013-2014 Encounter Data
Validation (EDV) Study
 Data collection and analysis, continued
– Medical record review
• Procurement of selected sample records
– Two-hour technical assistance call with all participating
MCOs
– HSAG to review project and procurement protocols
– Able to accommodate a variety of procurement
methods:
» Faxing
» Hardcopy submissions
» Electronic submission via secure file transfer protocol
– Note: HSAG applies strict protocols to ensure security
and confidentiality of members’ medical records
51
SFY 2013-2014 Encounter Data
Validation (EDV) Study
 Data collection and analysis, continued
– Medical record review
• HSAG procurement and abstraction tool
– Data collection, management, and reporting system
• HSAG reviewers are experienced:
– Clinical nurses
– Nurse coders
• Procurement and abstraction process
– Based on established policies and procedures
– Continually monitored to ensure validity and accuracy
» Inter-rater reliability testing & Rater-to-standard testing
» All reviewers must achieve 95% accuracy rate
» Variety of reports will be generated, i.e., medical record compliance
rates
52
SFY 2013-2014 Encounter Data
Validation (EDV) Study
 Data collection and analysis, continued
– Medical record review – analysis of cases
• Compare electronic encounter data to medical record data
• Analyze record completeness and the accuracy of coding
• Four primary indicators for data completeness and accuracy
1.
2.
3.
4.
Medical Record Agreement
Medical Record Omission (surplus)
Encounter Record Omission (missing)
Erroneous
53
SFY 2013-2014 Encounter Data
Validation (EDV) Study
 Reporting and recommendations
– Prepare draft report of findings including:
• Indicator results
• Sub-analysis findings
• Preparation of supplemental findings
for future evaluation by MCOs
– Presented for statewide and MCO-specific results
– Actionable recommendations for improvement
54
Objectives
1. Learn why Encounter Data Validation
studies are important.
2. Identify the core evaluation components
outlined in CMS’ protocols for validating the
quality of encounter data.
3. Understand the proposed scope of work
for Florida Medicaid’s SFY 2013-2014
encounter data validation study.
55
Questions