Presented by:
Kristin L. Carman, Ph.D.
American Institutes for Research
Presented at:
Academy for Health Services Research
Annual Meeting, San Diego, CA June, 2004


Project Team
 Roger Levine, PhD
Managing Research Scientist, AIR
 Karen K. Shore, PhD
Senior Social Scientist
 Margarita Hurtado, PhD
Principal Research Scientist
 Kristin L. Carman, PhD
Principal Research Scientist
 Judy Mitchell, MS
Senior Research Scientist
 Steven A. Garfinkel, PhD
Managing Research Scientist
 San Keller, PhD
• Principal Research Scientist


Funding
 Agency for Healthcare Quality and
Research
• Part of the CAHPS II grant
Partners
 HMSA, Hawaii
 Humana, Chicago

 To develop an A-CAHPS Survey
 To use Critical Incident data in novel ways to address issues related to:

Instrumentation

Quality improvement

Reporting, and

Cultural comparability

 Critical Incident
 “Incident”=an observable, specific behavior
 “Critical”=means incident was crucial to the outcome of interest



Organized structure for data collection; focuses on observable behavior
Used to collect and analyze reports of behaviors associated with specific outcomes
Qualitative method; in-depth interviews


200 interviews; 40 providers and 260 patients

Patient respondents divided equally among four different racial/ethnic groups

Open-ended responses are transcribed

Each interview usually generates 10 incidents; we’ll have 2000+ incidents


Gathering extensive data

Using qualitative software to create and manage a very complex data base

Developed a very careful data processing protocol
 Raw data is open-ended responses, transcribed
 Data is transformed into “incident write ups”
 These two types of data are the sources for analyses


Develop a complete taxonomy of the components of quality ambulatory health care, based on both patient and clinician perspectives
 Confirm that the domains measured by the draft
CAHPS® instrument are salient to patients and providers and can be assessed by patients
 Determine whether the domains are salient and measurable for both men and women, for individuals with different levels of education, and across a range of racial and ethnic groups
 Identify additional domains that should be measured


Identify CAHPS® item content that can result in a spread of scores at the high end of the score distribution to minimize ceiling effects
 Generate objective patient reports of health care experience
 Create items for Ambulatory CAHPS® for which a positive rating would be rare


Developing the taxonomy
 Randomly select at least 200 incidents
 Two teams classify incidents into major categories, then subcategories
 Iteratively validate and refine the taxonomy with additional set of incidents


Using incidents as the unit of analysis, then respondents, we investigate statistical associations between personal characteristics and the way people conceive of quality of care
 Tabulation of respondent characteristics by taxonomy categories which are quality of care themes (e.g., coordination of care)
 Regress taxonomy categories on respondent characteristics


Logistic Regression
 Simple and multiple

Dependent Variable=taxonomy category

Independent Variables=
 Individual demographics
 Respondent: provider or patient


Identify physician behaviors associated with excellent and poor quality of care based on experiences of patients and clinicians
 Identify combinations or co-occurrences of behavior
 Identify key facilitators and barriers to quality of care, specifically related to CAHPS domains
 Create tools for QI interventions to improve
CAHPS scores





Analyze interview and CI files
Conduct additional coding of data
Focus on which behaviors or actions by clinicians co-occur to create positive (or negative) experiences for patients
Compare findings by respondent characteristics



Identify variations in taxonomic structure for different racial/ethnic groups
If differences exist, identify the implications for:
 Supplemental domains, concepts and items
 CAHPS domain labels and explanatory vignettes
 Culturally appropriate interventions to improve care


Identify narratives (phraseology) that clearly and effectively explains CAHPS measures in reports


Complete interviews

Complete analyses

Disseminate findings

Kristin L. Carman, PhD
Principal Research Scientist
American Institutes for
Research
1000 Thomas Jefferson
Washington, DC 20007
(202) 342-5090
Kcarman@air.org
www.air.org
Karen K. Shore
Senior Research Scientist
American Institutes for
Research
ADDRESS
Palo Alto, CA (ZIP)
PHONE NUMBER kshore@air.org
www.air.org

Transcript
[CI 1-1]
[CI 1-3]
[CI 1-2]
[CI 2-1]
[CI 2-2]
Episode 2
[CI 3-1]
Episode 3
[CI 3-2]
Critical Incident behaviors
Database Creation
CIs for Interview
Critical Incident
Forms
Episode 1
CI 1-1
Separate CIs
CI 1-1
CI 1-2
CI 1-3
CI 2-1
CI 1-2
CI 1-3
CI 2-1
Verbatim Transcript
Marked by interviewer for Episode # and for CI behaviors (not numbered —only numbered here for illustration) in Atlas/ti
Links are logical only: episode # in CI # allows analyst to trace back to transcript
CI Forms filled out by cut and paste from
Transcript, numbered to reflect source episode.
This takes place in Word
CI forms cut into separate text files for sorting into
Taxonomy

Both transcript and concatenated CI files are assigned to doc families for respondent demographics
(See next page for detail)
Transcript
[CI 1-1]
[CI 1-3]
[CI 1-2]
[CI 2-1]
[CI 2-2]
Episode 1
Final Atlas/ti Database
Concatenated CIs
P-01-M-JM-1-1-A
TAX-Communicates
CI 1-1
P-01-M-JM-1-2-B
TAX-Clarifies
CI 1-2
Episode 2
Taxonomy codes are assigned by autocoding
CI 1-3
The ID code includes #’s for episode and CI.
Since ID #’s include identifiers for referents, they are assigned to demographic code families for referents
(See next page for detail)
[CI 3-1]
Episode 3
CI 2-1
[CI 3-2]
At this stage, the finalized concatenated CI file, with taxonomy codes embedded, is imported into Atlas/ti. Taxonomy codes are
“autocoded” by searching for
Taxonomy codewords. At this point, hypertext links can be created between CI forms and episodes in the transcript. If desired, Taxonomy codes may be applied manually to the transcript.

RESPONDENT DEMOGRAPHICS
Respondent demographics are represented by placing
“primary documents” in “PD Families” or sets:
Example:
Gender::Male = {PD1; PD4; PD5; PD8…}
Gender::Female = {PD2; PD3; PD6; PD7…}
Age::20s = {PD1; PD2…}
Age::30s = {PD4; PD8…}
Age::40s = {PD3…}
Age::50s = {PD6…}
Age::60s = {PD5; PD7…}
Imported in a table:
PD1
PD2
PD3
PD4
PD5
PD6
PD7
PD8
…
Gender
Male
Female
Female
Male
Male
Female
Female
Male
Age
20s
20s
40s
30s
60s
50s
60s
30s
Dataset can be parsed according to Boolean combinations of set-memberships, for example, to restrict a query to documents that belong in both the Male and 20s sets.
REFERENT DEMOGRAPHICS
Referent demographics are represented by placing CI ID codes into “Code Families” or sets. The logic is the same as for PD families, but at present there is no table import feature and assignment is made with the “code family manager” tool in Atlas/ti.
Again, set memberships can be used to focus queries on different classes of referents.
Strategies for associating data at episode or interview levels
Layered IDs:
Since ID’s are layered, a hierarchy of ID codes can be created in
Atlas/ti. For example, a hierarchy could be structured as follows:
Interview ID
+--Episode ID
+--CI ID
+--Referent ID
Code Families:
A code family that included all the CI ID codes for a given episode would enable searching by episode in the concatenated CI files.
Hypertext:
Hypertext links can be created from the CI forms to the episodes in the transcript, or even to the specific descriptions of behaviors from which the CIs are derived.