The Use of Cognitive Interviews to Revise the Quality of Trauma Care Patient-Reported Experience Measure (QTAC-PREM)
Online Resource 2
Quality of Life Research Journal
Authors:
Niklas Bobrovitz, MSc1,2
Maria J Santana, MPharm, PhD2, 3
Theresa Kline, PhD4
John Kortbeek, MD5
Henry T. Stelfox, MD, PhD2, 6,7
1. Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, United Kingdom
2. Department of Community Health Sciences, University of Calgary, Calgary, Alberta, Canada
3. W21C Research and Innovation Centre, University of Calgary, Calgary, Alberta, Canada
4. Department of Psychology, University of Calgary, Calgary, Alberta, Canada
5. Department of Surgery, University of Calgary, Calgary, Alberta, Canada
6. Department of Critical Care Medicine, University of Calgary, Calgary, Alberta, Canada
7. Department of Medicine, University of Calgary, Calgary, Alberta, Canada
E-mail: niklas.bobrovitz@phc.ox.ac.uk
Table 3. Items that participants did not have the information to answer
Original Survey/Item #
Specific Issue/Interpretation
Acute 1, 2, 3: All three items
 Difficulty distinguishing phases due to recall issues
consisted of one core question
(pre-hospital, emergency, ICU) and distinctions too
with five sub-components
fine to make (in-hospital rehab, hospital ward)
addressing: before hospital care,
emergency department,
intensive care unit, hospital unit,
in-hospital rehabilitation.
Post 1, 2, 3: All three items
 Difficulty distinguishing sub-components
consisted of one core question
with two sub-components
addressing: out-patient
rehabilitation and follow-up.
Acute 2, Post 2: During your
 Participants not able to identify specific components
care for this injury, how often
of medical skill to evaluate
were your healthcare providers
 Providers trusted to be skilled and competent,
(doctors, nurses, therapists, etc)
aspects of communication and interpersonal
skilled and competent in helping
characteristics of providers evaluated instead
you recover from your injury?
Acute 11: How often were you
 Participants could not assess how other patients were
treated fairly by all healthcare
treated in order to compare
providers and treated the same
 Prompts necessary to direct evaluation of
as other injured people around
discrimination
you?
Action Taken
Revised. All three items were collapsed
and subsequently underwent further
cognitive testing.
Revised. All three items were collapsed
and subsequently underwent further
cognitive testing.
Deleted.
Revised.
Acute 30: How often were you treated
unfairly because of your age, ethnicity,
gender, or personal characteristics?
Table 4. Items that had ambiguous terminology or were inconsistently interpreted
Original Survey/Item #
Specific Issue/Interpretation
Acute 1: How well-timed was the  "Well-timed" too ambiguous
care you received (did you receive  Multiple interpretations: responsiveness to care request;
care when it was needed?)
waiting times in the emergency department, for
diagnostic imaging, for surgery, and for transfers
Acute 3: How often were you told  "What you needed to know" too confusing
what you needed to know in a way  Participants evaluated clarity of all the information they
you could understand?
received regardless of perceived need
Acute 5: How often was your care
well organized? (all of your tests,
treatments, and visits from
different healthcare providers)
Acute 6: People with injuries are
often moved from one unit or
hospital to another during their
injury care. If this happened to
you, how often did the moves go
well?
 Unclear what participants considered "organized" care
 Multiple interpretations: the clarity of information during
transfers (where and why); prompt responses to care
requests; consistency of information from providers
 Multiple interpretations: being told where they were
being transferred to; being kept comfortable during
transfers; oriented to new hospital units upon arrival
Acute 8b: How often did
members of your care team (e.g.,
nurses, doctors, other healthcare
providers) provide kind and
friendly care?
Acute 8c: How often did the
care team deal well with your
concerns or frustrations?
 Multiple interpretations: whether providers introduced
themselves; explained their role in care; provided a lot of
information; were respectful
 "Deal well" too ambiguous
 "Taking action" to deal with concerns proved better as
it was more consistently interpreted as provider
acknowledgment of and responsiveness to requests
Action Taken
Deleted. Item on responsiveness effectively
covered on the HCAHPS. Unable to develop an
item on wait times for different aspects of care.
Revised.
Acute 9: How often did your healthcare
providers (e.g. doctors, nurses, therapists, etc.)
explain things in a way you could understand?
Deleted. Content covered by QTAC-PREM
and HCAHPS items.
Revised. Item split into the three components
participants identified. List for acute item 17
were key landmarks described by participants.
Acute 15: During your transfers did the
hospital staff or healthcare providers clearly
explain where you were being transferred to?
Acute 16: During your transfers, how often
were you kept comfortable?
Acute 17: When you arrived to a new hospital
unit did a healthcare provider explain where
important landmarks were in the unit? (e.g. call
button, bathroom, nurse's station, water/ice
machine)
Deleted. Content covered by survey or
HCAHPS items.
Revised.
Acute 25: When you expressed concerns or
frustrations about your care how often did
your healthcare provider take action to deal
with them?
Acute 10e: How often was your
personal hygiene treated well?
Acute 10h: How often was your
dignity treated well?
Acute 10j: How often were your
cultural, religious, and spiritual
needs treated well?
Post 1: How well-timed was the
care you received (did you
receive care when it was
needed?)
 Hygiene being "treated well" nonsensical and
ambiguous
 Answers given even though family (not providers)
helped with hygiene
 Explored overlap with HCAHPS item on help getting
to the bathroom and using a bedpan
 Getting to the bathroom not the only aspect of hygiene
 "Hygiene" included: frequency of showers/sponge
baths; cleaning defecation and excoriation
 Dignity being "treated well" nonsensical and
ambiguous
 Dignity distinct from being treated kindly
 Being physically covered up and not being exposed a
key part of dignity
 Needs being "treated well" nonsensical and ambiguous
 Key issue was respecting preferences rather than
considering or accommodating needs
 Respecting preferences prompted participants to think
about providing access to spiritual staff and not
discriminating based on personal preferences
 "Well-timed" too ambiguous
 Key aspect of care timing post-discharge was
promptness and availability of follow-up appointments
Post 3: How often were you told
what you needed to know in a
way you could understand?
 "What you needed to know" too confusing
 Evaluations were of the clarity of information about
specific injuries in follow-up appointments
Post 5: How often was your care
well organized? (all of your
tests, treatments, and visits from
 Unclear what participants considered "organized" care
 Interpretations included: ease of getting appointments;
family doctors receiving information from the hospital
Revised. Providers that help to maintain
hygiene listed to direct participants to think
of the care provided by staff, not family.
Acute 23: How often did your nurses or
other hospital staff help you maintain your
personal hygiene?
Revised.
Acute 27: How often was your dignity
considered by the healthcare providers?
Revised.
Acute 29: How often were your cultural,
religious, or spiritual preferences respected
by the healthcare staff and religious or
spiritual staff?
Revised.
Post 8a-8c: Did you have difficulty getting
follow-up appointments when you wanted
them with....
A. a trauma doctor, surgeon, or specialist?
B. A family doctor or general practitioner?
C. A physio, rehabilitation, or occupational
therapist?
Revised.
Post 12: During your follow-up
appointments, how often did your healthcare
providers explain things about your injuries
in a way you could understand?
Deleted. Content covered by other items on
survey.
different healthcare providers)
Post 9c: When you first met the
care team how often did they
deal well with any concerns or
frustrations?
 "Deal well" too ambiguous
 "Taking action" to deal with concerns proved better as
it was more consistently interpreted as provider
acknowledgment of and responsiveness to requests
Revised.
Post 13: During your follow-up
appointments, when you expressed concerns
or frustrations how often did your healthcare
providers take action to deal with them?
Table 5. Items not measuring intended constructs
Original
Survey/Item #
Acute 8a: How
often did the care
team introduce
themselves and
explain what they
do?
Acute 9: How often
were you included
as part of your care
team?
Acute 10a: How
often was your
shortness of breath
treated well?
Acute 10b: How
often were your
feelings of agitation
treated well?
Acute 10f: How
often were your
emotional needs
considered?
Specific Issue/Interpretation
Action Taken
 "What they do" limited to describing only their specific medical
responsibilities
 The phrase "their role in your care" was preferred because it prompted
patients to assess if a provider explained all aspects of their involvement with
the patient (i.e. treatments they will deliver, how often they will be seeing the
patient)
 The intended construct was the degree of inclusion in decision making
however, participants thought of whether their questions were answered and
whether treatment and care options were explained, which are aspects of
communication and do not reflect if patient preferences were incorporated
into the treatment plan
 Shortness of breath interpreted as the type of breathing experienced during
panic attacks, not as difficulty breathing due to injured chest/abdomen
 Participants identified situations when providers were not clinically
successful at treating the symptom
 Participants suggested evaluation of the effort to treat, not treatment outcome
 Agitation sometimes confused with pain
 After adding "irritability" to the item, participants could differentiate from
pain
 Participants identified situations when providers were not clinically
successful at treating the symptom
 Participants suggested we evaluate the effort to treat, not treatment outcome
 Most would recall whether emotional support was offered although those not
requiring emotional support would answer no even if support was offered
 Many thought consideration of emotional needs was only necessary for
critically injured patients or those subjected to violence
 Important to know if emotional support was available regardless of need
 Support staff and nurses are the ones to provide this support, not physicians
Revised.
Acute 24: When meeting a new
healthcare provider for the first time how
often did they introduce themselves and
clearly explain their role in your care?
Deleted. Content covered by other
items.
Revised.
Acute 20: How often did the healthcare
providers do everything they could to
help you with your difficulty breathing?
Revised.
Acute 21: How often did the healthcare
providers do everything they could to
help you with your agitation or
irritability?
Revised. Reworded to identify whether
the support was offered, regardless of
need.
Acute 26: Did a healthcare staff member
(e.g. psychologist, social worker, nurse)
offer to speak with you about your
emotional needs.
Post 7: Was your
regular doctor well
informed about your
injuries and
treatments?
 Intended to measure coordination between hospital and family doctor but
there was confusion about who should be informing the physician: some
rated their own explanations to their physicians, rather than the transfer of
information from the hospital or attending acute care physician
Revised.
Post 16: In your opinion, how much
information about your hospital stay was
communicated to your family physician
or general practitioner on a scale of zero
to ten, zero being no information and ten
being all the information?
Table 6. Items that included assumptions about healthcare processes
Original Survey/Item #
Specific Issue/Interpretation
Acute 7b: Did you get the
 Providers often do not know how a patient's
information you needed in
injuries could affect them after they leave the
a way you could understand
hospital but participants said providers would
about how the injuries
discuss it even if they couldn't give exact details
might affect your life?
 Item too long
Acute 7c: Did you get the
 Providers often cannot give an exact timeframe
information you needed in
for recovery or know exactly what will be
a way you could understand
involved in recovery
about how long it would
 Participants valued an approximate timeframe and
take you to feel better and
some discussion about how to care for injuries
what would be involved in
post-discharge
recovery?
 Item too long and double barreled: addressing
recovery timeline and activities
Post 7b: Did you get the
 At the time of follow-up providers still may not
information you needed in
be able to give an exact timeframe for recovery
a way you could understand  Approximate timeline still important
about how long it would
 "What would be involved in recovery" often
take you to feel better and
interpreted solely as physical restrictions on
what would be involved in
activities
recovery?
 We wanted a broader item assessing information
about recovery
 Item too long and double barreled: addressing
recovery timeline and activities
Post 8b: Did you get the
 Assumptions about the services patients receive
information you needed in
 All should get discharge information but not all
a way you could understand
necessarily get homecare information
about discharge and
 Many had difficulty recalling if verbal
homecare?
information was given prior to discharge
Action Taken
Revised.
Acute 11: Did your healthcare discuss how the injuries
might affect you after you leave the hospital?
Revised. Item split into two components.
Acute 12: Did your healthcare providers provide
instructions on how you should care for your injuries
after you leave the hospital?
Acute 13: Did your healthcare providers discuss how
long it might take you to recover from your injuries?
Revised. Item split into two components.
Post 9: During your follow-up appointments, did your
healthcare providers explain the next steps in your
recovery from injury for example, activities you should
or should not do, necessary medications, tests and
treatments, or other follow-up appointments?
Post 10: During your follow-up appointments, did your
healthcare providers explain approximately how long it
would take you to recover?
Revised.
Post 2: Before leaving the hospital, did your hospital
healthcare providers give you written instructions on
how to care for your injuries after being discharged?
Table 7. Items measuring non-priority aspects of care
Original Survey/Item #
Specific Issue/Interpretation
Post 9a: When you first met the care team
 Not relevant because patients have already
how often did they introduce themselves and
met most of the follow-up providers (i.e.
explain what they do?
family doctor, surgeons from acute care
centre)
Post 9b: When you first met the care team
 Not a crucial element of follow-up care
how often did they provide kind and friendly  Support through information provision and
care?
encouragement more relevant
Post 10a-e: How often were the following
 Not priority aspects of follow-up care
treated well...
 Participants more concerned with activities
A. emotional needs?
needed to recover
B. respect?
 Not grammatically correct and nonsensical to
C. dignity?
"treat" respect, dignity, values and personal
D. values and personal preferences for care?
preference, cultural needs
E. cultural, religious, and spiritual needs?
Post 11: Were you treated fairly by all
 Participants suggested this was a low-priority
healthcare providers and treated the same as
issue in the post-acute setting
the other injured people around you?
 Additionally, participants do not observe
interactions between providers and other
patients in the post-acute setting and
therefore had difficulty making comparisons
necessary to provide an evaluation
Action Taken
Deleted.
Deleted.
Deleted.
Deleted.
Table 8. Items with redundant content and those that overlapped with content on the HCAHPS survey
Original Survey, Item #
Specific Issue/Interpretation
Action Taken
Acute 7a: Did you get the
 Overlap with item on clarity of information Revised. Refocused on the unique part of
information you needed in a
participants' interpretations.
 Unique interpretation of this item was
way you could understand
Acute 10: Did your healthcare providers clearly
whether providers explained all of the
about your injuries?
explain all your injuries to you in a way you could
patients' injuries
understand?
 Participants evaluated whether providers
explained what was injured and
anatomically how it occurred
Acute 10c: How often were
 Participants interpreted anxiety the same as Deleted.
your feelings of anxiety
agitation/irritability
treated well?
 Overlap with item on agitation/irritability
Acute 10d: How often was
Revised. Two well-validated pain items from the
 Overlap with pain items on the HCAHPS
your pain treated well?
HCAHPS survey were added to replace our single
survey
pain item.
 Items were compared and participants
Acute 18: How often was your pain well controlled?
preferred the HCAHPS items
Acute 19: How often did the healthcare providers do
 Pain management one of the most
everything they could to help you with your pain?
important components of the measure
Acute 10g: How often were
 Overlap with HCAHPS item about courtesy Deleted.
you treated with respect?
and respect
Acute 10i: How often were
Deleted.
 Overlap with two items: “concerns and
your values and personal
frustrations” and "cultural, religious,
preferences for care
spiritual needs"
considered by the healthcare
providers?
Acute 12: How often was
 Evaluations were made of room cleanliness Deleted.
the hospital kept clean,
not overall hospital cleanliness, suggesting
comfortable, and
overlap with HCAHPS item on clean room
maintained?
and bathroom
Table 9. Additional content identified as relevant for injury patients
Survey/Item #
Reason for Item Generation
(New) Acute 14: How often was the information you  Participants highlighted the importance of consistent information from
received from your various healthcare providers
providers
consistent?
 Patient distress associated with getting inconsistent information because
of the resulting confusion about what was "correct"; taken as an
indication of poor coordination
(New) Acute 22: When the healthcare providers
 Patients complained about being moved forcefully, being jerked, or
rolled you, turned you over in bed, or helped you get
jolted; causes concern, distress and discomfort
out of bed and move around, how often did they do it  Instances of forceful movement perceived as unsafe care
carefully?
(New) Acute 28: How often did you experience care
 Safety question added to obtain a quantitative evaluation of the number
that was unsafe? (e.g. given the wrong medication,
of patients who perceived unsafe events in-hospital
wrong test, wrong treatment, etc.)
(New) Post 1: Where did you go after being
 Not all patients went home after discharge
discharged from [Hospital Name]?
 Item added to map patient injury care and to identify if there are
different issues faced by patients on different care pathways
(New) Post 4: After being discharged from the
 Participants identified a gap in pain management after leaving hospital
hospital, did you have enough pain medication to
 Although prescriptions are filled in-hospital prior to discharge, limited
control your pain well?
mobility or difficulty getting appointments with family doctor makes
re-filling prescriptions difficult
(New) Post 3: Did the written instructions you
 The amount and quality of written discharge instructions important for
received provide you with enough information to
patients to know how to care for their injuries at home
help you care for your injuries after being
discharged?
(New) Post 5: After being discharged from the
 Amount of support services, specifically home care, social work, and
hospital, did you get all of the support services that
psychological and psychiatric counseling an important issue, especially
you wanted, for example home care, social work, or
for brain injured patients and those unable to immediately return to work
counseling?
(New) Post 6a-6c: Since being discharged from the
 Used as screener item to direct participants to applicable items
hospital, have you attended an appointment to
 Added to identify which providers are being evaluating
follow-up about your injuries with....
 General practitioner added because some participants didn't have regular
A. a trauma doctor, surgeon, or specialist?
family physicians and attended walk-in clinics for follow-up
B. A family doctor or general practitioner?
C. A physio, rehabilitation, or occupational therapist?
(New) Post 7a-7c:
Are you scheduled for an appointment to follow-up
about your injuries with...
A. a trauma doctor, surgeon, or specialist?
B. A family doctor or general practitioner?
C. A physio, rehabilitation, or occupational
(New) Post 11: During your follow-up appointments,
did you get all of the information that you wanted
from the healthcare providers?
(New) Post 14: After being discharged from the
hospital, how often was the information you received
from your different healthcare providers consistent?
(New) Post 15: Since being discharged from the
hospital, how often have you experienced healthcare
that was unsafe?
(New) Post 17: How well were you guided through
the recovery process by your healthcare providers
after being discharged from the hospital on a scale of
zero to ten, zero being poor guidance and ten being
excellent guidance?
 Added to map patient care
 Most participants expressed receiving a lack of information during
appointments
 Item developed to assess the overall amount and quality of information
received during appointments
 Participants identified an issue with consistency of information from
different follow-up providers most commonly about what to do during
recovery including activities that could be tolerated and whether or not
to attend physiotherapy
 Safety question added to obtain a quantitative evaluation of the number
of patients who perceived unsafe events post-discharge
 Many participants (especially those with brain injuries and those not
returned to work) expressed a lack of guidance post-discharge
 Item developed to get an overall rating of guidance
 Participants evaluated: the amount and quality of information about
recovery and next steps; the ease of communication access to providers;
whether providers answered their questions during appointments