Cognitive testing CIS 2013:
Methodological guidelines
Petra Andries, Anneleen Bruylant and Machteld Hoskens
Presented to the representatives of the Belgian Federal Science Policy Office and
of the Direction Générale Opérationnelle de l‘Economie, de L‘Emploi et de la
Recherche de la Région Wallonne on March 7, 2012
What to expect
•
•
•
•
•
•
Some background
General Guidelines
Examples
Putting theory into practice
Our CIS question
Observations and Discussion
Some background
• Willis, Gordon B. (2005). Cognitive Interviewing: A tool for
Improving Questionnaire Design
• Van Cruysen, Adriana and Arundel, Anthony (2010). Guide to
conducting cognitive interviews of survey questionnaires
General goal
• Obtaining insight in possible problems (both overt and covert)
a questionnaire might have.
Major sources of error
• Errors of non-observation:
Are we surveying the right people?
• Errors of observation:
Are we getting the right answer?
• Post-observation error:
Are we doing the right things with the data?
Are we getting the right answers?
• Interview error:
Interviewer reads the question incorrectly, or registers the
answers incorrectly, …
• Response error:
Question characteristics and the way respondents process the
question
Question characteristics
• Conflicting, inaccurate or complicated instructions
• Question is lengthy, awkward, ungrammatical, or contains
complicated syntax
• Technical terms are undefined, unclear or complex
• Vague question: there are various ways to interpret the
question or to decide what is to be included or excluded
• Reference periods are missing, not well specified or in conflict
Assumptions
• Inappropriate assumptions are made about the respondent or
about his/her situation
• Assuming constant behavior/experience for situations that
vary
• Double-barreled questions: contains more than one question
Respondents’ processing
• Respondent’s may not know the answer to the question, or
may not remember the information asked for
• They may not have the attitude/be in the situation they are
being asked about
• The question may require a difficult mental calculation
• The question may ask about sensitive issues
• Socially acceptable responses may be implied
• Overlapping response categories
General Guidelines
• Generally, a relevant sample is chosen between 5-15
respondents
• Trained interviewers are used to conduct the test
• If problems become apparent early on during the testing,
changes can be made to the questions accordingly, and
testing continues using the altered version
General Guidelines (ii)
• Think-aloud technique: the subject has to report everything
he/she thinks about as they work their way to an answer
• Verbal probing: the use of probes to determine whether the
subject understands the question as intended, if the different
technical terms are understood, etc.
• A combination of both techniques will be used during our
testing of the questions
General Guidelines (iii)
• The interview may never exceed an hour
• Keep written notes of the interview, even if the interview is
being recorded, and record what time you start and end the
interview
• Prepare the probes to be used, but feel free to ask any
additional questions you may have, but don’t feel like you
have to probe everything
• This exercise is meant to find out whether the questions
obtain the information we aim for, whether the questions are
understood as we designed them, whether any part of the
question suggests bias, etc.
General Guidelines (iv)
• Interviewers have to understand the questions themselves, so
they are able to answer any questions the respondents may
have
• Keep your introduction short
• Keep the pace of the interview relaxed, but steady
• Do not defend or criticize the questions, and keep the tone of
your voice neutral, as well as the information you give (so as
not to bias the respondent)
• Probe proactively, even if there is no overt sign of trouble
General Guidelines (vi)
• Interviewers have to stress the fact that they did not design
the questions themselves, so any criticism is welcome
• Our goal is to find overt and covert problems of the questions,
so we are not interested in the actual answer the respondent
gives
• Start the interview with a short introduction, mentioning why
we conduct this test, and explaining what will happen during
the test. Use the windows-question to see whether your
instructions are clear to the respondent. End the interview
with thanking the respondent for his/her time, and offering
the opportunity to mention anything else he/she feels like
mentioning
Examples
• Q: How many times have you talked to a doctor in the last 12 months?
• A: I guess that depends on what you mean with “talked”. I talk to my
neighbor, who is a doctor, but you probably don’t mean that. I go to my
doctor about once a year for a general checkup, so I would count that. I’ve
also probably been to some type of specialist a couple more times in the
past year – once to get a bad knee diagnosed, and I also saw an ENT about
a chronic coughing thing, which I’m pretty sure was in the past year,
although I wouldn’t swear to it. I’ve also talked to doctors several times
when I brought my kids in to the pediatrician – I assume you don’t want
that included, although I can’t really be sure. Also, I saw a chiropractor,
but I don’t know if that’s a doctor in the sense you mean. So, what I’m
saying, overall, is that I’m not sure what number to give you- not sure
what you want.
Examples (ii)
• Q: How often do you, or the person who shops for your food, buy items
that are labeled ‘low salt’?
• Problems:
– One subject thought of low fat, not low salt.
– Several subjects said ‘yes’, but they meant that they get foods that they know are low in
salt, but it isn’t necessarily mentioned on the label.
– Different people have different behaviors (‘My wife always checks, I never do.’)
– How do you interpret an answer as ‘seldom’? (they rarely buy the types of food that
contain sodium, or they buy lots of such foods, but not the low-sodium ones.)
– One subject just read the mg of sodium, does that count as a label of ‘low salt’?
• Probes:
Who shops for your food?
Tell me in your own words how (you/they) decide how to choose foods that contain salt.
Examples (iii)
• Suggested rephrasing of the question:
Q: THE NEXT QUESTIONS ARE ABOUT SALT OR SODIUM IN FOODS.
a) Who shops for most of the food you eat: you or someone else?
b) When you/they shops for food, how often do you/they buy low-salt
or low-sodium varieties of foods, when you/they have a choice?
c) How do you/they usually select a food that is low in salt? Does it say
‘low salt’ on the label, or do you/they just know it is low in salt?
Putting theory into practice: intro
• Thank you for taking out the time to see me. First of all, I would like to
explain why we are doing this exercise. We are testing some questions
that have given us quite a few problems, as quite a few companies seem
to answer them incorrectly. We suspect the questions are not understood
by the companies the way we intended them.
• To give you an idea of what we’re going to do: I will ask you two questions,
which you will answer by telling me out loud everything that crosses your
mind. I may interrupt you sometimes with additional questions, and I will
be writing down your answers. Do not hesitate to ask for clarifications if
something is not clear, or if it is too difficult to answer. I did not design the
questions myself, so any comment or criticism is welcome! Also, we are
not interested in the answer you give in itself, but rather, we want to
discover whatever is wrong with these questions, and why companies
have a hard time answering them.
Putting theory into practice: Getting warmed up
• We will try out a simple question first, so you get an idea of what the test
will be like:
– Imagine you are at home. Now tell me how many windows there are. As you count the
number of windows, tell me what you see and what you’re thinking about.
– Was this question: very difficult, difficult, a little difficult, or not difficult at all? Why?
Putting theory into practice: Our CIS question
Combien votre entreprise a-t-elle dépensé en 2012 pour les activités ou les achats suivants? Veuillez également
donner une estimation de la part de ces acquisitions (ou d’activités) qui était prévue pour l’innovation de
produits ou de procès comme défini ci-dessus.
Activité
Dépenses 2012
Estimation de la part de l’activité
prévue pour l’innovation de
produit ou de procès
•
R&D interne
_____________€
____%
(y compris les dépenses en capital relatives aux
bâtiments et aux équipements spécifiquement
consacrés aux activités de R&D coûts de personnel
et coûts liés)
•
Acquisition de R&D externe
_____________€
____%
•
Acquisition de terrains, édifices,
_____________€
____%
machines et équipement (à l'exclusion des dépenses
pour des équipements destinés à la R&D)
•
Acquisition de logiciel
_____________€
____%
•
Acquisition de connaissances externes
_____________€
____%
(y compris licences)
•
Formation payée par l’entreprise
_____________€
____%
•
Publicité, campagnes de publicité,
_____________€
____%
•
études de marché
•
Activités de design (contrats interne et
_____________€
____%
externe)
• What problems can you foresee? What might not be
understood?
• Suggestions for probes we can ask? Remember, probes are
more effective if they are specific, rather than diffuse and
general!
• Let’s try it out!
Observations and discussion
• Observe as we try out the questions on each other.
• Any questions?