STRATEGIES FOR GROWTH
SM
"CONSULTANTS TO THE SERVICES INDUSTRY"
Comparative Advantages/
Disadvantages of Phone, Mail &
E-Mail/Web-Based Surveys
Prepared by:
William K. Pollock
President
SM
Strategies For Growth
Westtown, Pennsylvania
© 2004 Strategies For GrowthSM
P.O. Box 1024 / Westtown, PA 19395 / (610) 399-9717 / (610) 399-9718 (Fax) / www.s4growth.com / wkp@s4growth.com
Comparative Advantages/Disadvantages of Phone, Mail & E-Mail/Web-Based Surveys
Item/Area
Telephone Survey
Mail Survey
E-Mail/Web-Based Survey
Comparative Cost
ƒ Moderate to High - dependent on
number of interviews, respondent
incidence rate, length of
questionnaire, number of openended questions, degree of
customer segmentation, etc.
ƒ Low to moderate - varies by
complexity/sample design; heavily
dependent on number of pages/
inserts (i.e., weight); cost of outbound/
inbound postage and questionnaire/
envelope printing is high, but is
sometimes offset by internal printing/
posting capabilities; return postage
from outside U.S. adds cost.
ƒ Low - majority of costs are related to
questionnaire design/programming,
software for mass distribution and
receipt of completed e-questionnaires; but virtually no cost for
paper, printing or postage. Data
processing costs may actually be
lower through the use of embedded
data collection/processing tools.
Geographic Coverage
ƒ No problem covering all time
zones in North America - covering
French-speaking Canada and
Spanish-speaking Mexico would
add some incremental cost for
additional native-language trained
interviewers. Must be conducted
in native languages, at local times.
ƒ No problem mailing to all of North
America - mailing outside of U.S. adds
cost for return postage; mailing to
French-speaking Canada and
Spanish-speaking Mexico would add
some cost for translated questionnaires. Generally must be conducted
on an individual country basis.
ƒ No problem e-mailing to targeted
respondents on a worldwide basis;
generally conducted on a language(and cultural) specific basis, rather
than on a geographic basis.
Turnaround Time
ƒ Fast turnaround time once the
questionnaire has been
programmed and the sample has
been processed and entered into
the system - also allows for
interim “topline” printouts at any
time during the course of the
survey.
ƒ Slow turnaround time - preparatory
design, printing and envelope stuffing
may take a few weeks or more; may
then take up to 4 - 6 weeks “on the
street” before returns diminish; may
require a “reminder” mailing (adding
time/expense).
ƒ Fast turnaround time once the
questionnaire has been preprogrammed, the sample has been
processed and entered into the
system, and the data processing
specs have been developed generally allows for ad hoc detailed
printouts at any time during the
course of the survey.
Screening of
Respondents
ƒ Best method for screening and
managing respondent quotas in
terms of respondent relevance, as
well as per category/cell.
ƒ Limited screening capability for a
small sample mailout (e.g., no
guarantee that the questionnaire will
reach the targeted respondent);
however, not a problem if the entire
universe (e.g., 20,000 customers) is
targeted.
ƒ Limited screening capability - due to
the nature of the methodology,
questionnaires can only be
disseminated to potential
respondents that have valid e-mail
addresses; this may add significant
“skewing” and, as a result, may not
be applicable in many situations.
Page 2 of 5.
P.O. Box 1024 / Westtown, PA 19395 / (610) 399-9717 / (610) 399-9718 (Fax) / www.s4growth.com / wkp@s4growth.com
Comparative Advantages/Disadvantages of Phone, Mail & E-Mail/Web-Based Surveys
(continued)
Item/Area
Telephone Survey
Mail Survey
E-Mail/Web-Based Survey
Sample Respondents
ƒ Good, for named respondents
with current telephone numbers not as good when the interviewer
must use a “generic” screen to
reach unnamed respondents.
ƒ Good, for named respondents with
current home/business addresses but not good at all for unnamed
respondents (i.e., low likelihood of
having the mail questionnaire passed
on to the appropriate individual).
ƒ Good only for named respondents
with current e-mail addresses - but
not good at all for those without PC
access, e-mail addresses, or any
unnamed respondents (i.e., low
likelihood of having the e-mail
questionnaire passed on to the
appropriate individual).
Overall Response Rate
ƒ Best of all methodologies typically about 50% of the
“qualified” customer base
responds (depending on the
overall “quality” of the customer
list); provides the most management control in terms of meeting
individual segment/cell quotas.
ƒ Very low percentage - typically on the
order of 15% - 25% (depending on the
availability of a “clean”/“qualified”
customer list).
ƒ Very low percentage (but highly
dependent on the specific target
segments) - typically on the order of
10% - 20% (depending on the
availability of a “clean”/“qualified” ecustomer list). Highest likelihood of
response comes from segments that
reflect the highest continuous use of
PCs, the Internet, and e-mail.
Non-Response Bias
ƒ Small - reducible by prior
notice/alerts, callbacks, etc.;
controllable by interviewer.
ƒ Moderate to large - reducible by
incentives, but would involve a high
expense for large mailings;
respondent can selectively skip
questions, or misread skip patterns,
leading to incomplete responses.
ƒ Large - due to the fact that not all
targeted respondents have e-mail
addresses, or easy access to a PC
or the Internet; therefore, the nonresponse bias will generally be
significant.
Validity of Response
ƒ Relies on memory - little chance
to “look it up” while interview is
underway; however, calls made
soon after the event are more
likely to obtain current information.
ƒ Relies on memory - but may be aided
by easily accessible/handy records/
memos, where applicable; depends
on question, and ease of access to
back-up information.
ƒ Relies on memory - but may be
aided by easily accessible/handy
records/memos, where applicable;
depends on question, and ease of
access to back-up information.
Accessibility of
Respondent
ƒ Variable - depends on availability
and contact data; better controlled
through having respondent name
and phone number available, and
through scheduled callbacks.
ƒ Variable - depending on quality and
accuracy of the customer list (i.e.,
name, title, address, etc.).
ƒ Variable - depending on quality and
accuracy of the e-customer list (i.e.,
name, title, physical address, e-mail
address, etc.).
Page 3 of 5.
P.O. Box 1024 / Westtown, PA 19395 / (610) 399-9717 / (610) 399-9718 (Fax) / www.s4growth.com / wkp@s4growth.com
Comparative Advantages/Disadvantages of Phone, Mail & E-Mail/Web-Based Surveys
(continued)
Item/Area
Telephone Survey
Mail Survey
E-Mail/Web-Based Survey
Type of Questionnaire
ƒ No limitation beyond interviewer
skill - can ask scripted “probing”
follow-up questions such as “Why
do you say that?”, “How often has
that happened in the past 12
months?”, “For what reason?”,
etc.; however, limited ability to ask
complex questions (e.g., attribute
rankings, responses totaling to
100%, etc.).
ƒ Generally limited to check-off and fillin responses with only limited openended questions; no further ability to
probe or clarify response.
ƒ Generally limited to check-off and fillin responses with only limited openended questions; no further ability to
probe or clarify response (unless built
into the web-based or e-questionnaire
through a series of intricate “skip
patterns”).
Questionnaire Revisions
ƒ Highly flexible - open to
immediate changes “on the fly”
based on preliminary responses.
ƒ None, once mailed - requires pilot
interviews and pre-tests.
ƒ Somewhat flexible for Web-based;
but not so for e-mail surveys - Webbased surveys are open to impromptu
changes “on the fly”; e-mail surveys
are fairly “fixed” once e-distributed.
Alternate Question Sets
ƒ Full opportunity to add questions,
or skip patterns, as required, at
any time.
ƒ Not recommended - adds too much
complexity/pages and skip patterns to
the printed questionnaire.
ƒ Full opportunity to add questions, or
skip patterns, as required, for webbased (but not for e-mailed) surveys.
Question Order
ƒ Totally controlled by ComputerAided Telephone Interviewing
(CATI) system (including
staggered starts for attribute
testing, etc.).
ƒ No control - respondent can skip
questions and come back later, or
change answers to earlier questions
based on responses to later
questions.
ƒ Highly controlled for web-based
surveys, but not for e-mail surveys respondents must answer web-based
surveys in sequence, but may answer
e-mail surveys in any order.
Length of Interview
ƒ 15 minutes is the conventional
maximum for business interviews.
ƒ 4 - 8 pages is the conventional
maximum for business questionnaires
(but, regardless of length, must always
look extremely professional, and
should be printed on a single sheet of
paper (e.g., 11x 17, 11x 25.5) or in a
bound “booklet” (i.e., not stapled).
ƒ Equivalent of 4 - 8 pages is the
conventional maximum for business
questionnaires (but, regardless of
length, must always look extremely
professional, and should be easy to
follow, back-up and correct, pause,
etc. (i.e., must be user-friendly).
Interviewer Bias
ƒ
ƒ
ƒ
Exists to a small degree - but
generally not an issue.
None - no interviewer is involved.
None - no interviewer is involved.
Page 4 of 5.
P.O. Box 1024 / Westtown, PA 19395 / (610) 399-9717 / (610) 399-9718 (Fax) / www.s4growth.com / wkp@s4growth.com
Comparative Advantages/Disadvantages of Phone, Mail & E-Mail/Web-Based Surveys
(continued)
Item/Area
Telephone Survey
Mail Survey
E-Mail/Web-Based Survey
Pace of Interview
ƒ Set by interviewer - usually
attempts to move as quickly as
possible to avoid respondent
“disconnect”/termination.
ƒ Set by respondent - some risk that
multiple starts and stops will lead to a
“termination”.
ƒ Set by respondent - should be
programmed not to terminate if
respondent pauses too long (e.g., to
take a telephone call, etc.).
Ease of Callback/
Follow-up
ƒ Easiest - quick, generally
inexpensive and can be used to
increase final response rates.
ƒ Difficult, if not impossible on a
respondent-specific basis - too much
of a time lag exists; also, some risk
that the respondent may not believe
his/her responses have been kept
confidential if he/she is recontacted, or
contacted by telephone.
ƒ Not very difficult - if you adequately
track the e-mailouts (or e-invitations to
participate) and responses; easy to
follow-up with “e-reminders”, but some
risk that respondent may not believe
his/her responses have been kept
confidential if recontacted by e-mail.
Supervisory Control
ƒ Centralized, direct and immediate
- remote monitoring is available
for listening in during “live” calls.
ƒ None - respondent is on his or her
own.
ƒ None - respondent is on his or her
own.
Interviewer Training
Requirements
ƒ Full interviewer training and
specific project briefing is
conducted - continuous technical
assistance for interviewers is
available, as needed.
ƒ None - not applicable.
ƒ None - not applicable.
Capturing Sensitive
Information
ƒ More difficult than by mail - since
only limited anonymity may be
perceived by respondents.
ƒ Best mode for sensitive information entirely anonymous (if return envelope
contains no tracking info).
ƒ Mixed mode for sensitive information since e-mail address (or respondent
code/ID number) is traceable.
Interviewer/Respondent
Error
ƒ Depends on interviewer skill and
complexity of questionnaire however, errors are greatly
minimized through the use of the
CATI system.
ƒ Respondent error may be moderate to
high - for respondents answering
without reading/following instructions,
answering too quickly, or skipping
questions, etc.
ƒ Respondent error may be moderate to
high - for respondents answering
without reading/following instructions,
answering too quickly, or skipping
questions, etc.
Respondent Instructions/
Assistance
ƒ Interviewer provides respondent
with initial instructions, scales,
etc., but may not lead or prompt
respondent in any way.
ƒ Cover letter and instructions must be
very clearly articulated - additional
assistance may be made available
through an 800 number or on the web.
ƒ Cover e-letter and instructions must
be very clearly articulated - additional
assistance may be made available
through an 800 number or on the web.
Page 5 of 5.
P.O. Box 1024 / Westtown, PA 19395 / (610) 399-9717 / (610) 399-9718 (Fax) / www.s4growth.com / wkp@s4growth.com