FOLIO
Facilitated Online Learning as an Interactive Opportunity
Designing, Conducting and Analysing Surveys and
Questionnaires (ASQ)
Choosing the question type (part 2)
Closed Questions, Open-Ended Questions & Likert Scales
Introduction
This is the second of our briefings on choosing question types. Below we explore the use of closed
questions, open-ended questions and Likert scales.
Closed Questions
Closed questions limit the respondents to a range of fixed responses on particular points
to gather quantitative information about students' opinions (1). For example:
1. 'Do you find the mediated literature search service to be useful?'
(select one answer):
'yes'
'no'
'don't know'
2. 'Having completed the critical appraisal course, I feel that I know more about the
topic'
(select one answer):
Strongly disagree
Disagree
No opinion
Agree
FOLIO
Facilitated Online Learning as an Interactive Opportunity
Strongly agree
Closed questions are useful because they enable questionnaire responses to be
analysed easily (for e.g. the results of closed questions can be analysed simply by
adding up the numbers of respondents that replied 'Yes' or 'No' to a question). From the
point of view of respondents, they are also very quick and easy to answer (1). However,
because responses are limited to a set of alternatives, closed questions have the
potential to be biased. (2) In addition, respondents may feel the need to provide an
explanation about their answer, but are unable to due to the restrictive nature of the
question type.
Open-Ended Questions
Open-ended questions are qualitative and allow the exploration of many themes as respondents
can write about any issues they wish in response to the question. It is argued that open-ended
questions are the most potentially valuable type of survey question as it is usually the only part of a
questionnaire where respondents are free to address the issues that concern them, rather than the
issues that concern those conducting the survey (3). For example, if you are evaluating user
satisfaction of your service, the open-ended questions often tell you what aspects of service are of
most concern to your users. Whereas you might find that if you are asking respondents to rate
aspects of service on a scale, they might rate four or five items as low. With an open-ended
version of the question, respondents may only have strong opinions about one aspect of the
service - this then tells you where to start in improving the service. (3) In answering open-ended
questions, respondents may well raise issues that you have not anticipated, or even thought of (4).
However open-ended questions can also create some problems precisely because of the freedom
with which the respondents answer. For example, some respondents may answer ambiguously or
in a contradictory fashion. Some may have misunderstood the question and answer something
unrelated, or someone's answer could get misunderstood by the person analysing the
questionnaires. (5) Also you may find that those respondents without strong opinions on a
particular question may write very little or nothing at all. Answers to open-ended questions are
both time consuming for respondents to complete and for you to analyse when you are collating
your data (2). If questionnaires responses are handwritten, you may find that some are illegible
and therefore will be wasted (4).
Tips on open-ended questions


Open-ended questions allow for thoughts, feelings and experiences to be recorded (4).
It is a good idea to use open-ended questions when it is not possible to compile a
comprehensive list of alternative options for respondents to choose from (6).
FOLIO
Facilitated Online Learning as an Interactive Opportunity





Open-ended questions should begin with words such as "why" and "how" or phrases such
as "What do you think about..." (5)
Make sure you leave enough space for people to fill in their responses to questions,
however also be wary of leaving too much blank space as respondents may find this
daunting and be discouraged from responding (4).
Open-ended questions can be used as a follow-up to closed questions in order for
respondents to qualify their answers (2). For example you may ask respondents to rate an
aspect of the library and information service on a scale of 1-5 and then ask an open-ended
question for their reasons why they have rated the service at this point on the scale.
Be wary of including too many open-ended questions on one questionnaire as you may find
that this negatively affects the response rate (7).
If when you design your questionnaire you find that you only have open-ended questions,
consider instead asking them via a semi-structured interview. If time and/or resources
permit this, reconsider your questions and try to restructure some of them into closed
questions. (4)
Examples of open-ended questions



[On an information skills training course evaluation] Why did you enrol on this course?
[On a library and information service user survey] Are there any services not currently
available that you would like the library to provide?
[At the end of a questionnaire] Are there any additional comments you would like to make?
Likert Scales
The Likert scale was developed by Rensis Likert in 1932. Questions using Likert scales usually
present a statement and respondents are asked to express agreement or disagreement on a scale,
generally consisting of five points as below:
Strongly Agree
1
Agree
2
Neutral
3
Disagree
4
Strongly Disagree
5
The number assigned to each response becomes the value for that response. (8) This scale can
also be reversed so that 1 = Strongly Disagree and 5 = Strongly Agree (9). This type of scale may
also be replaced by a combination of written and numerical such as:
Strongly Agree 1............2............3..............4.............5 Strongly Disagree
FOLIO
Facilitated Online Learning as an Interactive Opportunity
However, bear in mind that there is evidence that more reliable and valid data is yielded if all the
points on the scale are labelled. It also helps to ensure constancy of meaning, as different
respondents may attach different meanings to unlabelled points. (10) Also note that the numbers
do not have to appear on the questionnaire that the respondents fill in, it is acceptable for the scale
to consist solely of strongly agree, agree etc.
As shown above, Likert scales give an idea of how strongly respondents feel about something, and
therefore are potentially more useful than a "yes/no" type question (11). As well as "agree v
disagree" type dimensions, Likert scales can also be used for:




Satisfied v Dissatisfied
Important v Unimportant
Always v Never (10)
Successful v Unsuccessful
Questions using Likert Scales are relatively simple to construct and they are easy for respondents
to complete (8), so you may find the response rate to these type of questions is higher than openended questions. When analysing the responses from Likert scales, the data can be treated as
quantative data and is therefore easier to analyse statistically (11).
Note that Likert scales are sometimes referred to as summated scales or summated rating.
Some issues with Likert Scales
There is some debate over whether or not to include a neutral point on the scale - some
researchers avoid respondents "sitting on the fence" by giving an even number of choices (i.e. 4 or
6) (9), this is know as a "forced choice" method (12).
The reliability of the Likert scale tends to increase with the number of items used. However as the
number of items increases, so does the time taken to complete the question and this may
demotivate your respondents. There is no hard and fast rule to determine the final number of items
in a scale and this will reflect the nature and complexity of the question (9), but it is generally
accepted that between 5 and 7 points on a scale is optimal (10).
Likert scales may be subject to distortion from several causes. Respondents may:



Avoid using extreme response categories, i.e. points 1 and 5 on the scale (known as central
tendency bias).
Agree with statements as they are presented (known as acquiescence response bias) - i.e.
if the statement is positive (e.g. "I find the library and information service useful")
respondents are more likely to answer in the affirmative.
Try to present themselves/their opinions in a more favourable light (known as social
desirability bias). (12)
FOLIO
Facilitated Online Learning as an Interactive Opportunity
Examples of Likert Scales
[On an information skills training course evaluation]
How difficult did you find this training session? (1=very easy, 5= very difficult):
(Please circle your choice) 1
2
3
4
5
[On a library and information service user survey]
Please indicate on the scale your opinion of the following statement:
The inter-library loans service provided by the library is important to me.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
Examples of library user surveys:





Evidence Base. Library user survey
London School of Economics. Information skills training evaluation form
Middlesex University. Information Skills Training Evaluation Form
University of Manchester Business School. Database training evaluation form
Essex County Council. 'Speakup' Library Services Questionnaire
References
1. Soroka, S.N. (2002) Number of Responses and the Most Important Problem [online] [Accessed
May 2006]
2. Galloway, K. (1997) Advantages and Disadvantages of Closed and Open-Ended Questions
[online] [Accessed May 2006]
3. Fitzgerald, J. (1996) The Best Questions to Ask [online] [Accessed May 2006]
4. Boynton, P.M. (2002) CHAIN Conference, Sunningdale - February 2002 [online] [Accessed May
2006]
FOLIO
Facilitated Online Learning as an Interactive Opportunity
5. Cano, V. (2000) Questionnaire Design [online] [Accessed May 2006]
6. Leung WC. How to design a questionnaire. StudentBMJ 2001; 9: 187-9. [online] [Accessed May
2006]
7. Van Bennekom, F. (2006) CRM applications, solutions, advice and tips [online] [Accessed May
2006]
8. Page-Bucci, H. (2003) The value of Likert scales in measuring attitudes of online learners
[online] [Accessed May 2006]
9. University of Salford (2006) Attitude Scales [online] [Accessed May 2006]
10. University of Surrey (2005) Questionnaire Design 2: Attitudes, Question order, Filtering etc.
[online] [Accessed May 2006]
11. Holah, M. (2006) Rating Scales [online] [Accessed May 2006]
12. Wikipedia (2006) Likert scale [online] [Accessed May 2006]
Further Reading
Farrell, S., Bannister, J., Ditton, J. & Gilchrist, E. (1997) Open and Closed Question. Social
Research Update Issue 17 (Summer 1997) [online] [Accessed May 2006]
Jamieson, S. (2004) Likert scales: how to (ab)use them. Medical Education; 38 (12): 1212-1218.
Mogey, N. (1999) So You Want to Use a Likert Scale? [online] [Accessed May 2006]