Thursday 30 th October 2014

• In addition to the online course extract from
Aldridge and Levine (2001), there are useful chapters in Oppenheim (1992) and De Vaus
(2001).
• While the research instrument in ‘face-to-face’ surveys is arguably more correctly referred to as the ‘interview schedule’, the literature often incorporates such instruments under the heading
‘questionnaire design’.

(i) “Mapping the semantic domain”
This stage involves identifying for each of the concepts fundamental to the proposed research its components (i.e. sub-concepts), context (i.e. related and ‘similar but different’ concepts), and parameters (i.e. the conditions which must be present for the concept to apply).
(ii) Formulating the question ideas
This stage involves identifying the range of issues on which specific questions need to be asked, i.e. it identifies that a question has to be asked relating to a particular sub-concept: “We need to ask a question about...”. At this stage one is moving from concepts towards indicators.
(iii) Writing the questions
At this stage the specific wording of the questions is determined
(iv) Assembling the questionnaire
At this stage the questions are assembled into a coherent, structured questionnaire.
Source: Halfpenny, P., Parthemore, J., Taylor, J. and Wilson, I. 1992. ‘A knowledge based system to provide intelligent support for writing questionnaires’. In Westlake, A. et al. (eds)
Survey and Statistical Computing. London: North Holland.

De Vaus refers to
(i) Measures of the dependent variables
(ii) Measures of the independent variables
(iii) Measures of “test” variables (i.e. variables that intervene between the independent and dependent variables, or which are temporally prior and possibly causally related to both)
(iv) Background measures (i.e. ‘demographic’ variables)

• Unstructured discussions with potential respondents/‘key informants’. The idea of this qualitative work is to give the researcher a more complete picture of the topic, and the language and perspectives of respondents.
• A search for existing questions on the topic, (e.g. via the ‘Question Bank’). However, inheriting measures from previous research carries a risk of reflecting the pre-conceptions of earlier researchers.
• Piloting the draft questionnaire, to check for problems and to allow questions to be refined.

• “Avoid putting ideas into the respondent’s mind early in the interview if we need spontaneous responses later on”
(Oppenheim).
• Beware of asking questions that unwittingly reveal the researcher’s attitude to the topic.
• “A question that strikes the respondent as rude or inconsiderate may affect not only his [sic] reply to that particular question but also his [sic] attitude to the next few questions and to the survey as a whole” (Oppenheim)
• Make the questionnaire attractive to the respondent by making it interesting and of obvious relevance to its stated purpose.

Oppenheim outs forward the following, chronological framework (derived from a
Gallup schema):
• Respondent’s awareness of issue.
• Respondent’s general feelings about issue.
• Respondent’s views on specific aspects of issue.
• Reasons for respondent’s views.
• Strength of respondent’s views.

• ‘Open’ questions, where the respondent’s verbatim answer to the question is recorded in full, are easy to ask, less easy to answer and difficult to analyse. The emphasis is on the respondent’s perspective, but there is still the possibility that answers will reflect what is uppermost in respondents’ minds.
• ‘Closed’ questions are easier to quantify, but result in a loss of spontaneity and expressiveness, and the ‘forced’ choice of answers may result in bias (and shift the balance towards the researcher’s perspective).
• A compromise is to ask an ‘Open’ question and then a similar
‘Closed’ question later in the questionnaire/interview.
• Oppenheim comments that “All closed questions should start their careers as open ones, except those where certain alternatives are the only ones possible”.

•
• Awareness of issue
Strength of views on issue

Closed,
•
• General feelings on issue

Open,
• Views on specific aspects of issue

Closed,
Reasons for views on issue

Open or Closed,

Closed.
Source: De Vaus (1986), adapted from Gallup.

• A Likert scale: e.g. a range of answers from ‘Strongly agree’ through to ‘Strongly disagree’.
• A semantic differential scale: A range of positions between two extremes of a continuum (e.g. ‘Caring’ through to
‘Uncaring’).
• A checklist: e.g. a list of leisure activities.
• A ranking of items: e.g. placing the most important attributes of a potential partner in order.
• A choice between statements: e.g. a choice of responses to the acquisition of the knowledge that one’s best friend’s partner is being unfaithful to them.

•
•
•

• Researchers sometimes want to measure some latent characteristic of their respondents, e.g. whether they have a
‘traditional’ or a ‘modern’ viewpoint on couple relationships.
• This is often done by asking a number of questions which each tap that characteristic and, when aggregated, collectively do so in more reliable way.

• How does one know that a measure constructed by aggregating various items to give a scale is measuring the ‘right’ quantity, i.e. is a valid measure?
• How does one ensure that what a measure is measuring is unidimensional, i.e. that it is not a composite of measures of two or more underlying concepts?
• How does one assess which items need to be included to maximise the reliability of the measure?
• And how does one assess the overall reliability of the scale?

• For a discussion of assessing various forms of
validity, see Oppenheim (1992).
• Unidimensionality can be assessed using a technique called factor analysis, see DeVellis
(2003).
• Reliability can be assessed using a measure called Cronbach’s alpha (see De Vaus, 2001;
DeVellis, 2003).

Factor analysis generates a set of underlying factors which successively maximise the amount of (remaining) variation in the items that they can explain. If a scale is working properly unidimensionally, then the first factor will explain a high proportion of the variation, and the subsequent factors similar, small amounts.

• According to DeVellis (2003: 95), “Alpha is an indication of the proportion of variance in the scale scores that is attributable to the true score”.
• Items are chosen for inclusion so as to maximise that proportion, and if they have relatively high correlations with the rest of the items within the scale (viewed collectively).
• De Vaus (2001) suggests that a value of at least 0.7 is preferable.

• Should all the items in the scale be treated as of equal importance? Or should their values be added in such a way as to increase/ decrease the relative importance of some items?
• Are the gaps between the values that a variable can take uniform in meaning?