Experiments for Survey Question Evaluation in Multiple-Country Settings Questionnaire administered to the LISS panel Version 1.0 Date Author July 2014 Marije Oudejans T: +31 13 466 8327 E: m.oudejans@uvt.nl P.O. Box 90153, 5000 LE Tilburg, the Netherlands Warandelaan 2, Building T, 2nd floor © CentERdata, Tilburg, 2014 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of the publisher. Table of contents 1 Project description ........................................................................... 2 2 Introduction ................................................................................... 3 3 Codebook ....................................................................................... 4 4 Descriptives .................................................................................. 23 Experiments for Survey Question Evaluation in Multiple-Country Settings 1/26 1 Project description Title: Experiments for Survey Question Evaluation in Multiple-Country Settings Datafile: jw14a Funding sources: MESS Project Investigator: H. Silber, T.H. Stark, J.A. Krosnick, A. Blom Project description: This research explores whether the principles of question design, which are based mostly on American data, can be generalized across countries. This codebook describes the study that has been fielded in the Netherlands. Sample: random sample of 2,796 panel members, aged 16 years and older (pilot participants were excluded) Overview of the response: Selected number of household members 2,796 (100%) Non-response: 539 (19.3%) Response: 2,257 (80.7%) Complete: 2,253 (80.6%) Incomplete: 4 (0.1%) Date of data collection: January 2014 Titel: Experiments for Survey Question Evaluation in Multiple-Country Settings Databestand: jw14a Financiering: MESS Project Onderzoeker: H. Silber, T.H. Stark, J.A. Krosnick, A. Blom Projectbeschrijving: Deze studie onderzoekt of de principes rondom het ontwerp van vragen, welke voornamelijk zijn gebaseerd op Amerikaanse data, kunnen worden gegeneraliseerd naar andere landen. Dit codeboek beschrijft de studie die is uitgevoerd in Nederland. Steekproef: willekeurige steekproef van 2.796 panelleden van 16 jaar en ouder (deelnemers aan de proefstudie zijn uitgesloten) Responsoverzicht: Selectie aantal huishoudleden: 2.796 (100%) Nonrespons: 539 (19,3%) Respons: 2,257 (80,7%) Compleet: 2.253 (80,6%) Incompleet: 4 (0,1%) Datum van dataverzameling: januari 2014 Experiments for Survey Question Evaluation in Multiple-Country Settings 2/26 2 Introduction In January 2014 the LISS panel1 was presented the questionnaire “Social issues”. In October 2013, 165 respondents completed a pilot version of the questionnaire. These panel members were excluded from the selection in January. The questionnaire was presented to a random sample of 2,796 panel members, and it was fully completed by 2,253 respondents (response percentage 80.6%). 1 Reference to LISS panel data in text: This paper makes use of data of the LISS (Longitudinal Internet Studies for the Social sciences) panel managed by CentERdata (Tilburg University, The Netherlands). Experiments for Survey Question Evaluation in Multiple-Country Settings 3/26 3 Codebook This codebook contains the questionnaire as fielded in the LISS panel. The variable names are printed in bold and correspond to the names in the dataset. The questionnaire routing is printed in italics for each variable concerned. open: answer box; no limit to the length of the answer string: answer box that accommodates a maximum amount of characters (255 is standard) empty: question may be left unanswered With numerical variables, wherever the range within which the respondent could choose an answer was not visible to the respondent, this is printed in italics in the codebook. Wherever no limits applied to the range within which to choose an answer, this is indicated in the codebook as ‘integer’. The so-called ‘fills’ (variable text) are indicated between straight brackets []. Variables between curly brackets {} are not part of the database, but the associated questions or texts were part of the questionnaire. nomem_encr2 Number of household member encrypted jw14a_m Year and month of the field work period jw14a001 Random variable A - Version Response Order Experiment 1 1 version 1 2 version 2 3 version 3 4 version 4 jw14a002 Random variable B - Version Response Order Experiment 2 1 version 1 2 version 2 3 version 3 4 version 4 2 Use nomem_encr als key variable to merge the data with data of other LISS panel studies in the LISS Panel Data Archive: http://www.lissdata.nl/dataarchive/ Experiments for Survey Question Evaluation in Multiple-Country Settings 4/26 jw14a003 Random variable C - Version Response Order Experiment 3 1 version 1 2 version 2 3 version 3 4 version 4 jw14a004 Random variable D - Version Acquiescence Experiment 1 1 version 1 2 version 2 3 version 3 4 version 4 jw14a005 Random variable E - Version Acquiescence Experiment 2 1 version 1 2 version 2 3 version 3 4 version 4 jw14a006 Random variable F - Version Acquiescence Experiment 3 1 version 1 2 version 2 3 version 3 4 version 4 jw14a007 Random variable G - Version Acquiescence Experiment 4 1 version 1 2 version 2 jw14a008 Random variable H - Version Question Wording 1 version 1 2 version 2 jw14a009 Random variable I - Version Response Order Experiment 4 1 version 1 2 version 2 Experiments for Survey Question Evaluation in Multiple-Country Settings 5/26 jw14a010 Random variable J - Version No Opinion Experiment 1, 2 en 3 1 version 1 2 version 2 3 version 3 4 version 4 jw14a011 Random variable K - Version Question Balance Experiment 1 1 version 1 2 version 2 3 version 3 4 version 4 jw14a012 Random variable L - Version Question Balance Experiment 2 1 version 1 2 version 2 3 version 3 4 version 4 jw14a013 Random variable M - Version Question Order Experiment 1 1 version 1 2 version 2 jw14a014 Random variable N - Version Question Order Experiment 2 1 version 1 2 version 2 jw14a015 Random variable O - Version Response Order Experiment 5 1 version 1 2 version 2 jw14a016 Random variable P - Version Response Order Experiment 6 1 version 1 2 version 2 Experiments for Survey Question Evaluation in Multiple-Country Settings 6/26 The respondent was not obliged to answer any of the questions. If the respondent clicked on the ‘Next’ button without answering the question, the following soft check was displayed: If you do not want to answer this question, that is ok. However, we would really appreciate it if you would be willing to answer it, because your response to this question will help our study a lot. Please feel free to either answer the question or continue to the next screen. Thank you in advance. {intro} In this questionnaire you will be presented with some general questions about Dutch society. We would like to hear your personal opinion about these issues. Question text dependent on Random variable A jw14a017 if jw14a001=1 Some people say that we will have plenty of oil 25 years from now. Others say that at the rate we are using our oil, it will all be used up in 15 years. Which of these ideas would you guess is most nearly right? 1 There will be plenty of oil in 25 years 2 Oil will be used up in 15 years if jw14a001=2 Some people say that that at the rate we are using our oil, it will be all used up in 15 years. Others say that we will still have plenty of oil 25 years from now. Which of these ideas would you guess is most nearly right? 1 Oil will be used up in 15 years 2 There will be plenty of oil in 25 years if jw14a001=3 Which of these ideas would you guess is most nearly right? We will still have plenty of oil 25 years from now, or at the rate we are using our oil, it will all be used up in 15 years. 1 There will be plenty of oil in 25 years 2 Oil will be used up in 15 years if jw14a001=4 Which of these ideas would you guess is most nearly right? At the rate we are using our oil, it will be all used up in 15 years, or we will still have plenty of oil 25 years from now. we zullen over 25 jaar nog ruimschoots olie hebben. 1 Oil will be used up in 15 years 2 There will be plenty of oil in 25 years Experiments for Survey Question Evaluation in Multiple-Country Settings 7/26 Question text dependent on Random variable B jw14a018 if jw14a002=1 Do you agree or disagree with this idea of how gasoline and oil prices are decided? Each company sets its own prices to meet the competition. 1 Agree 2 Disagree if jw14a002=2 Do you agree or disagree with this idea of how gasoline and oil prices are decided? The oil companies get together and set prices for their products. 1 Agree 2 Disagree if jw14a002=3 Which of these statements comes closest to your own idea of how gasoline and oil prices are decided? Each company sets its own prices to meet the competition, or the oil companies get together and set prices for their products. 1 Each company sets its own prices 2 Oil companies get together if jw14a002=4 Which of these statements comes closest to your own idea of how gasoline and oil prices are decided? The oil companies get together and set prices for their products, or each company sets its own prices to meet the competition. 1 Oil companies get together 2 Each company sets its own prices Experiments for Survey Question Evaluation in Multiple-Country Settings 8/26 Question text dependent on Random variable C jw14a019 if jw14a003=1 Some people feel the government should see to it that all people have adequate housing, while others feel each person should provide for his or her own housing. Which comes closest to how you feel about this? 1 Government should see to adequate housing 2 Let each person provide for own housing if jw14a003=2 Some people feel each person should provide for his or her own housing, while others feel the government should see to it that all people have adequate housing. Which comes closest to how you feel about this? 1 Let each person provide for own housing 2 Government should see to adequate housing if jw14a003=3 Which comes closest to how you feel about this? The government should see to it that all people have adequate housing, or each person should provide for his or her own housing. 1 Government should see to adequate housing 2 Let each person provide for own housing if jw14a003=4 Which comes closest to how you feel about this? Each person should provide for his or her own housing, or the government should see to it that all people have adequate housing. 1 Let each person provide for own housing 2 Government should see to adequate housing Experiments for Survey Question Evaluation in Multiple-Country Settings 9/26 Question text dependent on Random variable D jw14a020 if jw14a004=1 Which in your opinion is more to blame for crime and lawlessness in this country – individuals or social conditions? 1 Individuals more to blame 2 Social conditions more to blame if jw14a004=2 Which in your opinion is more to blame for crime and lawlessness in this country - social conditions or individuals? 1 Social conditions more to blame 2 Individuals more to blame if jw14a004=3 Do you agree or disagree with this statement? Individuals are more to blame than social conditions for crime and lawlessness in this country. 1 Agree 2 Disagree if jw14a004=4 Do you agree or disagree with this statement? Social conditions are more to blame than individuals for crime and lawlessness in this country. 1 Agree 2 Disagree Experiments for Survey Question Evaluation in Multiple-Country Settings 10/26 Question text dependent on Random variable E jw14a021 if jw14a005=1 Do you think the government should or should not see to it that every person has a job and a good standard of living? 1 Government should see to it 2 Government should not see to it if jw14a005=2 Do you agree or disagree with this statement? The government should see to it that every person has a job and a good standard of living. 1 Agree 2 Disagree if jw14a005=3 Do you agree or disagree with this statement? The government should not see to it that every person has a job and a good standard of living. 1 Agree 2 Disagree if jw14a005=4 Do you think the government should not or should see to it that every person has a job and a good standard of living? 1 Government should not see to it 2 Government should see to it Experiments for Survey Question Evaluation in Multiple-Country Settings 11/26 Question text dependent on Random variable F jw14a022 if jw14a006=1 Do you agree or disagree with this statement? Most men are better suited emotionally for politics than are most women. 1 Agree 2 Disagree if jw14a006=2 Do you agree or disagree with this statement? Most men are not better suited emotionally for politics than are most women. 1 Agree 2 Disagree if jw14a006=3 Would you say that most men are better suited emotionally for politics than are most women, that men and women are equally suited, or that women are better suited than men in this area? 1 Men are better suited 2 Men and women equally 3 Women are better suited if jw14a006=4 Would you say that most women are better suited emotionally for politics than are most men, that men and women are equally suited, or that men are better suited than women in this area? 1 Women are better suited 2 Men and women equally 3 Men are better suited Experiments for Survey Question Evaluation in Multiple-Country Settings 12/26 Question text dependent on Random variable G jw14a023 if jw14a007=1 Do you agree or disagree with this statement? Sometimes politics and government seem so complicated that a person like me can’t really understand what’s going on. 1 Agree 2 Disagree if jw14a007=2 Would you say that sometimes politics and government seem so complicated that a person like you can’t really understand what’s going on, or that you really can understand what’s going on? 1 Sometimes so complicated can't understand 2 Can understand all the time Question text dependent on Random variable H jw14a024 if jw14a008=1 There are some people who are against all churches and religion. If such a person wanted to make a speech in your city/town/community against churches and religion, should he be allowed the freedom to speak, or not? 1 Yes, allowed to speak 2 No, not allowed if jw14a008=2 There are some people who are against all churches and religion. If such a person wanted to make a speech in your city/town/community against churches and religion, should he be allowed to speak, or not? 1 Yes, allowed to speak 2 No, not allowed Question text dependent on Random variable I jw14a025 if jw14a009=1 As far as you know, would you say that average temperatures around the world have been higher in the last three years than before that, lower, or about the same? 1 Higher 2 Lower 3 About the same if jw14a009=2 As far as you know, would you say that average temperatures around the world have been lower in the last three years than before that, higher, or about the same? 1 Lower 2 Higher 3 About the same Experiments for Survey Question Evaluation in Multiple-Country Settings 13/26 Question text dependent on Random variable J jw14a026 if jw14a010=1 In general, do you think the courts in this area deal too harshly or not harshly enough with criminals, or do you not have an opinion on that? 1 Too harshly 2 Not harshly enough 3 No opinion if jw14a010=2 In general, do you think the courts in this area deal too harshly or not harshly enough with criminals, or don’t you have enough information about the courts to say? 1 Too harshly 2 Not harshly enough 3 Don’t have enough information to say if jw14a010=3 In general, do you think the courts in this area deal too harshly or not harshly enough with criminals, or don't you know? 1 Too harshly 2 Not harshly enough 3 Don’t know if jw14a010=4 In general, do you think the courts in this area deal too harshly or not harshly enough with criminals? 1 Too harshly 2 Not harshly enough Experiments for Survey Question Evaluation in Multiple-Country Settings 14/26 Question text dependent on Random variable J jw14a027 if jw14a010=1 Do you feel that almost all of the people running the government are smart people, or do you think that quite a few of them don’t seem to know what they are doing, or do you not have an opinion on that? 1 Are smart 2 Don’t know what they’re doing 3 No opinion if jw14a010=2 Do you think that quite a few of the people running the government don’t seem to know what they are doing, or do you feel that almost all of them are smart people? 1 Don’t know what they’re doing 2 Are smart if jw14a010=3 Do you feel that almost all of the people running the government are smart people, or do you think that quite a few of them don’t seem to know what they are doing, or don't you know? 1 Are smart 2 Don’t know what they’re doing 3 Don’t know if jw14a010=4 Do you feel that almost all of the people running the government are smart people, or do you think that quite a few of them don’t seem to know what they are doing? 1 Are smart 2 Don’t know what they’re doing Experiments for Survey Question Evaluation in Multiple-Country Settings 15/26 Question text dependent on Random variable J jw14a028 if jw14a010=1 Do you think that quite a few of the people running the government are corrupt, not very many are, hardly any of them are corrupt, or do you not have an opinion on that? 1 Quite a few 2 Not many 3 Hardly any 4 No opinion if jw14a010=2 Do you think that quite a few of the people running the government are corrupt, not very many are, hardly any of them are corrupt, or do you not have enough information about the government to say? 1 Quite a few 2 Not many 3 Hardly any 4 Don’t have enough information if jw14a010=3 Do you think that quite a few of the people running the government are corrupt, not very many are, or hardly any of them are corrupt, or don't you know? 1 Quite a few 2 Not many 3 Hardly any 4 Don’t know if jw14a010=4 Do you think that quite a few of the people running the government are corrupt, not very many are, or hardly any of them are corrupt? 1 Quite a few 2 Not many 3 Hardly any Experiments for Survey Question Evaluation in Multiple-Country Settings 16/26 Question text dependent on Random variable K jw14a029 if jw14a011=1 If there is a serious fuel shortage this winter, do you think there should be a law requiring people to lower the heat in their homes, or do you oppose such law? 1 Should be a law 2 Oppose such a law if jw14a011=2 If there is a serious fuel shortage this winter, do you think there should be a law requiring people to lower the heat in their homes? 1 Yes, should be a law 2 No, not a law if jw14a011=3 If there is a serious fuel shortage this winter, do you think there should be a law requiring people to lower the heat in their homes, or do you oppose such a law because it would be too difficult to enforce? 1 Should be a law 2 Oppose such a law if jw14a011=4 If there is a serious fuel shortage this winter, do you oppose that there should be a law requiring people to lower the heat in their homes, or do you think there should be such law? 1 Oppose such a law 2 Should be a law Experiments for Survey Question Evaluation in Multiple-Country Settings 17/26 Question text dependent on Random variable L jw14a030 if jw14a012=1 If there is a union at a particular company or business, do you think that all the workers there should be required to be union members, or are you opposed to this? 1 Yes, all in union 2 No, oppose if jw14a012=2 If there is a union at a particular company or business, do you think that all the workers there should be required to be union members? 1 Yes 2 No if jw14a012=3 If there is a union at a particular company or business, do you think that all the workers there should be required to be union members, or should it be left to the individuals to decide whether or not he wants to be in the union? 1 All should be required 2 Left to individuals if jw14a012=4 If there is a union at a particular company or business, are you opposed to all the workers there being required to be union members, or do you think that all the workers there should be required to be union members? 1 Oppose 2 All in union Experiments for Survey Question Evaluation in Multiple-Country Settings 18/26 Question text dependent on Random variable M jw14a031 if jw14a013=1 Do you think it should be possible for a pregnant woman to obtain a legal abortion if she is married and does not want any more children? 1 Yes 2 No if jw14a013=2 Do you think it should be possible for a pregnant woman to obtain a legal abortion if there is a strong chance of serious defect in the baby? 1 Yes 2 No Question text dependent on Random variable M jw14a032 if jw14a013=1 Do you think it should be possible for a pregnant woman to obtain a legal abortion if there is a strong chance of serious defect in the baby? 1 Yes 2 No if jw14a013=2 Do you think it should be possible for a pregnant woman to obtain a legal abortion if she is married and does not want any more children? 1 Yes 2 No Experiments for Survey Question Evaluation in Multiple-Country Settings 19/26 Question text dependent on Random variable N jw14a033 if jw14a014=1 Do you think labor unions should be permitted to spend their money to help elect or defeat candidates for political offices? 1 Yes 2 No if jw14a014=2 Do you think businesses should be permitted to spend their money to help elect or defeat candidates for political offices? 1 Yes 2 No Question text dependent on Random variable N jw14a034 if jw14a014=1 Do you think businesses should be permitted to spend their money to help elect or defeat candidates for political offices? 1 Yes 2 No if jw14a014=2 Do you think labor unions should be permitted to spend their money to help elect or defeat candidates for political offices? 1 Yes 2 No Experiments for Survey Question Evaluation in Multiple-Country Settings 20/26 Question text dependent on Random variable O jw14a035 if jw14a015=1 Generally speaking, would you say that most people can be trusted or that you can’t be too careful in dealing with people? 1 Most people can be trusted 2 Can’t be too careful if jw14a015=2 Generally speaking, would you say that you can’t be too careful in dealing with people or that most people can be trusted? 1 Can’t be too careful 2 Most people can be trusted Question text dependent on Random variable P jw14a036 if jw14a016=1 Do you agree or disagree that inequality continues to exist because it benefits the rich and powerful? Would you say... 1 Strongly agree 2 Agree 3 Neither agree nor disagree 4 Disagree 5 Strongly disagree if jw14a016=2 Do you disagree or agree that inequality continues to exist because it benefits the rich and powerful? Would you say… 1 Strongly disagree 2 Disagree 3 Neither agree nor disagree 4 Agree 5 Strongly agree Experiments for Survey Question Evaluation in Multiple-Country Settings 21/26 jw14a037 – jw14a041 Note: Please complete the questionnaire fully until you are returned to the initial screen. Only then will the system register the questionnaire as fully completed. Finally. What did you think of this questionnaire? 1 = certainly not 5 = certainly yes jw14a037 Was it difficult to answer the questions? jw14a038 Were the questions sufficiently clear? jw14a039 Did the questionnaire get you thinking about things? jw14a040 Was it an interesting subject? jw14a041 Did you enjoy answering the questions? 1 certainly not 22 33 44 5 certainly yes jw14a042 Starting date questionnaire jw14a043 Starting time questionnaire jw14a044 End date questionnaire jw14a045 End time questionnaire computed variable jw14a046 Duration of questionnaire in seconds Experiments for Survey Question Evaluation in Multiple-Country Settings 22/26 4 Descriptives N jw14a_m Year and month Minimum Maximum Mean Std. Deviation 2257 201401 201401 201401,00 ,000 2257 1 4 2,50 1,121 2257 1 4 2,49 1,122 2257 1 4 2,47 1,109 2257 1 4 2,51 1,130 2257 1 4 2,52 1,129 2257 1 4 2,46 1,134 2257 1 2 1,50 ,500 2257 1 2 1,51 ,500 2257 1 2 1,51 ,500 2257 1 4 2,50 1,116 2257 1 4 2,52 1,112 2257 1 4 2,51 1,111 2257 1 2 1,50 ,500 of the field work period jw14a001 Random variable A - Version Response Order Experiment 1 jw14a002 Random variable B - Version Response Order Experiment 2 jw14a003 Random variable C - Version Response Order Experiment 3 jw14a004 Random variable D - Version Acquiescence Experiment 1 jw14a005 Random variable E - Version Acquiescence Experiment 2 jw14a006 Random variable F - Version Acquiescence Experiment 3 jw14a007 Random variable G - Version Acquiescence Experiment 4 jw14a008 Random variable H - Version Question Wording jw14a009 Random variable I - Version Response Order Experiment 4 jw14a010 Random variable J - Version No Opinion Experiment 1, 2 en 3 jw14a011 Random variable K - Version Question Balance Experiment 1 jw14a012 Random variable L - Version Question Balance Experiment 2 jw14a013 Random variable Experiments for Survey Question Evaluation in Multiple-Country Settings 23/26 M - Version Question Order Experiment 1 jw14a014 Random variable 2257 1 2 1,51 ,500 2257 1 2 1,49 ,500 2257 1 2 1,50 ,500 2257 1 2 1,50 ,500 2255 1 2 1,48 ,500 2242 1 2 1,47 ,499 2247 1 2 1,48 ,500 2243 1 2 1,54 ,499 2248 1 3 1,82 ,450 2245 1 2 1,33 ,469 2242 1 2 1,15 ,353 2251 1 3 2,15 ,850 2247 1 3 2,17 ,410 2246 1 3 1,78 ,682 N - Version Question Order Experiment 2 jw14a015 Random variable O - Version Response Order Experiment 5 jw14a016 Random variable P - Version Response Order Experiment 6 jw14a017 dependent Question on text Random variable A jw14a018 dependent Question on text Random variable B jw14a019 dependent Question on text Random variable C jw14a020 dependent Question on text Random variable D jw14a021 dependent Question on text Random variable E jw14a022 dependent Question on text Random variable F jw14a023 dependent Question on text Random variable G jw14a024 dependent Question on text Random variable H jw14a025 dependent Question on text Random variable I jw14a026 dependent Question on text Random variable J jw14a027 dependent Question on text Random Experiments for Survey Question Evaluation in Multiple-Country Settings 24/26 variable J jw14a028 Question dependent on text 2248 1 4 2,57 ,971 2243 1 2 1,57 ,495 2244 1 2 1,69 ,461 2248 1 2 1,18 ,385 2241 1 2 1,24 ,429 2238 1 2 1,89 ,316 2234 1 2 1,86 ,346 2244 1 2 1,49 ,500 2248 1 5 2,88 1,129 2253 1 5 2,42 1,313 2253 1 5 3,88 1,049 2253 1 5 3,25 1,162 an 2253 1 5 3,61 1,061 enjoy 2253 1 5 3,68 1,018 time 2257 00:00:07 23:59:47 16:11:16 04:26:28,293 time 2253 00:00:04 23:59:54 16:14:47 04:30:20,476 Random variable J jw14a029 Question dependent on text Random variable K jw14a030 Question dependent on text Random variable L jw14a031 Question dependent on text Random variable M jw14a032 Question dependent on text Random variable M jw14a033 Question dependent on text Random variable N jw14a034 Question dependent on text Random variable N jw14a035 Question dependent on text Random variable O jw14a036 Question dependent on text Random variable P jw14a037 Was it difficult to answer the questions? jw14a038 Were the questions sufficiently clear? jw14a039 Did questionnaire the get you thinking about things? jw14a040 Was it interesting subject? jw14a041 Did you answering the questions? jw14a043 Starting questionnaire jw14a045 End questionnaire Experiments for Survey Question Evaluation in Multiple-Country Settings 25/26 jw14a046 Duration of 2253 47 1709621 7955,45 84186,090 questionnaire in seconds Valid N (listwise) 2112 Experiments for Survey Question Evaluation in Multiple-Country Settings 26/26