NEW PROCEDURES FOR
CODING OPEN-ENDED
SURVEY DATA
Matthew K
BERENT
University of Michigan
Jon A
KROSNICK
Stanford University
Arthur LUPIA
University of Michigan
IN THIS WEBINAR,
YOU WILL LEARN ABOUT…
 How and why open-ended questions are used in surveys
IN THIS WEBINAR,
YOU WILL LEARN ABOUT…
 How and why open-ended questions are used in surveys
 How we recognized problems with conventional practice
IN THIS WEBINAR,
YOU WILL LEARN ABOUT…
 How and why open-ended questions are used in surveys
 How we recognized problems with conventional practice
 Newly developed best practices
IN THIS WEBINAR,
YOU WILL LEARN ABOUT…
 How and why open-ended questions are used in surveys
 How we recognized problems with conventional practice
 Newly developed best practices
 An example of implementing these practices
IN THIS WEBINAR,
YOU WILL LEARN ABOUT…
 How and why open-ended questions are used in surveys
 How we recognized problems with conventional practice
 Newly developed best practices
 An example of implementing these practices
 Why being transparent about your procedure enhances your
survey’s public value
QUESTION T YPES
1. Numbers.
QUESTION T YPES
1. Numbers.
2. Categorical questions with unlimited universes.
QUESTION T YPES
1. Numbers.
2. Categorical questions with unlimited universes.
3. Knowledge quiz questions.
QUESTION T YPES
1. Numbers.
2. Categorical questions with unlimited universes.
3. Knowledge quiz questions.
 Much literature: These types of measurements are done more
accurately with open-ended questions than with closed -ended
questions.
QUESTION T YPES
1. Numbers.
2. Categorical questions with unlimited universes.
3. Knowledge quiz questions.
 Much literature: These types of measurements are done more
accurately with open-ended questions than with closed -ended
questions.
 Categories 2 and 3 require coding.
ANES: OPEN-ENDED QUESTIONS
 Is there anything in par ticular about JOHN MCCAIN that might
make you want to vote for him?
ANES: OPEN-ENDED QUESTIONS
 Is there anything in par ticular about JOHN MCCAIN that might
make you want to vote for him?
 Why do you think Barack Obama won the Presidential
election?
ANES: OPEN-ENDED QUESTIONS
 Is there anything in par ticular about JOHN MCCAIN that might
make you want to vote for him?
 Why do you think Barack Obama won the Presidential
election?
 What do you think is the most impor tant political problem
facing the United States today ?
T YPICAL PRACTICE
The interviewer transcribes responses
T YPICAL PRACTICE
The interviewer transcribes responses
The coder later:
T YPICAL PRACTICE
The interviewer transcribes responses
The coder later:
 Is given a list of coding categories but little or no
instructions.
T YPICAL PRACTICE
The interviewer transcribes responses
The coder later:
 Is given a list of coding categories but little or no
instructions.
 Reads transcribed answers.
T YPICAL PRACTICE
The interviewer transcribes responses
The coder later:
 Is given a list of coding categories but little or no
instructions.
 Reads transcribed answers.
 Assigns each answer to a category.
T YPICAL RESULT
Most Important Problem – CBS/NYT June, 2013
Economy/Jobs
Budget/National debt
Health care
Immigration
Religious values
Partisan politics
Big government
Values/Moral values
Miscellaneous government issues
Politicians
Barack Obama
Education
War/Peace
Miscellaneous economic issues
Guns
Business ethics
Community/Lack of unity
Terrorism
34%
6%
5%
3%
3%
3%
3%
3%
2%
2%
2%
1%
1%
1%
1%
1%
1%
1%
Republicans
Income gap
Miscellaneous government programs
Welfare
Taxes
Crime
Miscellaneous social issues
Foreign policy
Poverty
Foreign aid
Defense
Personal finances
Natural disasters
Environment
Nothing
Other
Don't know/No answer
1%
1%
1%
1%
1%
1%
1%
1%
1%
1%
1%
1%
1%
1%
1%
9%
3%
PROBLEMS
WITH ANES CODING
LUSKIN (2002)
“The verdict is stunningly, depressingly
clear:
LUSKIN (2002)
“The verdict is stunningly, depressingly
clear:
most Americans know very little about
politics…”
CONCLUSION BASED ON OPEN QUESTIONS
 ANES:
 “What job or political office does William Rehnquist hold?”
CONCLUSION BASED ON OPEN QUESTIONS
 ANES:
 “What job or political office does William Rehnquist hold?”
 Correct answers:
12% of the respondents.
PUBLIC POLLS
IMPLICATIONS FOR POLITICS
Elected representatives should ignore
public opinion?
IMPLICATIONS FOR POLITICS
Elected representatives should ignore
public opinion?
Why bother spending money on surveys
measuring the public’s opinions about
policy issues?
Should we believe this?
SCHOLARS COULD NOT CHECK THE
VALIDIT Y OF THIS RESULT
ANES didn’t release to the public:
SCHOLARS COULD NOT CHECK THE
VALIDIT Y OF THIS RESULT
ANES didn’t release to the public:
 the coding instructions
SCHOLARS COULD NOT CHECK THE
VALIDIT Y OF THIS RESULT
ANES didn’t release to the public:
 the coding instructions
or
the answer transcriptions.
ANES INSTRUCTIONS TO THE CODER
“We are strict regarding acceptable answers.
We will accept ONLY ‘Chief Justice.’
‘Justice’ alone is definitely not acceptable.
The court must be ‘the Supreme Court’.
“Chief Justice of the Court” won’t do.
IN FACT …
In addition to the 12% who said “Chief
Justice” and “Supreme Court”, …
IN FACT …
In addition to the 12% who said “Chief
Justice” and “Supreme Court”, …
30% identified Rehnquist as a Supreme
Court justice.
26% of respondents:
 said that Rehnquist was a judge
or
 said that he was on the Supreme Court
THE IMPACT OF INSTRUCTIONS
No wonder we characterize most Americans as
knowing little or nothing about politics.
ONE LESSON LEARNED ALREADY
Because ANES was not transparent about
its coding instructions,
those surveys may have misled scholars
about the extent of Americans’ knowledge
about politics.
ANOTHER INDICATION
THAT STANDARD
CODING MAY NOT BE
OPTIMAL
Dif ferences
Between
Organizations
Pew Research Center - Most Important Problem
01 Economy (unsp)
02 Unemployment/lack of jobs/ Job cuts
03 Risky bank loans/sub prime loans/people taking on too
much debt
04 Inflation/difference between wages/costs
05 Drop in retirement accounts (401K)/stock market
06 National debt/budget/deficit/balanced budget
07 The bailout
8 The stimulus
10 Taxes (unsp)
11 Financial crisis/credit crunch/banking situation
12 Housing market/foreclosures
13 Wallstreet/corporate America
14 Education/schools/affording education
15 A/O economic issues mentions
16 Finances
18 Crime/Violence/gangs/justice system (unsp)
20 Drugs/alcohol
25 Abortion
26 Social Security
27 Issues relating to the elderly
28 Welfare abuse
29 Race relations/Racism/Racial profiling
31 Morality/religion/family values (unsp)
33 Homelessness
34 Poverty/Hunger/Starvation
37 Healthcare (costs/accessibility)
43 Dissatisfaction with government/politics/Scandal/corruption in
govt
46 Partisanship/the parties
47 Immigration/immigration situation/foreigners
50 Environment/pollution/Global warming
52 A/O social/domestic issues mentions
56 Too much foreign aid/spend money at home
57 Defense issues/national and homeland security/military &
defense spending
58 Trade/Jobs moving overseas
59 US foreign policy/International relations Americas image
overseas
60 Iraq / War in Iraq
61 China
62 Terrorism
67 A/O international/foreign issues mentions
70 Recession/Slowing of the economy
76 Energy costs/Rising gas/heating prices
77 Middle East situation
78 Many things/Everything
79 Uneven distribution of wealth
80 Peace in the world/Peace
86 More regulation of markets
89 Oil dependence/Energy policy and alternatives
90 Obama
91 Govt control/overreach - socialism
Gallup Organization - Most Important Problem
6 Economy (General)
7 Unemployment-Jobs
8 Federal Budget Deficit-Debt
9 Taxes
10 Foreign Trade-Trade Deficit
11 Cost of Living-Inflation
12 Recession
13 Other Economic
14 Crime-Violence
15 Health Care-Hospitals
16 Drugs
17 Poverty-Hunger-Homelessness
18 Ethical-Moral-Religious Decline
19 Access to Education
20 AIDS
21 Medicare Increases-Senior Citizen
Insurance
22 International Problems-Foreign Affairs
23 Government-President Clinton-CongressPoliticians
24 Foreign Aid-Focus Overseas
25 Race Relations
26 Immigration-Illegal Aliens
27 Welfare
28 Environment-Pollution
29 Iraq-Saddam Hussein
30 Kosovo-Serbia-Yugoslavia-Milosevic
31 School Shootings
32 Gun Laws too Weak-Availability of Guns
33 Gun Control-Gun Laws Too Strong
34 Abortion Issues
35 Social Security Issues
36 Children’s Needs
37 Lack of a Military/Defense
38 Fuel/Oil Prices
39 Dishonesty/Lack of Integrity
40 Judicial System/Courts/Laws
41 National Security
42 Gap Between the Rich and Poor
43 Care for the Elderly
44 Wage Issues
45 Fear of War
46 Breakdown of the Family
47 Cancer/Diseases
48 Child Abuse
49 Advancement of computers/technology
50 Lack of energy Sources
51 Gun control/Guns
52 Costs associated with health insurance
53 Overpopulation
54 Way children are raised
55 Presidential choices/election year
56 Lack of respect for each other
57 Poor leadership/corrupt
58 Abuse of power
59 Lack of money
60 Education
61 The media
62 Election/Election Reform
63 Unifying the Country
64 Situation with China
65 Energy crisis
66 Terrorism
67 Feeling of Fear in this Country
68 Corporate Corruption
69 Space Shuttle Disaster
70 Iraq
71 Gay marriage/Homosexual rights
72 Licenses for the Undocumented
73 Natural disaster response
75 War-Conflict in the Middle East
MOST IMPORTANT PROBLEM: ECONOMY
MOST IMPORTANT PROBLEM: ECONOMY
MOST IMPORTANT PROBLEM: ECONOMY
MOST IMPORTANT PROBLEM: ECONOMY
ANOTHER INDICATION
THAT SURVEY PRACTICE
MAY NOT BE OPTIMAL
What
Psychologists
Do
PSYCHOLOGY
Multiple independent coders
PSYCHOLOGY
Multiple independent coders
Elaborate written instructions
PSYCHOLOGY
Multiple independent coders
Elaborate written instructions
Assess levels of agreement
PSYCHOLOGY
Multiple independent coders
Elaborate written instructions
Assess levels of agreement
Negotiate disagreements
ANES AND EVERY OTHER MAJOR SURVEY
ORGANIZATION WE TALKED TO
One coder
ANES AND EVERY OTHER MAJOR SURVEY
ORGANIZATION WE TALKED TO
One coder
Given coding categories and can add
categories if needed.
ANES AND EVERY OTHER MAJOR SURVEY
ORGANIZATION WE TALKED TO
One coder
Given coding categories and can add
categories if needed.
Little or no written instructions
ANES AND EVERY OTHER MAJOR SURVEY
ORGANIZATION WE TALKED TO
One coder
Given coding categories and can add
categories if needed.
Little or no written instructions
No monitoring of accuracy other than spot
checking by a supervisor.
POTENTIAL INNOVATIONS
Coding of text by computers.
Many such techniques require human coding.
Techniques being tested that do not require
human coding are very new.
CONFERENCE OF
EXPERTS
The ANES
Open-Ended
Coding Project
2008 CONFERENCE
(ANN ARBOR)
 Presenters:
 Experts on coding of text in many settings (e.g., content analysis of
news stories).
 Survey scholars who have studied coding.
 Experts on computer coding of text.
CODING CONFERENCE DETAILS
For more information search “ANES open-ended coding conference”:
h t t p: / / w w w. elec tionst udi es.org /con ferenc es/20 08 Met h ods/Met hodsCo nferenc e. ht m
 Transcripts
 Slides
OUR PROPOSAL
The ANES
Open-Ended
Coding Project
FOUR PARTS OF OPTIMAL CODING
FOUR PARTS OF OPTIMAL CODING
1. Code frame
FOUR PARTS OF OPTIMAL CODING
1. Code frame
2. Instructions
FOUR PARTS OF OPTIMAL CODING
1. Code frame
2. Instructions
3. Production
FOUR PARTS OF OPTIMAL CODING
1. Code frame
2. Instructions
3. Production
4. Evaluation
FOUR PARTS OF OPTIMAL CODING
Part 1
The Code Frame
CODE FRAMES
Definition: An organized and limited set of
categories used to classify or describe answers to
open-ended questions.
A SIMPLE CODE FRAME
Question: “Is there anything about Barack Obama that
would make you vote for him?”
A SIMPLE CODE FRAME
Question: “Is there anything about Barack Obama that
would make you vote for him?”
Category
Subcategory
Definition
Experience
Political experience
Political experience, work the candidate has done as a
politician or elected official, “experienced” or
“inexperienced” in general
Non-political experience
Any mentioned of something the candidate did in the past
that does not fit the political experience category
Honesty
The candidate's honesty, integrity, consistency,
predictability, sincerity, truthfulness
Leadership
The candidate's ability to lead, get people to work together,
make people want to follow, inspire people, inspire the
respondent, motivate people, motivate the respondent
All other answers
Any statement that does not fit one of the other categories
Personality
Other
GOOD CODE FRAMES
Characteristics of good categories and subcategories
GOOD CODE FRAMES
Characteristics of good categories and subcategories
Mutually exclusive
GOOD CODE FRAMES
Characteristics of good categories and subcategories
Mutually exclusive
Clearly defined
GOOD CODE FRAMES
Characteristics of good categories and subcategories
Mutually exclusive
Clearly defined
Collectively exhaustive
GOOD CODE FRAMES
Characteristics of good categories and subcategories
Mutually exclusive
Clearly defined
Collectively exhaustive
Objectively derived
BUILDING A GOOD CODE FRAME
RECOMMENDATION 1
Build a code frame around questions,
not answers
BUILDING A GOOD CODE FRAME
RECOMMENDATION 1
Build a code frame around questions, not answers.
Question: “What job or political office does Nancy Pelosi now
hold?”
BUILDING A GOOD CODE FRAME
RECOMMENDATION 1
Build a code frame around questions, not answers.
Question: “What job or political office does Nancy Pelosi now
hold?”
BUILDING A GOOD CODE FRAME
RECOMMENDATION 1
Build a code frame around questions, not answers.
Question: “What job or political office does Nancy Pelosi now
hold?”
Answers to the question
Speaker of the House of Representatives
Congresswoman
Majority leader in the Senate
CEO of Hewlett Packard
BUILDING A GOOD CODE FRAME
RECOMMENDATION 1
Build a code frame around questions, not answers.
Question: “What job or political office does Nancy Pelosi now
hold?”
Answers to the question
Speaker of the House of Representatives
Congresswoman
Majority leader in the Senate
CEO of Hewlett Packard
Answers to a different question
She is from California
I can’t stand that woman
She’s a Democrat
She has brown hair
BUILDING A GOOD CODE FRAME
RECOMMENDATION 1
Build a code frame around questions, not answers.
BUILDING A GOOD CODE FRAME
RECOMMENDATION 1
Build a code frame around questions, not answers.
Use an “Other” category for all responses that are not answers
to the question.
BUILDING A GOOD CODE FRAME
RECOMMENDATION 1
Build a code frame around questions, not answers.
Use an “Other” category for all responses that are not answers
to the question.
 Respondent answers are “on the same page”.
BUILDING A GOOD CODE FRAME
RECOMMENDATION 1
Build a code frame around questions, not answers.
Use an “Other” category for all responses that are not answers
to the question.
 Respondent answers are “on the same page”.
 Narrows the universe of possible code frame categories.
FOUR PARTS OF OPTIMAL CODING
1.
Code frame
1. Build a code frame around questions, not answer s
BUILDING A GOOD CODE FRAME
RECOMMENDATION 2
Learn from others
BUILDING A GOOD CODE FRAME
RECOMMENDATION 2
Learn from others
 Evaluate existing code frames
BUILDING A GOOD CODE FRAME
RECOMMENDATION 2
Learn from others
 Evaluate existing code frames
 Properties of a “good” code frame?
BUILDING A GOOD CODE FRAME
RECOMMENDATION 2
Learn from others
 Evaluate existing code frames
 Properties of a “good” code frame?
 Commonalities among code frames?
BUILDING A GOOD CODE FRAME
RECOMMENDATION 2
Learn from others
 Evaluate existing code frames
 Properties of a “good” code frame?
 Commonalities among code frames?
 Consult with experts
BUILDING A GOOD CODE FRAME
RECOMMENDATION 2
Learn from others
 Evaluate existing code frames
 Properties of a “good” code frame?
 Commonalities among code frames?
 Consult with experts
 Ask people who use similar data.
BUILDING A GOOD CODE FRAME
RECOMMENDATION 2
Learn from others
 Evaluate existing code frames
 Properties of a “good” code frame?
 Commonalities among code frames?
 Consult with experts
 Ask people who use similar data.
 Read what people have written about the question topic.
PROPERTIES OF A GOOD CODE FRAME?
QUESTIONABLE!
2004 ANES code frame (154 categories)
000. Inap, no further mention; no problems
401. WAGE AND PRICE CONTROLS/GUIDELINES; freezing prices; control of business profits
539. Other specific references to Russia/Detente/Eastern Europe, etc. (including changing site/boycotting 1980 Moscow Olympics); threat of/preventing war with Russia (exc. 714)
001. General reference to domestic issues; repairing/maintaining the nation's infrastructure (roads, bridges, dams, etc)
403. High price of food, all mentions (exc. 100)
540. FIRMNESS IN FOREIGN POLICY; maintenance of position of MILITARY/DIPLOMATIC STRENGTH (not 710-712)
005. POPULATION; any mention of population increase; reference toover-population/birth control
404. High price of other specific items and services
550. U.S. FOREIGN (MILITARY) INVOLVEMENT/COMMITMENT, extent of U.S. Foreign involvement; military assistance/aid (exc. 524 and 525)
006. DAY CARE; child care
405. MINIMUM WAGE, any mention; any mention of wage levels
560. U.S. FOREIGN (ECONOMIC) INVOLVEMENT/COMMITMENTS; extent of U.S.(foreign) economic aid; "foreign aid"
010. UNEMPLOYMENT; the number of people with jobs; unemployment rate/compensation; job retraining
407. Food shortages; economic aspects of food shortages, e.g., price of sugar (other references, code 120)
570. Prevention of war; ESTABLISHMENT OF PEACE; any reference (war in Iraq coded as 525)
013. CREATE JOBS/RECRUIT INDUSTRY in specific area/region/ state
408. Fuel shortages; "energy crisis"; oil companies making excessive profits; depressed condition of the oil industry; high price of gasoline(do not use code 404 for gas)
585. Obligation to TAKE CARE OF PROBLEMS AT HOME before helping foreign countries
020. EDUCATION; financial assistance for schools/colleges/students; quality of education/the learning environment/teaching; the high cost of college
410. RECESSION, DEPRESSION; prosperity of the nation; economic growth; GNP (see code 496 for "the economy")
599. Other specific mention of foreign affairs problems
030. AGED/ELDERLY; social security benefits; administration of social security; medical care for the aged; medicare benefits; insuring against catastrophic illness; prescription drug program for elderly (Must say Medicare or Elderly/Aged healthcare, ot herwise code under 040.)
411. MONETARY RESTRAINTS/CONTROLS; level of interest rates; availability of money/the money supply
700. NATIONAL DEFENSE; defense budget; level of spending on defense
035. Social Security won't be around in the future; paying into a system which won't benefit me/them
413. Future vision of what America will be like; a need to look to the future
710. DISARMAMENT; general reference to ENDING OF THE ARMS RACE; test ban treaty (not 540); SALT; INF treaty
040. HEALTH PROBLEMS/COST OF MEDICAL CARE; quality of medical care; medical research/training of doctors and other health personnel; hospitals; National Health insurance program
415. Against (increased) government spending; balancing of the (national) budget; against government stimulation of the economy; the size of the budget deficit
711. For DISARMAMENT; for extension of test ban treaty; support toward ending of arms race; against (additional) expenditures on military/arms development; SALT; SDI ("Star Wars"); INF treaty
045. PRO-ABORTION; pro-choice; the right of a woman to control her body
416. TAXES; general reference to tax structure; tax surcharge (NA R's direction); tax reform; other specific tax reference
712. Against (increased) policy of DISARMAMENT; against test ban treaty; for additional WEAPONS DEVELOPMENT; missile program; scientific/technological development in weapons/strategy; atomic bomb testing; increased DEFENSE BUDGET, increased arms expenditure(not
540); SALT; increased pay for military personnel; SDI ("Star Wars"); INF treaty
046. ANTI-ABORTION; pro-life; "abortion"--NFS
417. For tax cuts; against tax surcharge; for tax reform
713. General or specific references to functioning and performance of defense; waste, inefficiency (not codable in 710-712)
048. Other specific references to health problems; AIDS; Stem cell research, biologic research
418. Against tax cuts; for tax surcharge; against tax reform
050. HOUSING; providing housing for the poor/homeless; ability of young people to afford to buy homes/find homes to buy
424. PRODUCTIVITY of American industry; "giving a day's work for a day's pay"; revitalizing American industry
715. Security of nuclear secrets; Dept of Energy/Los Alamos nuclear security; Wen Ho Lee or Energy Secretary Richard compromising security; supposed Chinese spying
060. POVERTY; aid to the poor/underprivileged people; help for the (truly) needy; welfare programs (such as ADC); general reference to anti-poverty programs; hunger/help for hungry people in the U.S.
425. STOCK MARKET/GOLD PRICES; all references to gold prices, stock brokers, stock fluctuations, etc.
740. The space program; space race (not 711,712)
090. SOCIAL WELFARE PROBLEMS; "welfare"--NFS
427. VALUE OF THE DOLLAR; strength/weakness of the dollar against other currencies
750. MORALE OF NATION; Patriotism; National spirit; national unity; greed, selfishness of people
091. For general or other social welfare programs; "we need to help people more"
433. Large businesses taking over small businesses
760. BENEFITS FOR VETERANS; general reference
092. Against general or other social welfare programs; "too many give away programs for the people who don't deserve it"; for welfare reform
434. Corruption in corporate settings. Fraud, greed, mismanagement of corporate assets.
765. Allowing/accepting GAYS IN THE MILITARY (see code 385 for gay rights)
099. Other specific mentions of social welfare problems
440. Class oriented economic concerns--middle class, working class (pro);
799. Other specific mention of national defense problems ISSUES RELATING TO THE FUNCTIONING OF GOVERNMENT
100. FARM ECONOMICS; payment for crops/price of feed/cost of farming
441. Class oriented economic concerns--big business, monied interests (anti) too powerful
800. POWER OF THE (FEDERAL) GOVERNMENT; power of/control exercised by the federal government
103. SUBSIDIES/crop payments/government aid to farmers
442. Concern for inequitable distribution of wealth; gap between the rich and the poor; concentration of wealth in the hands of a few
810. (LACK OF) HONESTY IN GOVERNMENT; (LACK OF) ETHICS IN GOVERNMENT--general reference (exc. 811)
120. WORLD FOOD PROBLEMS; food shortages/starvation/famine (not 406 or 407)
451. For the regulation of interstate commerce, transportation, air travel, railways, government auto safety regulations; in favor of increased government regulation of business; mention of problems caused by deregulation
811. LACK OF PERSONAL ETHICS/morality of persons related to or part of government
150. CONSERVATION OF NATURAL RESOURCES; conservation, ecology; protecting the environment/endangered species
452. Against (increased) regulation of interstate commerce, transportation; AIR TRAVEL, RAILWAYS, etc.
812. The President lied/didn't tell the truth; covered up personal involvement with a young woman (also see more general code 874)
151. Controlling/REGULATING GROWTH or land development; banning further growth/development in crowded or ecologically sensitive areas; preserving natural areas
453. Solvency/stability/regulation/control of the nation's FINANCIAL INSTITUTIONS. [1990] Savings and Loan scandal
813. The President has low morals; had affair with a young woman (also see more general code 874)
153. POLLUTION; clean air/water
460. IMMIGRATION POLICY; establishing limits on how many people from any one nation can enter the U.S.; prohibiting specified types of persons from entering the U.S. (All mentions of Elian Gonzales or "the Cuban child" go here. If power of Federal government mentioned, see
codes 801 and 887.)
820. CAMPAIGN DONATIONS/PUBLIC FINANCING OF ELECTIONS; any mentions; campaign finance reform
154. Disposal of RADIOACTIVE/TOXIC waste (dumps, landfills)
463. Problems relating to the influx of political/economic refugees (Cubans, Haitians, Mexicans, etc.)
407. Food shortages; economic aspects of food
shortages, e.g., price of sugar (other references,
code 120)
714. Nuclear war; the threat of nuclear war; nuclear proliferation; the growing number of minor nuclear powers
830. CONFIDENCE/TRUST in political leaders/system; wisdom, ability, responsiveness of political leaders; quality of leadership provided by political leaders
160. DEVELOPMENT OF NATURAL RESOURCES /ENERGY SOURCES; harbors, dams, canals, irrigation, flood control, navigation, reclamation; location, mining, stock-piling of minerals; water power, atomic power; development of alternative sources of energy(includes mentions of
solar or nuclear power)
491. Economics--general; "Economics"--NFS
833. QUALITY/EFFICIENCY of public employees, diplomats, civil service; SIZE OF THE GOVERNMENT BUREAUCRACY; COST OF GOVERNMENT
199. Other specific mentions of agriculture or natural resources problems
492. International economics--general; economic problems in specific countries or regions
836. COMPENSATION; all references to the compensation of government employees, officials, congressmen, judges, local politicians/bureaucrats
200. LABOR/UNION PROBLEMS; union practices; job security provided workers; job safety issues; working conditions
493. U.S. foreign trade, balance of payments position; foreign oil dependency
837. Waste in government spending; keeping tabs on where money goes; pork barrel legislation
220. Anti-union; unions too powerful
494. Control of FOREIGN INVESTMENT IN U.S.; mention of foreigners buying U.S. assets (businesses, real estate, stocks, etc)
838. Government BUDGET PRIORITIES are wrong; Congress/President is spending money in the wrong areas/not spending money on the right things
299. Other specific mention of labor or union-management problems
120. WORLD FOOD PROBLEMS; food
shortages/starvation/famine (not 406 or 407)
495. PROTECTION OF U.S. INDUSTRIES; imposition of tariffs/reciprocal restrictions on foreign imports; limitation of foreign imports; mention of problems in specific industries competin with foreign manufacturers
840. SIZE OF FEDERAL GOVERNMENT; the (large) size of government/civil service/bureaucracy; the number of government departments/employees/programs
300. CIVIL RIGHTS/RACIAL PROBLEMS; programs to enable Blacks to gain social/economic/educational/political equality; relations between Blacks and whites
496. The economy--not further specified (code specific mention if R clarifies by saying "inflation", etc.; also see 400)
850. Adhering to the Consitution (see also legal reform, code 361, and also 887)
302. PROTECTION (expansion) OF WHITE MAJORITY; maintenance of segregation; right to choose own neighborhood; right to discriminate in employment
497. International competitiveness; outsourcing; loss of jobs to foreign competition; moving jobs abroad; modernizing plants/equipment/management techniques to meet foreign competition; matching the quality of foreign goods
853. POWER OF CONGRESS--general reference
304. Discrimination against whites; preferred treatment given to minorities
498. Mention of "twin problems" of a large national debt/budget deficit and unfavorable balance of trade/import-export ratio
856. POWER OF THE SUPREME COURT, all other references to the Supreme Court except 857, 858
499. Other specific mention economic or business problems
859. Other specific references to the (federal) balance of power; legislative gridlock in Washington
320. NARCOTICS; availability of drugs; extent of drug/alcohol addiction in the U.S.; interdiction of drugs coming to the U.S. from foreign countries; alcohol or drug related crime; drug laws
500. FOREIGN RELATIONS/FOREIGN AFFAIRS; foreign policy/relations, prestige abroad
862. FAIR ELECTION PROCEDURES; prevention of vote manipulation; curbing of political "bosses", smear campaigns
330. WOMEN'S RIGHTS; references to women's issues; economic equality for women; ERA
504. Relations with the Third World (no specific country or region mentioned)
869. Other specific references to problems of representation; term limitations for members of Congress
505. Relations with WESTERN EUROPE; Great Britain, France, Germany; our allies
874. Lack of support for the President; any anti-President comments, negative reference to the PRESIDENT's quality, style, etc. (also see codes 812, 813)
360. LAW AND ORDER; respect for the law/police; support for the police; death penalty; tougher sentences for criminals; need for more prisons
507. U.S. Foreign policy actions without UN or allied approval. Unilateral action by the U.S.; less concern about world opinion
875. Impeachment proceeding; mention of the Special Prosecutor; Republicans attempting to get the President out of office/Democrats attempting to keep the President in office (also see codes 812, 813)
361. Legal reform; Tort reform--general (for mentions regarding specific issues, see specific issue); see also Adhering to the Constitution(code 850)
510. VIETNAM; general reference to "the war," Indochina, Cambodia; aid
878. Mention of a specific CANDIDATE or relative of a candidate -- NFS
363. Police brutality; police not doing their job properly
514. Latin America, South America--any references; reference to war/situation in Nicaragua; U.S. support of the Contras
881. New president/administration getting started; other references specific to the President; support for the President; "let him get on with his job"
367. Against unregistered ownership of guns; legislative control of guns; "CONTROL OF GUNS"-NFS
515. Iran; mention of American hostages in Teheran; arms deal
885. PUBLIC APATHY/disinterest--all references
368. For gun ownership; right to have guns; against gun control
516. African countries; developing areas in Africa (not 518) --any mention; U.S. response to apartheid in South Africa
886. References to unfair/undeserved/excessive criticism by the media. All media mentions should go here: "The news can't be trusted"
370. EXTREMIST GROUPS/TERRORISTS; terrorist bombings/hostage-taking; political subversives; revolutionary ideas/approaches; Homeland Security mentions; the war against Al Quaeda; Islamic terrorists or extremists
519. Other specific countries/areas/trouble spots (exc. 520's, 530's)
887. Extending/protecting EQUAL RIGHTS, basic freedoms, human rights of all citizens; protecting the Bill of Rights
375. Euthenasia; right to die
524. MIDDLE EAST-- support or aid to Israel/Arab states; Arab/Israeli conflict; Iran-Iraq war; hostages in Lebanon/Middle East. [1990]Iraqi aggression in the Persian Gulf ; Saddam Hussein; war in/occupation of Iraq
899. Other specific mention of problems relating to the functioning of government
380. General mention of MORAL/RELIGIOUS DECAY (of nation); sex, bad language, adult themes on TV
525. The U.S. occupation of Iraq; the war to topple Saddam Hussein
990. Other specific mentions of important problems
381. Family problems--divorce; proper treatment of children; decay of family (except 006); child/elder abuse (incl. sexual); family values
527. Clash of religions; Christianity and Islam in conflict (code Islamic terrorists under 370).
995. "There were no issues"; "there were no issues, just party politics"
383. Problems of/with YOUNG PEOPLE; drug/alcohol abuse amoung young people; sexual attitudes; lack of values/ discipline; mixed-up thinking; lack of goals/ambition/sense of responsibility
530. RUSSIA/Eastern Europe; relations with Russia/the Communist bloc; detente/trade/negotiations with Russia-- NA whether 531 or 532
996. "There was no campaign in my district"
384. Religion (too) mixed up in politics; prayer in school
531. For PEACEFUL RELATIONS with Russia/Detente/Eastern Europe; for increased TRADE with Russia; talking/resuming negotiations with Russia on arms control/reduction (reaching/concluding a treaty is 711)
998/888. DK
385. HOMOSEXUALITY; protecting civil rights of gays and lesbians; accepting the lifestyle of homosexuals; granting homosexual couples the same rights and benefits as heterosexual couples, gay marriage (gays in the military go under code 765)
532. Against policy of Detente with Russia; COLD WAR; threat of external Communism; need to oppose/be wary of Russia
999/999. NA
399. Other specific mention of racial or public order problems; other mention of domestic issues
533. Prevention of Russian (Communist) expansion; mention of Soviet invasion and occupation of Afghanistan-- any reference; references to Soviet activity in Central America/Nicaragua)
400. INFLATION; rate of inflation; level of prices; cost of living
535. US/NATO involvement in the Balkans; US/NATO led air war to contain Serbia
310. Technology; mentions that are specific to technology; e.g., the Year 2000 computer (Y2K) problem
340. CRIME/VIOLENCE; too much crime; streets aren't safe; mugging, murder, shoplifting; drug related crime
CODE FRAME COMMONALITIES
Same question – Different code frames
AP-IPSOS
Gallup
New York Times
Quinnipiac
Pew
1 The president, George W. Bush in office
1 Other
01 Abortion
Economy general
01 Economy (unsp)
2 Government, politicians (other mentions)
2 DK
02 Poverty
Unemployment/Jobs
02 Unemployment/lack of jobs/ Job cuts
3 Morals, moral decay
3 REF
03 Immigration
Stock market
03 Risky bank loans/sub prime loans/people taking on too much debt
4 Don't care for others, people don't care
4 None
04 Guns
Cost of living
04 Inflation/difference between wages/costs
5 Race relations, racism
5 NA
05 Environment
Business leaving/Outsourcing
05 Drop in retirement accounts (401K)/stock market
6 Not family oriented, family values, breakdown of family unit
6 Economy (General)
06 Racism
Economy other
06 National debt/budget/deficit/balanced budget
7 Youth, no respect for elders, how kids behave
7 Unemployment-Jobs
07 AIDS
War (Iraq)
07 The bailout
8 Other (morality) mentions
8 Federal Budge Deficit-Debt
08 Child care
Energy costs/Crisis general
8 The stimulus
9 Violence, domestic violence
9 Taxes
09 Intolerance
Gas prices
10 Taxes (unsp)
10 Teen violence (e.g. violence at schools, gangs)
10 Foreign Trade-Trade Deficit
10 Misc. Govt. Programs
Oil prices/Crisis
11 Financial crisis/credit crunch/banking situation
11 Crime (unspecified)
11 Cost of Living-Inflation
11 Education
Energy crisis/Gas prices general
12 Housing market/foreclosures
12 Drugs, issues related to drug use, abuse
12 Recession
12 Elderly
Healthcare/Costs/Insurance/HMO's (health maintenance organizations)
13 Wallstreet/corporate America
13 Other (crime) mentions
13 Other Economic
13 Welfare
Terrorism general
14 Education/schools/affording education
14 Problems with public schools
14 Crime-Violence
14 Medicare/Medicaid
Security/Safety
15 A/O economic issues mentions
15 Education (unspecified)
15 Health Care-Hospitals
15 Public Works / infrastructure
Politicians/Campaigns general
16 Finances
16 More structure in schools, standardized
16 Drugs
16 Social Security
Political corruption
18 Crime/Violence/gangs/justice system (unsp)
17 Uneducated people (e.g. literacy issues)
17 Poverty-Hunger-Homelessness
17 Health care / health insurance
Budget general
20 Drugs/alcohol
18 Lack of funding for schools
18 Ethical-Moral-Religious Decline
18 Drug coverage/ prescription
Budget deficits
25 Abortion
19 Other (education) mentions
19 Access to Education
19 War/Peace (general)
Education general
26 Social Security
20 Oil shortage
20 AIDS
20 Misc. Foreign Affairs
Immigration
27 Issues relating to the elderly
21 Energy (unspecified)
21 Medicare Increases-Senior Citizen Insurance
21 Foreign Policy
Lack of ethics/morality
28 Welfare abuse
22 Gas prices (oil prices)
22 International Problems-Foreign Affairs
22 Nuclear
Lack of religion
29 Race relations/Racism/Racial profiling
23 Other (energy crisis) mentions
23 Government-President Clinton-Congress-Politicians
23 Defense / Military
Crime/Violence general
31 Morality/religion/family values (unsp)
24 Environment, pollution (unspecified)
24 Foreign Aid-Focus Overseas
24 War in Afghanistan/Taliban
Environment/Pollution
33 Homelessness
25 Other (environmental problem) mentions
25 Race Relations
25 War in Iraq
Family breakdown/Family values
34 Poverty/Hunger/Starvation
26 Abortion
26 Immigration-Illegal Aliens
26 Foreign aid
Foreign affairs/policy general
37 Healthcare (costs/accessibility)
27 Health care, coverage
27 Welfare
27 North Korea
Taxes general
43 Dissatisfaction with government/politics/Scandal/corruption in govt
28 High cost of prescription drugs/ drug coverage for seniors
28 Environment-Pollution
28 Specific County (Other)
Other
46 Partisanship/the parties (MERGED WITH CODE 43 IN TOPLINE)
29 Poverty, hunger, homelessness
29 Iraq-Saddam Hussein
29 Impeach GWBush
Don't know/No answer
47 Immigration/immigration situation/foreigners
30 Social Security
30 Kosovo-Serbia-Yugoslavia-Milosevic
30 Misc. Government Issues
50 Environment/pollution/Global warming
31 Social services (welfare, childcare)
31 School Shootings
31 Judicial branch
52 A/O social/domestic issues mentions
32 Government spending/ deficit/ budget
32 Gun Laws too Weak-Availability of Guns
32 Politicians /corruption/morals
56 Too much foreign aid/spend money at home
33 High cost of prescription drugs/ drug coverage for seniors
33 Gun Control-Gun Laws Too Strong
33 Government / gov't reform
57 Defense issues/national and homeland security/military & defense spending
34 Wars, unrest throughout the world
34 Abortion Issues
34 Democrats
58 Trade/Jobs moving overseas
35 War with Iraq (any mentions about Iraq)
35 Social Security Issues
35 Partisan politics
59 US foreign policy/International relations Americas image overseas
36 Too much focus on other countries, need to be tougher on oth
36 Children’s Needs
36 Republicans
60 Iraq / War in Iraq
37 Immigration, letting too many people in, lack of security
37 Lack of a Military/Defense
37 Campaign fund-raising /Reform
61 China
38 Terrorism, terrorist attacks
38 Fuel/Oil Prices
38 The President / George W. Bush / leadership
62 Terrorism
39 Improve defense, military readiness
39 Dishonesty/Lack of Integrity
39 Big government
67 A/O international/foreign issues mentions
40 USA losing credibility in the world/ problems with our allie
40 Judicial System/Courts/Laws
40 Misc. Moral Values
70 Recession/Slowing of the economy
41 Other (foreign affairs) mentions
41 National Security
41 Community
76 Energy costs/Rising gas/heating prices
42 Economy (unspecified)
42 Gap Between the Rich and Poor
42 Religious Values
77 Middle East situation
43 Taxes, taxes are too high
43 Care for the Elderly
44 Family breakdown / single parents
78 Many things/Everything
44 Unemployment, lack of jobs
44 Wage Issues
45 Moral values / Family Values
79 Uneven distribution of wealth
45 Cost of living is too high
45 Fear of War
50 Misc Economic Issues
80 Peace in the world/Peace
46 Corporate irresponsibility, corruption, fraud, deception, ac
46 Breakdown of the Family
51 Economy
86 More regulation of markets
47 Stock market uncertainty, devaluation
47 Cancer/Diseases
52 Budget / Deficit
89 Oil dependence/Energy policy and alternatives
48 Other (economy) mentions
48 Child Abuse
53 Jobs and Unemployment
90 Obama
49 OTHER
49 Advancement of computers/technology
54 Taxes
91 Govt control/overreach - socialism
50 Nothing, nothing else
50 Lack of energy Sources
55 Income gap
96 Other
51 (DK/NS)
51 Gun control/Guns
56 Foreign Trade
98 None
52 Costs associated with health insurance
57 Personal finances
99 Don’t know/refused
53 Overpopulation
58 Agriculture
54 Way children are raised
59 Stock Market
55 Presidential choices/election year
60 Housing
56 Lack of respect for each other
61 Crime / Violence
57 Poor leadership/corrupt
62 Drugs
58 Abuse of power
63 Police problems
59 Lack of money
65 Criminal policy
60 Education
66 Abuse
61 The media
67 Business ethics
62 Election/Election Reform
70 Youth (other)
63 Unifying the Country
71 Youth Crime
64 Situation with China
72 Youth Drugs
65 Energy crisis
73 Parenting
66 Terrorism
74 Youth values
67 Feeling of Fear in this Country
75 Natural Disasters
68 Corporate Corruption
76 Fuel costs
69 Space Shuttle Disaster
77 Electoral college / voting methods
70 Iraq
78 Hurricane
71 Gay marriage/Homosexual rights
79 Oil dependency/energy policies
72 Licenses for the Undocumented
80 Misc Social Issues
73 Natural disaster response
81 Media
75War-Conflict in the Middle East
82 Gay marriage
96 No additional mentions
83 Stem cell research
85 Natural Disasters/Tsunami
86 Israel/Palestine/Arab-Israeli Conflict
87 Middle East (general)
88 Bailout
90 Terrorism (general)
91 Osama bin Laden/Al Qaeda
92 Anthrax
92 Barack Obama/President-Elect
93 Biochemical/small pox
94 Homeland Security
96 Everything
97 Nothing
98 Other
99 DK/NA
CODE FRAME COMMONALITIES
Different code frames – Similar categories
AP-IPSOS
Gallup
New York Times
Quinnipiac
Pew
1 The president, George W. Bush in office
1 Other
01 Abortion
Economy general
01 Economy (unsp)
2 Government, politicians (other mentions)
2 DK
02 Poverty
Unemployment/Jobs
02 Unemployment/lack of jobs/ Job cuts
3 Morals, moral decay
3 REF
03 Immigration
Stock market
03 Risky bank loans/sub prime loans/people taking on too much debt
4 Don't care for others, people don't care
4 None
04 Guns
Cost of living
04 Inflation/difference between wages/costs
5 Race relations, racism
5 NA
05 Environment
Business leaving/Outsourcing
05 Drop in retirement accounts (401K)/stock market
6 Not family oriented, family values, breakdown of family unit
6 Economy (General)
06 Racism
Economy other
06 National debt/budget/deficit/balanced budget
7 Youth, no respect for elders, how kids behave
7 Unemployment-Jobs
07 AIDS
War (Iraq)
07 The bailout
8 Other (morality) mentions
8 Federal Budge Deficit-Debt
08 Child care
Energy costs/Crisis general
8 The stimulus
9 Violence, domestic violence
9 Taxes
09 Intolerance
Gas prices
10 Taxes (unsp)
10 Teen violence (e.g. violence at schools, gangs)
10 Foreign Trade-Trade Deficit
10 Misc. Govt. Programs
Oil prices/Crisis
11 Financial crisis/credit crunch/banking situation
11 Crime (unspecified)
11 Cost of Living-Inflation
11 Education
Energy crisis/Gas prices general
12 Housing market/foreclosures
12 Drugs, issues related to drug use, abuse
12 Recession
12 Elderly
Healthcare/Costs/Insurance/HMO's (health maintenance organizations)
13 Wallstreet/corporate America
13 Other (crime) mentions
13 Other Economic
13 Welfare
Terrorism general
14 Education/schools/affording education
14 Problems with public schools
14 Crime-Violence
14 Medicare/Medicaid
Security/Safety
15 A/O economic issues mentions
15 Education (unspecified)
15 Health Care-Hospitals
15 Public Works / infrastructure
Politicians/Campaigns general
16 Finances
16 More structure in schools, standardized
16 Drugs
16 Social Security
Political corruption
18 Crime/Violence/gangs/justice system (unsp)
17 Uneducated people (e.g. literacy issues)
17 Poverty-Hunger-Homelessness
17 Health care / health insurance
Budget general
20 Drugs/alcohol
18 Lack of funding for schools
18 Ethical-Moral-Religious Decline
18 Drug coverage/ prescription
Budget deficits
25 Abortion
19 Other (education) mentions
19 Access to Education
19 War/Peace (general)
Education general
26 Social Security
20 Oil shortage
21 Energy (unspecified)
22 Gas prices (oil prices)
23 Other (energy crisis) mentions
24 Environment, pollution (unspecified)
25 Other (environmental problem) mentions
Economy (General)
Economy General
20 AIDS
20 Misc. Foreign Affairs
Immigration
27 Issues relating to the elderly
21 Medicare Increases-Senior Citizen Insurance
21 Foreign Policy
Lack of ethics/morality
28 Welfare abuse
22 International Problems-Foreign Affairs
22 Nuclear
Lack of religion
29 Race relations/Racism/Racial profiling
23 Government-President Clinton-Congress-Politicians
23 Defense / Military
Crime/Violence general
31 Morality/religion/family values (unsp)
24 Foreign Aid-Focus Overseas
24 War in Afghanistan/Taliban
Environment/Pollution
33 Homelessness
25 Race Relations
25 War in Iraq
Family breakdown/Family values
34 Poverty/Hunger/Starvation
26 Abortion
26 Immigration-Illegal Aliens
26 Foreign aid
Foreign affairs/policy general
37 Healthcare (costs/accessibility)
27 Health care, coverage
27 Welfare
27 North Korea
Taxes general
43 Dissatisfaction with government/politics/Scandal/corruption in govt
28 High cost of prescription drugs/ drug coverage for seniors
28 Environment-Pollution
28 Specific County (Other)
Other
46 Partisanship/the parties (MERGED WITH CODE 43 IN TOPLINE)
29 Poverty, hunger, homelessness
29 Iraq-Saddam Hussein
29 Impeach GWBush
Don't know/No answer
47 Immigration/immigration situation/foreigners
30 Social Security
30 Kosovo-Serbia-Yugoslavia-Milosevic
30 Misc. Government Issues
50 Environment/pollution/Global warming
31 Social services (welfare, childcare)
31 School Shootings
31 Judicial branch
52 A/O social/domestic issues mentions
32 Government spending/ deficit/ budget
32 Gun Laws too Weak-Availability of Guns
32 Politicians /corruption/morals
56 Too much foreign aid/spend money at home
33 High cost of prescription drugs/ drug coverage for seniors
33 Gun Control-Gun Laws Too Strong
33 Government / gov't reform
34 Wars, unrest throughout the world
34 Abortion Issues
34 Democrats
35 War with Iraq (any mentions about Iraq)
35 Social Security Issues
35 Partisan politics
36 Too much focus on other countries, need to be tougher on oth
36 Children’s Needs
36 Republicans
37 Immigration, letting too many people in, lack of security
37 Lack of a Military/Defense
37 Campaign fund-raising /Reform
38 Terrorism, terrorist attacks
38 Fuel/Oil Prices
38 The President / George W. Bush / leadership
Economy (Unspec)
57 Defense issues/national and homeland security/military & defense spending
58 Trade/Jobs moving overseas
59 US foreign policy/International relations Americas image overseas
60 Iraq / War in Iraq
61 China
62 Terrorism
39 Improve defense, military readiness
39 Dishonesty/Lack of Integrity
39 Big government
67 A/O international/foreign issues mentions
40 USA losing credibility in the world/ problems with our allie
40 Judicial System/Courts/Laws
40 Misc. Moral Values
70 Recession/Slowing of the economy
41 Other (foreign affairs) mentions
41 National Security
41 Community
76 Energy costs/Rising gas/heating prices
42 Economy (unspecified)
42 Gap Between the Rich and Poor
42 Religious Values
77 Middle East situation
43 Taxes, taxes are too high
43 Care for the Elderly
44 Family breakdown / single parents
78 Many things/Everything
44 Unemployment, lack of jobs
44 Wage Issues
45 Moral values / Family Values
79 Uneven distribution of wealth
45 Cost of living is too high
45 Fear of War
50 Misc Economic Issues
80 Peace in the world/Peace
46 Corporate irresponsibility, corruption, fraud, deception, ac
46 Breakdown of the Family
51 Economy
86 More regulation of markets
47 Stock market uncertainty, devaluation
47 Cancer/Diseases
52 Budget / Deficit
89 Oil dependence/Energy policy and alternatives
48 Other (economy) mentions
48 Child Abuse
53 Jobs and Unemployment
90 Obama
49 OTHER
49 Advancement of computers/technology
54 Taxes
91 Govt control/overreach - socialism
50 Nothing, nothing else
50 Lack of energy Sources
55 Income gap
96 Other
51 (DK/NS)
51 Gun control/Guns
56 Foreign Trade
98 None
52 Costs associated with health insurance
57 Personal finances
99 Don’t know/refused
53 Overpopulation
58 Agriculture
54 Way children are raised
59 Stock Market
55 Presidential choices/election year
60 Housing
56 Lack of respect for each other
61 Crime / Violence
57 Poor leadership/corrupt
58 Abuse of power
59 Lack of money
60 Education
61 The media
62 Election/Election Reform
Economy
62 Drugs
63 Police problems
65 Criminal policy
66 Abuse
67 Business ethics
70 Youth (other)
63 Unifying the Country
71 Youth Crime
64 Situation with China
72 Youth Drugs
65 Energy crisis
73 Parenting
66 Terrorism
74 Youth values
67 Feeling of Fear in this Country
75 Natural Disasters
68 Corporate Corruption
76 Fuel costs
69 Space Shuttle Disaster
77 Electoral college / voting methods
70 Iraq
78 Hurricane
71 Gay marriage/Homosexual rights
79 Oil dependency/energy policies
72 Licenses for the Undocumented
80 Misc Social Issues
73 Natural disaster response
81 Media
75War-Conflict in the Middle East
82 Gay marriage
96 No additional mentions
83 Stem cell research
Economy (Unspecified)
85 Natural Disasters/Tsunami
86 Israel/Palestine/Arab-Israeli Conflict
87 Middle East (general)
88 Bailout
90 Terrorism (general)
91 Osama bin Laden/Al Qaeda
92 Anthrax
92 Barack Obama/President-Elect
93 Biochemical/small pox
94 Homeland Security
96 Everything
97 Nothing
98 Other
99 DK/NA
PROPERTIES OF A GOOD CODE FRAME?
YES
 Bureau of Labor Statistics (BLS) code frames.
 Code frame created prior to data collection.
 Rule for adding and subtracting codes.
 Categories are mutually exclusive and collectively exhaustive.
 The BLS code frames should be used.
FOUR PARTS OF OPTIMAL CODING
1.
Code frame
1. Build a code frame around questions, not answer s
2. Learn from other s
BUILDING A GOOD CODE FRAME
RECOMMENDATION 3
Look outside the survey world for
other categorization systems
BUILDING A GOOD CODE FRAME
RECOMMENDATION 3
Look outside the survey world for other categorization systems
 Government
BUILDING A GOOD CODE FRAME
RECOMMENDATION 3
Look outside the survey world for other categorization systems
 Government
 Science
BUILDING A GOOD CODE FRAME
RECOMMENDATION 3
Look outside the survey world for other categorization systems
 Government
 Science
 Business and Industry
BUILDING A GOOD CODE FRAME
RECOMMENDATION 3
Look outside the survey world for other categorization systems
 Government
 Science
 Business and Industry
 Religion
A GOVERNMENT
CATEGORIZATION SYSTEM
Federal Budget Categories




















N a t i o n a l d e fe n s e
I n te r n a t i o n a l a f f a i r s
G e n e r a l s c i e n c e , s p a c e a n d te c h n o l o g y
Energy
N a t u r a l r e s o u r c e s a n d e nv i r o n m e n t
Agriculture
Commerce and housing credit
Tr a n s p o r t a t i o n
Community and regional development
E d u c a t i o n , t r a i n i n g , e m p l oy m e n t , a n d s o c i a l s e r v i c e s
Health
Medicare
Income security
Social security
Ve te r a n s b e n e fi t s a n d s e r v i c e s
Administration of justice
G e n e r a l g ov e r n m e n t
N e t i n te r e s t
A l l o wa n c e s
Undistributed offsetting receipts
A PSYCHOLOGY
CATEGORIZATION SYSTEM
The “Big Five” Personality Traits
Openness to experience
Conscientiousness
Extraversion
Agreeableness
Neuroticism
FOUR PARTS OF OPTIMAL CODING
1.
Code frame
1. Build a code frame around questions, not answer s
2. Learn from other s
3. Look outside the sur vey world for other categorizati on systems
FOUR PARTS OF OPTIMAL CODING
Part 2
The Instructions
INSTRUCTIONS
The rules governing how people
categorize answers
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 1
Instructions should always be written,
never oral
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 1
 Instructions should always be written, never oral
 Create a detailed coding manual
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 1
 Instructions should always be written, never oral
 Create a detailed coding manual
 Modify the manual to address unanticipated problems
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 1
 Instructions should always be written, never oral
 Create a detailed coding manual
 Modify the manual to address unanticipated problems
 Distribute updated manuals simultaneously to all coders
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 1
 Examples from an instruction manual:
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 1
 Examples from an instruction manual:
“Try to identify as many ideas in an answer as possible. Any single
word, phrase, sentence, or group of sentences that could be an idea
should be treated as if it is an idea.”
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 1
 Examples from an instruction manual:
“Try to identify as many ideas in an answer as possible. Any single
word, phrase, sentence, or group of sentences that could be an idea
should be treated as if it is an idea.”
“Assign a code for each idea in a response even if it is repeated within
the response.”
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 1
 Examples from an instruction manual:
“Try to identify as many ideas in an answer as possible. Any single
word, phrase, sentence, or group of sentences that could be an idea
should be treated as if it is an idea.”
“Assign a code for each idea in a response even if it is repeated within
the response.”
“List the codes you assign in the order in which the ideas appear in an
answer”.
FOUR PARTS OF OPTIMAL CODING
1.
2.
Code frame
1. Build a code frame around questions, not answer s
2. Learn from other s
3. Look outside the sur vey world for other categorizati on systems
Instructions
1. Instructions should always be written, never oral
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 2
Clearly defined categories
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 2
 Clearly defined categories
 Assume coders are unfamiliar with a category
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 2
 Clearly defined categories
 Assume coders are unfamiliar with a category
 Define category inclusions
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 2
 Clearly defined categories
 Assume coders are unfamiliar with a category
 Define category inclusions
 Identify category exclusions
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 2
 Clearly defined categories
 Assume coders are unfamiliar with a category
 Define category inclusions
 Identify category exclusions
 Provide keywords
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 2
 Example from an instruction manual
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 2
 Example from an instruction manual
Policy -poor people
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 2
 Example from an instruction manual
Policy -poor people
 Definition: What the candidate will do about government programs to help
poor people. The candidate’s policy, stand, views, position, or emphasis on
government programs to help poor people.
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 2
 Example from an instruction manual
Policy -poor people
 Definition: What the candidate will do about government programs to help
poor people. The candidate’s policy, stand, views, position, or emphasis on
government programs to help poor people.
 Inclusions: This code should only be assigned to ideas that mention helping
poor people or government programs to help poor people.
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 2
 Example from an instruction manual
Policy -poor people
 Definition: What the candidate will do about government programs to help
poor people. The candidate’s policy, stand, views, position, or emphasis on
government programs to help poor people.
 Inclusions: This code should only be assigned to ideas that mention helping
poor people or government programs to help poor people.
 Exclusions: Any idea that mentions how much the candidate cares about poor
people should be assigned code 34 Groups.
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 2
 Example from an instruction manual
Policy -poor people
 Definition: What the candidate will do about government programs to help
poor people. The candidate’s policy, stand, views, position, or emphasis on
government programs to help poor people.
 Inclusions: This code should only be assigned to ideas that mention helping
poor people or government programs to help poor people.
 Exclusions: Any idea that mentions how much the candidate cares about poor
people should be assigned code 34 Groups.
 Keywords: welfare, food stamps, Medicaid, Aid for Families with Dependent
Children (AFDC), and public housing.
FOUR PARTS OF OPTIMAL CODING
1.
2.
Code frame
1. Build a code frame around questions, not answer s
2. Learn from other s
3. Look outside the sur vey world for other categorizati on systems
Instructions
1. Instructions should always be written, never oral
2. Clearly defined categories
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 3
Include examples
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 3
 Include examples
 Include examples of the coding rules
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 3
 Include examples
 Include examples of the coding rules
 Include examples of actual coding
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 3
 An example of a coding rule
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 3
 An example of a coding rule
Rule for coding comparative statements
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 3
 An example of a coding rule
Rule for coding comparative statements
 Ideas with comparisons should be coded as one idea. Here is
an example, “Jobs are more important than the environment”.
The answer compares “jobs” to “the environment”. You must
decide which of the two ideas is more important to the
respondent. In this case, the respondent said “Jobs are more
important”. This means you should code the “jobs” part of the
comparison and NOT the environment part.
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 3
 An example of actual coding
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 3
 An example of actual coding
Coding the idea “The government wastes too much money”
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 3
 An example of actual coding
Coding the idea “The government wastes too much money”
 The code that best fits this idea is 46 Budget priorities. The
description for code 46 includes “wasteful spending”. The
idea states “the government wastes too much money”, and
the “wasteful spending” description in 46 is closer to this idea
than any of the other code descriptions. You should assign
code 46 to the first idea.
FOUR PARTS OF OPTIMAL CODING
1.
2.
Code frame
1. Build a code frame around questions, not answer s
2. Learn from other s
3. Look outside the sur vey world for other categorizati on systems
Instructions
1. Instructions should always be written, never oral
2. Clearly defined categories
3. Include examples
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 4
Test instructions
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 4
 Test instructions
 At least two people independently code sample data
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 4
 Test instructions
 At least two people independently code sample data
 Identify instances in which independent work disagreed
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 4
 Test instructions
 At least two people independently code sample data
 Identify instances in which independent work disagreed
 Determine if disagreement rate is unacceptably high
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 4
 Test instructions
 At least two people independently code sample data
 Identify instances in which independent work disagreed
 Determine if disagreement rate is unacceptably high
 Determine reasons for disagreements
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 4
 Test instructions
 At least two people independently code sample data
 Identify instances in which independent work disagreed
 Determine if disagreement rate is unacceptably high
 Determine reasons for disagreements
 Address causes for disagreements
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 4
 Test instructions
 At least two people independently code sample data
 Identify instances in which independent work disagreed
 Determine if disagreement rate is unacceptably high
 Determine reasons for disagreements
 Address causes for disagreements
 Repeat testing process with a new data
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 4
 Example of testing instruction
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 4
 Example of testing instruction
 “ What job or political of fice does Nancy Pelosi now hold?”
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 4
 Example of testing instruction
 “ What job or political of fice does Nancy Pelosi now hold?”
 Round 1
 Identical coding for 74% of answer sample.
 Disagreements involving answers with “and” and “or”.
 Instructions revised.
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 4
 Example of testing instruction
 “ What job or political of fice does Nancy Pelosi now hold?”
 Round 1
 Identical coding for 74% of answer sample.
 Disagreements involving answers with “and” and “or”.
 Instructions revised.
 Round 2
 Identical coding for 66% of new answer sample.
 Disagreements involve two code categories.
 Code category definitions and instructions revised.
DEVELOPING GOOD INSTRUCTIONS
RECOMMENDATION 4
 Example of testing instruction
 “ What job or political of fice does Nancy Pelosi now hold?”
 Round 1
 Identical coding for 74% of answer sample.
 Disagreements involving answers with “and” and “or”.
 Instructions revised.
 Round 2
 Identical coding for 66% of new answer sample.
 Disagreements involve two code categories.
 Code category definitions and instructions revised.
 Round 3
 Identical coding for 91% of answer sample.
FOUR PARTS OF OPTIMAL CODING
1.
2.
Code frame
1. Build a code frame around questions, not answer s
2. Learn from other s
3. Look outside the sur vey world for other categorizati on systems
Instructions
1. Instructions should always be written, never oral
2. Clearly defined categories
3. Include examples
4. Test instructions
FOUR PARTS OF OPTIMAL CODING
Part 3
Production
PRODUCTION
Assigning codes to all the answers
EFFECTIVE PRODUCTION
RECOMMENDATION 1
Partition answers into chunks if
needed
EFFECTIVE PRODUCTION
RECOMMENDATION 1
 Partition answers into chunks if needed
 Original answer: His record as a war hero, country first stance.
I like his economic policies better, and I think he's got a
better handle on foreign af fairs.
EFFECTIVE PRODUCTION
RECOMMENDATION 1
 Partition answers into chunks if needed
 Original answer: His record as a war hero, country first stance.
I like his economic policies better, and I think he's got a
better handle on foreign af fairs.
 Answer chunks:




His record as a war hero
(His) country first stance
I like his economic policies better
I think he's got a better handle on foreign affairs
EFFECTIVE PRODUCTION
RECOMMENDATION 1
 Create chunking instruction manual using same procedures
for creating coding instruction manual
EFFECTIVE PRODUCTION
RECOMMENDATION 1
 Create chunking instruction manual using same procedures
for creating coding instruction manual
 Include detailed instructions for all chunking tasks
EFFECTIVE PRODUCTION
RECOMMENDATION 1
 Create chunking instruction manual using same procedures
for creating coding instruction manual
 Include detailed instructions for all chunking tasks
 Include examples
EFFECTIVE PRODUCTION
RECOMMENDATION 1
 Create chunking instruction manual using same procedures
for creating coding instruction manual
 Include detailed instructions for all chunking tasks
 Include examples
 Test chunking instructions
EFFECTIVE PRODUCTION
RECOMMENDATION 1
 Production chunking
EFFECTIVE PRODUCTION
RECOMMENDATION 1
 Production chunking
 Two people work independently to partition all answers into codable
chunks.
EFFECTIVE PRODUCTION
RECOMMENDATION 1
 Production chunking
 Two people work independently to partition all answers into codable
chunks.
 Identify the answers for which chunkers disagreed how to partition.
EFFECTIVE PRODUCTION
RECOMMENDATION 1
 Production chunking
 Two people work independently to partition all answers into codable
chunks.
 Identify the answers for which chunkers disagreed how to partition.
 Independent chunkers discuss disagreements and converge on
partitioning that both agree is correct.
EFFECTIVE PRODUCTION
RECOMMENDATION 1
 Example from chunking answers to “Is there anything about
Barack Obama that would make you want to vote for him?”
EFFECTIVE PRODUCTION
RECOMMENDATION 1
 Example from chunking answers to “Is there anything about
Barack Obama that would make you want to vote for him?”
 Identical chunking for 80% of sample answers
EFFECTIVE PRODUCTION
RECOMMENDATION 1
 Example from chunking answers to “Is there anything about
Barack Obama that would make you want to vote for him?”
 Identical chunking for 80% of sample answers
 Number of answer = 1,495
EFFECTIVE PRODUCTION
RECOMMENDATION 1
 Example from chunking answers to “Is there anything about
Barack Obama that would make you want to vote for him?”
 Identical chunking for 80% of sample answers
 Number of answer = 1,495
 Number of answer chunks = 4,018
FOUR PARTS OF OPTIMAL CODING
1.
2.
3.
Code frame
1. Build a code frame around questions, not answer s
2. Learn from other s
3. Look outside the sur vey world for other categorizati on systems
Instructions
1. Instructions should always be written, never oral
2. Clearly defined categories
3. Include examples
4. Test instructions
Producti on
1. Par tition answer s into chunks if needed
EFFECTIVE PRODUCTION
RECOMMENDATION 2
Independent coding
EFFECTIVE PRODUCTION
RECOMMENDATION 2
 Independent coding
 At least two coders independently code all answers (or answer
chunks)
EFFECTIVE PRODUCTION
RECOMMENDATION 2
 Independent coding
 At least two coders independently code all answers (or answer
chunks)
 No contact with each other or with a third party other than a single
authority
EFFECTIVE PRODUCTION
RECOMMENDATION 2
 Independent coding
 At least two coders independently code all answers (or answer
chunks)
 No contact with each other or with a third party other than a single
authority
 All coding questions are written and directed to a single authority
EFFECTIVE PRODUCTION
RECOMMENDATION 2
 Independent coding
 At least two coders independently code all answers (or answer
chunks)
 No contact with each other or with a third party other than a single
authority
 All coding questions are written and directed to a single authority
 A single authority distributes written answers to questions to all
coders simultaneously.
EFFECTIVE PRODUCTION
RECOMMENDATION 2
 Why independent coding?
 If two people reach the same conclusion, following the same
instructions for the same answer, you can be more confident that the
conclusion is “correct”.
FOUR PARTS OF OPTIMAL CODING
1.
2.
3.
Code frame
1. Build a code frame around questions, not answer s
2. Learn from other s
3. Look outside the sur vey world for other categorizati on systems
Instructions
1. Instructions should always be written, never oral
2. Clearly defined categories
3. Include examples
4. Test instructions
Producti on
1. Par tition answer s into chunks if needed
2. Independent coding
EFFECTIVE PRODUCTION
RECOMMENDATION 3
Reconciliation
EFFECTIVE PRODUCTION
RECOMMENDATION 3
 Reconciliation
 Identify coding disagreements from independent coding
EFFECTIVE PRODUCTION
RECOMMENDATION 3
 Reconciliation
 Identify coding disagreements from independent coding
 Written instructions explaining how to reconcile disagreements
EFFECTIVE PRODUCTION
RECOMMENDATION 3
 Reconciliation
 Identify coding disagreements from independent coding
 Written instructions explaining how to reconcile disagreements
 Coders discuss disagreements and determine “correct” coding
EFFECTIVE PRODUCTION
RECOMMENDATION 3
 Reconciliation example
EFFECTIVE PRODUCTION
RECOMMENDATION 3
 Reconciliation example
 An answer to the question: “congresswoman spokeswoman”
EFFECTIVE PRODUCTION
RECOMMENDATION 3
 Reconciliation example
 An answer to the question: “congresswoman spokeswoman”
 Coder 1 assigned 1 code: Speaker of the congress
EFFECTIVE PRODUCTION
RECOMMENDATION 3
 Reconciliation example
 An answer to the question: “congresswoman spokeswoman”
 Coder 1 assigned 1 code: Speaker of the congress
 Coder 2 assigned 2 codes: Speaker, Congresswoman
EFFECTIVE PRODUCTION
RECOMMENDATION 3
 Reconciliation example
 An answer to the question: “congresswoman spokeswoman”
 Coder 1 assigned 1 code: Speaker of the congress
 Coder 2 assigned 2 codes: Speaker, Congresswoman
 After reconciliation: Speaker, Congresswoman
FOUR PARTS OF OPTIMAL CODING
1.
2.
3.
Code frame
1. Build a code frame around questions, not answer s
2. Learn from other s
3. Look outside the sur vey world for other categorizati on systems
Instructions
1. Instructions should always be written, never oral
2. Clearly defined categories
3. Include examples
4. Test instructions
Producti on
1. Par tition answer s into chunks if needed
2. Independent coding
3. Reconciliati on
FOUR PARTS OF OPTIMAL CODING
Part 4
Evaluation
EVALUATION
Determining how well the coding
process worked
EVALUATION
RECOMMENDATION 1
Quantify inter-coder reliability
EVALUATION
RECOMMENDATION 1
 Quantify inter-coder reliability
 Determine how often independent coding produced the same results
EVALUATION
RECOMMENDATION 1
 Quantify inter-coder reliability
 Determine how often independent coding produced the same results
 Percent agreement - Does not take into account chance agreement
levels
EVALUATION
RECOMMENDATION 1
 Quantify inter-coder reliability
 Determine how often independent coding produced the same results
 Percent agreement - Does not take into account chance agreement
levels
 Many standard inter-coder reliability indices account for chance
levels
EVALUATION
RECOMMENDATION 1
 Standard reliability measures that account for chance
agreement










Cohen's Kappa
Holsti's method
Krippendorff's Alpha
Scott's Pi
Perreault-Leigh's Ir
Tinsley-Weiss's T
Bennett-Alpert-Goldstein's S
Lin's concordance coefficient
Hughes-Garrett’s Generalizability Theory approach
Rust-Cooil's PRL approach
EVALUATION
RECOMMENDATION 1
 What does a reliability index mean?
EVALUATION
RECOMMENDATION 1
 What does a reliability index mean?
 Theoretical - Agreement above and beyond what you would expect by
chance
EVALUATION
RECOMMENDATION 1
 What does a reliability index mean?
 Theoretical - Agreement above and beyond what you would expect by
chance
 Practical (Landis and Koch, 1977)
 Reliability > .80 is “almost perfect”
 Reliability > .60 and <.80 is “substantial”
EVALUATION
RECOMMENDATION 1
 An illustration of why you should compute an index that
accounts for chance
EVALUATION
RECOMMENDATION 1
 An illustration of why you should compute an index that
accounts for chance
 Reliability for Vote Choice Timing coding
EVALUATION
RECOMMENDATION 1
 An illustration of why you should compute an index that
accounts for chance
 Reliability for Vote Choice Timing coding
 Percent agreement = 83%
EVALUATION
RECOMMENDATION 1
 An illustration of why you should compute an index that
accounts for chance
 Reliability for Vote Choice Timing coding
 Percent agreement = 83%
 Krippendorff’s α = .81
FOUR PARTS OF OPTIMAL CODING
1.
2.
3.
4.
Code frame
1. Build a code frame around questions, not answer s
2. Learn from other s
3. Look outside the sur vey world for other categorizati on systems
Instructions
1. Instructions should always be written, never oral
2. Clearly defined categories
3. Include examples
4. Test instructions
Producti on
1. Par tition answer s into chunks if needed
2. Independent coding
3. Reconciliati on
Evaluation
1. Quantify inter -coder reliability
EVALUATION
RECOMMENDATION 2
Enable external evaluation
EVALUATION
RECOMMENDATION 2
 Enable external evaluation
 Allow others to critically review your procedures and results
EVALUATION
RECOMMENDATION 2
 Enable external evaluation
 Allow others to critically review your procedures and results
 Allow others to replicate your procedures
EVALUATION
RECOMMENDATION 2
 Fully document and make available all material
EVALUATION
RECOMMENDATION 2
 Fully document and make available all material
 The generation of the code frame
EVALUATION
RECOMMENDATION 2
 Fully document and make available all material
 The generation of the code frame
 Instructions
EVALUATION
RECOMMENDATION 2
 Fully document and make available all material
 The generation of the code frame
 Instructions
 Coding
EVALUATION
RECOMMENDATION 2
 Fully document and make available all material
 The generation of the code frame
 Instructions
 Coding
 Chunking
EVALUATION
RECOMMENDATION 2
 Fully document and make available all material
 The generation of the code frame
 Instructions
 Coding
 Chunking
 Reconciling
EVALUATION
RECOMMENDATION 2
 Fully document and make available all material
 The generation of the code frame
 Instructions
 Coding
 Chunking
 Reconciling
 Description of production coding
EVALUATION
RECOMMENDATION 2
 Fully document and make available all material
 The generation of the code frame
 Instructions
 Coding
 Chunking
 Reconciling
 Description of production coding
 Source data: original or redacted
EVALUATION
RECOMMENDATION 2
 Fully document and make available all material
 The generation of the code frame
 Instructions
 Coding
 Chunking
 Reconciling
 Description of production coding
 Source data: original or redacted
 Inter-coder reliability
FOUR PARTS OF OPTIMAL CODING
1.
2.
3.
4.
Code frame
1. Build a code frame around questions, not answer s
2. Learn from other s
3. Look outside the sur vey world for other categorizati on systems
Instructions
1. Instructions should always be written, never oral
2. Clearly defined categories
3. Include examples
4. Test instructions
Producti on
1. Par tition answer s into chunks if needed
2. Independent coding
3. Reconciliati on
Evaluation
1. Quantify inter -coder reliability
2. Enable external evaluation
FOUR PARTS OF OPTIMAL CODING
1.
2.
3.
4.
Code frame
1. Build a code frame around questions, not answer s
2. Learn from other s
3. Look outside the sur vey world for other categorizati on systems
Instructions
1. Instructions should always be written, never oral
2. Clearly defined categories
3. Include examples
4. Test instructions
Producti on
1. Par tition answer s into chunks if needed
2. Independent coding
3. Reconciliati on
Evaluation
1. Quantify inter -coder reliability
2. Enable external evaluation
WHAT IS THE VALUE OF
SURVEY RESEARCH?
Social decisions often rely on
evaluations
of past actions.
THERE ARE LOTS OF EVALUATIONS
WHICH EVALUATIONS
SHOULD WE BELIEVE?
EVALUATION CRITERIA
 CREDIBLE
 the quality of being believable or trustworthy
 LEGITIMATE
 in accordance with recognized or accepted standards or principles
WHAT IS THE VALUE OF
SURVEY RESEARCH?
It can be a source for credible and legitimate evaluations .
Richard Feynman (1974 –
Caltech Commencement
Address)
“[Scientific
integrity]
corresponds to a
kind of utter
honesty—a kind of
leaning over
backwards….
Richard Feynman (1974 –
Caltech Commencement
Address)
“…the idea is to give
all of the
information to help
others judge the
value of your
contribution;
not just the
information that
leads to judgment
in one particular
direction...”
THREE SOURCES OF CREDIBILIT Y
 Data Access
 Analytic Transparency
 A finding’s meaning often depends on the human actions that
produced it
 Production Transparency
 The meaning of data often depends on the human actions that
produced it
WHAT IS THE VALUE OF
SURVEY RESEARCH?
It can be a
source for
credible and
legitimate
evaluations.
WHAT IS THE VALUE OF
SURVEY RESEARCH?
Surveys allow increased honesty in many kinds of evaluation
THREATS TO CREDIBILITY
 In every survey, a complex decision sequence converts capital
into data points.
 Many elements of this path are not public.
OUR GOAL
Increase
Procedural transparency
Documentational rigor
Credibility of measures & inferences
PRINCIPLE
A basic expectation is to
document,
archive,
and share all data and methodology
so that they are available for careful scrutiny
by other scientists.
LEGITIMATING PRACTICES
 Increased procedural transparency
 Increased documentation at all stages
 Increased attention to coder interpretation of
instructions.
 Evaluation at many stages
HOW WE DID IT
 Theoretical Framework
 Developed with expert committees & recorded
 Code Frame
 Verified with expert committees & recorded
 Chunking
 Developed in cooperation with vendor & recorded
 Coding
 Executed by vendor with rigorous evaluation & recorded
OUR ARGUMENT
 In the case of open-ended data, limited introspection about,
and documentation of, procedures
 undermines the credibility and legitimacy
 and limits the value
 of even the most famous surveys.
CONCLUSION
While it is true that documentation and
validation can be time consuming and
expensive.
Science and society benefits from rigorous
public accounts of how we produce our data.
SUMMARY