Conducting Experiments, Survey
Construction and Data Collection
RCS 6740
7/18/05
Practical Considerations of
Research
 This lecture will focus on some of the
practical aspects of conducting research
studies such as:
 How do you select research participants
 What should you consider when deciding how to
manipulate an independent variable
 How do you measure a variable
 What should you consider when constructing a
survey
 What should you consider when collecting data
 How do ensure control in your study
Selecting Research Participants
 Define your participants
 People with Aphasia
 Women with Breast Cancer
 Rats with 5 Legs
 Define your population
 Everyone in the United States
 People in RCS 6740
 Consumers at “The Center” (CIL)
Selecting Research Participants
Cont.
 Define your sampling technique
 Probability sampling
 Random sampling
 Stratified random sampling
 Systematic sampling
 Cluster sampling
 Non-probability sampling
 Quota sampling
 Determine your sample size
 Remember, larger sample sizes may result in
more significant data
Manipulating the Independent
Variable


Construct the Operational Definition of your variables
 Exercise as defined by jogging for 20 minutes, 3 times a week, for
6 weeks
 Depression as defined by the Geriatric Depression Inventory
Set the Stage
 Provide the participants with Informed Consent
 Usually a document that entails the name of the study,
purpose(s) of the study, potential benefits and risks, funding
sources, contact information, and a statement that they can
withdraw at any time
 Explain to the participants in more detail why the study is being
conducted
 Rarely are hypotheses given to participants
 Tell them the study looks at employment when the
hypothesis looks specifically at bias towards the
employment of people with disabilities
 Can be completely truthful and precise
 Deception may be used
 When deception is used, it is imperative that participants
are told the truth during a debriefing at the conclusion of
the study
Types of Manipulations
 Straightforward Manipulations
 Presenting written, verbal, or visual material to
participants
 Examples:
 Having participants read material on substance abuse
prevention
 Read a passage to a participant and ask them
questions later to test recall (memory)
 Have a computer program that presents different
words that someone needs to pronounce to measure
the level of aphasia
 Most research is done in a straightforward manner
 Researchers can vary the difficulty of material,
motivational levels, the way questions are asked, or
the characteristics of the participants
Types of Manipulations Cont.

Staged Manipulations


Also called Event Manipulations
Used for 2 reasons


The researcher may need to create some sort of psychological
state (anxiety)
The researcher may need to stage a manipulation to recreate a
real-world scenario


Staged manipulations usually employ a confederate



Having a participant do one task and then having them do more
tasks at the same time
Also called an accomplice
A confederate is someone who appears to be another
particpant in an experiment but is really a part of the
experiment
Example: Someone who purposely insults a participant in a
study in order to provoke anger or frustration
Types of Manipulations Cont.
 Another example of the use of confederates:
 Asch (1956) study on conformity
 Which line is bigger?
 1)-------------------- 2)---------------------------- 3)-------------------
 Right before a participant had to choose which line
was the longest, a confederate announced an
incorrect answer
 Repeatedly, Asch found that people conformed to the
confederate’s incorrect response
Types of Manipulations Cont.
 Straightforward Manipulations Vs.
Staged Manipulations:
 It is easier to recreate real-world situations
using Staged manipulations
 Staged manipulations are more extensive in
terms of cost and set up procedures
 It is difficult for other researchers to
replicate staged manipulations
 Data and results of Straightforward
manipulations are easier to interpret due to
a greater sense of control
Strength of Manipulations
 The strength of a manipulation can effect
how results are interpreted
 Example: In a study on the use of a new
medication to treat diabetes, it is important that
the researcher uses a sufficient dosage of the
drug in order to see an effect
 5 mgs of Diabetex does not reduce blood sugar
levels
 10 mgs of Diabetex reduces blood sugar levels
by 3%
 20 mgs of Diabetex reduces blood sugar levels
by 10%
Strength of Manipulations Cont.
 A strong manipulation maximizes the
differences between groups and increases the
chance that the independent variable will have
a significant effect on the dependent variable
 Strong manipulations are especially important
in the early stages of research (Why?)
 Although you want a manipulation to be as
strong as possible, ethical considerations
should be examined
 You do not want to make someone so
anxious that they decide to jump off a
building
Cost of Manipulations
 Most researchers are limited in terms
of the funding they have for a
research project
 Confederates and elaborate
manipulations may cost a great deal
 The straightforward approach is
usually more budget friendly
Measuring the Dependent Variable
Types of Measures
 Self-report measures
 Behavioral Measures
 Physiological Measures
Measuring the Dependent Variable
Cont.
Self-report Measures
 Used to measure attitudes, judgments
about oneself or another, emotional states,
confidence in oneself or others, and other
aspects of human thought and behavior
 Example: Please use the following 7 point
scale to answer this question. I feel that I
am the best student in this class:
Strongly Disagree _ _ _ _ _ _ _ Strongly Agree
Measuring the Dependent Variable
Cont.
Behavioral Measures
 Direct observations of behaviors such as:
 How many errors someone makes on a test
 How many times a person can lift a 10lb object
 How may free throws a person makes in 30 minutes
 Things to consider with behavioral measures
 Rate: How many times a behavior occurs
 Reaction Time: How quickly a response occurs after a
stimulus
 Duration: How long a behavior lasts
Measuring the Dependent Variable
Cont.
Physiological Measures
 Recordings of responses of the body
 Galvanic Skin Response (GSR): Measures the
electrical conductance of the skin to look for arousal
or anxiety
 Electromyogram (EMG): Measures muscle tension to
look for stress
 Electroencephalogram (EEG): Measures activity of
brain cells such as looking at what area of the brain is
active while dreaming
 Magnetic Resonance Imaging (MRI): A technique
used to gather detailed images of the inner body
 Functional Magnetic Resonance Imaging (fMRI):
Looking at the inner body while someone performs a
physical or a cognitive task
Measuring the Dependent Variable
Cont.
What Type of Measure(s) to use?
 Basically, what type(s) of measure a researcher will use
depends on the nature of the study, cost, and possible
ethical considerations
 If at all possible a researcher should use a combination
of all three types:
 To study anxiety, a researcher can administer a selfreport measure asking how nervous a person thinks
they are, a behavioral measure that looks at how
many times a person’s voice cracks, and a
physiological measure looking at fluctuations in
someone’s heart rate
 When administering multiple measures, it is important to
remember to account for a possible order effect
 Example: A self-report measure on how nervous a
person is might cause them to become more nervous
thus increasing their heart rate
Measuring the Dependent Variable
Cont.
Sensitivity of the Dependent Variable
 The dependent variable should be sensitive enough to
detect differences
 Do you like your boss (not sensitive)
 On a scale of 1 to 10, how much do you like your
boss (very sensitive)
 Beware of the Ceiling Effect and the Floor Effect
 Ceiling Effect: Task is too easy so everyone performs
well
 Lifting a 1lb object
 Floor Effect: Task is too difficult so everyone performs
poorly
 Lifting a 200lb object
Information on Survey Construction
 DESIGNING SURVEYS
 A good question is one that produces
answers that are reliable and valid
measures of something that we want
to describe
 2 types of question evaluations:
 Those aimed at evaluating reliability
 Those aimed at assessing the validity of
answers
Survey Construction Cont.
 5 process standards of questions and
answers
 Questions need to be consistently understood.
 Questions need to be consistently administered
or communicated to respondents.
 What constitutes an adequate answer should be
consistently communicated.
 Unless measuring knowledge is the goal of the
question, all respondents should have access to
the information needed to answer the question
accurately.
 Respondents must be willing to provide the
answers called for in the question.
Survey Construction Cont.
 Possible steps to assess the
extent to which questions meet
process standards
 Focus group discussions
 Intensive or cognitive interviews
 Field pre-tests under realistic conditions
Survey Construction Cont.
 Some general rules for designing good survey
instruments






The strength of survey research is asking people about
their firsthand experiences: what they have done, their
current situations, their feelings and perceptions.
Questions should be asked one at a time.
A survey question should be worded so that all respondents
are answering the same question.
If a survey is to be interviewer administered, wording of
the questions must constitute a complete and adequate
script such that when the interviewer reads the question as
worded, the respondent will be fairly prepared to answer
the question.
All respondents should understand the kind of answer that
constitutes an adequate answer to a question.
Survey instruments should be designed so that the tasks of
reading questions, following instructions, and recording
answers are as easy as possible for interviewers and
respondents.
Survey Construction Cont.
 Ways of addressing validity in
surveys
 Deriving questions from relevant
literature
 Expert panel
Data Collection Tips








Use more than one survey or measure
Come prepared

Extra measures, pens, equipment

Important not to sway a response

(999=missing, 888=don’t know)

How to score it, the factors, the subscales
Be prepared to answer questions about the survey
Follow up to ensure all items were filled out
Know your survey
If possible, have a limited amount of researchers collect the
data
Enter data as quickly as possible and use a trusting
computer program (SPSS, ECXEL)
Have two people enter the data (one reads, one enters)
Control Considerations
Participant Expectations
 When a participant knows too much about a study,
they may expect to perform in a certain manner
 Demand Characteristics: any feature of an experiment
that might inform participants of the true purpose of a
study
 Once someone is aware of the hypothesis of a study,
the usually do whatever it takes to confirm the
hypothesis
 Hypothesis: People who take a drug will lessen their
anxiety levels. Once a participant knows this, they may
choose to do whatever they can do become less anxious
(e.g., exercise, see a counselor, meditate).
Control Considerations Cont.





Ways to Control Participant Expectations
Use deception to cover up what is really being studied
Administer irrelevant measures
Add filler questions to measures
 Filler questions are unrelated questions on a survey
that should disguise the true purpose of a study
Include a Placebo group in a study
 Placebo effect: When it is not know if results of a
treatment can be attributed to the treatment itself or
the expectancies of the treatment
Placebos may consist of giving someone a pill that is
made of sugar or injecting someone with saline solution
 It is important to remember ethical consideration of
using a placebo group. For example, if someone in
the placebo group becomes very ill, you may want to
administer treatment to them
Control Considerations Cont.
Experimenter Expectations
 Sometimes experimenters who know the purpose of a
study may expect a participant to act or perform a
certain way
 Also known as expectancy bias
 There are 2 potential sources of experimenter bias
 When a researcher treats participants in the control
group differently than participants in the treatment
group
 Spending more time with the treatment group
 Researchers may record or analyze observations in a
biased manner
 Well, even though he spoke 10 words clearly, I thought
he was close on another so I will give him 11/20
Control Considerations Cont.
Ways to Control Experimenter Expectations
 Researchers should be well trained and highly ethical
 Run all conditions simultaneously so the
experimenter’s behavior is the same for all
participants
 Use experimenters who are unaware of the true
hypothesis of the study
 Double-Blind Study: Neither the participant nor the
experimenter knows whether a placebo or the actual
treatment is given
Once again, it is highly unethical for experimenters
to influence participants or manipulate data!!!
Additional Tips for Conducting
Experiments
 Once a study is designed, there are certain ways to
improve it
 Write up a thorough Research Proposal
 Includes a literature review, purpose of the study,
procedures of the study, copies of measures, a
budget, and the manner with which the data will be
analyzed
 Used often for grants, Institutional Review Boards,
and Ethics Committees
 Conduct a Pilot Study
 A trial run of a study with small sample sizes
 Used to “get the kinks” out
 Used to gather participant feedback
Additional Considerations of
Experiments Cont.



Manipulation Checks
 A manipulation check is a way to measure if the independent
variable has an effect on the dependent variable
 Checks if the independent variable manipulation is a
successful operationalization of the conceptual variable
being studied
 Provides evidence for Construct Validity
 Allows researchers to change something in a study before
it is too late
 Allows researchers to see if they are getting significant
results
Example: In a study looking at whether people are nicer to
attractive people, a researcher will determine whether the person
who they chose to be attractive really is being viewed as so by the
participants. If they are not viewed as attractive, the researchers
can hire another model.
Manipulation Checks may consist of any type of measure (i.e., selfreport, behavioral, physiological)
Additional Considerations of
Experiments Cont.
 Debriefing
 Debriefing is the act of meeting with the
participants after the conclusion of the study to
discuss the true purpose of the study, ethical
considerations of the study, and educational
implications of the study
 Can also be used to ensure that the participants
do not share any information with those still
involved in the study
Can also be beneficial as researchers may gain
valuable information about how the study
looks and feels from the “inside”
Analyzing Data
 After a quantitative study has
concluded, all data should be
analyzed statistically
Communicating Research to Others
 Once a researcher has finished a study,
they should communicate their results and
findings to others
 Professional Meetings and Conferences (NRA)
that present the opportunity to share results in a
verbal, written, or poster format
 Journal Articles also are a good way to share
results
 Peer Review is the process of a paper being
accepted or declined entry into a journal based
on reviews from peers
 About 90% of papers submitted to the more
prestigious journals are rejected  but do not
let this get you down 