INTRODUCTION TO
RESEARCH:
MEASUREMENT
Part 2 of 3
By: Danielle
Davidov, PhD
&
Steve Davis,
MSW, MPA
OUTLINE
 1) Threats to research studies
 2) Steps in the research design process
 3) Identifying and defining variables
 4) Validity and reliability of measurement
RESEARCH DESIGN
Starts After the research question has been
developed and refined
The who, what, when, where, and how of
research
It comprises the Materials and Methods and
Limitations sections of publications
WHY IS DESIGN IMPORTANT?
 The goal of research design is to provide the most
valid and correct answer to the question
 i.e., we want to make sure we are “doing it right”
 This is done by minimizing the threats to the
soundness of your study’s conclusion(s):
 CHANCE
 BIAS
 CONFOUNDING
STUDY THREATS: CHANCE
The threat that the study’s findings are merely
the result of random processes (chance)
i.e., the findings are a “fluke”
We can’t do much to control random error
Also referred to as:
 Type 1 Error
 Random Error
 Unsystematic Error
STUDY THREATS: BIAS

The threat that the study’s results are
due to an unfair preference given to one
group or a set of outcomes in a study
 We can try to control bias in our study design and
subject recruitment
Also referred to as:
 Systematic Error
STUDY THREATS: CONFOUNDING
The threat that the association or relationship
observed in the study is influenced by or
related to another variable
We can control for this in our study design,
subject recruitment, and data analysis techniques
STEPS IN RESEARCH DESIGN
We try to minimize the three main threats
during all stages of design process, which are:
1)
2)
3)
4)
5)
Identifying and Defining Variables*
Selecting Measurement Methods*
Selecting (Sample) Subjects
Selecting a Research Design
Establishing an Analysis Plan
*We will be talking about steps 1 and 2 in this presentation
IDENTIFY & DEFINE VARIABLES
 What do you want to measure? (Identify)
 Ex) Patient satisfaction levels with ultrasound vs. history and physical exam only
 How do you want to operationalize “patient satisfaction?” (Define)
 Ex) Answers of “Good”, “Very Good”, or “Excellent” on a survey given to
patients about the care they received in the emergency department
IDENTIFICATION & DEFINITION OF
VARIABLES
Classifying Variables:
 Independent Variable
 The variable that has an effect on or influences the dependent
variable. This is the FACTOR/INTERVENTION
 i.e.) History and Physical Exam or Ultrasound + H & P
 Dependent Variable
 The variable that is affected by, or dependent upon, the
independent variable. This is the OUTCOME
 i.e.) Patient Satisfaction
IDENTIFICATION & DEFINITION OF
VARIABLES
Classifying Variables (continued)
 Confounding Variables –a variable that is related to both the
independent and the dependent variable
 CONFOUNDER or CONTROL variable
 Common confounders/controls in medical research:




Age
Gender
Race
Severity of Illness
IDENTIFICATION & DEFINITION OF
VARIABLES
Controlling for Confounding Variables
 Not adequately controlling for confounding variables can have
disastrous consequences on your research
 Identify and define as many as possible
 From previous literature
 From clinical observations
 From theory
What if we didn’t consider these important variables
when examining the relationship between the
Independent and Dependent variables???
IDENTIFICATION & DEFINITION OF
VARIABLES
 Operationalizing variables
 The process of defining variables in a measurable way.
IDENTIFICATION & DEFINITION OF
VARIABLES
 Levels of Measurement (NOIR)
 Nominal
Lower
 Ordinal
 Interval
 Ratio
Higher
NOMINAL LEVEL DATA
 Characteristic data that cannot be rank ordered
 This data is “categorical” – made up of “categories”,
not “levels” or “increments”
 Ex) Ice cream flavors—vanilla is not “better” or “more” than
chocolate
 Examples: Gender, Race, Student,
Marital Status, State or Country of Residence,
Insurance Status, Discharge Status, etc.
 Yes/No Responses are Nominal
 This type of data is usually “descriptive”
 Used to describe a population or sample
ORDINAL LEVEL DATA
 Data that can be rank ordered but that do not have measurable
distances between each level of rank
 Likert Scales - Strongly Disagree to Strongly Agree
 Class rank - Freshman, Sophomore, Junior, Senior; PGY-I, PGY-II, PGY-III
 Degree of illness: None, Mild, Moderate, Severe
 Senior is a higher rank than Freshman, but there
is no way to quantify how much higher Senior
is vs. “Freshman” or how much “more” illness
those with a severe illness have compared to
those with a mild illness
INTERVAL/RATIO DATA
 Data that can be ordered and that have a measurable
distance between each level
 The Interval Scale - Distances between positions are equal, but "0" is
an arbitrary assignment. For example, with temperature, each degree
is equally distant from another, but "0" does not mean that there is
no temperature. It is simply a reference point on the scale.
 The Ratio Scale - All positions are equally distant and "0" means that
the value is truly "0". If you have "0" money, you have none. But if you
have $200, you have twice as much as a friend who has $100.
 Examples of Interval/Ratio Data:





Age
Height
Weight
Many Clinical Serum Levels
Blood Pressure
LEVELS OF MEASUREMENT AND
POWER
 Defining variables at higher levels of measurement allows the
use of statistical tests that have more Power
 Power = the probability of finding a true relationship of difference if
it genuinely exists
 It is usually better to collect data at higher levels of
measurement and then collapse into categories later
 Ex) Age
 What is your age? ____ (best)
vs.
 What is your age? vs.
 18 – 25
 26 – 35
 36 – 45 etc.
vs.
 Under 40 & over 40
SELECTING A MEASUREMENT
METHOD
 Once you have defined and operationalized your research
question’s variables, you must decide how to measure them
and/or what measurement tool you will use.
 There are two forms of error that we must minimize when
selecting measurement methods and/or tools:
 Random error (CHANCE)
 Nonrandom error (BIAS AND CONFOUNDING)
RELIABILITY
 To minimize random error we choose a tool or method that is
RELIABLE
 Reliability – The extent to which a measurement method or tool
produces the same results over several measurements
 AKA precision
 Threats to Reliability
 Observer error: different measurements from the same or different
observers (i.e., blood pressure readings)
 Instrument error: different measurements from the instrument itself
due to extraneous environmental factors
 Subject error: different measurements from the natural biological
variability among humans
ASSESSING & MAXIMIZING
RELIABILITY
 How to assess Reliability:
 Repeat measurements on the same subject.
 Give a survey at two different time points
 Take blood pressure at two different time points
 Use more than one observer.
 Assess inter-rater agreement
 Have two different people take blood pressure
 How to maximize Reliability:
 Standardize the measurement methods
 Choose surveys and instruments that have been proven to be reliable
 Train observers
 Refine & update instruments
 Repetition
 Averaging the measures can cancel out error.
VALIDITY
 To minimize nonrandom error we choose a measurement
method and/or tool that is VALID
 Validity – The extent to which a measurement method and/or tool
measures what is sets out to measure
 AKA Accuracy
 Threats to Validity
 Observer bias: conscious or unconscious distortion in the perception
and/or reporting of the measurement
 Subject bias: bias-distortion of self-reported measurements due to
subjects beliefs and biases
 Hawthorne Effects and Social Desirability
 Instrument bias: consistently biased or inaccurate measurements
due to such things as worn parts or mechanical malfunction
 Lack of a clear gold standard: No “best” instrument out there
 Abstract/behavioral variables: These things are difficult to measure
 Patient satisfaction, pain, quality of life, intelligence
MAXIMIZING VALIDITY
 Strategies for maximizing Validity
 Blinding
 Ex) Do not allow physician who is taking blood pressure readings to know
which subjects are receiving blood pressure medication
 Deception
 Ex) Do not allow subjects to know which “group” they are in
 Give placebos
 Instrument Calibration
 Make sure instruments are working properly
 Use standardized/validated surveys and assessment tools
 Find these from literature searches
 Usually better to use “pre-made” surveys or instruments than creating one
from scratch
IN SUMMARY
 Identify and define your variables at the VERY beginning of
your study
 Don’t forget your control or confounding variables!
 Using higher levels of measurement is better!
 Choose instruments and data collection tools that are:
 RELIABLE – produce the same results over time (precise)
 VALID – produce results that represent “the truth” (accurate)
NEXT STEPS
 Go through “Introduction to Research Part 3: Sampling and
Design”
REFERENCES
Hulley SB, Cummings SR, Browner WS, Grady D, Hearst N,
Newman TB. Designing Clinical Research. 2nd ed.
Philadelphia, PA: Lippincott Williams & Wilkins; 2001:37-49
Spector PE. Research Designs. Newbury Park, CA: SAGE
Publications, Inc.; 1981 . ISBN: 0 -8039-1709-0
http://www.research-assessment-adviser.com/levels-ofmeasurement.html