Data Collection, Measurement, &
Data Quality in Quantitative and
Qualitative Research
1
Data Collection Methods
• Without appropriate data collection methods, the
validity of research conclusions is easily
challenged
2
Data Collection Methods
• Using New Data
– Collect own data for the
study
3
Data Collection Methods
• Using Existing Data
– Historical research
• Use records and other
documents from the past
– Secondary analysis
• Use of data gathered in a
previous study
4
Key Dimensions of
Data Collection Methods
• Structure
– The data collection should be very structured and consistent
• Quantifiability
– Able to be analyzed statistically
• Obtrusiveness
– Degree to which people are aware that they are being studied
• Objectivity
– Try to be as objective as possible
5
Data Collection
Quantitative Research
6
Types of Data Collection
• Self-Reports
• Observation
• Biophysiologic Measures
7
Types of Data Collection
• Self-Reports
–
–
–
–
–
–
Interviews
Questionnaires
Scales
Vignettes
Projective techniques
Q-sorts
8
Types of Data Collection: SelfReports
Interviews and Questionnaires (Structured)
• Participant's responses to questions by researcher
• Data is usually collected by means of a formal, written
document (instrument)
• Uses an interview schedule for questions that are asked
orally (face to face or via phone)
• Uses a questionnaire when participants complete the
instrument themselves
9
Types of Data Collection: SelfReports
Interviews and Questionnaires (Structured)
– Closed-ended questions (fixed alternative questions)
•
•
•
•
•
•
•
Response alternatives are specified by the researcher
Ensures comparability of responses
Facilitates analysis
Easy to administer
More efficient time use
Difficult to develop
Could lead to overlooking something important
10
Types of Data Collection: SelfReports
Interviews and Questionnaires (Structured)
– Open-ended questions
• Allows participants to respond to questions in their own
words
• Allows for richer, fuller information
11
Types of Data Collection: SelfReports
Interviews and Questionnaires
(Structured)
– Instrument Construction
• Develop outline of content of
research
• Design questions
• Pretest
– Trial run to determine if
instrument is free of biases,
errors, etc
12
Types of Data Collection: SelfReports
• Interviews Vs. Questionnaires
– Advantages of questionnaires
•
•
•
•
Less costly
Require less time and effort to administer
Can be completely anonymous
No biases relating to the researcher being present
13
Types of Data Collection: SelfReports
• Interviews Vs. Questionnaires
– Advantages of Interviews
• Response rate is higher in face to face interviews
• Effective for those that can not complete questionnaires
(children, blind, ESL, elderly)
• Questions are less likely to be misinterpreted than
questionnaires
• Interviews can produce additional information through
observation
14
Types of Data Collection: SelfReports
• Interviews Vs. Questionnaires
– Interviews are considered to be superior to questionnaires
15
Types of Data Collection: SelfReports
Types of Self-Reports (Structured)
•
•
•
•
Composite Scales (social - psychological)
Vignettes
Projective techniques
Q sorts
16
Types of Data Collection: SelfReports
Composite Scales (social - psychological)
– Scale: assigns a numeric score to people to place them on a
continuum with respect to attributes being measured
17
Types of Data Collection: SelfReports
Composite Scales (social - psychological)
– Likert scale
– Semantic Differential scale
– Visual Analog scale
18
Types of Data Collection: SelfReports
Composite Scales (social - psychological)
– Likert scale (summated rating scales)
• Consists of several declarative statements that express a
viewpoint
• Participant indicates the degree to which they agree to
disagree
• Able to summate the scores allowing for discrimination
among people with different viewpoints
19
Types of Data Collection: SelfReports
Composite Scales (social - psychological)
Example Likert Scale:
AU nursing students are very well prepared for working within the
current healthcare system
Strongly agree
Agree
Neutral
Disagree
Strongly disagree
20
Types of Data Collection: SelfReports
Composite Scales (social - psychological)
– Semantic Differential
• Participants rate a concept on a series of bipolar adjectives
• Can measure any concept
– Visual Analog Scale
• The scale is a straight line with anchors which are the
extreme limits of the experience or feeling
• Measures subjective experiences
21
Types of Data Collection: SelfReports
– Semantic Differential
Example
AU nursing graduates are:
Competent
Incompetent
Intelligent
Dim
– Visual Analog Scale
Example
On a scale of 0 to 10 how would you rate your pain if 10 was
the worst pain you have even experienced and 0 was no pain
22
Advantages of Scales
– Scales allow researchers to efficiently quantify the
strength and intensities of individual characteristics
– Discriminates among people with different attitudes,
fears, motives, perceptions, personality traits, needs
– Good for group and individual comparisons
– Can be implemented either verbally or in writing
23
Disadvantages of Scales
Response set biases
– Social Desirability Response Set Bias
• Participants give answers that are common social views
– Extreme Response Set Bias
• Participants express attitudes or feelings in the extreme (always, never)
– Acquiescence Response Set Bias
• Participants agree with all statements (yea-sayers or nay-sayers)
24
Disadvantages of Scales
– Ways to Reduce Response Set Biases
• Counterbalancing: positively and negatively worded
statements
• Developing sensitively worded questions
• Creating a permissive, nonjudgmental atmosphere
• Guaranteeing confidentiality
25
Types of Data Collection: SelfReports
Vignettes
– Brief description of events or situations to which
participants are asked to react
– Information about perceptions, opinions, or knowledge
– Questions post vignettes may be open-ended or closeended
– Economical to administer
– May contain response biases
26
Types of Data Collection: SelfReports
Projective Techniques
– Verbal self reports to obtain psychological
measurements
– Seek minimal participants’ conscious cooperation
– Ambiguous or unstructured stimuli elicits participants
needs, motives, attitudes, personality traits
i.e. Inkblot test, word association, role playing, drawing
– Useful in children, hearing or speech impaired
27
Types of Data Collection: SelfReports
Q Sorts
– Uses a set of card with words, phrases or statements
– Participant sorts cards along a bipolar dimension
(agree/disagree)
28
Advantages of Self-Reporting
Methods
• Most common method of data collection used by
nurses
• Reveal information that is difficult to obtain by
other means
• Can gather retrospective and prospective data
• Can measure psychological characteristics
29
Disadvantages of Self-Reporting
Methods
• Questionable validity and
accuracy
• Biases
30
Types of Data Collection:
Observation
• Observational Methods
– An alternative to self-reports
– Can be used to gather information such as characteristics,
condition of individuals, verbal communication, nonverbal
communication, activities, environmental conditions
31
Types of Data Collection:
Observation
Observational Methods
• Researcher has flexibility in the following areas:
– The focus of observation
• What events are to be observed
– Concealment
– Duration of observation
– Method of recording observations
32
Types of Data Collection:
Observation
Observational Methods (structured)
– Categories and checklists
– Rating Scales
33
Types of Data Collection:
Observation
Categories and Checklists
• Category system:
– attempts to designate information in a systematic, quantitative
manner
– Clear definition of behaviors and characteristics to be observed is
necessary
– Lists all behaviors or activities the observer wants to observe and
records occurrences
• Checklist:
– instrument to record observations
• Rating Scales:
– Are tools that require the observer to rate some phenomena along34a
descriptive continuum
Types of Data Collection:
Observation
– Observational Sampling
• Time sampling
– Selection of time periods for observations
• Event sampling
– Selects behaviors or events for observation
35
Evaluation of Observational
Methods
• Advantages
– Provides depth and variety of information
– Some problems are better suited to observation
• Disadvantages
–
–
–
–
Potential ethical issues
Lack of consent to be observed
Participants reaction to be observed
Biases
• Faulty inferences
36
Types of Data Collection
Biophysiologic
37
Types of Data Collection:
Biophysiologic
Types of Biophysiologic Measures
– In vivo
• Measures performed directly within or on living organisms
– i.e. blood pressure, temperature
– In vitro
• Data gathered from participants by extracting some
biophysiologic material from them for lab analysis
– i.e. blood work, microbiologic measures, cytology and histological
measures
38
Advantages of Biophysiologic
Measures
–
–
–
–
Are relatively accurate and precise
Are objective
Provide valid measures of targeted variables
Equipment is readily available
39
Disadvantages of Biophysiologic
Measures
– Measuring tool may affect variables it is attempting to
measure
– Interferences may create artifact
– Energy must often be applied to the organism when
taking measurements
40
• Measurement and Assessment of Data
41
Measurement
• Involves rules for assigning numeric values to
qualities
• Determines how much of an attribute is present
• Quantification
– Communicates the amount in numbers
42
Advantages of Measurement
–
–
–
–
Removes guesswork in gathering information
Tends to be objective
Obtains precise information
Can differentiate among people who possess different
degrees of an attribute
– Common language
43
Errors of Measurement
– Always the potential for error in all tools
– Extraneous factors affect measurement and distort
results
• Obtained score – is observed score
• True score – true score if no errors
• Error of measurement – the different between the true and
obtained scores
44
Factors Contributing to
Errors of Measurement
– Situational contaminants
» People’s awareness of observer, environmental
factors
– Response set biases
– Transitory personal factors
» Fatigue, mood, hunger (temporary)
– Administration variations
» Alterations in data collection methods
– Item sampling
» Errors introduced as a result of sampling
45
Reliability of Measuring Instruments
Reliability
– Refers to the consistency with which an instrument
measures the attribute
– The less variation in repeat measures the higher its reliability
46
Reliability of Measuring Instruments
Reliability
– Aspects of reliability
• Stability
• Internal consistency
• Equivalence
47
Reliability of Measuring Instruments
• Stability
• The extent to which the same scores are obtained when
the instrument is used with the same people on separate
occasions
• To assess stability: Test-retest reliability
– researcher administers the same measure to a sample of
people on two occasions and then compares the scores
48
Reliability of Measuring Instruments
• Internal Consistency
• Reliable to the extent that all its subparts measure the
same characteristic
• To assess internal consistency: Split-half technique
– the items comprising the test or scale are split into two
groups and scored, compute reliability coefficient
49
Reliability of Measuring Instruments
• Equivalence
• Determines the consistency or equivalence of the
instrument by different observers or raters
• To assess equivalence – interrater (interobserver) reliability
– Has two or more trained observers make simultaneous,
independent observations, compete reliability coefficient
50
Reliability of Measuring Instruments
• Reliability Coefficients
– A quantitative statistic that estimates how reliable an
instrument is
• Determine an instrument’s quality
• Low reliability makes it difficult to adequately test research
hypothesis
• If sample too homogeneous, the lower reliability coefficient
will be (instruments are designed to measure differences)
51
Validity of Measuring Instruments
Validity
– Is the concern whether the measurement tools actually
measure what they are supposed to measure
52
Validity of Measuring Instruments
Aspects of Validity
•
•
•
•
Face validity
Content validity
Criterion-related validity
Construct validity
53
Validity of Measuring Instruments
– Face validity
• Whether the instrument
looks as though it is
measuring the
appropriate construct
54
Validity of Measuring Instruments
– Content Validity
• Concerned with
adequacy of coverage
of the content area
being measured
– Tests of knowledge
– Psychosocial traits
– Based on judgment
55
Validity of Measuring Instruments
– Criterion-Related
Validity
• Wants to establish the
relationship between
scores on an instrument
and some external
criterion
– Compute a validity
coefficient –
correlates scores on
the instrument with
scores in the
criterion variable
56
Validity of Measuring Instruments
– Construct Validity
• Concerned with what
construct is the
instrument actually
measuring
57
Validity of Measuring Instruments
• To assess construct validity–
– known-groups technique
» Groups that are expected to differ on certain
attributes are administered the instrument then
scores are compared
– Factor analysis
» Statistical procedure
– Examination of relationships based on theoretical
predictions
58
Reliability of Measuring Instruments
• If a measuring device is not reliable, it can not be valid
– High reliability of an instrument provides no evidence of its
validity
– Low reliability is evidence of low validity
• An instrument can be reliable without being valid
Reliability consistently measures accurately
Validity measures what it is supposed to
59
Data Collection
Qualitative Research
60
Questions for Thought
• What are the systematic rules for analyzing
qualitative data?
61
Qualitative Data Collection
Types of Self-Reports - Unstructured
Self-Reports Methods (Unstructured)
• Interviews
• Diaries
• Observation
62
Qualitative Data Collection
Types of Self-Reports - Unstructured
Interviews
•
•
•
•
•
Flexible
Not directed by set questions
Interviews are conversational in nature
Usually interviews are long
Can be tape recorded or researcher may take notes
63
Qualitative Data Collection
Types of Self-Reports - Unstructured
• Completely unstructured interviews
• Start with broad (grand tour) questions
• Further questions are guided by initial responses – one
question's answer leads to the next question
• Focused or semi-structured interviews
• Researcher lists topics that must be covered in an interview
• Uses a topic guide to ensure all question areas are covered
64
Qualitative Data Collection
Types of Self-Reports - Unstructured
• Focus group interviews
• Interviews with groups of 5 to 15 people whose opinions and
experiences are solicited simultaneously
• Uses topic guide to guide questions
65
Qualitative Data Collection
Types of Self-Reports - Unstructured
• Life Histories
• Narrative self-disclosures about life experiences
• Has informants describe experiences in chronological order
• Orally or written
66
Qualitative Data Collection
Types of Self-Reports - Unstructured
Diaries
• Have informants
maintain daily logs of
some aspect of their
lives
67
Qualitative Data Collection:
Observational Methods - Unstructured
Observational Methods
• Unstructured observation
– Attempt to see the world as the participants see it
– Participant observation – data collector actually participates in the
group
» Participation can be from the role as an observer or totally
immersed in the social setting as a participant
» Researcher needs to gain entrée into the social group under
investigation
» Researcher needs to establish rapport and develop trust within
the group
68
Qualitative Data Collection:
Observational Methods - Unstructured
• Observational Data Collection
– Physical setting
» In what context is the human behaviour occurring
– Participants
» Information about the participants, what are their roles,
characteristics
– Activities
» What are the participants doing
– Frequency and duration
» Specific information about the activity
69
Qualitative Data Collection:
Observational Methods - Unstructured
– Process
» How is event occurring
– Outcomes
» Why is the activity occurring and what are the results
– Single positioning
» Staying in a single location
– Multiple positioning
» Involves moving around to observe behaviour from different
perspectives
– Mobile positioning
» Involves following a person throughout a given activity
70
Qualitative Data Collection:
Observational Methods - Unstructured
• Observational Data Recording
• Uses logs and field notes
» Log – records daily events
» Field notes – observer’s efforts to record information and
understand data
» Observational notes – descriptions of events and conversations
» Theoretical notes – interpretive attempts to attach meaning to
observations
» Methodologic notes – instructions about what observations that
need to be made
» Personal notes – comments about researcher’s own feelings
71
Assessment of Qualitative Data
– Do the measures used by the researcher yield data reflecting
the truth
– Qualitative research attempts to do this through establishing the
data’s trustworthiness
72
Assessment of Qualitative Data
Establish Trustworthiness by assessing:
–
–
–
–
1. Credibility
2. Dependability
3. Confirmability
4. Transferability
73
Assessment of Qualitative Data:
Trustworthiness
1. Credibility
– Confidence in the truth of the data
• Prolonged engagement and persistent observation
– Sufficient time to collect data, focus on the phenomena being studied
74
Assessment of Qualitative Data:
Trustworthiness
Triangulation
– Use of multiple referents to draw conclusions, attempts to distinguish true
information from errors
– Data Source Triangulation
» Multiple data sources (interviewing diverse informants on same topic)
– Investigator Triangulation
» Using more than one person to collect data
– Theory Triangulation
» Using multiple perspectives to interpret data
– Method Triangulation
» Using multiple methods (observation and interviews)
75
Assessment of Qualitative Data:
Trustworthiness
• External checks: Peer debriefing and member checks
– Peer debriefing – review and explore various aspects of inquiry with
objective peers
– Member checks – providing feedback to study participants and
assessing their reactions
• Searching for Disconfirming evidence
– Search for data that challenges the emerging conceptualization or
theory
• Researcher credibility
76
Assessment of Qualitative Data:
Trustworthiness
2. Dependability
– Data stability over time and over conditions
– Stepwise replication
• Having several researchers break into teams and evaluate
the data separately and then compare conclusions
– Inquiry audit
• Scrutiny of the data and supporting documents by an
external reviewer
77
Assessment of Qualitative Data:
Trustworthiness
3. Confirmability
– The objectivity or neutrality of the data, can other
independent people agree about data’s relevance
– Audit trail – documentation that allows an independent
auditor to come to the same conclusions about the
data
78
Assessment of Qualitative Data:
Trustworthiness
4. Transferability
– The extent to which the findings from the data can be
transferred to other settings or groups
79
Reference
• Loiselle, C. G., Profetto-McGrath, J., Polit, D. F., &
Beck, C. T. (2011). Canadian essentials of nursing
research. (Third Edition). Philadelphia: Lippincott,
Williams & Wilkins.
80