12. Are previous studies reviewed?
 Yes
Are the references relevant?
 Yes- they use the same variables: Vaccination and behavioural nudges/sciences
 Articles are from reputable journals and interdisciplinary publications such as:
Medical journals
 One articles is a RCT Level II evidence
 Are within current time from 2020-2021
13. From the studies reviewed - describe in your own words what is already known about
the research problem?
 Dai et al article: There is evidence that suggests that behavioural nudges may
increase COVID-19 vaccination compliance.
 That the pandemic increased the use of digital health care technology
 Health care systems now are able to utilize technology to increase health care
communication
 Nudges change the way information or choices are delivered that may impact
patient behaviour
Identify the gap and or conflict in knowledge about the problem.
GAP: the best way to design a text message to nudge patients to get vaccinated has not
been well examined.
And so how does this study intend to fill the gap/resolve the problem?
 Conduct a randomized control clinical trail to test 19 different text messages to promote
influenza vaccination rates from September 2020- March 2021.
 Text messages will be sent to patients before their upcoming appointment with
their primary care provider to see the effect on vaccination rates.

14. Research approach method:
 Quantitative
How can you tell?
 Uses the terms control and treatment group
 Uses the term random in relation to the sample
 Variable are measured by instruments
 Utilizes statistical analysis such as p-value, correlational coefficients= r, and alpha
15. Identify the study design as:
 Experimental - True
How can you tell?
 The study has Randomization, control, and manipulation
 In order to be true >> randomization of all subjects >> which is required which
this study has.
 There is a control group
 There is a manipulated group >> intervention group
Based on the study design, what is one limitation seen for this design?
 "Winners curse"
 History - internal validity
 Another event may effect the dependent variable >> Historical bias or issues
with specific marginalized groups of people

Measurement effects: "many different scientists developed
16. Describe the setting/environment for the study?
 Setting: field study = 90 primary care practices at two large health systems in
Northeastern US between September 2020 and march 2021
STUDY SUBJECTS:
17. What did the researchers use as the accessible population?
 All adults with a new or return in-person clinic appt with their primary care
provider at 90 care practices in two major health care systems in northeastern
USA.
What inclusion criteria were used in subject selection for the study? (gender, age,
diagnosis, occupation, etc).
 Eligible patients:
o adults with a new or return in-person clinic appt with their primary care
provider during the study period
o Ages 18 +
o Pts who canceled their appts less than 4 days in advance, Pt's who did not show
up to or who converted their appt to telemedicine, Pts who rescheduled their
original appt were included
o Clinicians who pts who were eligible were: physicians, resident physicians, NP's,
and physician assistants
 Excluded = based on electronic health records
o pt's received the influenza vaccine prior to the visit
o had a documented allergy or adverse event to the vaccine
o did not have a cell phone number to receive text messages to, or
o had previously opted out of receiving test message or participating in research.
Do the characteristics of the sample actually obtained reflect/represent the target
population?

Characteristics actually obtained:
o Sociodemographics
 Age, Mean (SD)
 Female sex, n (%)
 Race/ethnicity, n (%)
 White non-Hispanic Black non-Hispanic Asian non-Hispanic Hispanic
 Other
o Location, n (%)
 Penn Medicine
o Received flu shot in prior year, n (%)
o Clinician type during visit, n (%)
 Attending/Faculty physician Resident
 Physician assistant
 Nurse practitioner
Yes/no/somewhat?
Elaborate (hint: first scan descriptive statistics of sample and then compare the two or
more groups being studied) >> now look back at the target population you identified





Target population: Adults in the United States with an upcoming primary care
appointment
o Adults: 18+
o Clinician types are similar to the inclusion criteria
o Location: is within united states
The representativeness of a sample cannot be guaranteed without access to a
database about the entire population.
The demographics of the sample and control group have similar standards of deviation
The sample and target population are not extremely homogenous
Because it is difficult and inefficient to assess an entire population, the researcher must
employ sampling strategies that minimize or control for sample bias.
If an appropriate sampling strategy is used, the sample data will almost always
enable a reasonably accurate understanding of the phenomena under investigation.
18. What procedures were used to select the sample: Probability or non-probability? And
identify the specific sub-type?
 Probability >> as it was a randomized sample.
o Sub-type = Stratified Random Sampling
19. What is the sample size?
 N= 74,811
What is the reason given for this sample size? Any power analysis?
If no reason given >> attempt to infer why the sample is this size
 A general rule of sample size in quantitative studies is to always use a larger sample so
that it is more likely to be representative of the population >> it was conducted as a
megastudy
 Power analysis was estimated: at least 90% power to detect a difference of 3.45
percentage points in vaccination rates between intervention and control group.
o This assumed a baseline vaccination rate of 33.0% and a 2 sided alpha = 0.05 as
their threshold for statistical significance and our target sample of 4000
patients per group
o This means that a sample size of 4000 patients per group were required to
detect a significant effect >> meaning that the results of the study did not
occur by chance
o These assist with supporting the researchers hypothesis and to decreased a type
II error (accepting null when it should be rejected)
o Assures that the researcher will decide correctly to reject the null hypothesis
when it is actually false
Indicate sample mortality
 There is no sample mortality indication as the sample size is the same at the end of the
study as it was at the start
20. How did the researcher protect human rights?
 The trial protocol was approved by the University of Pennsylvania Institutional
Review Board >> which granted a waiver of informed consent because this



study was an evaluation of a health system intervention that posed minimal
risk to clinicians and patients.
Neither clinicians nor patients were compensated for participation.
The control group was given….
Consent* wasnt obtained**
Who sponsored/funded this study? And does this concern you?
 This study was supported in part by the National Institutes of Health under Award
P30AG034532, the Bill and Melinda Gates Foundation, the Flu Lab, the Penn Center for
Precision Medicine Accelerator Fund, the Robert Wood Johnson Foundation, and the
University of Pennsylvania Health System through the Penn Medicine Nudge Unit.
Support for this research was also provided in part by the AKO Foundation, John Alexander, Mark J. Leder, and Warren G. Lichtenstein
 I am somewhat concerned as the study revealed multiple conflicts of interest in which
three physicians have either been funded or are a part of boards that may affect the
outcome of the study.
21. What type of data were collected?
 Just circle this
 Outcomes of vaccination rates
 Digital health records of participants
 Sample characteristics of the patient
22. What was the data collection method/tool used?
 Procedure/tx/intervention
o Primary outcome measure: influenza vaccination on the day of the
primary care visit or in the three days leading up to the visit.
 Existing Data (pt charts and records)
 Questionnaire >> page 3

Outline the procedures used for data collection from start to completion >> point form
Under interventions
23. Were methods used to ensure the reliability of the data collected? >> differences
between raters, differences between measurement times and so on
Yes/No/Not discussed?
 Mega study >> a field experiment in which many interventions developed by different
teams of scientists were tested in the same population and on the same outcome

What level of reliability is reported?
The level of reliability was not discussed
Is it acceptable?
Yes/no?
Elaborate:
 Cannot elaborate when reliability of the study is not discussed at all. There is no
indication of any reliability tests within the article
Were methods used to ensure the validity of the data collected? (expert review?)
Yes/No/Not Discussed
If yes >> what did the researcher use?
 Not Discussed: since reliability was not discussed >> without reliability the instrument
cannot have validity
 The trial used the CONOSORT >> seen in supplement 1
o This trial followed the Consolidated Standards of Reporting Trials (CONSORT)
reporting guidelines (supplement 1)
24. What was the response rate? Only if applicable
 NOT APPLICABLE >> qualitative
 BUT IF I WAS TO GIVE AN EXMAPLE: Response rate: 74811/ 92277 >> 81.07%