SB 822: Quantitative Methods for Program Evaluation Research
Massachusetts College of Art and Design
Let’s Talk About Sex, Baby Evaluation Proposal
Boston University School of Public Health
Ariela Braverman Bronstein
Halla Daoui
Chana Leah Rosenbaum
Charlotte Vieira
Section 1: Intervention Overview
What is the public health problem?
The public health problem the intervention addresses is unsafe sex behaviors and nonconsensual
sex on college campuses. The following is national data:

1 in 5 women is sexually assaulted while in college1

75-80% of the time, the victim knows their attacker1

40% of college sexual assault survivors fear their attacker will retaliate1

52.6% of college students report using a condom or other protective barrier for vaginal
intercourse and less used a condom for oral and anal sex2

Individuals at ages 15-24 are at a higher risk of acquiring STDs2
Massachusetts College of Art and Design (MassArt) is an independent public college located in
Boston, Massachusetts. As a school receiving federal funding, it must comply with the Campus
Sexual Violence Elimination (SaVE) Act and Title IX3, which protects people from
discrimination based on sex in education programs that receive federal funding. The Campus
SaVE Act seeks to address violence against women on college campuses. It does this by
requiring the following steps be taken by colleges:4

Incidents of domestic violence, dating violence, sexual assault, and stalking must be
included in annual crime statistic reports.

All victims must be provided with written rights regarding how to change their living
arrangements, transportation, or working situations to avoid an unsafe environment, and
receive information regarding mental health or legal counseling, and other support
services.

Minimum standards for dealing with disciplinary procedures with regards to domestic
violence, dating violence, sexual assault, and stalking must be in place.

Programming and education for students and employees about domestic violence, dating
violence, sexual assault, and stalking must be provided.
Many college students are uneducated about safer sex and/or choose not to engage in safer sex
for a variety of reasons. This is an important concern, and needs to be addressed. While MassArt
does not report an above-average amount of sexual assaults, domestic violence, dating violence,
or stalking, this topic needs to be addressed to prevent it from becoming a bigger problem. By
creating this program, MassArt is also complying with the Campus SaVE Act.
What is the intervention? What stage is the intervention (planned or already implemented)?
The intervention is a combination of an educational workshop taught to MassArt undergraduate
students and a mass media campaign. The media campaign includes a common hashtag
(#massart4consent), social media presence (instagram and twitter), flyers, and magnets and
luggage tags (with the hashtag). The intervention is currently in its pilot year. After the pilot
year, the workshop will be evaluated and mandated for all freshmen in addition to the Campus
Clarity online program.
What are the intervention’s outcome objectives?
Students will lead healthier sex lives and engage in healthy relationships by trying to prevent
STIs and unwanted pregnancies, decrease sexual assaults, and gain consent before engaging in
any sexual act.

Discussing STI testing and results with partner(s)

Gaining and giving consent

Choosing whether or not to use contraceptives together (individual and partner(s))

Regular STI testing

Practicing healthy communication with partner(s)
What are the program’s activities?

Social media promotion
o
Weekly tweets with the hashtag and links to popular websites promoting consent
and safer sex

Moss graffiti

Photo campaign

Flyers posted on campus

Distribute magnets and luggage tags with the hashtag

Deliver a one-hour workshop using trust building activities and information about safer
sex and consent (only freshman) (See Appendix 1).
o
Myths vs. facts exercise to discuss common misperceptions around sex, consent,
and other related topics.
o
Fishbowl activity to demonstrate nonverbal communication and the importance of
actively listening to and appreciating your partner.
How will activities be implemented?

Graduate student intern working with the Counseling and Wellness Center will teach the
workshop to undergrads with or without co-facilitation of student leaders. The workshop
is taught in residence halls and to student groups.

The weekly tweets are released through the Student Development twitter account.

The flyers and photos are posted throughout the campus by the graduate student intern
and work study students within the Counseling and Wellness Center.

Magnets and luggage tags are located in the Counseling and Wellness Center at all times
for anyone to take, and given out at Counseling and Wellness events and during this
program.
What agency is delivering the intervention? Where will the intervention be implemented?
The Counseling and Wellness Center, within MassArt, is delivering this intervention.

The moss graffiti hashtag is placed in two main buildings on the MassArt campus (Tower
and Kennedy lobbies).

The tweets are released through the Student Development Twitter account, and students
are encouraged to tweet with the hashtag.

Magnets and luggage tags are given out during the program and at other health and
wellness events (e.g., The Colleges of the Fenway Health and Wellness Fair).
Who is the primary target population? Who is the secondary target population(s)?
MassArt undergraduate students are the primary target population. The secondary target
populations are other members of the Colleges of the Fenway Consortium.
Input
Output
Resources
Activities
Learning Obj.
Performance Obj.
Promote program through
hashtag
o Photo campaign
o Flyers posted on
campus
o Moss graffiti
o Distribute magnets
and luggage tags
o Weekly tweets using
the hashtag
Explain importance of consent
and provide examples of how to
gain and give consent
Reach out to supports when in
need or confused about
situations regarding consent and
sexual health
Marketing resources
o Twitter
o Magnets
o Luggage tags
o Moss
o Flyers
Staff/student time
o Twitter feed
o Maintain moss
Formative research with
students
o Incentives
o Marketing materials
o Assistance with promoting the
campaign
MassArt Departments
o Counseling and
Wellness Center
Interactive, educational
program to inform
students about topics
such as
o Consent
o Healthy communication in
relationships
o Getting tested
o General safer sex
o
o
Housing and
Residence Life
o Provide location and student
audience for consent and
safer sex programming
o
Student Development
Tweets are
released from this twitter
account
o
All flyers are
approved by this department
o
o
Facilities
o Provide plywood and
placement for moss graffiti
o Approve locations for flyers
and magnets on campus and
moss
Fishbowl exercise
o Two concentric circles of
participants facing each
other
o Demonstrate nonverbal
communication and the
importance of actively
listening to your partner(s)
and appreciating them
o All activities and marketing
tactics for this program are
sent from this department
o
Myths vs. facts
exercise
o Discuss common
misperceptions around sex,
consent, and other related
topics
Outcomes/Objectives
Describe healthy communication
skills with partner(s)
o Discuss STI testing and results
o Gaining and giving consent
o Choosing whether or not to use
contraceptives together
Explain what being sexually
healthy entails
o Regular STI testing
o Practicing healthy communication with
partner(s)
Increase self-efficacy skills
regarding
Gaining consent before engaging
in sexual acts
o Engaging in healthy
communication
o Reaching out for help when
needed
o Identifying and respecting when
their partner(s) is consenting to
any sexual or intimate acts
o
Describe how not gaining
consent can have legal
consequences for perpetrators
and emotional consequences for
victims
Describe what “safer sex” entails,
including the following:
o
o
o
Consistent use of contraception
Regular STI testing
Healthy communication with
partner
Maintain healthy communication
with sexual/intimate partner(s)
o Discuss enjoyable acts with
partner(s)
o Relay STI testing results with
partner to ensure healthy
statuses
o Gain consent with partner(s)
before engaging in sexual
acts
Consistently use one or more
forms of contraception when
engaging in sexual acts
Get tested regularly for STIs
Health & Quality
of Life
Ultimate Obj.
Students will lead healthier
sex lives and engage in
healthy relationships by trying
to prevent STIs and unwanted
pregnancies, decrease sexual
assaults, more gain consent
before engaging in any sexual
act.
Section 2: Process Evaluation
Evaluation Questions
Were program staff trained to
implement the program as intended?
What percentage of students surveyed
completed questionnaires?
Indicators






What is the demographic makeup of
the student body at each participating
school?

Whom are the students who completed
questionnaires?

How many MassArt students attended
LTASB workshops?


What was the reach of the LTASB
marketing campaign?



# of hours of training
# of training sessions
Staff satisfaction and preparedness
post training
Data Sources


Performance Measures
Analyze program records
Survey of staff
- How prepared they felt to
perform jobs
- How satisfied they were with
trainings



Analyze survey response records
within each student group



# of hours of training
# of training sessions
% of staff who received training in
accordance with protocols
Average levels of staff satisfaction
Average levels of staff preparedness
% of MassArt freshman who
completed pre- and post-test surveys
% of MassArt upperclassman who
completed post-test surveys
% of BC students completed pre- and
post-test surveys
Demographic characteristics of all
student groups (race, income, age,
sex, city/country of origin)


Analyze anonymous student body
demographic data available through
the admissions office of participating
schools

Demographic characteristics of
students who completed
questionnaires (race, income, age,
sex, city/country of origin)
# of freshmen who attended LTASB
workshops
# of upperclassmen who attended
LTASB workshops
# of tweets posted per week
# of views for each tweet posted
# of tweets that are re-tweeted

Analyze demographic questions
within questionnaire

None (descriptive)

Analyze program records and
workshop sign in records

# of freshmen who attended LTASB
workshops
# of upperclassmen who attended
LTASB workshops
# of tweets posted per week
# of views for each tweet posted
# of tweets that are re-tweeted




Analyze program records, flyer
distribution plan, and twitter analytics
(analytics.twitter.com)



% of MassArt freshman who completed
pre- and post-test surveys
% of MassArt upperclassman who
completed post-test surveys
% of BC students completed pre- and
post-test surveys
None (descriptive)



What are the levels of student
awareness and self-reported exposure
to LTASB program components?







# of twitter followers
# of moss sites around campus
# of months during which moss is
maintained
# of types of flyers posted
# of total flyers posted
# of students who recognize hashtag
# of students that have used the
hashtag
# of students that have completed
workshop
# of students that have seen flyers
#of students that have seen moss




Analyze awareness and self-reported
exposure questions within
questionnaire







# of twitter followers
# of moss sites around campus
# of months during which moss is
maintained
# of types of flyers posted
# of total flyers posted
# of students who recognize hashtag
# of students that have used the
hashtag
# of students that have completed
workshop
# of students that have seen flyers
#of students that have seen moss
Section 3: Outcome Evaluation -- Evaluation Design
What are the outcome evaluation questions?
Three Behavioral Change-based Outcome Evaluation Questions:

Will the intervention lead to an increase in testing for transmitted STDs and STIs among
students?

Will the intervention lead to an increase in gaining and giving consent among students?

Will the intervention lead to an increase in contraceptive use among students?
Four Intermediate Objective Questions:

Can students describe sexual consent?

Can students distinguish between the multiple forms of contraception?

Can students discuss sexual preference and concerns with partner?

Can students list available resources for support/help on campus?
The one-hour education workshop is one of the primary components of the program
where certain skills will be taught to the students. The material delivered during the workshop is
intended to impact positive knowledge and skills regarding gaining and giving sexual consent,
increased contraceptive use, and perceived behavioral control regarding sexual consent and
ability to test for STDs. Many of the social media components are directed to serve as cues to
action.
What study design will be used?
It is a Pre-Post Control Group Design (Quasi Experimental). The following is a diagram the
design:
Fall semester:
X1: LTASB media - all year
X2: LTASB workshop - offered 3 times in fall semester
Sent in July, due in August:
O1: LTASB pretest
O2: Campus Clarity pretest
Second semester - after winter break, before spring break (mid-February):
O3: LTASB posttest
O4: Campus Clarity post-test
Why was this design selected? Remember that the strength of the design has to be weighed
against the feasibility of executing it.
The LTASB workshop is to be offered to all freshmen and the media campaign to the
entire student body. Randomization is impractical and therefore a quasi-experimental pre-posttest design is the best alternative. This design will allow evaluators to measure the potential
effects of the intervention by examining the difference in the pre-test and post-test results. By
having both groups (intervention and control), evaluators can control for the possibility that other
factors not related to the intervention are causing the observed differences.
What is the unit of allocation (individual, group, classroom, community, etc.)?
Individual level.
Who will serve as the control/comparison group? Why was this group selected? How will
comparability be assessed? What will be the follow-up period and why?
Intervention #1: MassArt freshmen exposed to LTASB and Campus Clarity.
Control #1: Freshmen from Boston College (BC) as they have no sexual health programs in
place.
Intervention #2: MassArt upperclassmen exposed to social media campaign (and possibly
previous LTASB pilot).
Control #2: Upperclassmen from BC as they have no sexual health programs in place.
Comparability:
Baseline demographics for each school’s student body will be collected via demographic
data from each admissions office. Demographic data specific to those participating in the
evaluation will be collected via the surveys. The choice to use BC for the control groups was
driven by the fact that BC has no sexual health programs, and the design required selection of a
school from the Boston area. There are obvious differences between the student bodies at each
school. Using the collected demographic data we can examine comparability between the groups
and statistically control for differences between the two schools.
MassArt freshmen will be surveyed twice prior to the workshop, once through the
Campus Clarity general risky behaviors pre-test and once through the LTASB pre-test. This will
be during the summer prior to the fall semester beginning. There will be a Campus Clarity posttest and LTASB post-test, as well. The BC freshmen and upperclassmen, as well as MassArt
upperclassman, will be provided with LTASB pre-tests during prior to the fall semester and
LTASB post-tests during spring semester of the same academic year.
The follow up period:
Post-tests for MassArt freshmen will be given the following semester (before spring
break) to assess long-term behavioral change and knowledge gain. Post-tests for MassArt
upperclassmen, BC upperclassmen, and BC freshmen will be given at the same time as MassArt
freshmen during spring semester. MassArt upperclassmen will be asked if they participated in
the LTASB workshop (to assess prior knowledge, comprehension of material, and long term
behavior change). Another assessment could take place during the fall semester of the following
year to see if those workshop participants could be followed-up with in order to get a sense of a
sustained long-term change.
To what extent might potential threats to internal validity operate within this design?
Which are more likely and why? Which are less likely and why?
How will you guard
against these threats?
Due to the selected study design’s lack of randomization, selection is the main threat to
internal validity. This threat could be reduced, yet not eliminated, by choosing a comparison
group similar to the intervention group. History, maturation, instrumentation, and testing are
controlled for given that each intervention group has a control group. In terms of internal
validity, interaction of selection-history, selection-maturation, and selection-testing are possible
threats. Compensatory rivalry and contamination are also possible threats. Refer to table 2 for
more details regarding this.
Previous encounters with various sexual partners and experiences could differ between
groups. Moreover, groups are maturing at different rates with respect to the measured outcome
(sexual behaviors and health practices). For instance, upperclassmen in both schools are expected
to have lower tendencies to experiment with multiple partners and engage in risky behaviors due
their advanced maturity over freshmen. Selection - testing and selection-instrumentation are also
possible threats to validity. The LTASB pre-test given to everyone, especially MassArt freshmen
that are also given Campus Clarity pre-test, could possibly "prime" the students differently. The
MassArt freshman could have learned differentially from the pretest (both LTASB and Clarity).
Threat
Threat to Validity
Explanation
History
Freshmen: No
It is controlled for because we have two-control group.
One for Freshmen and one upper classmen.
Notes:
Upper classmen: No
Instrumentation
Freshmen: No
It is controlled for because we have two-control group.
One for Freshmen and one upper classmen.
Upperclassmen: No
Maturation
Freshmen: No
Freshmen are given two different pre-tests
Clarity and LTASB while BC surveyed student
body is exposed to different structured tests.
It is controlled for because we have two-control group.
One for Freshmen and one upper classmen.
Upperclassmen: No
Regression
Freshmen: Maybe
Upper classmen: Maybe
Testing:
Freshmen: Maybe
It is a possibility to have a regression effect Not
because of the students selected but because their
scores might be extreme
LTASB and Clarity Pre-tests might “prime” freshmen
students.
The MassArt students could have learned
differentially from the pre-tests.
Upper classmen: No
Selection
Freshmen: Yes
Non-randomized selection posing greatest threat to
internal validity.
Upper classmen: Yes
Selection-History
Freshmen: Maybe
Upperclassmen: Maybe
Selection-
Freshmen: Maybe
Maturation
Upper classmen: Maybe
It is possible that there are different events happening at Students are impacted by certain event that
one school and not another. For instance, BC religious
might happen or have happened at their
counsels decide to ban all sexual health services at the
schools.
student clinic.
Could be an issue if age distributions in both samples in
the two schools are different. For instance, freshmen at
MassArt may be mainly 18 while freshmen majority at
BC are 20.
Therefore we would expect older students could have
Age difference here between students could
count for higher or lower tendency to engage in
risky behaviors.
tendency to be more careful about choosing sexual
partners and more conscious regarding healthy practice
of being tested.
Selection-Testing
Freshmen: No
Not a concern.
This would be an issue only if students were
not all on the same baseline
Upper classmen: No
Selection-
Freshmen: No
Instrumentation
Upper classmen: No
Attrition
Freshmen: Maybe
Even though Freshmen are giving Clarity test in
addition to the other survey that is given to all other
groups, this should not be an issue
Students in all groups taking the pre-test could drop out If any, the effect of this threat would be very
of school.
small.
Upper classmen: Maybe
Contamination
Freshmen: Yes
Upper classmen: Yes
Hawthorne Effect
Freshmen: Maybe
Upper classmen: Maybe
Resentful
Demoralization
Compensatory
Rivalry
Maybe: on behalf of BC
or students, especially
freshmen.
Given that a pilot program was administrated on a
previous group (now upperclassmen), interaction
between this group and freshmen student body could
happen, thus lead to contamination.
The previous pilot may have influenced
upperclassmen’s behaviors.
Students who take the survey and also given the
curriculum program may realize they are going to be
observed and tested again and as a result may report
change of behaviors due to that.
BC students might be motivated as the “underdogs” to
outperform MassArt students who are known to be
receiving an intervention
BC freshmen students are not getting any
treatment, which may be a motive.
Section 4: Outcome Evaluation -- Sampling
How will the sample be selected and recruited? (Describe selection procedures for
intervention and control/comparison groups.)
Freshmen:

The intervention group will be all freshmen at MassArt. LTASB workshop attendance
will be mandatory so participants will be exposed to both interventions: the mass media
campaign and the workshop, in addition to the Campus Clarity program. We will conduct
a baseline questionnaire during the summer before the workshop, so before classes start
students will have had to complete both the LTASB questionnaire and the Campus
Clarity questionnaire. After the workshop, freshmen will be asked to complete a second
questionnaire during the spring semester (February), and they will have to complete
another questionnaire for Campus Clarity, as well. All of the questionnaires will be
mandatory.

The control group will be a random sample of freshmen at Boston College. They will
receive the same web-based LTASB questionnaire as the intervention group during the
summer before classes start and then those who answer the questionnaire will be asked to
complete the second LTASB questionnaire in February. We will offer incentives for
students who complete both the pre-test and the post-test questionnaires.
Upperclassmen:

The intervention group will be all upperclassmen at MassArt. This group will have been
exposed to the mass media campaign, and the ones who participated in the pilot group of
the workshop would have been exposed to that. In addition, the workshop will be offered
to upperclassmen that wish to take it, attendance will not be mandatory so not all of the
upperclassmen will be exposed to the workshop. These students will complete the same
LTASB questionnaires as the freshmen for our intervention, at the same calendar time as
the freshmen. However, this will not be mandatory for them, so we will offer incentives
for those who complete both questionnaires.

The control group will be a random sample of upperclassmen from Boston College.
These students will be reached via email and asked to complete the LTASB questionnaire
during the summer before classes start as a pre-test and then those who answer the first
one will be asked to complete another LTASB questionnaire during the spring semester
as a post-test. Students will also be incentivized to answer both questionnaires.
What eligibility criteria will be used?
Students ages 18-23 at MassArt and students within the same age group at Boston
College are eligible for our study. Any student is eligible regardless of gender, sexual
orientation, or sexual activity.
What are the strengths and weaknesses of these sampling procedures in terms of
generalizability?
One of the strengths of our study is that, to our knowledge, there is no “safe sex” program
at Boston College. This situation will allow us to compare the effectiveness of both interventions
(the mass media campaign and the workshop) together by comparing the freshmen groups, and
the mass media campaign alone by comparing the upperclassmen groups.
One of the weaknesses is that not all the groups have the same selection process. For the
intervention group, it will be mandatory so we expect less attrition from that group than the rest.
Since participation will not be mandatory for the other 3 groups, we plan to offer incentives to
people who complete both questionnaires to increase the level of participation.
We could also expect some resistance from our intervention group since they will be asked to
complete 4 questionnaires in total, 2 for LTASB and 2 for Campus Clarity, which might seem to
them as unnecessary work.
We know that the characteristics of the students enrolled in MassArt might differ from
other college students, which might pose an issue for the generalizability of our results.
Section 5: Outcome Evaluation – Measurement
Outcome Evaluation Question
Dependent Variable
Outcome Objectives
Will the intervention lead to an increase in testing
for transmitted STDs and STIs among students?
STI testing
Will the intervention lead to an increase in gaining
and giving consent among students?
Consent practices
Will the intervention lead to an increase in
contraceptive use among students?
Contraceptive use
Intermediate Objectives
Can students describe sexual consent?
Attitudes and beliefs about consent
Can students distinguish between the multiple
forms of contraception?
Attitudes and beliefs about safer sex and
contraception
Can students discuss sexual preference and
concerns with partner?
Healthy sexual communication skills
Self-efficacy
Can students list available resources for
support/help on campus?
Knowledge of support services at MassArt
Selection of Measures
The Sexual Health Indicators survey, created to measure indicators of sexual health
among 16-24 year olds, is self-administered and contains 75 questions. Included are the
following five dimensions of sexual health: 1) physical, mental, and social well-being in relation
to sexuality; 2) approach to sexuality; 3) sexual relationships; 4) sexual experiences; and 5)
discrimination, coercion, and violence. Items are measured using a Likert scale ranging from 1
(strongly disagree) to 5 (strongly agree). The Sexual Health Indicators survey was developed for
use and tested among college-aged respondents, and has demonstrated high internal reliability
(Cronbach’s alpha >0.7) when used with this age group.
The survey has also been tested construct validity in the following areas: condom use at
last vaginal sex, protection self-efficacy, STI/HIV testing self-efficacy, and sexual orientation.
The relationships between opposing participant scores in each of these areas were found to be
stable across gender, age, size of area of residence, and survey language.5
For the purposes of the LTASB survey, a subset of 20 questions will be chosen based on
their relevance to the overall outcome evaluation questions. It will also be important to choose
questions that are conservative and deemed appropriate by both MassArt and Boston College.
We acknowledge that this may limit the reliability and validity of this measure. In order to
address this, we will build reliability and validity testing measures into our evaluation plan.
The Sexual Consent Scale-Revised (SCS-R) is a 39 item, self-administered questionnaire
developed to assess attitudes, beliefs, and behaviors in relation to sexual consent. All items are
measured using a Likert scale ranging from 1 (strongly disagree) to 7 (strongly agree). The
overall magnitude of correlation for the SCS-R was .87 which indicates a very high level of
reliability.
It has been administered in other settings to undergraduate students and has shown
positive correlation with awareness of consent and positive attitude toward establishing consent
and a negative correlation with lack of perceived behavioral control.6
Similar to our plan for the Sexual Health Indicators survey, we will be taking a subset of
10 questions based on overall relevance and appropriateness. As referenced previously, this may
limit the reliability and validity of the measure, an issue that will be addressed by incorporating
reliability and validity testing into our evaluation plan.
In addition, the surveys will include some general demographic questions, including age,
race/ethnicity, gender, sexual orientation, grade, etc.
Dependent Variable
Measure
Potential Measurement Error
STI testing
Self-report, in pre- and post-test survey (MassArt and
BC):
Several questions in the Sexual Health Indicators survey
asses STI testing self-efficacy.5
See “Selection of Measures” above.
Consent practices
Contraceptive use
Attitudes and beliefs about
consent
Sample questions include:
 “I feel confident I could ask my doctor or
health-care provider for STI testing”
 “I feel confident I could ask my partner to get
tested for STIs”
Self-report, in pre- and post-test survey (MassArt and
BC):
The Sexual Consent Scale-Revised (SCS-R) contains
questions regarding consent practices.6
Sample questions include:
 “I have not asked for sexual consent (or given
my consent) at time because I felt that it might
backfire and I wouldn’t end up having sex”
 “Typically I communicate sexual consent to my
partner using nonverbal signals and body
language”
 “I always verbally ask for consent before I
initiate a sexual encounter”
Self-report, in pre- and post-test survey (MassArt and
BC):
The Sexual Health Indicators survey contains questions
regarding contraceptive use.5
Sample questions include:
 “I feel confident in my ability to use protection
on myself and/or my partner”
 “If I wanted to practice “safer-sex” with
someone, I would insist on doing so”
 “I feel confident in my ability to suggest using
protection with a new partner”
Self-report, in pre- and post-test survey (MassArt and
BC):
The Sexual Consent Scale-Revised (SCS-R) contains
questions regarding attitudes and beliefs about consent.6
Sample questions include:
 “Not asking for sexual consent from time-totime is okay”
 “I would have difficulty asking for consent
because it would spoil the mood”
 “I feel that verbally asking for sexual consent
should occur before proceeding with any sexual
activity”
See “Selection of Measures” above.
See “Selection of Measures” above.
See “Selection of Measures” above.
Attitudes and beliefs about
safer sex and contraception
Healthy sexual
communication skills
Self-efficacy
Knowledge of support
services at MassArt
Self-report, in pre- and post-test survey (MassArt and
BC):
The Sexual Health Indicators survey contains questions
regarding safe sex and contraceptive use attitudes and
behaviors.5
Sample questions include:
 “I feel confident I could bring up the topic of
protection with my health-care provider”
 “I feel confident I could purchase protection
without feeling embarrassed”
 “I feel confident I could easily ask my partner if
he/she had protection (or tell them that I didn’t)”
Self-report, in pre- and post-test survey (MassArt and
BC):
The Sexual Health Indicators survey contains questions
regarding healthy sexual communication skills.5
Sample questions include:
 “I am direct about voicing my sexual needs and
preferences”
 “If I were to have sex with someone, I’d tell my
partner what I like”
 “I feel confident in my ability to suggest using
protection with my new partner”
Self-report, in pre- and post-test survey (MassArt and
BC):
The Sexual Health Indicators survey contains questions
regarding overall sexual self-efficacy.5
Sample questions include:
 “I feel confident I would be able to choose when
and where to engage in sexual activity”
 “I feel confident I would be able to refuse sexual
activity with someone I’m not comfortable with”
 “I feel confident I would be able to go out with
someone without feeling obligated to engage in
sexual activity”
Self-report, in pre- and post-test survey (MassArt only):
Since this measure is specific to resources at MassArt,
both pre- and post- test surveys for MassArt students will
include 10 questions to assess overall knowledge of
support services available to students on campus. Most
items will be measured using a seven-point Likert scale
ranging from 1 (strongly disagree) to 7 (strongly agree),
with the exception of some dichotomous (yes/no) answer
options.
Respondents will be asked the following questions:
 “If I needed advice about my sexual health, I
would know where to go on campus for support”
 “If I felt that I was in a difficult position, I
would know whom to speak to at MassArt”
 “I know where to get condoms on campus”
 “I have attended a LTASB workshop”
See “Selection of Measures” above.
See “Selection of Measures” above.
See “Selection of Measures” above.
Since we are not relying on a validated
measure for questions related to
knowledge of support services at
MassArt, we must acknowledge the
possibility for some level of random or
systematic error.
In order to minimize potential
measurement error, we plan to pilot
test this question set with a small
group of students. We will also
conduct a focus group of students in
order to assess reactions to and
comprehension of questions.
Section 6: Outcome Evaluation – Power Analysis
Why conduct a power analysis?
In order to decrease the probability of type II error, it is important to determine the
sample size of students needed from each school (MassArt and BC), this will also help us to
increase the precision of our study.
Select one dependent variable for the power analysis. Why was it selected?
The dependent variable we selected is ‘attitudes and beliefs of consent and legal and
emotional consequences’. For the purpose of sample size calculation this will be classified as a
dichotomous variable (yes/no), and it will be measured through the surveys based on
standardized questionnaires. Of the 5 questions asked on sexual consent attitudes and beliefs
those who agree with the right answer on at least 3 of them will be considered yes and those who
agree on less than 3 will be considered no.
We chose this variable because we are proposing a 1 year evaluation and a behavioral
change would usually take longer to be detectable than knowledge. In addition, at the time we
are implementing the intervention and both tests we do not know if the all the participants are
sexually active and it would make it harder to evaluate a behavior.
What is the intervention’s expected magnitude of change? How was this estimate derived?
What is the expected baseline prevalence of the selected variable? How was this estimate
derived?
There is not enough data to know for certain the prevalence of knowledge on sexual
consent. For this reason we decided to use the most conservative approach, and assigned a
baseline prevalence of 50%. Since we are evaluating the intervention after 1 year, we are
expecting a 10% increase in the knowledge of sexual consent.
Based on the expected baseline prevalence and expected magnitude of change, do you
expect to recruit a large, medium, or small sample? Why?
Based on the estimated sexual consent prevalence of 50% among college students, and an
expected increase of 10%, we would need a large sample size (305 students in each group) to
have an 80% power to detect a statistically significant change. The sample size previously
calculated (305 students per group) is assuming no attrition. Considering that this is a
longitudinal study, we expect to have some attrition so the sample size would have to be larger.
Section 7: Outcome Evaluation – Analysis
Why employ statistical tests?
It is crucial to employ statistical tests in order to make sense of data. Statistical tests
allow evaluators to make quantitative decisions about the selected intervention. Statistical tests
also help determine whether there is enough evidence to support or reject the null hypothesis.
How will change be assessed for both dependent variables?
Analysis of Variance, an ANOVA test will be performed to analyze the differences
between group means for the different variables tested. When using ANOVA, certain specific
tests such as Tukey would indicate which group variations were significantly different.
What variables may confound the relationship between the independent and dependent
variable?
Age, gender, alcohol, and pre-testing are all possible confounders.
Given the design and possible confounders, will bivariate or multivariate statistics be used?
Why?
Following ANOVA, multivariate analysis will be performed which will examine
relationships among multiple variables at the same time. Multivariate statistics allows for
understanding which variables are creating difference while taking into account the possible
impact of one or more confounding factors.7 Multivariate analysis will control for potential
confounders by showing the relative contribution of each independent variable (given the
presence of the other independent variables) to variation in the dependent variable.7
Section 8: Summary
Outline the strengths and weaknesses of your evaluation plan with regards to potential
threats to validity, logistics, and other considerations.
Strengths:

Because LTASB is in the pilot year, there is flexibility to alter it along with the
evaluation plan, as evaluations are best when created alongside the program.

Surveys by email are a simple, quick, and inexpensive way to collect data.

Boston College does not allow programs around sexual health, which provides a control
group with less contamination possibilities than other schools.

Two control groups at Boston College allows us to compare the freshmen and
upperclassmen at both Boston College and MassArt separately to better determine how
the intervention effects different ages.

History, instrumentation, and maturation are all controlled for because there are two
control groups (upperclassmen and freshman).
Weaknesses:

Selection is the major threat to validity, because of the lack of randomization in the
control and experimental groups. However, we are going to have an internal (MassArt
upperclassmen) and two external (Boston College freshmen and upperclassmen)
comparison groups to best compare intervention results.

Selection-history and selection-maturation may be threats to validity because different
events could be occurring at each school, thus altering one student body’s actions, and the
sample of students at both schools may be innately different.

Contamination is also a threat because there has been a pilot year of the LTASB program
at MassArt, which could lead to interactions between students who attended this program
in its first year and those who are starting it afterwards.

Testing may pose a threat to validity with the freshman at MassArt, because they receive
both the LTASB and Clarity pre-tests, which could “prime” them to answer differently
than they would have without the pre-tests.

Compensatory rivalry is also a potential threat, because BC students may feel the need to
outperform MassArt students to show they don’t need the intervention to accomplish the
same goals.

The characteristics of the student body at MassArt poses an issue for generalizability of
our results.

Lastly, Hawthorne effect may be a threat to validity because students, who have taken the
survey and attended a LTASB program, may realize they are being observed and then
tested again, which could lead to them reporting changes in behavior due to that.
References
1. Centers for Disease Control and Prevention. (2013, August 20). Sexually Transmitted
Diseases (STDs): Adolescents and Young Adults. Retrieved February 23, 2014, from
Center for Disease Control and Prevention: http://www.cdc.gov/std/life-stagespopulations/adolescents-YoungAdults.htm.
2. STDs in Adolescents and Young Adults. (2014, January 7). Centers for Disease Control
and Prevention. Retrieved February 19, 2014, from
http://www.cdc.gov/std/stats12/adol.htm.
3. White House Task Force to Protect Students From Sexual Assault. (2014). Not Alone.
Washington, D.C. : U.S.
4. Clery Center. (2012). The Campus Sexual Violence Elimination (SaVE) Act. Retrieved
October 27, 2014, from Clery Center for Securtiy On Campus:
http://clerycenter.org/campus-sexual-violence-elimination-save-act
5. Smylie, L., Clarke, B., Doherty, M., Gahagan, J., Numer, M., Otis, J., … Soon, C. (2013).
The development and validation of sexual health indicators of Canadians aged 16-24
years. Public Health Reports (Washington, D.C.: 1974), 128 Suppl 1, 53–61.
6. Humphreys, T. P., & Brousseau, M. M. (2010). The sexual consent scale-revised:
development, reliability, and preliminary validity. Journal of Sex Research, 47(5), 420–
428. doi:10.1080/00224490903151358
7. Boston University Modules. 2014. Multivariable Methods Modules. Retrieved from:
http://sphweb.bumc.bu.edu/otlt/MPHModules/BS/BS704_Multivariable/BS704_Multivariable_print.html
Appendix 1: Program Outline
Let’s Talk About Sex, Baby
Materials Needed
o Poster board, sticky wall pad, or board to write on
o Markers
o Contraception
o Paper for participants emails
Time: about 1 hour
Program Timeline:
1. Program facilitator(s) introduction
2. Explain that this program comes out of MassArt’s Counseling and Wellness Center, and
then provide information about MassArt Student Health Services and the Counseling and
Wellness Center.
 Student Health Services
o Harvard Vanguard Medical Associates, located on the 2nd floor of the
Treehouse Residence Hall
 Sick visits and physicals, check for and treat STIs, and
gynecological visits
 Bring student ID and insurance card
o Harvard Vanguard’s Kenmore location
 Counseling and Wellness Center
o Programming throughout the year, such as Sex in the Dark and stress
reduction groups
o Free short-term therapy, evaluations, referrals, and other resources
o Can drop in or schedule an appointment over the phone
3. Ground rules: Have participants offer suggestions, and provide a few examples they do
not mention or if they are unwilling to speak up. Examples include:
 Confidentiality
 Use “I” statements
 No assumptions
 Respect
 Share at your own level
 E.L.M.O. (Enough Let’s Move On)
4. Provide a brief introduction about the program
 Teach about safer sex practices and healthy relationships
5. Icebreaker: Write “_ _ _ k” in front of the room and ask participants what 4-letter words
end in –k and are related to sex. They can volunteer answers, but if no one offers
suggestions, wait a bit and then give them the answer: “T-A-L-K.” The purpose of the
icebreaker would be used to establish trust, develop a high level of comfort, and break
down barriers to communication.
 Why did no one suggest this word?


Did anyone think of it?
Do people think about talking about sex before doing it, either right before or way
before?
 Explain why communication is key before engaging in any sexual act
 If you don’t talk, how will the other person know what you do or don’t like, and if
the act is consensual?
6. Main program activity #1: “fishbowl exercise”
 There will be two circles of participants, one inside the other. The inner circle will
talk to one person in the outer circle about a question the facilitator will ask about,
or leading to, sex and/or relationships, while the outer circle cannot speak. The
roles will then be reversed and the outer circle will speak and respond to the inner
circle, but the inner circle cannot speak. The idea is to understand how
communication works, with both nonverbal and verbal cues, and how much we
listen to each other.
 Choose at least 2 questions from each section:
o Where are you from, and have you been enjoying Boston while at
MassArt?
o What are you currently studying, and why did you choose that?
o Tell your partner about your family.
---------------o Tell your partner when you first learned about anything regarding
relationships. (how old were you, did you observe relationships or told by
someone else how they should be, such as gender)
o Describe your idea of the perfect relationship.
o What do you think pressuring someone to do something sounds or looks
like?
 Discussion afterwards
o How did it feel to talk to someone who wasn’t answering? Was it
uncomfortable, awkward, etc.?
o How was it to not be able to answer your partner?
o Did you notice nonverbal cues (such as nodding or smiling)?
o Do you see any connection between this exercise and intimate
relationships?
 What if you were only relying on nonverbal cues for consent, but
your partner was intoxicated?
 Would you enjoy always having a one sided relationship?
 You need to communicate to figure out what each other likes and
is comfortable with, if you are both “in the mood,” or similar
questions.
7. Main program activity #2: True or False
 Read aloud statements regarding sex, relationships, and consent, and have
participants decide whether or not they believe it to be true or false by standing on
one side of the room. (e.g. if you believe it is true, stand on the left side of the
room)
 Discuss why both sides believed a statement to be true or false by asking where
they had learned the information.

Statements: Only do as many as time permits to stay at about 1 hour for the whole
program.
o All STIs are preventable. --- True, if the proper precautions are taken. If you
use protection, get tested frequently (and know whether or not your partner
is clean), or abstain from sexual acts, the chances of contracting an STI are
extremely low. However, remember that certain STIs (crabs, scabies, and
herpes) only need skin-to-skin contact for contraction.
o You can contract an STI from the toilet. ---- False. STIs cannot live outside
or away from the body for a long period of time.
o If you are intoxicated, you can still proclaim you did not give consent. True
o You can contract AIDS from unprotected sex. --- False. If your partner has
HIV or AIDS, you can contract the infection if engaging in unprotected sex,
but you would get HIV first. HIV can turn into AIDS if your T-cell count (for
immune system) drops below a certain level or if you contract 2
opportunistic infections (e.g. pneumonia). However, you can now live for
decades without ever contracting AIDS and lead a normal life. Also, you
cannot get HIV/AIDS from hugging someone, if they cough on you, etc.
o MassArt can tell your parents about your physical and/or mental health if
you are on their insurance. --- False. Your information is completely
confidential and cannot be discussed with your parents unless you give
consent.
o If you’ve already engaged in any sexual act with someone, you are allowed
to say no in the future. ---- True. It doesn’t matter how many times you have
engaged in sexual acts with someone, because it is your choice each time.
Even married couples are legally allowed to say no to their spouse.
8. Debrief:
 Option 1: Go around the room and ask participants one thing they liked about the
program, and one thing they would change. Allow the option to pass if someone
does not feel comfortable speaking up. Distribute the evaluation afterwards.
 Option 2: Pass out post-its and have each participant write one comment about the
program (good, bad, or something they would like to change) and have them
place them in front of the room on the wall/board. Read them aloud and see if it
brings up any more discussion among participants. If there is not enough time,
then skip this debrief activity and give the evaluation.
9. End the program by telling participants that you will stay back for a few minutes if
anyone has any further questions or comments.
 Have a sheet of paper for participants to write their email if they would like to be
told about MassArt’s Counseling and Wellness Center future events.
 If someone asks a question about the topic or MassArt’s Health Services and you
do not know the answer, take down their email and tell them you will find out the
answer and email them as soon as you know.