ClickerPlus
Pilot Study
“Our challenge is to make sure every child in America is learning from an
effective teacher—no matter what it takes. So today, I ask you to join
President Obama and me in a new commitment to results that recognizes and
rewards success in the classroom ... “
Secretary of Education, Arne Duncan, July 2, 2009
ClickerPlus Pilot Study:
Applying Advertising Research Methods to the
Measurement of Student Learning Goals
Lilly Conference on College and University Teaching
Washington D.C.
3 June 2010
Prepared by: Brian Moore
Agenda
•
•
•
•
•
•
ClickerPlus?
Hypothesis: ClickerPlus made me a better teacher
Advertising research methods
Method
“Survey”
Results
– Pilot study
– Audience survey
• Discussion
4
What is ClickerPlus?
• Quantitative method
– What they know
– What they learn
• Pre/post
• Advertising research
• Clickers
5
Hypothesis
ClickerPlus made me a better teacher:
– Students learned (more)
– Students appreciated (more)
6
Advertising Research Methods
How do we know that
advertising is effective?
Case History:
Dove Deodorant
Tracking Study
(New)Dove Deodorant
Moisturizing cream
Soothes and beautifies the underarms.
9
Dove Deodorant Communication Goals
• Be recognized as the best brand at caring for the look
and feel of underarm skin
• Become the beauty icon in the deodorant market through
focusing on improving the look of underarm skin
• Ensure that perceived efficacy remains at parity with the
market average
– Source: Euro Effie Awards Case History, 2009
10
People have to “learn.”
“leaves underarm skin
soft and smooth”
Benchmark Survey
(pre-wave)
Wave 1 Survey
(post-wave)
X%
X+%
…
“helps me cope with
the problems of
shaving”
11
Y%
Y+%
Ads “teach.”
12
Results
Dove deodorant “owns” underarm beauty.
14
Papers & Case Studies –
World Advertising Research Council
ClickerPlus Pilot Study Results
Summer 2009
Prepared by
Brian Moore
Baruch College
July 2009
Method
• Construct true-false statements.
• Have students respond
– Clickers
– TurningPoint
• Analyze.
• Adjust.
• Clarify and reinforce.
• Re-administer.
• Compare.
• Adjust.
16
Beginning of the semester
June 1, 2009
End of the semester
July 2, 2009
Learning
True/False questions
Benchmarks
True/False questions
17
Changes over time
Sample Description and Timing
•
•
•
•
Place: Baruch College
Course: Advertising
Sample: Baruch undergraduates
Sample Sizes:
– Pre-Wave, n=38;
– Post-Wave, n= 25
• Timing:
– Pre-Wave:
June 1, 2009
– Interim Review:
June 18, 2009
– Post-Wave:
July 2, 2009
18
Limitations
• Merely directional.
• Sample is insufficient, i.e.
– Small
– Unequal
– Unrepresentative
• Getting the underlying concept ?
• Need for validation
19
Instructions to the students in the
classroom
This is not a test. You won’t be graded on
this. We just want to find out what you know.
First Slide
Based on everything you know or may have
heard about advertising, please indicate
whether you believe that the following
statements are true or false.
21
Audience:
Pick up your Clickers!
Based on everything you know or may
have heard about advertising, please
indicate whether you believe that the
following statements are true or false.
The most important thing good advertising
does is make an emotional connection with
consumers.
1. True
2. False
ls
e
0%
Fa
Tr
ue
0%
The production of advertising ideas follows a
clearly defined process.
1. True
2. False
ls
e
0%
Fa
Tr
ue
0%
Good advertising can
keep a bad product alive.
1. True
2. False
ls
e
0%
Fa
Tr
ue
0%
American Idol would be a good place to
advertise the new version of the iPhone.
1. True
2. False
ls
e
0%
Fa
Tr
ue
0%
What you say in advertising is more
important than how you say it.
1. True
2. False
ls
e
0%
Fa
Tr
ue
0%
Emotional advertising appeals are more
effective than rational advertising appeals.
1. True
2. False
ls
e
0%
Fa
Tr
ue
0%
Advertising increases the value of the
advertised product.
1. True
2. False
ls
e
0%
Fa
Tr
ue
0%
If your advertising grabs people’s attention,
it will probably work.
1. True
2. False
ls
e
0%
Fa
Tr
ue
0%
If you like a particular product, you will be
better prepared to create great advertising
for that product.
1. True
2. False
ls
e
0%
Fa
Tr
ue
0%
It is more effective to have your advertising
seen or heard by more people than it is to
have fewer people see it more often.
1. True
2. False
ls
e
0%
Fa
Tr
ue
0%
How did the students do?
Summary of before/after results
For all 10 Statements
Results vary.
Correct Answers
90
85
80
74
70
75
67
65
63
60
Percentage
75
74
73
57
57
50
40
39
36
35
36
30
22
20
21
13
10
12
9
0
1
2
3
4
5
6
7
Statements
Jun-09
36
Jul-09
8
9
10
Results for each statement
Percentage
The most important thing good advertising does is
make an emotional connection with consumers.
100
90
80
70
60
50
40
30
20
10
0
61
78
39
22
Jun-09
38
True
False
Jul-09
False: “If it doesn’t sell,
it is not creative.,” David Ogilvy
(Belch & Belch 285)
Percentage
The production of advertising ideas follows a
clearly defined process.
100
90
80
70
60
50
40
30
20
10
0
63
True
False
38
Jun-09
39
74
26
Jul-09
True: “The production of
ideas is
just as definite a
process as the
production of Fords.,”
James Webb Young
(Belch & Belch 260).
Percentage
Good advertising can keep a
bad product alive.
100
90
80
70
60
50
40
30
20
10
0
27
88
73
False: “Nothing will put a bad
product out of business faster
than a good advertising
campaign.,” David Ogilvy
13
Jun-09
40
True
False
Jul-09
Percentage
American Idol would be a good place to
advertise the new version of the iPhone.
100
90
80
70
60
50
40
30
20
10
0
35
65
9
Jun-09
41
True
False
91
Jul-09
False: It would be a waste of
mass media on an exclusive
product, i.e. a poor return on
investment (Belch & Belch 354).
Percentage
What you say in advertising is more
important than how you say it.
100
90
80
70
60
50
40
30
20
10
0
12
35
88
65
Jun-09
42
True
False
Jul-09
True: “What really decides
consumers to buy or not to
buy is the content of your
advertising, not its form.,”
David Ogilvy
(Belch & Belch 291).
Percentage
Emotional advertising appeals are more
effective than rational advertising appeals.
100
90
80
70
60
50
40
30
20
10
0
43
74
75
True
False
26
25
Jun-09
Jul-09
True: You should always seek
emotional benefits. They are
more compelling, and they are
easier to defend
(Belch & Belch 285).
Percentage
Advertising increases the
value of the advertised product.
100
90
80
70
60
50
40
30
20
10
0
67
85
33
15
Jun-09
44
True
False
Jul-09
True: The simple act of
advertising a product makes
it appear more
valuable. Good advertising can
increase value perceptions and
direct pricing strategies (Moore).
Percentage
If your advertising grabs people’s
attention, it will probably work.
100
90
80
70
60
50
40
30
20
10
0
43
79
57
21
Jun-09
45
True
False
Jul-09
False: “Advertising succeeds or
fails depending on how well it
communicates the desired
information and attitudes to the
right people at the right time and
at the right cost,” Russell Colley
(Belch & Belch 220).
Percentage
If you like a particular product, you will be better
prepared to create great advertising
for that product.
100
90
80
70
60
50
40
30
20
10
0
25
64
True
False
75
36
Jun-09
46
False: You may not be
representative of the target
consumer, and you won’t be
objective (Moore).
Jul-09
Percentage
It is more effective to have your advertising seen
or heard by more people than it is to have fewer
people see it more often.
100
90
80
70
60
50
40
30
20
10
0
43
64
True
False
57
36
Jun-09
47
Jul-09
False: It depends on your
objectives, e.g.; if your message
is complicated, or If your product
is unfamiliar to people,
you probably need more
frequency (Belch & Belch 336).
Am I an effective teacher?
Learn?
Appreciate?
Learn?: Yes and No.
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
The most important thing good advertising does is make an
emotional connection with consumers.
The production of advertising ideas follows a clearly defined
process.
Good advertising can keep a bad product alive.
American Idol would be a good place to advertise the new
version of the iPhone.
What you say in advertising is more important than how you say
it.
Emotional advertising appeals are more effective than rational
advertising appeals.
Advertising increases the value of the advertised product.
If your advertising grabs people’s attention, it will probably work.
If you like a particular product, you will be better prepared to
create great advertising for that product.
It is more effective to have your advertising seen or heard by
more people than it is to have fewer people see it more often.
49
Appreciate?: Yes!
Student Course and Faculty Evaluation
5
Mean Ratings on a 5 Point Scale
4.5
4
3.5
3
2.5
2
1.5
1
0.5
0
1
2
3
4
5
Course Content
6
7
9
Course Delivery
Fall 2008
50
8
10
11
12
13
Learning Environment
Summer 2009
14
15
16
17
Impact on you
Summer 2008
18
How did the audience do?
Potential ClickerPlus Benefits
•
•
•
•
•
•
•
•
•
•
Objective
Flexible
Infinitely scalable
Easy
Kindergarten to College
Anthropology to Zoology
Yields real-time results
Attainment of learning goals
Inspires new goals
Provides benchmarks for:
– Overall improvement
– Developing individual goals
• Preserves and protects academic freedom
52
Conclusions
• ClickerPlus made me a better teacher
– Learned
– Appreciated
• Need more data:
– More cases
– Student test scores
• Better samples
• Validate
– Qualitative research
– Search phrase algorithm
• Automate
53
Next Steps
• Awaiting response to our grant proposal from the U.S.
Department of Education
– More instructors
– Different subjects
– Different levels
– $$$
• …
• Brian.Moore@baruch.cuny.edu
• 646-312-3332
54
ClickerPlus
Pilot Study
55
Works Cited
• Belch, George, and Michael Belch. Advertising and
Promotion. New York: McGraw-Hill, 2009
• Georgia Tech School of Civil and Environmental
Engineering. [2000] Instructional objective writing
assistant
• Moore, Brian. Lectures on Advertising and Promotion,
Baruch College, New York, June 2009.
• Ogilvy, David. Various lectures, speeches and books.
• Pinker, Steven. The Blank Slate. New York: Penguin
Group, 2002
56
Appendix
Scatter diagram
LEARNING GOALS
At the end of this course, you will understand how global advertising
agencies propose integrated marketing communication programs to
global manufacturers of consumer package goods. You will learn
how to conduct research to define the target audience and its media
habits, brainstorm to generate messaging ideas, develop a creative
strategy, create executions derived from the strategy, produce the
executions, test whether or not the executions communicate
their intended messages, plan and buy the media to deliver the
messages to the target audience, and incorporate all of your
findings into a persuasive, visually-intensive, oral presentation.
You will learn how to work as part of a cross-functional team to
achieve the goals stated above.
58
The technique discriminates, and more students answered
the questions correctly at the end of the semester than they
did at the beginning of the semester.
Percent Correct
100
80
60
Jun-09
Jul-09
40
20
0
0 1 2 3 4 5 6 7 8 9 10
Statements
59
60
61
Formatting Learning Goals
• Student centered (e.g., “by the end of the
course, students will be able to…”),
• Outcome oriented (e.g. “write a scholarly
research paper…”), and
• Detailed (e.g., “using MLA-style referencing and
formatting…”).
Georgia Tech [2000]
COM3150 Course Review
62
My Learning Goals
1.
2.
3.
4.
Defend the primacy of strategic messaging.
Describe and apply the advertising planning process.
Recognize why good advertising cannot keep a bad product alive.
Analyze the efficiency of different media vehicles, using syndicated
research.
5. Explain why the content of your advertising is more persuasive than
its form.
6. Discuss why emotional benefits are more persuasive than rational
benefits.
7. Describe how and why advertising increases the value of the
advertised product.
8. Interpret and explain the hierarchy of effects model.
9. Identify target audiences by analyzing syndicated research.
10. Demonstrate the trade-offs between advertising reach and
frequency.
63
Less Effective Example
Effective Example
This course in
Engineering Graphics
will introduce students to
the use of computeraided-design software
At the end of this course
in Engineering Graphics,
you, the student, will be
able to draw a multi-view
representation of a solid
object using a computeraided-design software
Georgia Tech [2000]
64
The 10 Statements
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
The most important thing good advertising does is make an
emotional connection with consumers.
The production of advertising ideas follows a clearly defined
process.
Good advertising can keep a bad product alive.
American Idol would be a good place to advertise the new
version of the iPhone.
What you say in advertising is more important than how you say
it.
Emotional advertising appeals are more effective than rational
advertising appeals.
Advertising increases the value of the advertised product.
If your advertising grabs people’s attention, it will probably work.
If you like a particular product, you will be better prepared to
create great advertising for that product.
It is more effective to have your advertising seen or heard by
more people than it is to have fewer people see it more often.
65