Product Roadmap via A/B Testing
Market Needs +
Product Roadmap
Product
Growth Conversion
Data Scientist
or Analysts
Customer
Product Managers or
Roadmap into
Implementation via
Expectations +
business Analysts
Testable Chunks
A/B Testing CRO
Business Goals
01
Deconstruct
02
Business Analyst or
Developers or
UX Designers
CRO Analysts
03
Program Management
(Dedicated CRO program Manager)
04
05
A/B Testing Process
01
Define Goal (conversion)
Before your create a test, your
01
need to know what, exactly,
04
you’re hoping to accomplish.
Crown the New Champion
Once you’ve launched your
02
02
test, you need to let it run for a
05
long enough period of time.
Brainstorm/Generate Idea
CONTINUOUS
Once you’ve determined a
IMPROVEMENT
goal you want to accomplish,
you’ll need to generate ideas.
05
Test Until Statistically Significant
By this point, you’ve set a goal,
03
Implement Changes
You will be need to implement
changes with highest potential,
intuition and best practices.
03
04
select a page and determine
what you want to test.
A/B Testing Process
VARIATION A
30%
Conversion rate
50% visitors
see variation A
VARIATION B
16%
50% visitors
see variation B
Conversion rate
A/B Test Results
VARIATION
CONVERSION RATE
IMPROVEMENT
CONFIDENCE
CONV./VISITORS
STATUS
ORIGINAL
0.15% (+0.10%)
-
-
5/5,134
Stopped
VARIATION 01
0.25% (+0.20%)
+150.00%
97.01%
13/5,152
Winner
A/B Testing Line Graph
Total visitors
Average conversion rate
Original
Variation 01
A/B Test Tracking
OWNER (LIST
LANDING PAGE
NAME OF
NAME
DRI FOR TEST
STATUS
TEST
START
END
PREDICTION
RESULT
STATISTICALLY
SIGNIFICANT ?
NOTES
NEXT STEPS
HERE)
How to Create
Visuals for
Marketing
John
Smith
Demo
Amanda
Landing Page
Morgan
Call Sales
John
Landing Page
Smith
Completed
swapped with
5/17/17
5/20/17
subheader
Not Started
Different image on
B version
A is the title, B is
better because the
original sub header is
Success
✓
more actionable
5/27/17
6/20/17
background) will
Inconclusive
X
descriptive
better because it
was statistically significant at
setting up a/b on CTA
95% confidence. Declared
placement next
than A - not statistically
significant but declared B as
winner.
Version A will perform
7/6/17
Chose Variation B as winner,
Variation B performed 4% better
perform better
6/23/17
than the original version and
version B the winner
Version B (image with
Different Headline.
Not Started
Variation B performed 8% better
Version B will perform
LP Header
Inconclusive
X
matches promo copy
Chose Variation B. setting up
a/b test testing two different
types of images next
Version A performed 3% better,
Chose Variation A as winner,
not statistically significant. Test
running template-level A/B test
ran for 3 weeks.
next
Test "A" converted 33%better
2018
Predictions
Landing Page
Amanda
Morgan
In Progress
Copy bullets vs long
form paragraphs
6/17/17
7/12/17
version B will perform
better
Success
✓
than Test "B." We are100%
Chose Variation A as winner,
certain that the changes in Test
setting up a/b on CTA
"A" will improve your conversion
placement next
rate.
Get the full version of this
presentation and access our library
of hundreds of corporate resources
Join You Exec Plus today and take your career
to new heights. Receive premium corporate
resources in your inbox every week.
Start today