Split Test Highlights
TABLE OF CONTENTS
1.
2.
3.
4.
5.
6.
7.
8.
9.
Cart Tests
Store Tests
Product Page Tests & Split-Tests
Buy Box Tests
Site-wide Header & Sticky Header Tests
Presell Page Tests
Mobile SPLIT TESTS (Buy Box, Header, etc)
Miscellaneous Split-TestS (Social Proof, Price, Cart layout, etc)
Homepage Tests
CART TESTS
Conversion for Cart Upsell Split-Test
10/26/17
What Was Tested: Cart upsell vs no cart upsell
Result: Cart upsell wins
About This Test: This test prompted customers to upgrade a product in their carts (ex: a
customer buying BoomStick Color may be prompted to upgrade to BoomStick Trio). Test
results show significant conversion improvement when customers are presented with an
upgrade opportunity while adding items to cart. Conversion for Cart Upsell ranged from
about 54%-73% versus 44%-66% for the non-upsell Cart/Checkout.
Video Clip: Cart Upsell
Dynamic product upgrade cart test
11/9/17
INITIAL DYNAMIC UPGRADE
SECOND DYNAMIC UPGRADE BASED ON
PRIOR UPGRADE SELECTION
What Was Tested: Single product upgrade (Control) vs dynamic product upgrade (Variant)
Result: Dynamic upgrade wins
About This Test: The Control was a simple Cart Upsell of Boomstick Trio for customers
buying a single Boomstick Color. In this test, if a customer chooses a pre-purchase
upgrade, they are offered a second, larger upgrade based on what they just upgraded to
(ex: if a customer upgrades from Boomstick color to Trio, then they are offered a Boom
Bag bundle next). Offering this second dynamic Upsell improved conversion rate by about
2% with a 91% chance of winning.
Video Clip: Dynamic Product Upgrade (00:53-1:49)
Slide-Out Cart Test
1/30/19
REGULAR CART
SLIDE OUT CART
What Was Tested: Cart page redirect vs slide-out cart
Result: Cart page redirect wins
About This Test: A slide-out cart was tested against a normal cart that redirects to a cart
page (control). The slide-out cart allows customers to still have the upgrade option and keep
shopping. The control won this test, with average revenue per user (ARPU) dropping from
$10 to $8 at 1% confidence and a conversion rate decrease of 13.6% with the slide-out cart.
(However, store visits were down 10% during this test and the test was stopped early. Ezra
concluded by stating he wants to run this test again.)
Video Clip: Slide Out Cart Test (2:48-4:06)
Cart layout
9/15/19


CLASSIC CART (MOBILE)
CLASSIC CART (DESKTOP)

VARIANT CART (DESKTOP)

VARIANT CART (MOBILE)
Data showing +$2.50 ARPU for Cart Layout Variant
What Was Tested: Traditional cart layout vs user-friendly cart layout
Result: User-friendly cart layout wins
About This Test: For this test, the cart layout on both desktop and mobile was rearranged
in various ways to be more user-friendly. The "Checkout Now" button was placed in the
center and add-to-cart function was made easier to use. Testimonials remained and
upgrade CTAs were also made more user-friendly. The ARPU of the variant showed an
increase of over $2.50 with an overall revenue increase of $20,000.
Video Clip: Cart Layout (7:48-11:23)
Split Testing Upsells in Cart
2/11/21
What Was Tested: Cart upsell products (Boom Bright vs Boomsilk)
Result: Boom Bright
About This Test: This test was a simple comparison of placing different products in the
cart checkout upsell box. The two products in this test were Boom Bright and Boomsilk.
When customers took the action to checkout, they were prompted with an option to add
either Boom Bright or Boomsilk to their cart. Boom Bright significantly outperformed
Boomsilk, with nearly 12% higher conversion rate, $2.30 more per visit and over $5,000
more revenue despite fewer views.
Video Clip: Split Test Your Upsells
STORE TESTS
Category Page Redesign & Products Per Line
3/25/21
What Was Tested: Number of products displayed (4 rows vs 3 rows) plus visual features
Result: 4 product display with enhanced visual features wins
About This Test: On the store category page, 4 rows of products (with enhanced visual
features such as headlines, adjusted spacing, and larger buttons) were tested against 3
rows of Products (with no enhanced visual features). The 4 rows variant saw 2% more
product page views and 4% more cart views. Additionally, displaying more products per
line increased scroll depth and widened the funnel.
Video Clip: Store Grid Redesign & # Per Line (1:17-2:34)
STACKED vs CAROUSEL & Display number FOR MOBILE & DESKTOP
5/6/21 & 6/11/21
4 Across Desktop Display (Winner)
2 Across Mobile Display (Winner)
What Was Tested: Stacked carousel vs row/column format (with item amounts)
Result: 4 Across Desktop; 2 Across Mobile wins
About This Test: Store category page tests were run for both mobile and desktop for
stacked carousels vs row/column formats. On mobile, stacked carousels won over
columns. However, on desktop, 2-item rows won over carousels. Ezra was moving forward
beyond this test with further testing of specific item amounts for carousel vs row/column
formats.
Building on tests from the previous test segment, additional store category page formats
were tested. Columns of 4 and 2 items were tested on desktop and mobile,
respectively. These formats outperformed the control (4 across on desktop & a single
item column on mobile) as well as carousel options for desktop & mobile.
Video Clip: Initial Carousel vs Stacked Test & Further Item Display Amount Test
Product page tests
Visual storytelling
6/15/17


What Was Tested: Increased product imagery vs text heavy product page
Result: Increased product imagery wins

About This Test: This test ran increased & reorganized visuals on the product page. Visual
features included headline color for improved readability, highlight of core value
proposition, and increased product imagery. The improved visual feature Product Page
variant lifted revenue per visitor and revenue per conversion, and it showed a 57% chance
to beat the more text-heavy control.
Video Clip: Visual Storytelling
boom! simplified product page
10/12/17
Busier Product Page
Lighter Weight Product Page
What Was Tested: Light weight product page vs busier product page
Result: Light weight product page wins
About This Test: We tested simplifying the product page design on both desktop and
mobile. In the test, the hero shot and call to action are largely visible in the center of the
page with a price point and CTA Sales videos. When viewed in mobile, the featured
elements are even more simplified.
Video Clip: Product Page Simplification (00:00-1:21)
testimonial as Product title
7/11/18
Testimonial as Title
for Boom Mask
Testimonial as Title
for Boomsilk
What Was Tested: Testimonial as product title vs product name
Result: Testimonial as product title wins
About This Test: Because customers typically know the name of the product, a test was run
replacing the product title with a testimonial. Revenue per order increased from $7 to $41 at
100% confidence and conversions were up from 7% to 23%. However, additional testing was
done since the test was run during a sale and results were highly atypical. This time, the test
was run on Boomsilk. ARPU increased from $27 to $28 at 83% confidence. What pushed this
test into winner territory was conversion rate, which increased by 5.25% at 94% confidence,
outperforming the control.
Video Clip: Testimonial as Title pt. 1 ; Testimonial as Title pt. 2
BUY BOX TESTS
review stars in buy box
1/30/19
What Was Tested: Review stars in buy box vs in product title
Result: Review stars in buy box wins
About This Test: We moved the review stars from the product title to below the money
back guarantee. ARPU increased from $16.50 to $18 and conversions went up by 2.5%.
Based on results, this was applied to all products.
Video Clip: Review Stars in Buy Box (1:41-2:25)
buy box product upgrade
7/15/21 & 8/5/21
Control
Variant
What Was Tested: Buy box product upgrade vs upgrade redirect
Result: Buy box product upgrade wins
About This Test: This was a split test on the Boomstick Color page offering an upgrade
from a single Boomstick to a Boomstick Trio. The original page has the offer in form of a
redirect button while the test page presents an option to upgrade without redirecting
them to a different product page. On mobile, this test page was losing to the original but
the variant eventually pulled ahead. On Ddsktop, the test page was winning significantly.
Video Clip: Buy Box Product Upgrade Pt. 1 ; Pt. 2
pop-up modal
12/9/21
What Was Tested: Product page pop-up buy box modal vs standard ATC button
Result: Pop-up buy box modal wins
About This Test: In an effort to streamline the customer journey, a pop-up modal in the
product page buy box was tested so that customers have the option to continue shopping
or check out now. Customers also have an option to simply exit out of the pop-up to
continue shopping on the product page. The pop-up modal saw an increase in AOV of
over $3.
Video Clip: Pop-Up Buy Box Modal
Site-wide & sticky header TESTS
Pared down store header
7/27/17
What Was Tested: Pared down store header vs classic store header
Result: Pared down store header wins
About This Test: In the BOOM Store, the standard header included a big logo, tagline,
search box, phone number, account, menu, and CTA. In the test variant, the search box
and menu were removed, effectively forcing customers to more directly navigate toward
the product. The pared down header outperformed the control in revenue per visitor and
total revenue.
Video Clip: Pared Down Store Header
sticky footer atc + atc confirmation response
3/25/21
What Was Tested: Sticky add-to-cart buttons at footer (with confirmation message) vs
original add-to-cart button
Result: Sticky add-to-cart buttons at footer wins
About This Test: The sticky add-to-cart button showed a 9% lift in conversion rate at 95%
confidence. Offering a stick CTA on mobile product pages increases action and replacing
the cart redirect with a success message was a better experience for users who were still
browsing.
Video Clip: Sticky Footer ATC (6:39-8:17)
Presell PAGE tests
presale page layout
7/13/17
What Was Tested: Simplified presale page layout vs classic layout
Result: Simplified layout wins
About This Test: An easier to read and navigate variant of BOOM’s presale page was
tested against the long-time, classic presale page. While the variant contained almost all
of the same content, the layout was presented in a simplified way. While ARPU was dead
even, BOOM Club opt-ins went up nearly 30% at 100% confidence. Visits to the store
page also went up 194% at 100% confidence, and heatmaps showed significant increases
in CTA clicks.
Video Clip: Simplified Presale Page Layout
two-column presell
9/5/19



What Was Tested: Two-column presell on desktop & tablet vs single-column on mobile
Result: Two-column presell wins
About This Test: We tested a two-column presell presented on desktop and tablet vs a
single-column Presell on mobile. The two-column presell box includes BOOM’s top 2
selling products. On desktop, ARPU increased from $4 to $4.57, and on tablet it increased
from $1.52 to $1.77, both in a high 80% range of confidence. Conversion rate on desktop
increased from 4.3% to 5.5% at 100% confidence, and tablet saw an increase from 1.8% to
2.1% at 90% confidence. The aggregate conversion rate increase was 2.85% to 3.53% at
100% confidence. Visits to the product page increased from 25.3% to 33% on desktop and
15.4% to 22.9% on tablet.
Video Clip: Two Column Presell (00:46-7:27)
mobile presell font size
3/25/21
What Was Tested: 3 font sizes on mobile presell page (16pt vs 18pt vs 20pt)
Result: Size 20pt font wins
About This Test: Size 16 was the original font size for this test, and it was tested against
sizes 18 and 20. Size 20 worked best. There was no impact on bounce rate, but time on the
site increased by 3% and scroll depth and page views increased by 1%. Making the site
easier to read makes people stick around longer and navigate further. As a result of this
test, font size was increased to 20 across mobile and desktop.
Video Clip: Presale: Mobile Font Increase (2:36-3:24)
long-form copy vs pictures in-between on presell
12/8/22
What Was Tested: Long-form copy vs pictures in-between on presell
Result: Long-Form Copy
About This Test: With the intent of making the Rose Nude presell “more fun to read”,
pictures relating to the product were added between each tip to break up the long-form
copy. The long-form copy control was worth $0.75 more per visitor on mobile and $0.50
cents more per visitor across all traffic. Average order value was higher as well as
conversion rate.
Video Clip: Long-Form Copy vs Pictures In-Between
mobile split tests
pared down mobile product page
7/13/17 & 7/27/17
What Was Tested: Simplified mobile product page vs heavier original
Result: Heavier original wins
About This Test: We tested two variants with pared down presentation and visuals on the
mobile product page. The control features significantly more videos, imagery, and longform left-right content, and both variants removed most of the videos and all images.
Though results did not include large amounts of data at this time, the control showed an
80% chance of beating both variants. However, after running the test more and slightly
tweaking the variant by removing reviews and Yotpo toward the footer, the simplified
variant began outperforming the content-heavy Control.
Video Clip: Pared Down Mobile Product Page pt. 1 ; pt. 2
sticky atc on mobile Footer
6/10/21
What Was Tested: Sticky add-to-cart button on mobile footer vs none
Result: Sticky add-to-cart wins
About This Test: This test examined implementation of a sticky add-to-cart CTA button
visible while scrolling the product page on mobile. Results showed an increase of $0.46
more per visitor on 64% of sessions by having a sticky add-to-cart CTA on the mobile
footer vs none.
Video Clip: Mobile Sticky ATC (4:24-6:56)
miscellaneous
opt-in colored background
1/31/20
What Was Tested: Colored background for opt-in vs white background
Result: Colored background wins
About This Test: During a holiday giveaway, a colored backdrop behind the opt-in box
converted significantly better than text with a white backdrop. As a result of this, a similar
split test was run for the Boom Club opt-in page. Running this test on the Boom Club optin page, the colored backdrop variant outperformed the white backdrop control by over
4%. Additionally, the colored backdrop significantly outperformed in AOV, conversion
rate, and per session value.
Video Clip: Opt-in Colored Backdrop Test
shadows on buttons site-wide
3/25/21
What Was Tested: Shadows on buttons vs no shadows
Result: Box shadows wins
About This Test: Shadows were added to buttons to give them a 3D appearance. The
shadowed boxes won by a lot – revenue per session saw an $0.08 increase while bounce
rate was unaffected. Additionally, customers who stayed on the site spent more money
and conversions increased by 5% on tablet and 2% on desktop.
Video Clip: Sitewide Box Shadows on Buttons -- (0:00-1:16)
Free shipping threshold
4/21/22
What Was Tested: $60 free shipping threshold vs $55 free shipping threshold
Result: $55 free shipping threshold wins
About This Test: This test was run through the app ShipScout. As a result of the positive
impact from raising the free shipping threshold from $35 to $55 in the past, a raise to $60
free shipping threshold was tested. In addition to earning $0.13 less per visitor, the AOV
for $60 free shipping threshold dipped to $46.12 from the AOV for the $50 free shipping
threshold of $46.66.
Video Clip: Free Shipping Threshold