Proving ROI with A/B Testing

How you can be confident with your Instant Audiences results through data-driven A/B Tests

In today’s competitive eCommerce landscape, every marketing dollar counts. At Instant, we want our brands to be confident of the true impact of Instant, providing indisputable proof for the ROI Instant brings. This is why Instant is introducing built-in A/B testing—so that any brand can see objectively how Instant drives your brand's revenue growth.

The A/B test is easy to set up, and only some brands are eligible to run this test. Please reach out to your Customer Success Manager to get started. Once it is set up, the A/B test results are 100% self-serve with results displayed inside of the Instant Dashboard. The information displayed in the "A/B Test" section of the dashboard is designed to help you evaluate how much additional revenue Instant generates, with statistics in there to guide you through the Instant A/B Test logic and process. 

How the A/B Test Works:

1. Enable A/B Testing – A customer success manager (CSM) can activate an A/B test for a merchant directly from the Instant dashboard.
   
2. Split Audience Cohorts – Shoppers visiting the merchant’s store are randomly assigned to one of two groups - 50% of shoppers into each group:
   
   1. Group A (Instant Test): This group's user events are de-anonymised from Instant and sent to Klaviyo, powering Instant marketing flows in Klaviyo.
   2. Group B (Excluded): This group will not have their user events de-anonymised by Instant.
      
3. Track Key Metrics – The platform automatically monitors conversion events, revenue generated, and shopper engagement for both groups.
4. Analyze Revenue Impact – We strongly recommend the A/B test to run for be at least 4 to 6 weeks, and should not be shorter. This is because the larger the sample size is, the more accurate the results will be. 
   During the test, the Instant dashboard can present an objective side-by-side comparison of:
   1. Revenue generated in total, and by each cohort
   2. What each cohort's test results process looks like
   3. Revenue uplift between the two cohorts - bearing in mind this isn't reflective of the total revenue uplift you would see with Instant as only 50% of the eligible shoppers are going through the Instant experience.
5. Make Data-Driven Decisions – brands can use these results to validate the effectiveness of Instant’s platform and optimize their marketing strategies.

Accessing your A/B Test 

1. Navigate to the Instant Dashboard and log in.

2. After logging in, select A/B test in the left-hand panel

3. This will take you directly to your A/B test results

 

How to Understand your Instant A/B Test Dashboard and Data:

1. How we tested: consistent with how Instant Audiences usually functions, the A/B test only include Instant identified browsers arriving at your store. We exclude any browsers that Klaviyo or Shopify would have already identified. 

For example, in this A/B test Instant has identified 3,548 shoppers that Klaviyo and Shopify were unable to identify.

2. Results

This section of the report presents the primary test outcomes:

• Instant Revenue Uplift: The additional revenue generated from shoppers who received Instant emails compared to those who did not.

• Total Revenue: The combined revenue from both groups over the testing period. Noting that the is the revenue generated only from shoppers that were tracked by the A/B test.

• Revenue by Group: A breakdown showing how much revenue was generated by Group A vs. Group B.

• Expected Future Uplift: The estimated future revenue gain if Instant emails continue to be sent to all eligible shoppers.

Also included in this section is a time-series chart that helps you visualise and tracks revenue accumulation for both Group A and Group B throughout the testing period.

• The green line: Represents revenue from shoppers in Group A.

• The grey line: Represents revenue from shoppers in Group B.

• The gap between the lines: Indicates the incremental revenue difference generated by Group A vs Group B.

For example, in this test Instant’s emails generated an additional $17,835.04, an 17% revenue increase comparing Group A over Group B. If the merchant extends this strategy, they can expect an estimated future uplift of $35,670.08.

3. Test breakdown

The test breakdown diagram provides a visual representation of how the test was structured and the final results:

• Total Shoppers Tested: The number of shoppers split into two groups.

• Conversions Per Group: The number of shoppers who completed a purchase.

• Revenue Per Group: The revenue generated from each cohort.

• Percentage Uplift: The increase in conversions and revenue observed in the Test Group.

For example, in this test we analyzed 14,467 shoppers, split evenly between the two groups:

• Group A (Received Emails): 259 conversions, generating $123.4K in revenue.

• Group B (No Emails): 236 conversions, generating $105.6K in revenue.

• Result: A clear 17% increase in conversions and revenue that Group A has demonstrated over Group B.

---

Key notes about your A/B Test:

• Traffic Separation: The traffic is split into two separate groups, allowing you to clearly compare the value of Instant identifications. 
• No Impact on Existing Flows: The A/B test is run independently from your existing Klaviyo flows, so it will not interfere with your store’s regular revenue generation.
• No Email Triggered for 50% of Browsers: In the A/B test, 50% of the browsers that Instant identifies will not have their de-anonymised events sent to Klaviyof, allowing us to track their behavior separately.

---

Frequently Asked Questions:

1. How are the browsers assigned to Group A and Group B?

Browsers that Klaviyo identifies are excluded from the A/B test. For shoppers that are not identified by Klaviyo but are identified by Instant, 50% will be sent Instant events (Group B), and the other 50% will not (Group A). These are randomly selected. 

2. How do we measure the impact of Instant?

We measure conversion and revenue for each group (A and B). The results show how much additional revenue could be generated by using Instant events.

3. What happens if a browser identified by Klaviyo makes a purchase?

If a browser is identified by Klaviyo, their purchases are tracked as part of the standard Klaviyo metrics and do not impact the A/B test and they are excluded from the A/B test. 

4. Will My A/B Test Revenue Appear in My Analytics Page?

No, the A/B test revenue will not appear in your analytics page. Your analytics page shows how much revenue Klaviyo is reporting. The A/B test is comparing the overall revenue difference between the two randomly allocated cohorts (one we send events to Klaviyo, the other we don't).

Conclusion

This A/B testing feature provides definitive proof of Instant's value, eliminating skepticism and doubts related to your brand's ROI figures. Brands can now see exactly how much additional revenue they’re generating with Instant Audiences—helping them make data-driven decisions and supercharge their revenue!