Can I A/B test Instant to validate the results?
When a shopper purchases after engaging with an abandonment email triggered by Instant, we attribute that revenue to Instant. This attribution model mirrors how platforms like Klaviyo attribute revenue generated to individual email marketing flows.
Our research has found that the majority of revenue attributed to Instant is associated with merchants who would not have placed an order had they not received an abandonment email. However, given the seasonality of retail sales, it can be difficult to see exactly how Instant's incremental revenue increase has impacted a merchants' top-line revenue, especially when a merchant adopts Instant during a quieter sales month.
Case Study
For this merchant, we conducted an A/B test to determine exactly how much additional revenue Instant's superior abandonment email marketing could generate for them.
We assigned all shoppers identified by Instant, and not identified by Klaviyo or Shopify, to two cohorts:
- Control — for this segment of shoppers, despite our ability to identify them, we suppressed sending any events to Klaviyo. Therefore, shoppers in this group would not receive abandonment emails triggered by Instant, and if they were to purchase, this would have been purely of their own volition.
- Treatment — for this segment of shoppers, we sent the usual events to Klaviyo, triggering abandonment emails.
Over two weeks, we found:
- The control segment, who did not receive any Instant-triggered abandonment emails, generated $69.8K in revenue.
- The treatment segment, who did receive Instant-triggered abandonment emails, generated $84.2K in revenue.
This is a revenue uplift of $14.4K over two weeks. Had the A/B test not been running the merchant would've generated $28.8K, or $57.6K over a one-month period.
While we did find that Klaviyo attributed slightly more revenue than this to Instant Audience's flows, this revenue uplift is impressive and inline with the results published in our case studies, achieving a massive 23x return-on-investment for the customer.
A/B testing Instant
We're happy to conduct A/B tests for individual merchants who wish to better quantify the ROI they receive from Instant.
Similar to the case study above, in these tests, we assign all shoppers identified by Instant, and not identified by Klaviyo or Shopify, to two cohorts:
- Control — for this segment of shoppers, despite our ability to identify them, we suppress sending any events to Klaviyo. Therefore, shoppers in this group would not receive abandonment emails triggered by Instant, and if they were to purchase, this would have been purely of their own volition.
- Treatment — for this segment of shoppers, we sent the usual events to Klaviyo, triggering abandonment emails.
Over two weeks, we typically see the control segment, who did not receive any Instant-triggered abandonment emails, generate much less revenue.
We can estimate a total monthly increase by multiplying this number by two, to account for the test only running for two weeks, and again by two, to account for the control group that did not receive our events.
To qualify for an A/B test of Instant, merchants must have at least $500,000 in monthly website revenue and relatively high traffic. Smaller websites, unfortunately, do not have enough traffic to produce a statistically significant result from an A/B test of this nature.