Can I A/B test Instant to validate the results?
When a shopper purchases after engaging with an abandonment email triggered by Instant, we attribute that revenue to Instant. This attribution model mirrors how platforms like Klaviyo attribute revenue generated to individual email marketing flows.
Our research has found that the majority of revenue attributed to Instant is associated with merchants who would not have placed an order had they not received an abandonment email. However, given the seasonality of retail sales, it can be difficult to see exactly how Instant's incremental revenue increase has impacted a merchants' top-line revenue, especially when a merchant adopts Instant during a quieter sales month.
Case Study
For this merchant, we conducted an A/B test to determine exactly how much additional revenue Instant's superior abandonment email marketing could generate for them.
We assigned all shoppers identified by Instant, and not identified by Klaviyo or Shopify, to two cohorts:
- Control — for this segment of shoppers, despite our ability to identify them, we suppressed sending any events to Klaviyo. Therefore, shoppers in this group would not receive abandonment emails triggered by Instant, and if they were to purchase, this would have been purely of their own volition.
- Treatment — for this segment of shoppers, we sent the usual events to Klaviyo, triggering abandonment emails.
Over two weeks, we found:
- The control segment, who did not receive any Instant-triggered abandonment emails, generated $69.8K in revenue.
- The treatment segment, who did receive Instant-triggered abandonment emails, generated $84.2K in revenue.
This is a revenue uplift of $14.4K over two weeks. Had the A/B test not been running the merchant would've generated $28.8K, or $57.6K over a one-month period.
While we did find that Klaviyo attributed slightly more revenue than this to Instant Audience's flows, this revenue uplift is impressive and inline with the results published in our case studies, achieving a massive 23x return-on-investment for the customer.
A/B testing Instant
We're happy to conduct A/B tests for individual merchants who wish to better quantify the ROI they receive from Instant.
Similar to the case study above, in these tests, we assign all shoppers identified by Instant, and not identified by Klaviyo or Shopify, to two cohorts:
- Control — for this segment of shoppers, despite our ability to identify them, we suppress sending any events to Klaviyo. Therefore, shoppers in this group would not receive abandonment emails triggered by Instant, and if they were to purchase, this would have been purely of their own volition.
- Treatment — for this segment of shoppers, we sent the usual events to Klaviyo, triggering abandonment emails.
Over two weeks, we typically see the control segment, who did not receive any Instant-triggered abandonment emails, generate much less revenue.
We can estimate a total monthly increase by multiplying this number by two, to account for the test only running for two weeks, and again by two, to account for the control group that did not receive our events.
To qualify for an A/B test of Instant, merchants must have at least $500,000 in monthly website revenue and relatively high traffic. Smaller websites, unfortunately, do not have enough traffic to produce a statistically significant result from an A/B test of this nature.
Comparing results to Google Analytics
Instant Audiences and Instant AI both conform to Klaviyo's attribution model, because this is the standard for email marketing tools. Because this attribution model varies significantly from Google Analytics, many merchants using Google Analytics notice attribution variances.
Ensure all paid and organic channels can receive credit for conversions
By default, Google Analytics only attributes revenue to Google paid channels. This massively biases Google Analytics attribution towards Google channels.
For more fair attribution across non-Google channels, ensure all paid and organic channels can receive credit for conversions.
Changing this setting will not impact historic data, may take a few days to take effect, and may impact the conversions you create in Google Ads for bidding and reporting.
Tighten your attribution windows in Google Analytics
By default, Google Analytics has extremely broad attribution windows. This means top-of-funnel channels like Google Ads will get credit for conversions even if their only touchpoint event was months ago. This excessively dilutes attribution across channels, giving old and irrelevant events credit for recent conversions.
We recommend tightening your attribution windows in Google Analytics as follows:
- Acquisition key events (i.e., first_open, first_visit): 7 days (30 days is the default).
- Other key events: 30 days (90 days is the default).
- Engaged-view key events: 3 days.
Understand data-driven attribution
Google's default model is called data-driven attribution. This model is powered by AI. It is analyzing your web traffic, and, where necessary, guessing whether a conversion happened and why. This is necessary because Google Analytics isn't always are of the identity of a shopper (unlike Instant Attribution, thanks to our excellent identity resolution technology) and therefore their history.
This type of attribution works great across very large datasets, but can be extremely unreliable for smaller datasets. Any merchant turning over under $500,000 in monthly website revenue is unlikely to see accurate results from any AI-driven attribution model. These merchants should double check their results by switching to Google's paid and organic last click attribution model, which admittedly favors bottom-of-funnel channels like Instant AI, Instant Audiences, and Klaviyo.