ABConvert: A/B Testing

We’re sorry the difference between our analytics and billing measurements was not communicated clearly enough.

To clarify, there are two valid ways to measure orders during an A/B test. The first is the storewide method, which counts all orders processed by the store while a test is active. The second is the exposure-filtered method, which shows only orders attributed to visitors who were actually exposed to the test. In this case, the approximately 400 orders shown in analytics were exposure-filtered orders, while billing was based on approximately 8,000 storewide orders processed during the active test period. These figures represent different measurements and should not be expected to match. Storewide order volume is a common billing model among A/B testing platforms.

The exposure-filtered view was introduced as an analytics improvement and made the default to help merchants focus on test-related visitors and orders, reduce noise from unrelated store activity, and reach meaningful insights faster. It did not change the billing method or create additional orders. However, we recognize that we did not explain clearly enough that this improved analytics view was different from the storewide measurement used for billing. This lack of clarity caused the approximately 400 exposure-filtered orders to be mistaken for the total billable order volume.

Our team reviewed the account and responded within an hour. We examined both the analytics and billing records, explained the difference between the two measurements, and acknowledged that this distinction should have been clearer in our app and documentation. Although the overage was calculated according to the storewide billing method associated with the plan, we offered a full refund of the $236 overage as a one-time courtesy because we accept responsibility for not communicating the improvement clearly enough. We are also updating the app and documentation to prevent similar confusion.

Finally, the review’s claim of more than $400 in unjustified fees is factually incorrect. The actual disputed overage was $236—not more than $400—and we offered to refund it in full. The charge was calculated using the storewide order volume defined by the billing method; it was not an arbitrary or unexplained fee.

We take concerns like this seriously and remain committed to transparent billing, accurate analytics, and responsive support.