When I was looking at the results for an A/B test that was set up to split traffic 50/50, the “total impressions” volume for the variants were split closer to 60/40.
Can I assume that Builder split traffic 50/50? Or is it possible that the total impression volumes were indicative of the actual traffic split and the test at that point did not split traffic 50/50?
Note: I cannot provide an example because the data for the test disappeared (which I notified the team about separately). However, I have the absolute number of conversions from my own data and am trying to figure out if I can use that data to understand which variant performed better. If I can assume that there was a 50/50 traffic split that will work, if not, it would be inaccurate.
The sample size when I looked was ~5k total impressions, now it’s ~12k total impressions.
I don’t know what the current split is between the two variants (given the data disappearance issue). If you can’t see the splits on your end, I’ll post a new screenshot once it’s available again in the editor.
Hi John! We have a new A/B test flow that should be coming soon that will help prevent confusion like this. The new flow will ensure that the “control” group doesn’t receive any traffic before the tests groups, and if you want to add / remove test groups you can clone the content and start a new test.
Also, we’re working on some improvements to the Insights page that should fix the other issues with older content with a lot of traffic. We’ll let you know when we have something for you to test out!
In the meantime, I’d suggest going to beta.builder.io, cloning your content and starting a new test. That way you can rest assured that the traffic breakdown is correct as well regaining the ability to view the rest of the test results on the Insights tab.
Thanks for reaching out.