"Insufficient confidence" with 1400+ participants, 60+ completions and 2 variants

Open dingyu-tourcandy opened this issue 4 years ago • 2 comments

We started this A/B test about three weeks ago to find out which of the two form designs has the best conversion rate. So far we have 1400+ participants and 60+ completions, but split still gives us a "Insufficient confidence".

Screen Shot 2021-07-21 at 15 11 14

Is it normal? If not, what could be the cause?

Jul 21 '21 06:07 dingyu-tourcandy

Split calculates the ZScore based on the participant count and conversion rate. Only displays it when confidence is >= 90%

So I think the confidence level is still too low(<90%). If the conversion rate(completed count) increases a bit, it should work. For example, if the conversion rate on the second alternative hits 6.2% it should display 90% confidence (if my quick math here is not wrong)

https://github.com/splitrb/split/blob/main/lib/split/dashboard/helpers.rb#L29-L44

Jul 21 '21 10:07 andrehjr

@andrehjr Thanks for the explanation. But the conversion rate is still a bit far from 6.2% and probably never will be. Is there any other way to have a reasonable confidence level?

Jul 26 '21 05:07 dingyu-tourcandy