splitrb
Be able to call "finish" multiple times without resetting the test group
I would like to bucket users into two groups, and then measure the number of times each group performs a certain action.
Let's say group A has two users and group B has two users. Those users in group A perform the action 100 times in total. The users in group B perform the action 300 times in total.
This is what I want to track. Is there a reasonable way to do that with Split currently?
If I use reset: true, I can track multiple finishes per user, but the group choice is also reset each time, which I don't want. Once you're in group A, I want you to stay in group A.
If I don't use reset: true, they stay in their group, but only one finish is tracked per user.
I tried a hack where I store their choice in a cookie and use weights for the alternatives so they keep an alternative after the reset, but this didn't quite do what I want either: while it does preserve the group and track multiple finishes, it also increases the participant count each time. So if I perform the action 100 times, I will be tracked as 100 participants.
I think what we will do for now is to keep counters in Redis and look at both Split and these counters. Could be a nice use case for Split to support, though.
I think i also need what you're requesting, but I don't totally understand what you're asking. Could you let me know if this example is what you're looking for?
Facebook wants to see what color like button (red or blue) generates the most likes, so they would call finish every time a user likes, but they don't want the color to change for every post in their feed.
@Un3qual Yup! That sounds like the use case we had as well.
Our specific use case was changing button styling and wanting to see which style got the highest number of clicks across the site.
We need this thing to and I'm willing to discuss/implement it, but I want some clarity on the approach before wasting time. @andrew will you receive/accept a PR implementing this?
@henrik do you have anything in mind about how this feature should be exposed? New helper or new flag:
-
ab_finished(:new_user_free_points, keep_experimenting: true)(although I'm not sure how that would help with incrementing participants count) -
ab_reentrant_test(:name, 1,2,3)
@YurySolovyov Nothing in mind, sorry – haven't thought about this for a while! It's brilliant that you're working on this.
@henrik I kinda have a solution internally that introduces 2 new methods:
ab_reentrant_test
ab_complete_reentrant
with mostly the same signatures. I think I can get rid of ab_complete_reentrant and just go with
ab_complete_reentrant(:expetiment, reentrant: true)
I also have the same requirements as addressed in the issue. In addition, would love the functionality of explicitly setting the current test for a user as incomplete before starting a new one, but hopefully having some type of reentrant functionality will cover this need as well.
Willing to help out. For those that might have created a solution, is there any public facing repo for such code?
Ok, for anyone who wants to use it or even turn it into PR, here it is: https://gist.github.com/YurySolovyov/4f2a7af96b3ad4f22461133f7395e8cd
We placed it into rails initializers dir, so it was easy to remove, it monkey-patches some of the Split's methods, but at least it is self-contained.