mozilla
Create an A/B test for showing non-review warning for non-Firefox clients
If you visit any listing page on AMO using a non-Firefox client, you get a Download Firefox button instead of the usual Install button. Currently, Recommended add-ons benefit from a feature called Return to AMO that allows that Download flow to facilitate users to get the extension right after they install Firefox. We plan to expand this feature to work for all listed add-ons, and that brings up the question of what to do for add-ons that are not Recommended.
For Firefox clients we show a warning message on all listing pages of add-ons that are not Recommended (or fall into other special categories like By Firefox). However, we don't show this message for non-Firefox clients, so a user could do the whole Return to AMO flow without being notified like Firefox users are.
We should consider showing the same warning for non-Firefox clients. Since we're currently A/B testing the messaging for non-Firefox clients, we need to hold off on making any changes to this. We'll probably want to make some A/B testing for showing the warning, to make sure we're not significantly regressing conversion of non-Firefox users.
An alternative to this would be to show the warning in Firefox, the moment the user is going to install the add-on. This shouldn't have a big impact on conversion, but it wouldn't be a great experience if the same information wasn't available when browsing the site and choosing an add-on.
┆Issue is synchronized with this Jira Task
An alternative to this would be to show the warning in Firefox, the moment the user is going to install the add-on. This shouldn't have a big impact on conversion, but it wouldn't be a great experience if the same information wasn't available when browsing the site and choosing an add-on.
While I agree that ideally the user would see something similar when browsing AMO and in-client as part of the RTAMO workflow, the point I raised in the meeting was that displaying the warning for non-Firefox clients could discourage people from downloading Firefox at all, which I think we want to avoid. I think we'd be better off showing the warning only on the client as part of the RTAMO flow. It's true that a non-Firefox user wouldn't see the warning when browsing on AMO, but it's not exactly relevant to them, as they cannot install the add-on. Once they have downloaded Firefox, if they look at the listing on AMO (with Firefox) they would see the warning.
The counterargument to this is that if a user is downloading Firefox on an add-on page, it's likely that they are interested in getting the add-on and want to install Firefox because of the add-on. Showing the warning at the very last step of this process can feel like a bait-and-switch IMO, and could impact the activation rate for those users.
There are a couple of questions to answer here: (1) what's best for the user's interests, and (2) which solution leads to more active profiles. The first one is a UX problem (keeping users safe vs. being overly alarmist). The second one can be answered through experimentation.
Yes, I think we should A/B test showing the warning on listing pages for non-Firefox clients. The download funnel experiment is due to launch in a few weeks, so this could be the next thing we test. I suppose we could also run it before the funnel experiment, but we would need to set it up very quickly.
Yes, I think we should A/B test showing the warning on listing pages for non-Firefox clients. The download funnel experiment is due to launch in a few weeks, so this could be the next thing we test. I suppose we could also run it before the funnel experiment, but we would need to set it up very quickly.
It probably wouldn't be difficult to get it set up quickly, but it might need to include the fixes being discussed at https://github.com/mozilla/addons/issues/14196, and that would slow it down. Also, we should define exactly what we want to measure here. I can see it being any combination of:
- users who click the button to be directed to mozilla.org (AMO conversions)
- users who download Firefox from mozilla.org (mozilla.org conversions)
- users who install Firefox (installs based on telemetry data)
but it might need to include the fixes being discussed at mozilla/addons#14196, and that would slow it down
Yeah, I wouldn't want this to delay the funnel experiment. If we can do it quickly, great, otherwise it can wait.
Also, we should define exactly what we want to measure here
I think those 3 metrics would continue to be useful, but most likely the only perceived impact, if any, would be on the first one.
