How to prepare experiment and data parameters for BOTORCH_MODULAR?

[ayzwu]

Hi, I'm trying to use the module below to run Bayesian Optimization, based on my A/B testing data, which has train_x, train_y, and train_yvar. However, I couldn't find in the tutorial how to specify the first two parameters (experiment & data). Does anyone have any insights on this? Thanks!

model_bridge_with_GPEI = Models.BOTORCH_MODULAR( experiment=experiment, data=data, surrogate=Surrogate(FixedNoiseGP), # Optional, will use default if unspecified botorch_acqf_class=qNoisyExpectedImprovement, # Optional, will use default if unspecified )

[lena-kashtelyan]

Hi @ayzwu, is there a reason why you are looking to use Models.BOTORCH_MODULAR directly (pretty low-level Ax construct) instead of using our user-friendly Service API (see tutorial), which will still choose GPEI under the hood (or it might choose other versions of Bayesian optimization depending on your search space).

If there is no specific reason, it might be easiest to just follow the tutorial I linked above and use the Service API. If there is a specific reason, let us know what it is! It will still be most likely easiest to use the Service API instead of manually constructing experiment and data; you might just need to specify a custom generation strategy (tutorial) if you need to configure the model manually in some way.

[ayzwu]

Hi Kashtelyan,

Thanks for getting back to me!

My goal is to optimize parameter for A/B experiments. Since the metric is noisy, I need to use fixed noise GP with qNEI to get multiple candidate points to use for my next round of experiment. I would also need to consider multi-objective scenario to optimize two metrics simultaneously.

I have my experiment data available. What I don’t know is how to construct it in a way to call the API. That’s why I was trying BoTorch to assemble modules together. If there’s an easier way, please let me know! Thanks very much!

Ayzwu

On Sep 13, 2022, at 8:59 AM, Lena Kashtelyan @.***> wrote:

 Hi @ayzwu, is there a reason why you are looking to use Models.BOTORCH_MODULAR directly (pretty low-level Ax construct) instead of using our user-friendly Service API (see tutorial), which will still choose GPEI under the hood (or it might choose other versions of Bayesian optimization depending on your search space).

If there is no specific reason, it might be easiest to just follow the tutorial I linked above and use the Service API. If there is a specific reason, let us know what it is! It will still be most likely easiest to use the Service API instead of manually constructing experiment and data; you might just need to specify a custom generation strategy (tutorial) if you need to configure the model manually in some way.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[lena-kashtelyan]

Hi @ayzwu, just realized that you are probably looking to use BatchTrial-s for your A/B test, so you can't use the Service API. There is some information about Trial vs. BatchTrial here: https://ax.dev/docs/core.html#trial-vs-batch-trial. I can help you pick the correct setup if you tell me more about your experiment. For example, is it an online experiment (i.e. are you testing responses of users to treatments)? What are the metrics you are optimizing for? How many points will you evaluate?

My original response below won't be applicable if you need to use BatchTrial-s, so feel free to disregard it in that case. This tutorial for the Developer API will be a good place to start instead: https://ax.dev/tutorials/gpei_hartmann_developer.html; once you tell me more about your experiment, I'll be able to help more.

---

My original response:

If you use Ax Service API (see tutorial), it will automatically select the right surrogate model (so a type of the GP) and acquisition function for your problem –– you won't need to configure anything manually. Just make sure to pass the correct SEM via ax_client.complete_trial if the noise level in your problem is known (I'm assuming it is, if you wanted to use FixedNoiseGP). The example in the tutorial: https://ax.dev/tutorials/gpei_hartmann_service.html#4.-Run-optimization-loop passes 0.0 for SEM (from the output of the evaluate function) because there we are using a synthetic function, so we know it to be noiseless. You can instead specify the correct SEM for your metric(s).

If you don't know the SEM for your observations, you can just not pass any value for it or pass None, and Ax will infer the noise for you. In that case, it will select SingleTaskGP instead of FixedNoiseGP, since noise level in that case is unknown and thus not fixed.

To check which model was picked for you in AxClient, you can do the following after calling AxClient.create_experiment :

1. Check how many Sobol trials Ax will run by printing out ax_client.generation_strategy –– it should say something like "GenerationStrategy(Sobol for 5 trials, GPEI) for subsequent trials. Note that GPEI here does not mean that qEI will be used instead of qNEI –– we will only use qEI if you specifically set the noise level in your observations to 0.0; by default, GPEI will use qNEI.
2. Generate the Sobol trials via ax_client.get_next_trial and complete them with data via ax_client.complete_trial.
3. Generate the first Bayesian optimization trial via ax_client.get_next_trial.
4. Once the first Bayesian optimization trial has been generated, examine the model that was picked via ax_client.generation_strategy.model.model (it should be of type BotorchModel).
5. You can check ax_client.generation_strategy.model.model.model to see which BoTorch GP type it picked.
6. You can check ax_client.generation_strategy.model.model.acqf_constructor to see which acquisition function it will use.

---

Finally, if you really want to configure your model manually, see this part of my response above:

It will still be most likely easiest to use the Service API instead of manually constructing experiment and data; you might just need to specify a custom generation strategy (tutorial) if you need to configure the model manually in some way.

You can pass the generation strategy you construct as a constructor argument AxClient like so: ax_client = AxClient(generation_strategy=my_generation_strategy). To use Models.BOTORCH_MODULAR` in the generation strategy you can follow this piece of the modular BotAx tutorial: https://ax.dev/tutorials/modular_botax.html#5.-Utilizing-BoTorchModel-in-generation-strategies.

[HUST-Lee]

At presenr,do you know How to prepare experiment and data parameters for BOTORCH_MODULAR? please taech me! thank u;

[lena-kashtelyan]

@HUST-Lee, the easiest way will be to use Service API as suggested above. Alternatively, you can use developer API following this tutorial: https://ax.dev/tutorials/gpei_hartmann_developer.html –– this will show how to get experiment and data. You will use your generation strategy in place of any call to Models.<MODEL>, e.g. here: https://ax.dev/tutorials/gpei_hartmann_developer.html#5.-Perform-Optimization you will not use sobol.gen, but rather generation_strategy.gen.

Instructions for setting up your generation strategy are above.

Closing this as the issue has been answered; feel free to reopen if you have follow-ups (but please don't comment on a closed issue as we will likely not see the comment).

[cyrilmyself]

hi, @ayzwu, i have the same application Scenario, but i do not know how to get the train_yvar with A/B testing data, can you tell how to obtain it?