Update gen from multiple nodes to support generating multiple arms per node

[mgarrard]

Summary: This diff adds the functionality to generate multiple generator runs per node in a trial that uses multiple nodes.

In follow up diffs we will:

1. Support fixed features -- there is ongoing discussion about fixed features on lena-kashtelyan's recent diff, and we may be able to leverage the solution there/help build it out
2. merge this gen method with our standard gen method

Reviewed By: lena-kashtelyan

Differential Revision: D57398578

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D57398578

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 95.26%. Comparing base (d1164d7) to head (c057d91).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2469   +/-   ##
=======================================
  Coverage   95.26%   95.26%           
=======================================
  Files         485      485           
  Lines       47384    47424   +40     
=======================================
+ Hits        45138    45178   +40     
  Misses       2246     2246           

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D57398578

[facebook-github-bot]

This pull request has been merged in facebook/Ax@db26b23a91323c3ffbf4430397d422fbb1ca7fc7.