facebook
Speed up `get_pending_observation_features` by caching (on the experiment) which trials have received observations.
Summary: "Speed up get_pending_observation_features by caching (on the experiment) which trials have received observations."
"We would cache points that are non-pending, as forever non-pending, since we're not going to "lose" them from the data so they don't need to be checked again once they are there"
Change
This diff adds simple caching for pending metric names in the add_arm_to_pending_features method.
- For trials with status
is_deployed, non-pending metrics (i.e metrics with values in the output of trial.lookup_data) are cached inexperiment.not_pending_cache - When add_arm_to_pending_features is next called, if a trial has status
is_deployed, and all of the experiment's metrics are present in the not_pending_cache, evaluation of that trial is skipped - Additionally, for trials which are not
is_deployed, "lookup_data" is skipped entirely, as its outputs are not used.
Differential Revision: D54218989
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 94.84%. Comparing base (
a4bf3f1) to head (c50624c).
:exclamation: Current head c50624c differs from pull request most recent head fc6817d. Consider uploading reports for the commit fc6817d to get more accurate results
Additional details and impacted files
@@ Coverage Diff @@
## main #2225 +/- ##
=======================================
Coverage 94.83% 94.84%
=======================================
Files 467 467
Lines 46329 46357 +28
=======================================
+ Hits 43936 43965 +29
+ Misses 2393 2392 -1
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
