TIBCOSoftware
Model lifecycle management
Current behavior: When a Flogo app leverages the inference activity, the specified model cannot be changed without a restart of the process. The model must also be physically placed on the device, cloud storage, etc prior to the app being executed.
Expected behavior: Explore options to enable a more robust delivery of models and over the air update or models without requiring the engine to be restarted. Also consider support for champion/challenger model bakeoffs.
What is the motivation / use case for changing the behavior?
Enhance the usability in production scenarios
- where it is either not possible or very challenging to gain access to the device, storage location, etc to update a model
- where downtime is not an option to deliver new models
Additional information you deem important (e.g. I need this tomorrow):
Can I ask what you mean by "champion/challenger model bakeoffs"? Also, we can check timestamps of models to see if we need to reload? so we don't have to shutdown the engine.
@abramvandergeest thinking A/B testing. Basically load two models, split the inference load between models and eval the performance of each.
Also, kinda thinking we need some form of svc to push updated models to the running engine... I suppose we could also look at dropping models in a dir and the activity could just load the latest timestamp...
So I have been thinking of how to approach this. My thinking is that we can create an A/B testing activity that passes models/features/etc to the next activity (i.e. the inference activity). This activity would be built into the app in general with a config file set to only A, then we can add A/B options to the config file. I would suggest if we do this going with a pre-existing open source A/B testing framework. I don't know any off the top of my head but I would be surprised if one doesn't exist. Then we either use the svc or check dates to load the latest time stamp to actually change a model. If we do this in the context of the A/B test we would just change which output is used more. If that makes sense.
A google search provides this quora question on opensource A/B testing frameworks: https://www.quora.com/Which-are-the-best-open-source-A-B-testing-framework-to-use-in-your-projects