operatorai
Archives being overwritten if generated too quickly
I originally put this issue into the discussion area, but thought it might be a better fit as an issue.
I am migrating an existing database that is storing models to the modelstore and saw that if I added the models from my database to the modelstore too quickly (within the same second), the versions directory gets updated correctly BUT the timestamped archives get overridden.
I can resolve this for my use case by putting in a 1 second sleep in my import code to make sure there are no collisions, but the modelstore should detect that models might be overridden and fix it by checking for an existing timestamped versioned entry with the same timestamp and modifying the saved timestamp so that nothing gets overridden.
Since the timestamps are not in UTC time, I imagine that a directory containing the model data being accessed from another server in a different time zone may have the same issue with timestamp collisions.
I guess to resolve this the modelstore would need to :
- Timestamp in UTC time so that all timestamps would only increase over time (so no possibility of TS collision)
- Detect an existing archive with the same (within the same second) timestamp and pause or modify the generated timestamp to avoid a collision
Not sure if the new release will resolve this, please let me know if I can help or if there is some configuration setting that would resolve this.
Hello! Thanks for raising this. I had run into the problem myself when running integration tests, but had worked around it with a time.sleep() line - which is a hack. I agree that this should be fixed in the library itself 👍 and will look into it soon
@nlathia another solution to this may be to add a lock to the model store database.
You could potentially follow how terraform does this with an external dynamodb like table call that gets and sets a lock when someone accesses the model store. Or simply creating/deleting a lock file prior to writing to the db.
@shsnyder @cdknorow I've opened the PR linked above with a fix for this issue. Feedback welcome! Will aim to merge this today and then ship this with the next release of modelstore.
Thanks again for raising this!
This has been released as part of modelstore==0.0.77, thanks again for raising!
- https://github.com/operatorai/modelstore/pull/216