LAION-AI
Require better Transaction Handling
Retry on transactions must be only in the scope of Outer Commits Flush commands that raise SerializationFailure must let it pass for the outer Commit to handle retry Changes required in the decorators in database_utils.py
IMO it would also be good to detect nesting of decorated functions calls and not retry in nested inner scopes.
@melvinebenezer @andreaskoepf
Thanks for bringing this issue to our attention. I understand that you are facing some issues with the transaction handling in the database_utils.py file. From the information provided, it sounds like there is a need for retry logic to be implemented only within the scope of outer commits and that flush commands that raise a SerializationFailure should be allowed to pass for the outer commit to handle retry.
I propose that we implement the following solution to address this issue:
- Add a check to detect nesting of decorated function calls and prevent retry logic from being applied in nested inner scopes.
- Implement a retry mechanism within the scope of outer commits, specifically for cases where a SerializationFailure is raised.
- Update the decorators in the database_utils.py file to incorporate the above changes.
- Additionally, I would like to suggest that we also conduct some testing to ensure that these changes do not have any unintended consequences on other parts of the codebase.
Please let me know your thoughts on this proposal and if there are any additional details or concerns that I may have missed. I would also like to hear any additional suggestions or recommendations that you may have.
