datachain icon indicating copy to clipboard operation
datachain copied to clipboard

Add a way to persist dataset even if exception is thrown

Open amritghimire opened this issue 1 year ago • 2 comments

Refactoring upon the changes at #494 from the comments, this introduces the way to allow a way to persist dataset even if exception is thrown. With this change, the cleanup is now done at the active context. So, we can use the session context as checkpoints now. Example implementation is as:

https://github.com/iterative/datachain/blob/029e677e596c84bb8c2ecb56603d164adde03fe8/tests/scripts/feature_class_exception.py#L1-L34

In this file, local_test_datachain should only persist since it is the only dataset where the exception doesn't occur in current context.

Closes #500

amritghimire avatar Oct 09 '24 17:10 amritghimire

Deploying datachain-documentation with  Cloudflare Pages  Cloudflare Pages

Latest commit: 501ad01
Status: ✅  Deploy successful!
Preview URL: https://87fed5ff.datachain-documentation.pages.dev
Branch Preview URL: https://amrit-persist-error.datachain-documentation.pages.dev

View logs

Codecov Report

Attention: Patch coverage is 95.83333% with 1 line in your changes missing coverage. Please review.

Project coverage is 87.11%. Comparing base (e0c654e) to head (501ad01). Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
src/datachain/query/session.py 95.45% 0 Missing and 1 partial :warning:
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #504   +/-   ##
=======================================
  Coverage   87.10%   87.11%           
=======================================
  Files          92       92           
  Lines        9852     9832   -20     
  Branches     1350     1349    -1     
=======================================
- Hits         8582     8565   -17     
  Misses        917      917           
+ Partials      353      350    -3     
Flag Coverage Δ
datachain 87.08% <95.83%> (+<0.01%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Oct 09 '24 17:10 codecov[bot]

Approving to move forward, but this design Session -> Catalog -> Sessions looks a bit weird to me.

Thanks. Modified the chain logic.

amritghimire avatar Oct 15 '24 03:10 amritghimire