snowflake-connector-python icon indicating copy to clipboard operation
snowflake-connector-python copied to clipboard

SNOW-669650: Fix `write_pandas` atomicity when `overwrite=True`

Open sfc-gh-jdu opened this issue 3 years ago • 1 comments

Please answer these questions before submitting your pull requests. Thanks!

  1. What GitHub issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

    Fixes #1262

  2. Fill out the following pre-review checklist:

    • [x] I am adding a new automated test(s) to verify correctness of my new code
    • [x] I am adding new logging messages
    • [ ] I am adding a new telemetry message
    • [ ] I am modifying authorization mechanisms
    • [ ] I am adding new credentials
    • [ ] I am modifying OCSP code
    • [ ] I am adding a new dependency
  3. Please describe how your code solves the related issue.

    The original approach dropped/truncated the original table directly even if the underlying copy into query fails, which is not atomic. Now in this PR we create a table with random name first if overwrite=True. Then if the query succeeds, we drop the original table and rename this table back to the original table.

    I refactored some code in write_pandas to make my change simpler, which are necessary in my opinion. I listed them in the comment

sfc-gh-jdu avatar Oct 21 '22 23:10 sfc-gh-jdu

Codecov Report

Merging #1291 (2ccb6c2) into main (32b459e) will increase coverage by 0.12%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1291      +/-   ##
==========================================
+ Coverage   81.49%   81.62%   +0.12%     
==========================================
  Files          59       59              
  Lines        8562     8555       -7     
  Branches     1275     1271       -4     
==========================================
+ Hits         6978     6983       +5     
+ Misses       1277     1266      -11     
+ Partials      307      306       -1     
Impacted Files Coverage Δ
src/snowflake/connector/pandas_tools.py 88.67% <100.00%> (+6.37%) :arrow_up:
src/snowflake/connector/ocsp_snowflake.py 71.99% <0.00%> (+0.32%) :arrow_up:
src/snowflake/connector/errors.py 90.32% <0.00%> (+0.53%) :arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

codecov-commenter avatar Oct 22 '22 01:10 codecov-commenter