SNOW-669650: Fix `write_pandas` atomicity when `overwrite=True`
Please answer these questions before submitting your pull requests. Thanks!
-
What GitHub issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
Fixes #1262
-
Fill out the following pre-review checklist:
- [x] I am adding a new automated test(s) to verify correctness of my new code
- [x] I am adding new logging messages
- [ ] I am adding a new telemetry message
- [ ] I am modifying authorization mechanisms
- [ ] I am adding new credentials
- [ ] I am modifying OCSP code
- [ ] I am adding a new dependency
-
Please describe how your code solves the related issue.
The original approach dropped/truncated the original table directly even if the underlying copy into query fails, which is not atomic. Now in this PR we create a table with random name first if
overwrite=True. Then if the query succeeds, we drop the original table and rename this table back to the original table.I refactored some code in
write_pandasto make my change simpler, which are necessary in my opinion. I listed them in the comment
Codecov Report
Merging #1291 (2ccb6c2) into main (32b459e) will increase coverage by
0.12%. The diff coverage is100.00%.
@@ Coverage Diff @@
## main #1291 +/- ##
==========================================
+ Coverage 81.49% 81.62% +0.12%
==========================================
Files 59 59
Lines 8562 8555 -7
Branches 1275 1271 -4
==========================================
+ Hits 6978 6983 +5
+ Misses 1277 1266 -11
+ Partials 307 306 -1
| Impacted Files | Coverage Δ | |
|---|---|---|
| src/snowflake/connector/pandas_tools.py | 88.67% <100.00%> (+6.37%) |
:arrow_up: |
| src/snowflake/connector/ocsp_snowflake.py | 71.99% <0.00%> (+0.32%) |
:arrow_up: |
| src/snowflake/connector/errors.py | 90.32% <0.00%> (+0.53%) |
:arrow_up: |
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more