gobblin icon indicating copy to clipboard operation
gobblin copied to clipboard

[GOBBLIN-1714] Prevent silent failure of data copy if files don't exist in source

Open AndyJiang99 opened this issue 3 years ago • 1 comments

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

  • [ ] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
    • https://issues.apache.org/jira/browse/GOBBLIN-1714

Description

  • [ ] Here are some details about my PR, including screenshots (if applicable):
  • [ ]
  • [ ] Throws exception if no files found in source for data copy
  • [ ] Doesn't throw exception if no files found in target for data copy as the files will be created

Tests

  • [ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:
  • [ ] All current tests pass

Commits

  • [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

AndyJiang99 avatar Sep 23 '22 04:09 AndyJiang99

Codecov Report

Merging #3568 (71161bf) into master (78484e2) will decrease coverage by 0.14%. The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master    #3568      +/-   ##
============================================
- Coverage     46.93%   46.78%   -0.15%     
+ Complexity    10580    10567      -13     
============================================
  Files          2105     2111       +6     
  Lines         82149    82468     +319     
  Branches       9147     9174      +27     
============================================
+ Hits          38558    38586      +28     
- Misses        40050    40323     +273     
- Partials       3541     3559      +18     
Impacted Files Coverage Δ
...data/management/copy/RecursiveCopyableDataset.java 86.36% <0.00%> (ø)
...blin/data/management/copy/RecursivePathFinder.java 0.00% <0.00%> (-92.86%) :arrow_down:
...data/management/copy/extractor/EmptyExtractor.java 0.00% <0.00%> (-83.34%) :arrow_down:
...a/management/copy/hive/HiveLocationDescriptor.java 36.17% <0.00%> (-14.90%) :arrow_down:
...anagement/copy/hive/UnpartitionedTableFileSet.java 66.66% <0.00%> (-12.97%) :arrow_down:
...ata/management/copy/hive/HiveCopyEntityHelper.java 51.41% <0.00%> (-11.59%) :arrow_down:
...ache/gobblin/util/commit/DeleteFileCommitStep.java 34.04% <0.00%> (-8.52%) :arrow_down:
...a/management/copy/publisher/CopyDataPublisher.java 67.54% <0.00%> (-7.95%) :arrow_down:
.../hive/filter/LookbackPartitionFilterGenerator.java 78.94% <0.00%> (-5.27%) :arrow_down:
...ervice/modules/core/GobblinServiceGuiceModule.java 84.74% <0.00%> (-5.26%) :arrow_down:
... and 24 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

codecov-commenter avatar Sep 23 '22 04:09 codecov-commenter