gobblin
gobblin copied to clipboard
[GOBBLIN-1642] add debug logs in AsynchronousFork
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA
- [x] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1642
Description
- [x] Here are some details about my PR, including screenshots (if applicable): Input records are passed from Task to Fork via BoundedBlockingRecordQueue. APIs in this class has timeouts. I believe sometimes record may not be obtained from this queue and the request may time out. Adding some debug level logs to investigate when this happens.
Tests
- [x] My PR adds the following unit tests OR does not need testing for this extremely good reason: trivial changes
Commits
- [x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
- Subject is separated from body by a blank line
- Subject is limited to 50 characters
- Subject does not end with a period
- Subject uses the imperative mood ("add", not "adding")
- Body wraps at 72 characters
- Body explains "what" and "why", not "how"
Codecov Report
Merging #3503 (c1f4895) into master (3a49796) will increase coverage by
0.01%. The diff coverage is83.33%.
@@ Coverage Diff @@
## master #3503 +/- ##
============================================
+ Coverage 46.68% 46.69% +0.01%
- Complexity 10405 10412 +7
============================================
Files 2078 2079 +1
Lines 81288 81335 +47
Branches 9078 9084 +6
============================================
+ Hits 37947 37982 +35
- Misses 39843 39849 +6
- Partials 3498 3504 +6
| Impacted Files | Coverage Δ | |
|---|---|---|
| .../apache/gobblin/runtime/fork/AsynchronousFork.java | 89.65% <83.33%> (-2.02%) |
:arrow_down: |
| ...a/org/apache/gobblin/util/limiter/NoopLimiter.java | 40.00% <0.00%> (-20.00%) |
:arrow_down: |
| ...lin/util/filesystem/FileSystemInstrumentation.java | 92.85% <0.00%> (-7.15%) |
:arrow_down: |
| .../java/org/apache/gobblin/cluster/SleepingTask.java | 39.39% <0.00%> (-6.07%) |
:arrow_down: |
| ...lin/restli/throttling/ZookeeperLeaderElection.java | 70.00% <0.00%> (-2.23%) |
:arrow_down: |
| ...a/org/apache/gobblin/cluster/GobblinHelixTask.java | 60.21% <0.00%> (-2.16%) |
:arrow_down: |
| ...pache/gobblin/iceberg/writer/GobblinMCEWriter.java | 76.10% <0.00%> (-1.24%) |
:arrow_down: |
| ...main/java/org/apache/gobblin/util/HadoopUtils.java | 29.54% <0.00%> (-0.33%) |
:arrow_down: |
| ...bblin/iceberg/writer/GobblinMetadataException.java | 100.00% <0.00%> (ø) |
|
| ...apache/gobblin/hive/writer/HiveMetadataWriter.java | 0.00% <0.00%> (ø) |
|
| ... and 4 more |
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact),ø = not affected,? = missing dataPowered by Codecov. Last update 3a49796...c1f4895. Read the comment docs.
I found a way to overcome if timing out is the issue. One can increase fork.record.queue.timeout. If that does not help, will ask for review on this PR. Thanks!