gobblin icon indicating copy to clipboard operation
gobblin copied to clipboard

[GOBBLIN-1703] avoid double quota increase for adhoc flows

Open arjun4084346 opened this issue 3 years ago • 1 comments

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

  • [X] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
    • https://issues.apache.org/jira/browse/GOBBLIN-1703

Description

  • [X] Here are some details about my PR, including screenshots (if applicable): When a gaas flow request comes, resource handler checks the quota right there. However, if the flow has runImmediately=true, the quota will be checked again when the first job starts. This should be avoided.

Tests

  • [ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

  • [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

arjun4084346 avatar Sep 07 '22 01:09 arjun4084346

Codecov Report

Merging #3550 (30049f6) into master (ae2ea19) will decrease coverage by 0.89%. The diff coverage is 46.50%.

@@             Coverage Diff              @@
##             master    #3550      +/-   ##
============================================
- Coverage     47.76%   46.86%   -0.90%     
- Complexity     8557    10617    +2060     
============================================
  Files          1705     2111     +406     
  Lines         64976    82667   +17691     
  Branches       7036     9202    +2166     
============================================
+ Hits          31035    38742    +7707     
- Misses        31240    40363    +9123     
- Partials       2701     3562     +861     
Impacted Files Coverage Δ
.../org/apache/gobblin/service/ServiceConfigKeys.java 0.00% <ø> (ø)
...gobblin/runtime/spec_store/MysqlBaseSpecStore.java 89.16% <ø> (ø)
...lin/service/modules/flow/MultiHopFlowCompiler.java 67.25% <0.00%> (-1.62%) :arrow_down:
...odules/orchestration/AbstractUserQuotaManager.java 100.00% <ø> (+22.85%) :arrow_up:
...ervice/modules/flow/BaseFlowToJobSpecCompiler.java 60.68% <12.50%> (-3.54%) :arrow_down:
...e/modules/orchestration/MysqlUserQuotaManager.java 47.34% <31.38%> (-26.53%) :arrow_down:
...blin/service/modules/orchestration/DagManager.java 82.09% <50.00%> (-0.14%) :arrow_down:
...odules/orchestration/InMemoryUserQuotaManager.java 70.07% <73.03%> (+5.97%) :arrow_up:
.../modules/scheduler/GobblinServiceJobScheduler.java 64.62% <100.00%> (ø)
...e/gobblin/service/monitoring/GitConfigMonitor.java 95.23% <0.00%> (-4.77%) :arrow_down:
... and 412 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

codecov-commenter avatar Sep 07 '22 01:09 codecov-commenter

Functionality of checking quota for every job is left unchanged that can be done in the other PR. Handling parallel flow functionality is also left unchanged for the other PR.

arjun4084346 avatar Sep 24 '22 05:09 arjun4084346