[KYUUBI #3695] ADAPTIVE policy to dynamic expand or reduce engine pool size

[CHzxp]

Why are the changes needed?

As described in issues #3695: Kyuubi pool-share-level current policies can not dynamic expand or decrease engin according to engin load

How was this patch tested?

• [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

• [ ] Add screenshots for manual tests if appropriate

• [x] Run test locally before make a pull request

[CHzxp]

please review code

[pan3793]

Thanks @CHzxp, I will take a look ASAP, and would you please rebase your branch on upstream master first? there is conflict now.

[CHzxp]

Thanks @pan3793, i will deal conflicks ASAP.Thank you!

[pan3793]

@CHzxp I roughly understand what you are trying to do by reading the description and code. Basically, I think it's a good idea, but the code needs to be polished before getting in, and have an offline discussion w/ @cxzl25, he has a similar idea but sounds more mature.

Anyway, it's a good start, @cxzl25 would you please share more about your idea? Then we can discuss the design details and split it into multi PRs for implementation, this one could be part of the whole plan.

[CHzxp]

@cxzl25,@pan3793 how can get connection with @cxzl25 for discussion the issue.

[CHzxp]

@pan3793 , thank you review code, i will polish my code ASAP .

[CHzxp]

please help me review my code, thanks

[codecov-commenter]

Codecov Report

Merging #3697 (cfa270e) into master (18d9885) will increase coverage by 0.93%. The diff coverage is 44.44%.

:exclamation: Current head cfa270e differs from pull request most recent head c06ff56. Consider uploading reports for the commit c06ff56 to get more accurate results

@@             Coverage Diff              @@
##             master    #3697      +/-   ##
============================================
+ Coverage     51.84%   52.77%   +0.93%     
  Complexity       13       13              
============================================
  Files           506      498       -8     
  Lines         29007    28097     -910     
  Branches       3992     3869     -123     
============================================
- Hits          15038    14829     -209     
+ Misses        12548    11868     -680     
+ Partials       1421     1400      -21     

Impacted Files	Coverage Δ
...ain/scala/org/apache/kyuubi/engine/EngineRef.scala	66.89% <31.81%> (-6.96%)	:arrow_down:
.../engine/reblance/RebalancedSparkEnginePolicy.scala	31.81% <31.81%> (ø)
...in/scala/org/apache/kyuubi/config/KyuubiConf.scala	97.47% <100.00%> (-0.02%)	:arrow_down:
...e/kyuubi/engine/reblance/HandleSpaceRecorder.scala	100.00% <100.00%> (ø)
.../org/apache/kyuubi/session/KyuubiSessionImpl.scala	77.04% <100.00%> (+0.62%)	:arrow_up:
...g/apache/kyuubi/session/KyuubiSessionManager.scala	89.74% <100.00%> (+0.85%)	:arrow_up:
...apache/kyuubi/client/api/v1/dto/SessionHandle.java	53.84% <0.00%> (-10.86%)	:arrow_down:
...he/kyuubi/plugin/spark/authz/util/AuthZUtils.scala	54.71% <0.00%> (-9.18%)	:arrow_down:
...org/apache/spark/kyuubi/SQLOperationListener.scala	74.39% <0.00%> (-8.54%)	:arrow_down:
.../kyuubi/server/mysql/constant/MySQLErrorCode.scala	13.84% <0.00%> (-6.16%)	:arrow_down:
... and 58 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[CHzxp]

@pan3793 @cxzl25 please look at my code

[pan3793]

I left some comments but TBH I don't fully understand the implementation detail, because the variables/methods name is confusing to me.

[CHzxp]