[KYUUBI #6344] FlinkProcessBuilder prioritizes user configurations

[wForget]

:mag: Description

Issue References 🔗

This pull request fixes #6344

FlinkProcessBuilder specifies yarn.ship-files, yarn.application.name and yarn.tags configurations of kyuubi platform. Sometimes we also need to customize these configurations, so we should prioritize these user configurations.

Describe Your Solution 🔧

FlinkProcessBuilder prioritizes user configurations.

Types of changes :bookmark:

• [x] Bugfix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Behavior Without This Pull Request :coffin:

Behavior With This Pull Request :tada:

Related Unit Tests

added new unit test

---

Checklist 📝

• [X] This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

[pan3793]

the patch involves user-facing changes, to un-confuse users and future exploeres, can you elaborate more on the background and behavior change in PR description?

[link3280]

@wForget I have the same question with @pan3793 . Why do users need to override the yarn tags for flink applications and only flink applications? IIUC, the tags are mainly for internal usage which users rarely care about.

[wForget]

@wForget I have the same question with @pan3793 . Why do users need to override the yarn tags for flink applications and only flink applications? IIUC, the tags are mainly for internal usage which users rarely care about.

kyuubi may not directly face users, but be connected to the upper application platform. Just like kyuubi adds session id in yarn tags, we also hope to be able to add the upper application platform id into yarn tags, as well as the application name.

[wForget]

the patch involves user-facing changes, to un-confuse users and future exploeres, can you elaborate more on the background and behavior change in PR description?

In my scenario this is a bug fix, just before this change if we specified flink.yarn.ship-files=customShipFile it would override the flinkExtraJars specified in kyuubi, I think we should merge them.

[wForget]

the patch involves user-facing changes, to un-confuse users and future exploeres, can you elaborate more on the background and behavior change in PR description?

In my scenario this is a bug fix, just before this change if we specified flink.yarn.ship-files=customShipFile it would override the flinkExtraJars specified in kyuubi, I think we should merge them.

Submitted an issue #6344 and recorded error details.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 84.61538% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 58.43%. Comparing base (5cbbdc3) to head (feca972). Report is 1 commits behind head on master.

Files	Patch %	Lines
...ache/kyuubi/engine/flink/FlinkProcessBuilder.scala	77.77%	0 Missing and 2 partials :warning:

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #6342      +/-   ##
============================================
- Coverage     58.45%   58.43%   -0.02%     
  Complexity       24       24              
============================================
  Files           653      653              
  Lines         39865    39889      +24     
  Branches       5481     5482       +1     
============================================
+ Hits          23303    23310       +7     
- Misses        14073    14075       +2     
- Partials       2489     2504      +15     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[wForget]

Thanks, merged to master and branch-1.9.