[GOBBLIN-1398] load system specific hive config to support multiple h…

[jhsenjaliya]

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

• [x] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
  • https://issues.apache.org/jira/browse/GOBBLIN-1398

Description

• [x] Here are some details about my PR, including screenshots (if applicable):

GOBBLIN-1308 takes care of ability to connect to remote and secure hive metastore, but it still requires the management of hive-site to be provided to be container local file. 

When it comes to multiple hive clusters this manual management approach would not work, and requires special feature of providing system specific hive-site.xml without namespace collision.

This ticket aims to do following things

1. define a way to provide remote system configuration ( keep giving flat config is more cumbersome )
2. based on system config and feature flag, copy config files to container local path automatically.
3. when creating metastoreClient, pick up the right config for the requested system ( identified by the metastore URI )

 

Tests

• [x] My PR adds the following unit tests OR does not need testing for this extremely good reason: Needs to add unit test once reviewed.

Commits

• [x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

[codecov-io]

Codecov Report

Merging #3236 (6b2a3ee) into master (4fb83cc) will decrease coverage by 37.35%. The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master   #3236       +/-   ##
============================================
- Coverage     46.37%   9.02%   -37.36%     
+ Complexity     9935    1738     -8197     
============================================
  Files          2030    2030               
  Lines         78744   78849      +105     
  Branches       8759    8783       +24     
============================================
- Hits          36520    7117    -29403     
- Misses        38822   71036    +32214     
+ Partials       3402     696     -2706     

Impacted Files	Coverage Δ	Complexity Δ
.../java/org/apache/gobblin/hive/HiveConfFactory.java	0.00% <0.00%> (-73.34%)	0.00 <0.00> (-7.00)
...main/java/org/apache/gobblin/util/ConfigUtils.java	13.00% <0.00%> (-45.86%)	11.00 <0.00> (-30.00)
...rg/apache/gobblin/yarn/GobblinYarnAppLauncher.java	0.00% <0.00%> (-24.32%)	0.00 <0.00> (-12.00)
...che/gobblin/yarn/GobblinYarnConfigurationKeys.java	0.00% <ø> (-66.67%)	0.00 <0.00> (-1.00)
...c/main/java/org/apache/gobblin/util/FileUtils.java	0.00% <0.00%> (-100.00%)	0.00% <0.00%> (-4.00%)
...n/java/org/apache/gobblin/fork/CopyableSchema.java	0.00% <0.00%> (-100.00%)	0.00% <0.00%> (-2.00%)
...java/org/apache/gobblin/stream/ControlMessage.java	0.00% <0.00%> (-100.00%)	0.00% <0.00%> (-1.00%)
...va/org/apache/gobblin/dataset/DatasetResolver.java	0.00% <0.00%> (-100.00%)	0.00% <0.00%> (-3.00%)
...va/org/apache/gobblin/converter/EmptyIterable.java	0.00% <0.00%> (-100.00%)	0.00% <0.00%> (-1.00%)
...org/apache/gobblin/ack/BasicAckableForTesting.java	0.00% <0.00%> (-100.00%)	0.00% <0.00%> (-3.00%)
... and 1073 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 4fb83cc...6b2a3ee. Read the comment docs.

[autumnust]

Hi Jay, thanks for this patch. Starting to look at it. Can you also update the your PR description for more context of what you changed as well as replace all Gobblin-XXX with the actual ticket number ? Thanks.

[jhsenjaliya]

@autumnust , addressed all comments, can u pls take a look? about the testcase, we dont have much structure in place for it, but I will take a look at GobblinYarnAppLauncherTest and see if we can add it. Thanks

[codecov-commenter]

Codecov Report

Merging #3236 (1449172) into master (dd4c08d) will decrease coverage by 0.07%. The diff coverage is 3.77%.

@@             Coverage Diff              @@
##             master    #3236      +/-   ##
============================================
- Coverage     46.48%   46.40%   -0.08%     
+ Complexity    10021    10020       -1     
============================================
  Files          2037     2037              
  Lines         79292    79391      +99     
  Branches       8840     8866      +26     
============================================
- Hits          36855    36845      -10     
- Misses        39018    39123     +105     
- Partials       3419     3423       +4     

Impacted Files	Coverage Δ
...main/java/org/apache/gobblin/util/ConfigUtils.java	40.96% <0.00%> (-17.90%)	:arrow_down:
...rg/apache/gobblin/yarn/GobblinYarnAppLauncher.java	23.37% <0.00%> (-0.94%)	:arrow_down:
...che/gobblin/yarn/GobblinYarnConfigurationKeys.java	66.66% <ø> (ø)
.../java/org/apache/gobblin/hive/HiveConfFactory.java	57.69% <36.36%> (-15.65%)	:arrow_down:
...he/gobblin/source/PartitionAwareFileRetriever.java	48.14% <0.00%> (-7.41%)	:arrow_down:
...g/apache/gobblin/hive/orc/HiveOrcSerDeManager.java	60.46% <0.00%> (-3.49%)	:arrow_down:
...in/java/org/apache/gobblin/cluster/HelixUtils.java	34.71% <0.00%> (-3.31%)	:arrow_down:
...lin/restli/throttling/ZookeeperLeaderElection.java	70.00% <0.00%> (-2.23%)	:arrow_down:
...a/management/copy/publisher/CopyDataPublisher.java	74.17% <0.00%> (-1.33%)	:arrow_down:
...pache/gobblin/runtime/GobblinMultiTaskAttempt.java	62.30% <0.00%> (-0.40%)	:arrow_down:
... and 3 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update dd4c08d...1449172. Read the comment docs.