feat: add environments values inheritance

[Vince-Chenal]

Refers to this discussion: #726

Summary

Allow inheriting values defined in previous values items. There were some concerns about the risk of introducing a breaking change with this feature, that's why we added an environment variable feature flag (HELMFILE_LAYERED_ENV_VALUES, disabled by default) in order to isolate this behaviour and avoid breaking changes.

Default behaviour stays the same with environment values

Example

environments:
  default:
    values:
        - values1.yaml
        - values2.yaml.gotmpl <-- variables from values1.yaml are available in there

We added a dedicated test for EnvironmentValuesLoader.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[hileef]

Nice ! 💪 @mumoshu , with the additional layeredValues flag to guard against these changes being breaking, do you think we could potentially include this before moving to a major version ? 🙂

I like this implementation as it looks like a fairly simple way to reduce boilerplate while avoiding exec & loadYAML machinery 🙏

[mumoshu]

@Vince-Chenal @hileef Hey! First of all, this looks awesome! I'm definitely looking forward to merging this with only a few changes described below. Can you confirm?

---

My motivation for doing that is to make it work in concert with environmentTemplates and inherit feature proposed and discussed in https://github.com/helmfile/helmfile/pull/840#issuecomment-1528881556. Could you two read that thread, and my combined proposal below, and mind giving me feedbacks if any? 🙏

So what I'd like to do before merging this pull request is to change layedValues from a flag to another array of values.yaml files that can co-exist with the existing values so that it plays nicely with the environmentTemplates.

environmentTemplates:
  base:
    values:
    - common-static-values.yaml
    layeredValues:
    - common-dynamic-values-1.yaml
    - common-dynamic-values-2.yaml

environments:
  test:
    inherit:
      template: base
    values:
    - adhoc-static-values.yaml
    layeredValues:
    - adhoc-dynamic-values-1.yaml
    - adhoc-dynamic-values-2.yaml

This will result in the test environment to be:

environments:
  test:
    values:
    # (1) sees empty Values when rendering itself
    - common-static-values.yaml
    # (2) sees empty Values when rendering itself
    - adhoc-static-values.yaml
    layeredValues:
    # (3) sees (1)+(2)
    - common-dynamic-values-1.yaml
    # (4) sees (1)+(2)+(3)
    - common-dynamic-values-2.yaml
    # (5) sees (1)+(2)+(3)+(4)
    - adhoc-dynamic-values-1.yaml
    # (6) sees (1)+(2)+(3)+(4)+(5)
    - adhoc-dynamic-values-2.yaml

This way, we can inject values "before the common layedValues derived from the template, but after all static values" via adhoc-static-values.yaml. In other words, you can now pass inputs to the template-declared adhoc-dynamic-values-*.yaml files.

Not only each layedValues entry can see the latest merged values, the first layeredValues entry should see the values from common-static-values.yaml and adhoc-static-values.yaml for the input passing use-case I've described in the previous paragraph.

All in all, this design would give you all the benefits of environment templates along with layered values. Maximum reusability and DRY. WDYT?

[hileef]

Hello @mumoshu , thanks for the feedback !

I've had a look at #840 and your summary proposal here, IMHO I think it's great 💪 I like that we can maintain the requested features here in a terse way, while opening the door for supporting even more use cases ;

I'll open a PR targetting this one shortly to add the change you requested, I'm pretty sure that @Vince-Chenal will be ok with approving it 🙂 Once done, we can probably re-update this PR and request your review for merge once again 🙏

[hileef]

Hello again @mumoshu , it took me a while to have time to spend on this topic, but we've now updated this PR to propose an implementation for the design as proposed in your above comment 🙂

➡️ could you take a look and let us know if these updated changes fits your expectation ? cc. @Vince-Chenal keeping you in the loop

[hileef]

Hello @mumoshu , @yxxhero 👋 We've updated this PR to resolve the remaining conflicts against the main branch just now 🙏

It seems that you haven't had a chance to take a look at its contents yet, could you let us know if there are things we can do in the meantime to help move this forward ? 🙂

[yxxhero]

@mumoshu ping

[hileef]

@mumoshu anything we can do on our side to help move this forward ? 🙂

[stucki-stuck]

Hello, We have exactly the same needs in our company. Any updates? @mumoshu or @yxxhero

[yxxhero]

@mumoshu WDYT?