Automating generations of values.yml

[LLsq2]

BN2 creation requires a dynamically created values.yml file for helm to bring up an environment. This PR takes bootstrapped information and dynamically generates a values file with the correct NodeIDs, also allows for the env to be dynamically sized.

[SaveTheRbtz]

Any reason to not use the https://pkg.go.dev/gopkg.in/yaml.v3#Marshal ?

[sjonpaulbrown]

In general, I think we should take a different approach to the creation and generation of the values.yml file. Currently, we are creating the file in an "unstructured" manner by appending to the file itself. This is very difficult to maintain and could result in issues when modifying the data structure and inputs that need to be templated.

I would recommend that we take one of two possible approaches:

1. Model the data using structs and write the data as we do in the localnet bootstrap process
2. Standardize a given set of variables for templates and use go templating to inject them into a template that will be written

The advantage of using https://pkg.go.dev/text/template would be that we could leverage the templating engine to structure the data while injecting "standard" inputs that may or may not be used. This makes it very easy to create new templates that require no code changes, and subsequently, it allows for flexible definition of specific templates for different usecases.

As an example, if we define a "standard" set of inputs that is a combination of the bootstrap info and variables injected from the CLI, we can create different templates that have statically defined configuration rather than dynamically generated values. This sort of functionality would allow us to create different templates for different usecases such as BFT testing, performance testing, etc. Furthermore, it makes it incredibly flexible as we could "choose" the template by specifying a template name on the CLI.

Overall, I think either of the above approaches will result in a more flexible and maintainable solution. While some of the templating may eventually be able to go away, we may inevitably need to support the dynamic generation of different values.yml files, and it is important that the approach is flexible and maintainable.

[codecov-commenter]

Codecov Report

Merging #3305 (80c3847) into master (c2a415a) will decrease coverage by 0.04%. The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3305      +/-   ##
==========================================
- Coverage   55.37%   55.32%   -0.05%     
==========================================
  Files         763      770       +7     
  Lines       70151    70658     +507     
==========================================
+ Hits        38843    39094     +251     
- Misses      28110    28350     +240     
- Partials     3198     3214      +16     

Flag	Coverage Δ
unittests	55.32% <ø> (-0.05%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
consensus/hotstuff/pacemaker/timeout/config.go	78.75% <0.00%> (-16.34%)	:arrow_down:
module/mempool/epochs/transactions.go	90.32% <0.00%> (-9.68%)	:arrow_down:
fvm/environment/value_store.go	62.40% <0.00%> (-3.44%)	:arrow_down:
...ratelimit/internal/limiter_map/rate_limiter_map.go	76.00% <0.00%> (-2.67%)	:arrow_down:
fvm/transactionSequenceNum.go	71.15% <0.00%> (-2.54%)	:arrow_down:
fvm/state/state.go	60.73% <0.00%> (-1.96%)	:arrow_down:
module/profiler/profiler.go	71.27% <0.00%> (-1.89%)	:arrow_down:
ledger/complete/ledger.go	60.78% <0.00%> (-1.47%)	:arrow_down:
consensus/hotstuff/eventloop/event_loop.go	73.46% <0.00%> (-1.37%)	:arrow_down:
model/flow/transaction.go	50.80% <0.00%> (-1.05%)	:arrow_down:
... and 44 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[gomisha]

Replaced by https://github.com/onflow/flow-go/pull/3488