copier Support ChainableUndefined for jinja2 Environment

Jinja 2.11 supports setting undefined option on an Environment to ChainableUndefined. This allows the default filter to work on nested elements and attributes.

Arguably this could be a better default behavior for use in copier, but regardless of the default it would be nice to be able to configure this in copier.yml. The _envops configuration exists, but there's no way to set undefined to a python class in that YAML file (that I know of at least), so I think either a specific config value would be needed, or someway to lookup a class for the undefined key. I'm not sure which is the best or preferred way, but would like to turn this on to make the templates that use nested configuration easier to set their defaults.

Dec 02 '20 18:12 ppg

Would you please provide an example that demonstrates why this would be better?

Dec 03 '20 08:12 yajo

Yes, sorry I should have included one.

This doesn't have great support in copier (which is a different issue request), but if you have more complex configuration, say in .copier-answers.yml you have:

go:
  lint:
    changes_only: true

So that you can control when you generate a helper bash file to lint golang files whether to fail on any lint issues, or only ones that were introduced by a line your PR touches (i.e. a reviewdog pattern if anyone is interested). In my helper file to lint I want to do something like:

{%- if go.lint.changes_only|default(False) %}
golint ./...
{%- else %}
golint ./... | reviewdog -fail-on-error -f=golint -diff='git diff origin/master'
{%- endif%}

I would like that to also work for:

go: {}

or even no go key. If you can set the jinja2's Environment's undefined is set to ChainableUndefined it will, but with how copier has it you must do:

{%- if go and go.lint and go.lint.changes_only|default(False) %}
golint ./...
{%- else %}
golint ./... | reviewdog -fail-on-error -f=golint -diff='git diff origin/master'
{%- endif%}

Dec 03 '20 19:12 ppg

OK nice, I can see the usefulness then.

However, have you considered simplifying your questionary? Usually templates are meant to be filled by human answers, and although using complex answer types is perfectly supported, in my experience almost always is better to make simpler, more assertive questions. Example:

# this is copier.yaml
go_lint_changes_only:
  type: bool
  default: false

Dec 04 '20 10:12 yajo

We considered that, but we didn't like the flattened form for a few reasons:

It get's crowded quickly; you're showing an example of 1 option, but that looks much worse and harder to read when you have have go.lint with a number of options as well as go.<other_keys>.
There's a hierarchy when it's flattened, but now you're depending on humans to enforce the structure correctly vs. something that is meant to enforce/encode that structure such as JSON or YAML. For example, is it okay to have go_lint_changes_only, golang_vet_changed and go_linter_enabled? I would say no.
The hierarchy is ambiguous and I can't inspect it; for example, is go_lint_changes_only really go_lint:changes_only or go:lint_changes_only or go:lint:change_only; also if I want to predicate something on any go lint options, with it nested I can look at go and go.lint (or with the ChainableUndefined just go.lint|default(False)) with the hierarchy expressed correctly. With flat options I think this is impossible to discern.
You can't support an array with the flattened structure; this one basically cuts off the functionality of having an arbitrary number of say binaries we generate from a repo, packages, etc. I know you could do _0, _1, ... but if the argument against a complex structure is easier interactive human answers I don't think having a bunch of entries you need to know to ignore is a great UX either, especially as each entry itself has a nested schema.
We update repos much more than we create them, so while the nested answers are more difficult to answer interactively (at least with the current copier; that IMO should/could be an improvement too), the normal way for us to change answers is to edit the .copier-answers.yml (where we also put in documentation to make clear what to change) and then re-run update. Therefore, the interactive question answering is the least frequently used input method for us and if the options are too confusing one can always copy from another source or edit after creation and re-run update.

Obviously our use case may not be the normal one, but that's why we've chosen the non-flat options, for now at least.

Dec 08 '20 22:12 ppg

OK, well I was just telling a suggestion, but of course there are different needs and designs, yours may be perfectly good for your case. :wink:

Just a couple of comments.

the normal way for us to change answers is to edit the .copier-answers.yml (where we also put in documentation to make clear what to change) and then re-run update.

Remember this is an unsupported way of updating.

For example, is it okay to have go_lint_changes_only, golang_vet_changed and go_linter_enabled? I would say no.

Latest alpha release supports the when key on questions, which allows to skip questions that are not supposed to be asked, based on other questions. Still alpha though.

Dec 09 '20 08:12 yajo

Sorry, poor etiquette on my part; I didn't mean to seem ungrateful for the suggestion/feedback, I definitely appreciate it, so thank you! I was just trying to enumerate the considerations that went into our evaluation of that, nothing more.

Remember this is an unsupported way of updating.

Is 'this' editing .copier-answers.yml directly? Or re-running update? Or something else? I didn't think either was necessarily unsupported; obviously you need to edit .copier-answers.yml 'correctly', but copier --force update for example is allowed to be run and will read from that file if present, I thought the running of that command and the reading of that file were both officially supported.

Are there recommendations of how to apply that when key to say allow the user to specify N number of things that the configure? Like is the recommendation asking a yes/no to service_1 and then predicating the remaining service_1_<field> on service_1? Or to just have service_1_<field1> and predicate fields 2-N on that first one? We'd certainly want to adopt a standard if we went that route, but unsure if there are pros and cons for those choices.

All that being said, at some point we'll still probably look at extending the copier question engine to support a richer configuration schema and interactive answer flow. That will take a little up front development, but I don't think a ton: the configuration needs to be extended to describe nested schemas (perhaps add a properties key like swagger/OAS for when the type is yaml); and then when asking questions if it's an array there will need to be an interactive way to add another or stop adding; if it's an object to delve into the nested keys or not. But adding that seems fairly straightforward and it's then reusable, so the ROI seems pretty good and immediate.

Also, is there a timeline on v6? I see the alpha releases, but wasn't sure when to roughly expect it. Not trying to apply any sort of pressure, just curious if I would want to wait for it vs. doing additional work/workarounds in v5.

Dec 09 '20 19:12 ppg

Is 'this' editing .copier-answers.yml directly?

Yes. It is further explained in https://github.com/OCA/oca-addons-repo-template/issues/6#issuecomment-703503660.

I think I should state this clearly in the docs somewhere... as I see there's general confusion about it. :thinking:

Dec 10 '20 09:12 yajo

Ah sorry, quick muscle memory made me respond quickly, but I didn't answer all things... :laughing:

Are there recommendations of how to apply that when key to say allow the user to specify N number of things that the configure?

No recommendations so far, it's a fairly new feature, still not completely stable.

You can just do it as you please. Inside when value, you can put a jinja template that results in true when the conditions you want are met. This requirement is not very strict, so it's open to how you want to use it. Example:

using_github:
    type: bool

repo_name:
    type: str
    when: "{% if using_github %}true{% endif %}"

Also, is there a timeline on v6? I see the alpha releases, but wasn't sure when to roughly expect it.

Right now there isn't. You can follow https://github.com/copier-org/copier/projects/1 to see progress. ATM it has 2 main problems:

It's under a big refactor. Doing big changes right now is a bit hard because the codebase is a bit entangled. You can follow progress in #110. I have a WIP in #314 which is far from finished.
Lack of funding. Until latest stable, I was paid for development, and Copier evolved quickly. Right now I'm pushing for v6 mostly in my personal time. I think it's a very important milestone, but without funding it's gonna progress slowly. Sponsorship can be done from my profile, just in case you're interested. :grin:

If it were already refactored, new features would land faster. If I got paid, it would get refactored faster. In this very moment, I'm trying not to introduce new features until it's refactored, mostly because after that, it's gonna be way easier to do it.

Apart from that, I encourage you to test the latest alpha. Having alpha testers is also a good help!

Dec 10 '20 09:12 yajo

Is 'this' editing .copier-answers.yml directly?

Yes. It is further explained in OCA/oca-addons-repo-template#6 (comment).

I think I should state this clearly in the docs somewhere... as I see there's general confusion about it.

I read the linked comment, but I don't think that's how it works is it? At least not on v5.1.0. I routinely change .copier-answers.yml and run copier --force update and it reads the answers and then applies them. In the middle of that comment you're saying that somehow copier would know not to update the file from the template, I'm not understanding how it would know that, especially with the --force option? It seems like the only info copier has available is the .copier-answers.yml and the template, isn't it forced to render the template with the answers and see if it changes (which it would if you changed .copier-answers.yml)?

Dec 22 '20 18:12 ppg

I read the linked comment, but I don't think that's how it works is it? At least not on v5.1.0. I routinely change .copier-answers.yml and run copier --force update and it reads the answers and then applies them.

Believe me, I'm the maintainer, I know how this thing works. 😋

I merged recently #317 to clarify this misunderstanding. The fact it works for you is an accident because you didn't happen to hit any corner cases yet (or because you hit them but didn't notice), but yes, that's an unsupported way of updating, and it will always be because of technical reasons explained there.

If you have further questions regarding this subject, please open a new discussion. Let's keep the issue focused, I hope you don't mind. 😄

Dec 23 '20 09:12 yajo