Formalize the existence of external (read-only) stacks

[danielkza]

Currently, the ref and xref lookups always use the same AWS profile and region that Stacker is initialized with. That means they cannot access outputs in other accounts or regions. It is possible to create a stack that will just "pull" those outputs by adding locked: true to it. But that still does not make it possible to reference a stack outside of the current namespace, if one is configured. Additionally, options that should never take effect for a "read-only" stack were allowed, such as class_path or stack_policy_path.

To handle that use case, introduce and document the external: yes option for stacks. External stacks do not have any of the options related to creating or updating resources, such as class_path and template_path, and are used only as sources of values to other stacks (through outputs and lookups). Additionally, add an fqn option to external stacks to fix the namespace issue.

Internal changes

To ensure the separation has the desired effect, enable strict mode while parsing configuration into models, but only for non-top-level keys. Achieve it by manually filtering unrecognized top-level keys before forwarding the data for parsing.

To better separate external and managed stacks in config, rework the model hierarchy a little: add BaseStack that only contains the shared config options between both stack types, make it a superclass of Stack, and introduce ExternalStack. Unfortunately Schematics 2.0 has a bug where it did not validate models parsed using polymorphism, so upgrade to version 2.1.

Move decisions on whether a stack needs to be submitted or updated to the stacker.stack.Stack class, to simply the logic and allow it to differ between external and managed stacks. Introduce stacker.stack.ExternalStack with "empty" implementations of all the methods/properties related to templates, blueprints and updates.

Documentation updates

Rework the Stack documentation to list options allowed for all stacks, external and managed stacks separately. Document everything in YAML format, so that people can copy snippets to their configs from the different parts. Fix some typos and clarify some details in the process.

Open questions

The external/managed terminology is the best I could come up with, but I still feel it can be improved. Any suggestions of alternatives are welcome!

Notes

I ran functional tests manually on my all account and they passed - I even had to fix one that had stack_policy_path misspelled and was no longer accepted due to strict parsing.

[danielkza]

Should't the unit tests also be running on Python 3 now?

[danielkza]

Interesting note: I found 20+ usages of invalid options in stacks in my company's repo when using this PR (due to enabling strict parsing). Most of them were harmless (using enable instead of enabled), but a few were a bit more dangerous.

I can make a PR that just enables strict parsing before this one (which is larger and a bit more controversial). What do you think @ejholmes?

edit: PR here: https://github.com/cloudtools/stacker/pull/623

[phobologic]

@ejholmes @danielkza - I'm thinking we should spin a release of stacker this week, it's been quite a while, and we have a lot of features/updates already. Do you think we should wait for this? What's left for discussing here?

[danielkza]

I'm looking to revive this PR as I update our internal Stacker fork to the latest master. The strict parsing change has been integrated for a while already, any other concerns I should handle to get this merged? @ejholmes @phobologic

[danielkza]

Rebased to latest master. Was a bit harder than what I hoped, and I ended up having to make some changes to the build action tests.

I'll address your review comments in further commits @phobologic :)

[danielkza]

I also found some issues running the functional tests, good thing we have those!

[danielkza]

I'm circling back to this. Rebased to latest master again, but unfortunately I'm seeing failed unit tests locally due to #709.

[danielkza]

@phobologic Things looked up-to-date to me, but I took the opportunity to improve the wording on the docs a little bit and forced-pushed.

[phobologic]

Weird! I wonder why it is showing the additions of stack_name, which is already in the master branch. Ok, once tests pass I'll go ahead and merge :)

[danielkza]

I think it's because I moved it around in the file a little bit.