file lookup should not be nested

[xiaket]

Currently all lookups are nested, so after stacker has resolved some vars, it will load the resolved var and try to do another round of lookup until it cannot find any lookups in the produced value. However, this has brought some issues in my user case. I'm using file lookup to reference some file content for userdata, which in turn contains some bash logic. stacker is expected to resolve the first lookup, but is should not look inside my userdata script and try to replace the bash vars inside of it(which will fail and raise an exception).

I think nested lookup is a great feature, it is really powerful. However, for the case of file lookup, I think it should not try to resolve the loaded content. Or at least, stacker should provide an option not to go nested if the user want to.

I cannot find a way to implement this without a lot of change in the codebase. The lookup is nested because of the infinite loop in variables.py, the resolve_lookups function in registry.py has no way to reset self.lookups so the loop could be broken. Perhaps the lookup handlers could raise a special exception and we can handle the exception in variables.py and break there, but I think that's for the devs to decide.

My local setup:

config file:

CloudFormationInitConfigSetsFile: ${file plain:file://../files/userdata.json}

userdata.json

...
"/usr/local/bin/foobar export --namespace ${NAMESPACE} > /opt/foo/foo.conf\n\n",
...

It would produce the following error message:

stacker.exceptions.FailedVariableLookup: Couldn't resolve lookups in variable `CloudFormationInitConfigSetsFile`. Unknown lookup type: "None"

It happened because stacker had created a new lookup in runtime and it is:

set([Lookup(type=None, input='NAMESPACE', raw='NAMESPACE')])

[phobologic]

Thanks @xiaket - I can see how that would cause problems. Have you tried escaping the variables in the userdata? I'm actually not sure if that would help or not - the other option, which is definitely a work-around, is to not use the curly braces in the shell script.

If escaping doesn't help, I'm not sure what the right thing to do here is. People do find the nesting of lookups useful, and I could see someone wanting the file lookup to do further lookups, though it is a bit of a stranger case.

Anyway, let me know if any of the workarounds work - as you said, implementing the ability for some lookups to disable further lookups would be a significant change.

[xiaket]

Thanks @phobologic! I'm sure that escaping the variables in the userdata would work, but I think that does not solve the issue that we are facing: stacker lookup should not impose any requirements on the userdata file or any file in general.

I understand your concern that if we change the file lookup handler so that it does not go deeper, it could break someone's setup so it is not desirable. However, we could add another lookup type for file lookup here. For example, we could introduce plain-no-nest and registry.py could detect the -no-nest part of the type and add a flag for it. We could even do this for every lookup handler.