Wrong Order in Substitution Resolution Leads to Faulty Configuration

[sbachstein]

The substitution resolution is not performed in the order of the HOCON variable assignment.

Version: 0.3.58

Example:

pyhocon -i <INPUT_FILE>

Input file:

common = common
original = ${common}/original
result = ${original}
replaced = ${common}/replaced
result = ${replaced}
copy = ${result}

Result:

{
  "common": "common",
  "original": "common/original",
  "result": "common/replaced",
  "replaced": "common/replaced",
  "copy": "common/original"
}

Expected Result:

{
  "common": "common",
  "original": "common/original",
  "result": "common/replaced",
  "replaced": "common/replaced",
  "copy": "common/replaced"
}

Analyzing the substitution process, we see that the substitution is performed in the following order:

1. ${common} -> original = common/original
2. ${original} -> result = common/original
3. ${replaced} -> result = ${replaced} (HOWEVER: cannot be resolved because ${replaced} contains a substitution itself, thus the previous resolution stays for now! Here lies the issue!)
4. ${common} -> replaced = common/replaced
5. ${result} -> copy = common/original (Because the result = ${replaced} substitution could not be done until now, this apparently takes the previous substitution)

Now the next substitution iteration starts with the single left over substitution:

1. ${replaced} -> result = common/replaced

[polopi]

It seems to actually be the expected behavior from the specs.

Using the original JAVA library as a baseline and a java shell

import com.typesafe.config.Config
import com.typesafe.config.ConfigFactory

Config conf = Config(SimpleConfigObject({"common":"common","copy":${result},"original":${common}"/original","replaced":${common}"/replaced","result":${original},"result":${replaced}}))

conf = conf.resolve();
Config conf = Config(SimpleConfigObject({"common":"common","copy":"common/replaced","original":"common/original","replaced":"common/replaced","result":"common/replaced"}))

System.out.println(conf.getString("common"))
common

System.out.println(conf.getString("original"))
common/original

System.out.println(conf.getString("result"))
common/replaced

System.out.println(conf.getString("replaced"))
common/replaced

System.out.println(conf.getString("copy"))
common/replaced

The problem you see here is that some "merging" is also happening because it sees that you redefined result. reference

result = ${original}
result = ${replaced}

yields

result = ${replaced}

Then substitution happens with config

{
common = common
original = ${common}/original
result = ${replaced}  // merged
replaced = ${common}/replaced
copy = ${result}
}

The problem is because the result keys are defined at the same level.

[sbachstein]

I am not entirely sure what you are suggesting: Are you saying the actual behavior is according to spec or are you saying the behavior that I am expecting is according to spec?

I have checked this with https://hocon-playground.herokuapp.com/ which uses the java implementation and it confirms my expectations.

While this minimal example may look synthetic because you would not have duplicate keys within the same scope, it is absolutely possible for this to happen using includes or merging two configs.

For example when using a common configuration file that is merged with an environment-specific config file where values from the common configuration are potentially overwritten.

[afanasev]

Hi! I've created test based on your test data and managed to fix it and not break any other tests: https://github.com/chimpler/pyhocon/pull/284

[afanasev]

Still not good though, I see cases working wrong.

[afanasev]

Better now: https://github.com/chimpler/pyhocon/pull/285