Problems with Sequence template

[cloud-rocket]

I am trying to use confuse examples to dynamically update a Sequence, but dump function is not working after dynamic update:

servers_example.yaml:

servers:
  - host: one.example.com
  - host: two.example.com
    port: 8000
  - host: three.example.com
    port: 8080

test.py:

import confuse
import pprint
source = confuse.YamlSource('servers_example.yaml')
config = confuse.Configuration(__file__)
config.add(source)
template = {
     'servers': confuse.Sequence({
         'host': str,
         'port': 80,
     }),
 }
valid_config = config.get(template)
pprint.pprint(valid_config)


config.set({
     'servers': valid_config['servers'] + [
         {'host': 'four.example.org'},
         {'host': 'five.example.org', 'port': 9000},
     ],
})
updated_config = config.get(template)
pprint.pprint(updated_config)


print(config.dump(full=False).strip())

config.dump produces:

Traceback (most recent call last):
  File "test.py", line 26, in <module>
    print(config.dump(full=False).strip())
  File "/usr/local/lib/python3.8/dist-packages/confuse/core.py", line 717, in dump
    yaml_out = yaml.dump(out_dict, Dumper=yaml_util.Dumper,
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 290, in dump
    return dump_all([data], stream, Dumper=Dumper, **kwds)
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 278, in dump_all
    dumper.represent(data)
  File "/usr/lib/python3/dist-packages/yaml/representer.py", line 27, in represent
    node = self.represent_data(data)
  File "/usr/lib/python3/dist-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/usr/lib/python3/dist-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "/usr/local/lib/python3.8/dist-packages/confuse/yaml_util.py", line 108, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/usr/lib/python3/dist-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/usr/local/lib/python3.8/dist-packages/confuse/yaml_util.py", line 127, in represent_list
    node = super(Dumper, self).represent_list(data)
  File "/usr/lib/python3/dist-packages/yaml/representer.py", line 199, in represent_list
    return self.represent_sequence('tag:yaml.org,2002:seq', data)
  File "/usr/lib/python3/dist-packages/yaml/representer.py", line 92, in represent_sequence
    node_item = self.represent_data(item)
  File "/usr/lib/python3/dist-packages/yaml/representer.py", line 58, in represent_data
    node = self.yaml_representers[None](self, data)
  File "/usr/lib/python3/dist-packages/yaml/representer.py", line 231, in represent_undefined
    raise RepresenterError("cannot represent an object", data)
yaml.representer.RepresenterError: ('cannot represent an object', {'host': 'one.example.com', 'port': 80})

Any idea what am I doing wrong? or is it a bug?

[sampsyo]

Ah, that's a very funky problem! I believe Sequence is actually a red herring here. The problem is about the types that come out of validating the mapping. You can see it if you throw in this print statement:

print(type(valid_config['servers'][0]))

which reveals that the things that look like dicts in there are actually of type confuse.templates.AttrDict. That's a subclass of dict we use to make it possible to use m.port instead of m['port']. Apparently, PyYaml doesn't like this subclassing!

The problem goes away if you convert these back to plain dicts. Like so:

config.set({
     'servers': [dict(m) for m in valid_config['servers']] + [
         {'host': 'four.example.org'},
         {'host': 'five.example.org', 'port': 9000},
     ],
})

Maybe we should consider this a problem in PyYaml, and maybe it would go away if we switched to a modern library (#52). But maybe, if we stick with PyYaml, we should teach the dumper we use about AttrDict.

[sampsyo]

(Thank you, by the way, for the self-contained reproducible bug report. Having a program I could actually run made it easy to find the problem!)

[cloud-rocket]

Thanks @sampsyo for the detailed and fast response!

Maybe it's a different issue, but it's also not exactly clear how can you extend a Sequence dynamically (the way I did it with config.set does not seem to be the most Pythonic way). I tried using config.add and Python list append - but none of it seems to be working....

[sampsyo]

Oh sure, that's a very good question! You should be able to accomplish this by using add or set, but then, instead of using get (which just gets the "top" value for a view), try all_contents instead. This unit test shows how that works: https://github.com/beetbox/confuse/blob/2d7b6c8393aad515fd028d3b2088c4150d585b52/test/test_views.py#L230-L233

Namely, all_contents collects all the values from all the sequences from all the sources for the given point in the configuration tree. I hope that works for your use case!

[cloud-rocket]

@sampsyo - sorry, don't really understand how is it related.

I am trying to add extra members to Sequence dynamically instead of using set to manually add new items to previous items.

I tried the following things, but they are not working:

config.add({'servers': [
         {'host': 'six.example.org'}
     ]
})

config['servers'].add([
         {'host': 'six.example.org'}
     ]
)         

config['servers'].add(
         {'host': 'six.example.org'}
)        

config['servers'].add([
         OrderedDict([('host', 'six.example.org')])
     ]
)         

[sampsyo]

I suppose what I'm saying is that both of those things should work as-is, but you have to retrieve the data with config['servers'].all_contents() instead of config.get(). Can you give that a shot?

[cloud-rocket]

all_contents() creates a generator... Can you please give an example on how to add an item to an existing sequence, validate it and dump to yaml? Tnx

[sampsyo]

You can use list(...) to get a list out of a generator.

For broader context, I don't think there is a built-in template that can do this "flattening" without using all_contents. But I do think it could be written, probably using all_contents. It would be a bit of a project, but hopefully not too bad!