Merging YAML group sources

[treydock]

It looks like the design of the YAML group source loading is to fail if the same source exists in two files. Is this a behavior that can be changed? Below is an example of YAML files compiled for 3 clusters. Ideally I'd have the roles source merged from all 3 files as that's set as the default group source. I wasn't sure if changing this behavior is something that would be open to changing?

==> groups.d/oakley.yaml <==
roles:
    oakley-compute: 'n0[004-080,082-496,505-641,643-646,648-693]'
    oakley-login: 'oakley0[1-2]'
    oakley-gpu: 'n0[281-319,641,643-646,648-660,689-692]'

==> groups.d/owens.yaml <==
roles:
    owens-compute: 'o0[001-824]'
    owens-login: 'owens-login0[1-4]'
    owens-gpu: 'o0[649-802,805-808]'

==> groups.d/ruby.yaml <==
roles:
    ruby-compute: 'r[0001-0240,0501-0506,1001]'
    ruby-login: 'ruby0[1-2]'
    ruby-gpu: 'r0[201-220,506]'

# nodeset -ll
Traceback (most recent call last):
  File "/bin/nodeset", line 9, in <module>
    load_entry_point('ClusterShell==1.7.3', 'console_scripts', 'nodeset')()
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 378, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2566, in load_entry_point
    return ep.load()
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2260, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/usr/lib/python2.7/site-packages/ClusterShell/CLI/Nodeset.py", line 35, in <module>
    from ClusterShell.CLI.Error import GENERIC_ERRORS, handle_generic_error
  File "/usr/lib/python2.7/site-packages/ClusterShell/CLI/Error.py", line 30, in <module>
    from ClusterShell.CLI.Utils import GroupResolverConfigError  # dummy but safe
  File "/usr/lib/python2.7/site-packages/ClusterShell/CLI/Utils.py", line 30, in <module>
    from ClusterShell.NodeSet import NodeSet
  File "/usr/lib/python2.7/site-packages/ClusterShell/NodeSet.py", line 69, in <module>
    ILLEGAL_GROUP_CHARS)
  File "/usr/lib/python2.7/site-packages/ClusterShell/NodeUtils.py", line 501, in __init__
    self._parse_config(os.path.dirname(parsed[-1]))
  File "/usr/lib/python2.7/site-packages/ClusterShell/NodeUtils.py", line 558, in _parse_config
    self._sources_from_yaml(autosfn)
  File "/usr/lib/python2.7/site-packages/ClusterShell/NodeUtils.py", line 614, in _sources_from_yaml
    self.add_source(source)
  File "/usr/lib/python2.7/site-packages/ClusterShell/NodeUtils.py", line 371, in add_source
    group_source.name)
ValueError: GroupSource 'roles': name collision

[degremont]

Thanks for this report! First of all, ValueError is not catch and should be. We will fix this.

Regarding your proposal, this does not make sense for traditional upcall-based group sources. For this, we definitely need to forbid source name duplicate.

Regarding YAML-based group source, this is different. It could be interesting in some cases. We need to think about it, however, the patch is not straighforward due to the cache expiration management. Let us know if this is blocker for you.

[treydock]

@degremont Seeing YAML group sources merged would be useful but not a blocker for us. For the time being I've moved all our clusters into a single file so I can put necessary groups under the same source.