Bug report

We have a playbook that looks something like this:

- name: Example customization playbook
  hosts: 127.0.0.1
  connection: local
  gather_facts: true
  tasks:
    - name: Install roles and collections
      community.general.ansible_galaxy_install:
        type: both
        requirements_file: ./requirements.yml

    - name: Execute a role from collection installed from requirements.yml
      ansible.builtin.include_role:
        name: my_namespace.my_collection.role1

And we invoke that like this:

    ansible:
      install_method: pip
      package_name: ansible
      run_user: my_user
      pull:
        url: "<url>"
        playbook_name: "custom.yml"

Debug logs show that the collection is installed in the first task. However, it appears that ansible-pull can't find the role:

TASK [Execute a role] **********************************************************
        ERROR! the role 'my_namespace.my_collection.role1' was not found in /home/my_user/.ansible/pull/test/roles:/home/my_user/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/home/my_user/.ansible/pull/test

Indeed, ansible-pull doesn't seem to be looking in the collection install path (although it is clearly able to use the Community collection?). Executing ansible --version as a task in the playbook shows:

                "stdout_lines": [
                    "ansible [core 2.17.0]",
                    "  config file = None",
                    "  configured module search path = ['/home/imy_user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']",
                    "  ansible python module location = /home/my_user/.local/lib/python3.10/site-packages/ansible",
                    "  ansible collection location = /home/my_user/.ansible/collections:/usr/share/ansible/collections",
                    "  executable location = /home/my_user/.local/bin/ansible",
                    "  python version = 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (/usr/bin/python3)",
                    "  jinja version = 3.0.3",
                    "  libyaml = True"
                ]

And the collection is indeed installed to the first location in the "ansible collection location" path.

Things we've tried to solve the problem, but haven't:

Using an ansible.cfg file in the repository cloned to force the collection installation location, but it has made no difference.
Splitting the playbook into two plays, one to install the collection, the second to execute the role, but this hasn't helped
Both import_role vs include_role
Executing the role from "roles:" rather than as part of a task

Manually logging into the instance and running the custom.yml playbook completes as expected - showing both that the collection is installed and that the "system" Ansible can find it.

We don't know a priori if a requirements.yml file exists, which makes using the "galaxy/actions" more problematic, and it isn't clear from the documentation if installing from a requirements.yml file works:

  galaxy:
    actions:
      - ["ansible-galaxy", "install", "-r", "./requirements.yml"]

TL;DR It doesn't seem possible to use roles from within a collection installed as part of the playbook executed by the Ansible module.

Steps to reproduce the problem

See sample playbook and cloud-init config above

Environment details

Operating System Distribution: Ubuntu 20.04 and 22.04
Cloud-init version:

Ubuntu 20.04 - /usr/bin/cloud-init 24.1.3-0ubuntu1~20.04.1
Ubuntu 22.04 - /usr/bin/cloud-init 24.1.3-0ubuntu1~22.04.1

Cloud provider, platform or installer type: Openstack with Terraform launching Ubuntu cloud images

cloud-init logs

The relevant section appears to be:

2024-06-18 07:51:22,085 - util.py[WARNING]: Running module ansible (<module 'cloudinit.config.cc_ansible' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ansible.py'>) failed
2024-06-18 07:51:22,085 - util.py[DEBUG]: Running module ansible (<module 'cloudinit.config.cc_ansible' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_ansible.py'>) failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/config/modules.py", line 286, in _run_modules
    ran, _r = cc.run(
  File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 60, in run
    return self._runners.run(name, functor, args, freq, clear_on_fail)
  File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 156, in run
    results = functor(**args)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ansible.py", line 202, in handle
    run_ansible_pull(ansible, deepcopy(pull_cfg))
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ansible.py", line 258, in run_ansible_pull
    stdout = pull.pull(
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ansible.py", line 90, in pull
    stdout, _ = self.do_as([*self.cmd_pull, *args])
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_ansible.py", line 100, in do_as
    return self.distro.do_as(command, self.run_user, **kwargs)
  File "/usr/lib/python3/dist-packages/cloudinit/distros/__init__.py", line 1225, in do_as
    return subp.subp(
  File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 298, in subp
    raise ProcessExecutionError(
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['su', '-', 'my_user', '-c', 'env PATH=$PATH ansible-pull --url=https://<url> custom.yml']
Exit code: 1
Reason: -
Stdout: Starting Ansible Pull at 2024-06-18 07:51:15
        /home/ipuuser/.local/bin/ansible-pull --url=<url> custom.yml
        [WARNING]: Could not match supplied host pattern, ignoring: test
        localhost | CHANGED => {
            "after": "bd5cab726a44c0640f65b6607770bd29ffae3bb8",
            "before": null,
            "changed": true
        }
        [WARNING]: No inventory was parsed, only implicit localhost is available
        [WARNING]: provided hosts list is empty, only localhost is available. Note that
        the implicit localhost does not match 'all'
        [WARNING]: Could not match supplied host pattern, ignoring: test1
        ERROR! the role 'my_namespace.my_collection.role1' was not found in /home/my_user/.ansible/pull/test1/roles:/home/my_user/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/home/my_user/.ansible/pull/test1
        
        The error appears to be in '/home/my_user/.ansible/pull/test/custom.yml': line 54, column 15, but may
        be elsewhere in the file depending on the exact syntax problem.
        
        The offending line appears to be:
        
              ansible.builtin.include_role:
                name: my_namespace.my_collection.role1
                      ^ here
Stderr:

Jun 18 '24 09:06 jjo93sa

@jjo93sa It looks like the error is this:

              ansible.builtin.include_role:
                name: my_namespace.my_collection.role1
                      ^ here

And ansible.builtin.include_role is failing to find the role that you reference: m̀y_namespace.my_collection.role1. Unfortunately, we don´t have enough information to help you. Where is that role supposed to be coming from? Is it installed by some other part of your playbook? Did you build it into the image prior to launching the instance?

This doesn´t look like a cloud-init bug; it seems to me like you've got an ansible configuration error. If you disagree, please include more information.

Jun 25 '24 15:06 holmanb

Yes, as I tried to indicate in the pseudo playbook above, the role is part of a collection that is installed in the first task. The requirements.yml file used by ansible_galaxy_install is included in the same repository as the playbook executed by cloud-init.

    - name: Install roles and collections
      community.general.ansible_galaxy_install:
        type: both
        requirements_file: ./requirements.yml

Logging into the instance shows that the collection has been installed correctly.

What seems to be failing is ansible-pull (I believe that is the executable that cloud-init is calling?) finding the role from within the collection. I've tried various different playbook structures and configurations, but no matter what, the role is not found and playbook execution fails:

Using an ansible.cfg file in the repository cloned to force the collection installation location, and whilst it changed the installed location, it didn't encourage ansible-pull to find the role within the collection
Splitting the playbook into two plays, one to install the collection, the second to execute the role
Both import_role vs include_role fail similarly
Executing the role from "roles:" cf tasks fails
Using the collections: keyword vs specifying the FQN of the role in import/include_role makes no difference, role isn't found

If I manually login to the instance and run the playbook using ansible-playbook, it runs to completion.

I'm happy to provide more information, but I'm at a loss as to what else I can try at this point.

Jun 25 '24 15:06 jjo93sa

Have a similar case. Based on ansible source code (https://github.com/ansible/ansible/blob/devel/lib/ansible/playbook/play.py) we get collection list when initializing an object Play. So after installIing collection in the playbook, the collection list for this playbook does not change . Try to split into 2 playbooks: the first - install collection, the second - include role. Or try to import_playbook

Jul 23 '24 07:07 i-a-artemev

Ansible module doesn't find collections installed by the playbook

Bug report

Steps to reproduce the problem

Environment details

cloud-init logs