Any way to force update (pull) of role specified in requirements.yaml?

[boredomwontgetus]

Hello Guys!

I struggle a bit with an issue regarding roles and requirements.yaml.

So i write a role and put it in a new git-repo. I add it to requirements.yaml and my semaphore-task fetches the role before the playbook is executed. Later on, i do some changes to the role (a little fix e.g.), commit the change to git. I do not tag it and do not use versions in my requirements for most of these roles.

Now, semaphore always says there is not change in the requirements file so there is no need to pull roles or collections. Which, technically is true.

Is there any way to make semaphore force a pull on roles specified in requirements.yaml like i normally would do with ansible-galaxy install .... --force

Thanks and BR Tom

[AngelFreak]

This is also a major issue for me, as I am using git as a scm for roles.

Every time I update a role, i have to put a release, or hash tag in the requirements file. It's a really annoying process.

An option to force pull requirements, not cache roles, or even better check if it is the most recent version, would be great! The default behavior could be keept as is, and a checkbox for overriding it could be added in the task template.

[Ganzabahl]

I understand why you might want to fix the hash in production env. However I fully agree with you both, when developing on semaphore environment. Changing hash every time there is a change, destroys the workflow completely. I'm currently looking into, if there is an easy way of adding a button or variable, to override the hash-check when deploying. Since I'm not part of the semaphore-team, any hints or preferences from the team, would be appreciated. After all the end-goal should be a contribution that would merge into mainline ;)

[supabibz]

up on this issue : deployed 2.8.90 and the issue remains. I love this product but this issue is a showstopper.

[sparsick]

Question: Would it have some side effect, when forcing would be the default behavior in Semaphore?

Of course, having a force button or something else would be cleaner, but like, I understand, more work. So changing the default behavior would be a workaround that would work for me.

[Ganzabahl]

Question: Would it have some side effect, when forcing would be the default behavior in Semaphore?

Of course, having a force button or something else would be cleaner, but like, I understand, more work. So changing the default behavior would be a workaround that would work for me.

Well I guess you would want the option of having it as already designed. Always running with "latest" in production is a really bad idea, and therefore the manual process somehow makes sense. It however it annoying when developing new playbooks/roles etc.

I might be looking into the code again but got demotivated last time. Partly because of the vue-part and partly because I don't know if a pull request actually would be merged in.

[sparsick]

Well I guess you would want the option of having it as already designed. Always running with "latest" in production is a really bad idea, and therefore the manual process somehow makes sense. It however it annoying when developing new playbooks/roles etc.

In general, you are right. But in production I would use reference on tags, and they should be immutable by definition. If you reference on a branch, then you "accept" that there are changes. But of course a option on the UI would be the cleanest way

[dtufood-kihen]

I've investigated Semaphore's source code regarding an issue where updates to roles aren't recognized if the roles/requirements.yml file hasn't been changed. Semaphore appears to be checking this file for changes by comparing the current state of the file to a stored hash. If no changes are detected (i.e., the stored hash matches the current file), Semaphore skips the Ansible Galaxy installation process. This results in Semaphore not pulling updates for roles that have been changed in their respective repositories but haven't had their version/tag information updated in the requirements.yml file.

At present, there isn't a built-in option in Semaphore that would force the tool to always reinstall roles from the requirements.yml file. However, I've forked Semaphore and removed the condition that checks for changes in the requirements.yml file in the installRolesRequirements and installCollectionsRequirements functions found in services/tasks/runner.go. With this change, Semaphore should always run the Ansible Galaxy install process, effectively reinstalling roles regardless of changes to the requirements.yml file.

Here's the link to the forked repository: https://github.com/dtufood-kihen/semaphore/tree/always-refresh-requirements

Please note that this modification serves as a potential workaround to ensure Semaphore always pulls the latest role changes. However, this change may also reduce efficiency by reinstalling roles unnecessarily when there are no updates.

Furthermore, I believe that the robustness of Semaphore could be improved by better segregating the files in the /tmp/semaphore folder. A clearer separation and organization of inventories, templates, tasks, and projects could lead to easier troubleshooting and improved manageability of the resources.

Another concern I have, is that it appears that all roles, regardless of the project or repository they belong to, are currently installed into /tmp/semaphore/.ansible/roles. This can potentially cause conflicts. For example, consider a scenario where two playbooks require different branches or tags of the same role. In such a situation, there is currently no mechanism in place to manage these separate requirements properly.

A possible solution to these issues could be to create a separate folder for each task run, rather than using a global cache. This would create an isolated environment for each task, preventing conflicts between different roles, branches, or tags required by different playbooks. It would also allow Semaphore to retain a "snapshot" of the exact environment for each task run, aiding in debugging and reproducibility. This approach, however, would increase the disk space usage and might increase the execution time of tasks, especially if the roles and other resources need to be fetched from a remote repository. It would be beneficial to have a configuration option that allows the user to choose between using a global cache or isolated task folders, depending on their specific use case and resources.

[enpaul]

Just to add to this issue (which I also just encountered), I wanted to expand on the "add a hash to requirements.yml" idea for forcing reinstall. My implementation uses Pre-Commit to automatically increment the hash when (and only when) files in my repo collections are updated. These are the details below:

# sample directory tree
my-ansible-project/
├── .scripts/
│   └── update_requirements_hash.py
├── collections/
│   └── my_namespace/
│       ├── collection1/
│       │   └── ...
│       ├── collection2/
│       │   └── ...
│       └── requirements.yml
└── .pre-commit-config.yaml

# .scripts/update_requirements_hash.py
import sys
import argparse
from pathlib import Path
import uuid
import subprocess
import hashlib


def main():
    """Update the hash in the requirements.yml file"""

    parser = argparse.ArgumentParser(prog=__name__)
    parser.add_argument(
        "--requirements-file",
        type=Path,
        help="Path to the requirements file to update",
        required=True,
    )
    parser.add_argument(
        "changed_files", nargs="+", help="Paths that have been changed this commit"
    )
    args = parser.parse_args()

    commit = (
        subprocess.run(["git", "rev-parse", "HEAD"], capture_output=True)
        .stdout.decode()
        .strip()
    )

    hasher = hashlib.new("sha256")
    hasher.update(f"{commit}:{':'.join(args.changed_files)}".encode())

    with Path(args.requirements_file).open("r") as infile:
        lines = infile.readlines()

    comment = f"## [AUTO] Semaphore update hash: {hasher.hexdigest()}\n"

    for index, line in enumerate(lines):
        if line.strip().startswith("## [AUTO] Semaphore update hash:"):
            lines[index] = comment
            break
    else:
        lines = [comment] + lines

    with Path(args.requirements_file).open("w") as outfile:
        outfile.writelines(lines)


if __name__ == "__main__":
    sys.exit(main())```

# .pre-commit-config.yaml
---
repos:
  - repo: local
    hooks:
      - id: semaphore
        name: Enforce updated requirements hash
        entry: python
        language: system
        args:
        - .scripts/update_requirements_hash.py
        - --requirements-file
        - collections/requirements.yml
        files: "^collections/my_namespace/.*"

This results in a collections/requirements.yml that looks like below:

## [AUTO] Semaphore update hash: 1ea0c461d8a563aec595bb49e14a551a71fd037d3f8e8506f8a2362493a5d5db
---
collections:
- source: collections/my_namespace/
  type: subdirs

After creating the above files, install pre-commit, run pre-commit install and now anytime you commit changes to any file under collections/my_namespace/ the hash will be automatically changed to force reinstallation when the change gets pulled into Semaphore (just be certain you commit the change to requirements.yml too)