Server crashes when the main branch changes

[carlosjgp]

We have a case where the developers were using the dev branch and the main branch on Github because they were using a Gitflow approach

But now they are doing Continuous Deployment so their main branch has changed to master.

This has caused the server to crash every time that tries to go through that repository

We use detect-secrets-server as a cronjob on Kubernetes using a Persistent volume claim for the folder /repositories/

Scanning: [email protected]:my-company/my-app.git
Traceback (most recent call last):
  File "/usr/bin/detect-secrets-server", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/detect_secrets_server/__main__.py", line 39, in main
    return actions.scan_repo(args)
  File "/usr/lib/python2.7/site-packages/detect_secrets_server/actions/scan.py", line 37, in scan_repo
    scan_head=args.scan_head,
  File "/usr/lib/python2.7/site-packages/detect_secrets_server/repos/base_tracked_repo.py", line 133, in scan
    self.storage.fetch_new_changes()
  File "/usr/lib/python2.7/site-packages/detect_secrets_server/storage/base.py", line 84, in fetch_new_changes
    git.fetch_new_changes(self._repo_location)
  File "/usr/lib/python2.7/site-packages/detect_secrets_server/storage/core/git.py", line 71, in fetch_new_changes
    '--force',
  File "/usr/lib/python2.7/site-packages/detect_secrets_server/storage/core/git.py", line 213, in _git
    stderr=subprocess.STDOUT
  File "/usr/lib/python2.7/subprocess.py", line 223, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['git', '--git-dir', '/repositories/repos/179222fd2291a1a104c7750b95f5c3df3ae7f9c6325d25e5047059ffd25e47369c080091f58dd06007cc905ed31b60da447f7c2fd9f0fa1eded4740f3bf43760', 'fetch', '--quiet', 'origin', 'dev:dev', '--force']' returned non-zero exit status 128

[domanchi]

Hi @carlosjgp,

detect-secrets-server gets the "main" branch through this command: git rev-parse --abbrev-ref HEAD for the locally cloned bare repo.

Based on the subprocess traceback, it tried to fetch changes from dev branch.

Not sure how you changed the main branch to master, but you might have to change the master branch for the local git repo too. If you changed the master branch upstream, probably easier to rm -rf /repositories/repos/179222fd2291a1a104c7750b95f5c3df3ae7f9c6325d25e5047059ffd25e47369c080091f58dd06007cc905ed31b60da447f7c2fd9f0fa1eded4740f3bf43760, and track it again.

[carlosjgp]

I just did that

Hi @carlosjgp,

detect-secrets-server gets the "main" branch through this command: git rev-parse --abbrev-ref HEAD for the locally cloned bare repo.

Based on the subprocess traceback, it tried to fetch changes from dev branch.

Not sure how you changed the main branch to master, but you might have to change the master branch for the local git repo too. If you changed the master branch upstream, probably easier to rm -rf /repositories/repos/179222fd2291a1a104c7750b95f5c3df3ae7f9c6325d25e5047059ffd25e47369c080091f58dd06007cc905ed31b60da447f7c2fd9f0fa1eded4740f3bf43760, and track it again.

I just did that and also realised that I had to delete the same entry 179222....f3bf43760 from AWS S3... maybe too late because detect-secrets was running already and I got this...

Scanning: [email protected]:my-company/my-app.git
Traceback (most recent call last):
  File "/usr/bin/detect-secrets-server", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/detect_secrets_server/__main__.py", line 39, in main
    return actions.scan_repo(args)
  File "/usr/lib/python2.7/site-packages/detect_secrets_server/actions/scan.py", line 24, in scan_repo
    s3_config=getattr(args, 's3_config', None),
  File "/usr/lib/python2.7/site-packages/detect_secrets_server/repos/s3_tracked_repo.py", line 90, in load_from_file
    base_directory,
  File "/usr/lib/python2.7/site-packages/detect_secrets_server/repos/base_tracked_repo.py", line 100, in load_from_file
    data = cls.get_tracked_repo_data(storage, repo_name)
  File "/usr/lib/python2.7/site-packages/detect_secrets_server/repos/s3_tracked_repo.py", line 95, in get_tracked_repo_data
    output = super(S3TrackedRepo, cls).get_tracked_repo_data(storage, repo_name)
  File "/usr/lib/python2.7/site-packages/detect_secrets_server/repos/base_tracked_repo.py", line 112, in get_tracked_repo_data
    return storage.get(storage.hash_filename(repo_name))
  File "/usr/lib/python2.7/site-packages/detect_secrets_server/storage/s3.py", line 38, in get
    Filename=file_on_disk,
  File "/usr/lib/python2.7/site-packages/boto3/s3/inject.py", line 172, in download_file
    extra_args=ExtraArgs, callback=Callback)
  File "/usr/lib/python2.7/site-packages/boto3/s3/transfer.py", line 307, in download_file
    future.result()
  File "/usr/lib/python2.7/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/usr/lib/python2.7/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

running again now...

[carlosjgp]

This is where you can change the default branch on GitHub by the way

[carlosjgp]

It did work this time.

Is this something that is worth consider managing better?

[carlosjgp]

what do you think @domanchi?

[domanchi]

@carlosjgp : it might be possible to pass in the master branch name as a CLI argument, but it seems pretty edge case to me (only speaking from the workflows that I've been exposed to) :shrug:

If you think this is a larger issue that's worth the time to address, PRs are definitely welcome!