Support mirroring of LFS files

[hobti01]

It would be very nice if files managed with LFS were mirrored. Mirroring repositories that have LFS pointers results in the following error:

[2020-09-17T15:51:41Z ERROR git_mirror] Unable to sync repo [email protected]:some/repo.git -> [email protected]:some/repo-mirror.git (Command "git" "push" "[email protected]:some/repo-mirror.git" "+refs/tags/*:refs/tags/*" "+refs/heads/*:refs/heads/*" failed with exit code: 1, Stderr: remote: GitLab: LFS objects are missing. Ensure LFS is properly set up or try a manual "git lfs push --all".

We've worked around this with

    # Repositories with LFS managed files need special attention in order to mirror the file contents
    - |
      LFS_REPOS="repo"
      for r in $LFS_REPOS; do
        origin_dir="$MIRROR_DIR/git-gitlab-com-some-$r-git"
        mkdir -p "$origin_dir"
        git clone --mirror [email protected]:some/$r.git "$origin_dir"
        pushd "$origin_dir"
          git lfs fetch --all
          git lfs push [email protected]:some/$r-mirror.git --all
        popd
      done
    # Perform a mirror, reusing any existing cloned repos in $MIRROR_DIR
    - ./git-mirror -m "$MIRROR_DIR" -g some%2Fgroup -u ssh://[email protected] --refspec "+refs/tags/*:refs/tags/*" "+refs/heads/*:refs/heads/*"

But a more elegant approach would be welcomed!

[bachp]

@hobti01 Thank you for the suggestin. I think supporting Git LFS would be nice.

Maybe it would be sufficient to make sure git lfs is properly setup during the cloning.

Suggestions are welcome.

[hobti01]

Thanks for the feedback @bachp. In our case even with lfs setup large files are not pulled due to an .lfsconfig that ignores all by default. This is why git lfs fetch --all is needed to override the lfsconfig. I don't see an alternative to "get inside" the git-mirror copy loop other than coding it in, or coding in a "pre copy" option. I think the pre copy is more extensible, but also opens the door to possible security issues.

However, pre/post call outs might solve other open issues with a general solution. I look forward to your opinion.