Latest commits not reliably detected when branches are fetched

[cpt1gl0]

After pushing a commit and getting triggered by a GitLab webhook, GitLab branch source is not reliably detecting that latest commit. We are repeatedly encountering the issue where GitLab branch source reports "no changes detected" for the respective branch in the Jenkins multibranch pipeline event log and consequently does not trigger a build.

Doing some investigation I found the cause of the issue being the GitLab API itself not reliably giving the latest commit for existing branches. The reason behind this is a caching mechanism which was lately introduced into GitLab API when querying multiple branches.

[cpt1gl0]

Created related pull request #145.

[cpt1gl0]

I looked up the relevant parts of GitLab source code. There it can easily be seen that fetching all branches is being cached https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/branches.rb#L44-77 while for fetching a single branch there is no caching https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/branches.rb#L79-94.

[cpt1gl0]

@LinuxSuRen Could you please take a look, also at the related MR, as soon as possible so this can be fixed? At the moment GitLab Branch Source plugin will not reliably work with the latest version of GitLab because of this issue.

[cpt1gl0]

A related GitLab issue was just created: https://gitlab.com/gitlab-org/gitlab/-/issues/333548

[cpt1gl0]

Note that a similar trigger delay is used in GitHub Branch Source plugin to avoid the GitHub cache. I think having the delay proposed in PR #145 is a reasonable solution for this issue.

[cpt1gl0]

@MarkEWaite Hi Mark! Could you please check the related PR and merge it if you don't have any objections?

[MarkEWaite]

@cpt1gl0 I'm not a maintainer of this plugin

[cpt1gl0]

@cpt1gl0 I'm not a maintainer of this plugin I am sorry.

@baymac Could you please check my PR and merge it if there are no objections?

[baymac]

@cpt1gl0 I am no longer able to maintain this repository due to limited time. Also these changes can have other side effects and I cannot approve without testing these changes. I think @LinuxSuRen should be the right person to approve this.

Considering that @LinuxSuRen is not active, do let me know if you want to adopt this plugin, we can have a separate thread to make you an admin.

[Olcod]

@cpt1gl0 Thanks for your MR mate. We've merged this into our fork and installed it and it has been a HUGE improvement.

But just an FYI: the issue is not gone completely, as I still noticed some branches being deleted on Jenkins. I've only noticed this when there is an MR incoming or something has been merged, the plugin would (I assume) scan the repo and some of the branches are going missing. It's almost always the branches that are not receiving updated for a long time.

[cpt1gl0]

@Olcod

But just an FYI: the issue is not gone completely, as I still noticed some branches being deleted on Jenkins.

I think this is a different issue that should be treated separately. This issue is not about branches being deleted.

[cpt1gl0]

@baymac We will test the proposed changes on our build server to find out if they solve the issue and do not have any side effects. I will keep you updated about the results and ask you again for merging the PR if no issues occur.

I am not entirely sure if I want to adopt this plugin. I will think about it.

[drakenfly]

Wouldn't it be an easier solution to just fetch all merge requests after retrieving the list of OPENED ones?

List<MergeRequest> mrs = gitLabApi.getMergeRequestApi()
    .getMergeRequests(gitlabProject, Constants.MergeRequestState.OPENED);
mrs = mrs.stream().filter(mr -> mr.getSourceProjectId() != null)
    .map(mr -> {
        try {
            return gitLabApi.getMergeRequestApi().getMergeRequest(gitlabProject, mr.getIid());
        } catch (GitLabApiException e) {
            throw new RuntimeException(e);
        }
    })
    .collect(Collectors.toList());
request.setMergeRequests(mrs);

this should also have no other unforeseen side-effects.

[sb3lxb]

We run into this currently even with the trigger delay enabled and set to 30 seconds. For example, a multi-branch pipeline may be scheduled to run and a commit is made. The pipeline runs according to the commit it was triggered against however no additional pipeline is run for the newest commit that was just made.