atlantis icon indicating copy to clipboard operation
atlantis copied to clipboard
verified publisher icon runatlantis

Gitlab_client: Apply doesn't work on a branch cut from another until after making a new commit

Open jukie opened this issue 3 years ago • 4 comments

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

This is an unlikely edge case but I hit it nonetheless. I'm PoC'ing Atlantis at work and once I had a branch I was comfortable with I wanted to cut a new branch/PR so the output was easier to read. I could've also just deleted all the prior comments but this was faster. Upon doing so and opening the second PR without including any new changes, apply fails after a successful plan. I suspect this has something to do with the commit/branch lookup logic in the gitlab client.

Reproduction Steps

  1. Make a change on branch A
  2. Open Merge Request
  3. Run atlantis plan (may not be necessary)
  4. Close Merge Request
  5. On local cut new branch from A, e.g. git checkout -b newBranchB
  6. Push branch B and open Merge Request
  7. Run plan and apply

Logs

{"level":"info","ts":"2022-09-01T20:43:03.317Z","caller":"vcs/gitlab_client.go:105","msg":"determined GitLab is running version 15.2.2","json":{}}
{"level":"info","ts":"2022-09-01T20:43:03.469Z","caller":"server/server.go:869","msg":"Atlantis started - listening on port 4141","json":{}}
{"level":"info","ts":"2022-09-01T20:43:03.469Z","caller":"scheduled/executor_service.go:46","msg":"Scheduled Executor Service started","json":{}}
{"level":"info","ts":"2022-09-01T20:43:55.247Z","caller":"events/events_controller.go:533","msg":"parsed comment as command=\"plan\" verbose=false dir=\"\" workspace=\"\" project=\"\" flags=\"\"","json":{}}
{"level":"info","ts":"2022-09-01T20:43:55.579Z","caller":"events/project_command_builder.go:307","msg":"found no atlantis.yaml file","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:43:55.579Z","caller":"events/project_finder.go:57","msg":"filtered modified files to 4 .tf or terragrunt.hcl files: [aws/example-account/us-west-2/ebs-demo/ebs.tf aws/example-account/us-west-2/ebs-demo/locals.tf aws/example-account/us-west-2/ebs-demo/main.tf aws/example-account/aws-global-commons.tf]","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:43:55.580Z","caller":"events/project_finder.go:78","msg":"there are 2 modified project(s) at path(s): aws/example-account/us-west-2/ebs-demo, aws/example-account","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:43:55.580Z","caller":"events/project_command_builder.go:312","msg":"automatically determined that there were 2 projects modified in this pull request: [repofullname=jukie/atlantis-demo path=aws/example-account/us-west-2/ebs-demo repofullname=jukie/atlantis-demo path=aws/example-account]","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:43:55.584Z","caller":"events/project_command_context_builder.go:294","msg":"cannot determine which version to use from terraform configuration, detected 0 possibilities.","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:43:55.584Z","caller":"events/project_command_context_builder.go:294","msg":"cannot determine which version to use from terraform configuration, detected 0 possibilities.","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:43:55.704Z","caller":"events/project_locker.go:80","msg":"acquired lock with id \"jukie/atlantis-demo/aws/example-account/us-west-2/ebs-demo/default\"","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:43:59.915Z","caller":"models/shell_command_runner.go:156","msg":"successfully ran \"/usr/local/bin/terraform init -input=false -upgrade\" in \"/atlantis-data/repos/jukie/atlantis-demo/6/default/aws/example-account/us-west-2/ebs-demo\"","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:43:59.988Z","caller":"terraform/terraform_client.go:314","msg":"successfully ran \"/usr/local/bin/terraform workspace show\" in \"/atlantis-data/repos/jukie/atlantis-demo/6/default/aws/example-account/us-west-2/ebs-demo\"","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:44:08.463Z","caller":"models/shell_command_runner.go:156","msg":"successfully ran \"/usr/local/bin/terraform plan -input=false -refresh -out \\\"/atlantis-data/repos/jukie/atlantis-demo/6/default/aws/example-account/us-west-2/ebs-demo/default.tfplan\\\"\" in \"/atlantis-data/repos/jukie/atlantis-demo/6/default/aws/example-account/us-west-2/ebs-demo\"","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:44:08.566Z","caller":"events/instrumented_project_command_runner.go:53","msg":"plan success. output available at: https://gitlab.redacted.org/jukie/atlantis-demo/-/merge_requests/6","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:44:08.749Z","caller":"events/project_locker.go:80","msg":"acquired lock with id \"jukie/atlantis-demo/aws/example-account/default\"","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:44:08.778Z","caller":"models/shell_command_runner.go:156","msg":"successfully ran \"/usr/local/bin/terraform init -input=false -upgrade\" in \"/atlantis-data/repos/jukie/atlantis-demo/6/default/aws/example-account\"","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:44:08.879Z","caller":"terraform/terraform_client.go:314","msg":"successfully ran \"/usr/local/bin/terraform workspace show\" in \"/atlantis-data/repos/jukie/atlantis-demo/6/default/aws/example-account\"","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:44:08.977Z","caller":"models/shell_command_runner.go:156","msg":"successfully ran \"/usr/local/bin/terraform plan -input=false -refresh -out \\\"/atlantis-data/repos/jukie/atlantis-demo/6/default/aws/example-account/default.tfplan\\\"\" in \"/atlantis-data/repos/jukie/atlantis-demo/6/default/aws/example-account\"","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:44:09.088Z","caller":"events/instrumented_project_command_runner.go:53","msg":"plan success. output available at: https://gitlab.redacted.org/jukie/atlantis-demo/-/merge_requests/6","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:44:09.395Z","caller":"events/plan_command_runner.go:215","msg":"Running policy check for command=\"plan\" verbose=false dir=\"\" workspace=\"\" project=\"\" flags=\"\"","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:44:09.395Z","caller":"events/policy_check_command_runner.go:39","msg":"no projects to run policy_check in","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:44:36.781Z","caller":"events/events_controller.go:533","msg":"parsed comment as command=\"apply\" verbose=false dir=\"\" workspace=\"\" project=\"\" flags=\"\"","json":{}}
{"level":"warn","ts":"2022-09-01T20:44:36.948Z","caller":"events/apply_command_runner.go:101","msg":"unable to update commit status: POST https://gitlab.redacted.org/api/v4/projects/jukie/atlantis-demo/statuses/ae8fdf627b54a1fba8b916e01e01504c6f0e6865: 400 {message: Cannot transition status via :run from :running (Reason(s): Status cannot transition via \"run\")}","json":{"repo":"jukie/atlantis-demo","pull":"6"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:101\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:296"}
{"level":"error","ts":"2022-09-01T20:44:37.189Z","caller":"events/command_runner.go:427","msg":"PANIC: runtime error: invalid memory address or nil pointer dereference\nruntime/panic.go:221 (0x44a075)\nruntime/signal_unix.go:735 (0x44a045)\ngithub.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:209 (0xa5371a)\ngithub.com/runatlantis/atlantis/server/events/vcs/proxy.go:72 (0xa58824)\ngithub.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28 (0xa590c4)\ngithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:109 (0xc33dd5)\ngithub.com/runatlantis/atlantis/server/events/command_runner.go:296 (0xc38703)\nruntime/asm_amd64.s:1581 (0x4651a0)\n","json":{"repo":"jukie/atlantis-demo","pull":"6"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).logPanics\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:427\nruntime.gopanic\n\truntime/panic.go:1047\nruntime.panicmem\n\truntime/panic.go:221\nruntime.sigpanic\n\truntime/signal_unix.go:735\ngithub.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable\n\tgithub.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:209\ngithub.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable\n\tgithub.com/runatlantis/atlantis/server/events/vcs/proxy.go:72\ngithub.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus\n\tgithub.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28\ngithub.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:109\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:296"}

Stacktrace

runtime error: invalid memory address or nil pointer dereference
runtime/panic.go:221 (0x44a146)
runtime/signal_unix.go:735 (0x44a116)
github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:209 (0xa655ba)
github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72 (0xa6a5a4)
github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28 (0xa6ada4)
github.com/runatlantis/atlantis/server/events/apply_command_runner.go:109 (0xc32f15)
github.com/runatlantis/atlantis/server/events/command_runner.go:296 (0xc37843)
runtime/asm_amd64.s:1581 (0x465140)

Environment details

Self hosted gitlab using helm chart in EKS

Additional Context

jukie avatar Sep 01 '22 21:09 jukie

Issue is in mr.HeadPipeline.SHA at: https://github.com/runatlantis/atlantis/blob/71233ef3b6ed8081dc49dc33c45aceeceaaea753/server/events/vcs/gitlab_client.go#L209

jukie avatar Sep 02 '22 17:09 jukie

@jamengual now I'm wondering if this is a valid issue for Atlantis or if I should open another one at https://github.com/xanzy/go-gitlab? https://pkg.go.dev/github.com/xanzy/[email protected]?utm_source=gopls#MergeRequest.HeadPipeline

jukie avatar Sep 02 '22 17:09 jukie

we can leave this open and you can file in go-gitlab too and then report back here.

jamengual avatar Sep 02 '22 19:09 jamengual

After digging into it a bit more the issue is here and how Atlantis is using the client. In my example mr.HeadPipeline is nil so that just needs to be accounted for. Since no new commits have been pushed to branch-B, that MR doesn't have any directly associated pipelines. However, traversing to the pipelines from MR/branch-A will show a reference to MR/branch-b.

jukie avatar Sep 06 '22 17:09 jukie