Unrecoverable panic error on orchestrator

[prakarsh-dt]

git-sensor received an event of a new commit and published an event on nats, devtron orchestrator tried to take action on the published event but found that the ci pipeline is already deleted and got stuck in an unrecoverable panic error.

Checklist:

• [x] I've searched in the docs and FAQ for my answer
• [x] I've included steps to reproduce the bug.

Describe the bug

This is an edge case scenario which can arise in cases where a commit was done to a branch on which a ci pipeline was configured and the ci-pipeline was immediately deleted before the orchestrator could take an action on it.

Logs

{"level":"error","ts":1632929358.1785128,"caller":"events/EventBuilder.go:125","msg":"found error on payload build for cd stages, skipping this error ","material":null,"stacktrace":"github.com/devtron-labs/devtron/client/events.(*EventSimpleFactoryImpl).BuildExtraCDData\n\t/go/src/github.com/devtron-labs/devtron/client/events/EventBuilder.go:125\ngithub.com/devtron-labs/devtron/pkg/app.(*AppServiceImpl).WriteCDSuccessEvent\n\t/go/src/github.com/devtron-labs/devtron/pkg/app/AppService.go:268"}
{"level":"info","ts":1632929358.2569418,"caller":"argocdServer/Version.go:48","msg":"connected argocd","serverVersion":"v1.8.1+c2547dc"}
{"level":"info","ts":1632929358.2570548,"caller":"devtron/App.go:82","msg":"starting server on ","port":8080}
panic: runtime error: index out of range [0] with length 0

goroutine 113 [running]:
github.com/devtron-labs/devtron/pkg/pipeline.(*WorkflowDagExecutorImpl).buildWFRequest(0xc00122a780, 0xc00165ef70, 0xc001633320, 0xc00062ef00, 0x1, 0xc00010e430, 0x0, 0xc000c26f00)
	/go/src/github.com/devtron-labs/devtron/pkg/pipeline/WorkflowDagExecutor.go:473 +0x199a
github.com/devtron-labs/devtron/pkg/pipeline.(*WorkflowDagExecutorImpl).TriggerPostStage(0xc00122a780, 0xc001633320, 0xc00062ef00, 0x1, 0x0, 0x0)
	/go/src/github.com/devtron-labs/devtron/pkg/pipeline/WorkflowDagExecutor.go:399 +0x1f1
github.com/devtron-labs/devtron/pkg/pipeline.(*WorkflowDagExecutorImpl).HandleDeploymentSuccessEvent(0xc00122a780, 0xc000d70bd0, 0x28, 0x10, 0x0)
	/go/src/github.com/devtron-labs/devtron/pkg/pipeline/WorkflowDagExecutor.go:593 +0x127
github.com/devtron-labs/devtron/api/router/pubsub.(*ApplicationStatusUpdateHandlerImpl).Subscribe.func1(0xc0013c3260)
	/go/src/github.com/devtron-labs/devtron/api/router/pubsub/ApplicationStatusUpdateHandler.go:88 +0x7c4
github.com/nats-io/stan%2ego.(*conn).processMsg(0xc00058a000, 0xc0014fdb30)
	/go/src/github.com/devtron-labs/devtron/vendor/github.com/nats-io/stan.go/stan.go:835 +0x242
github.com/nats-io/nats%2ego.(*Conn).waitForMsgs(0xc000656000, 0xc0002498c0)
	/go/src/github.com/devtron-labs/devtron/vendor/github.com/nats-io/nats.go/nats.go:2099 +0x342
created by github.com/nats-io/nats%2ego.(*Conn).subscribe
	/go/src/github.com/devtron-labs/devtron/vendor/github.com/nats-io/nats.go/nats.go:2856 +0x4cc

AB#414

[nishant-d]

logs form other instance

{"level":"error","ts":1632931272.8374634,"caller":"events/EventBuilder.go:125","msg":"found error on payload build for cd stages, skipping this error ","material":null,"stacktrace":"github.com/devtron-labs/devtron/client/events.(*EventSimpleFactoryImpl).BuildExtraCDData\n\t/go/src/github.com/devtron-labs/devtron/client/events/EventBuilder.go:125\ngithub.com/devtron-labs/devtron/pkg/app.(*AppServiceImpl).WriteCDSuccessEvent\n\t/go/src/github.com/devtron-labs/devtron/pkg/app/AppService.go:268"}
panic: runtime error: index out of range [0] with length 0

goroutine 162 [running]:
github.com/devtron-labs/devtron/pkg/pipeline.(*WorkflowDagExecutorImpl).buildWFRequest(0xc00083e140, 0xc0005861a0, 0xc000825050, 0xc000562000, 0x1, 0xc00060ca90, 0x0, 0xc0047b65a0)
	/go/src/github.com/devtron-labs/devtron/pkg/pipeline/WorkflowDagExecutor.go:473 +0x199a
github.com/devtron-labs/devtron/pkg/pipeline.(*WorkflowDagExecutorImpl).TriggerPostStage(0xc00083e140, 0xc000825050, 0xc000562000, 0x1, 0x0, 0x0)
	/go/src/github.com/devtron-labs/devtron/pkg/pipeline/WorkflowDagExecutor.go:399 +0x1f1
github.com/devtron-labs/devtron/pkg/pipeline.(*WorkflowDagExecutorImpl).HandleDeploymentSuccessEvent(0xc00083e140, 0xc0020c30e0, 0x28, 0x10, 0x0)
	/go/src/github.com/devtron-labs/devtron/pkg/pipeline/WorkflowDagExecutor.go:593 +0x127
github.com/devtron-labs/devtron/api/router/pubsub.(*ApplicationStatusUpdateHandlerImpl).Subscribe.func1(0xc0038f4420)
	/go/src/github.com/devtron-labs/devtron/api/router/pubsub/ApplicationStatusUpdateHandler.go:88 +0x7c4
github.com/nats-io/stan%2ego.(*conn).processMsg(0xc000546c40, 0xc0038c2a00)
	/go/src/github.com/devtron-labs/devtron/vendor/github.com/nats-io/stan.go/stan.go:835 +0x242
github.com/nats-io/nats%2ego.(*Conn).waitForMsgs(0xc0007fa000, 0xc0002c1140)
	/go/src/github.com/devtron-labs/devtron/vendor/github.com/nats-io/nats.go/nats.go:2099 +0x342
created by github.com/nats-io/nats%2ego.(*Conn).subscribe
	/go/src/github.com/devtron-labs/devtron/vendor/github.com/nats-io/nats.go/nats.go:2856 +0x4cc