bug: graceful shutdown is not so graceful

[StewartJingga]

Is your feature request related to a problem? Please describe. Compass panics on graceful shutdown with below stacktrace

panic: Drain() is not implemented

goroutine 56 [running]:
google.golang.org/grpc/internal/transport.(*serverHandlerTransport).Drain(0xc000e340a0?)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/transport/handler_server.go:439 +0x27
google.golang.org/grpc.(*Server).GracefulStop(0xc000821880)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1765 +0x3f4
github.com/odpf/compass/internal/server.Serve.func1()
	/home/runner/work/compass/compass/internal/server/server.go:157 +0x1bb
created by github.com/odpf/compass/internal/server.Serve
	/home/runner/work/compass/compass/internal/server/server.go:143 +0xaae

Checking from this issue, this looks like the expected behaviour when calling grpcServer.GracefulStop().

Compass calls grpcServer.GracefulStop() as part of graceful shutdown process as seen here.

Compass should not panic on graceful shutdown.

Describe the solution you'd like

1. Remove calling GracefulStop

We can remove calling grpcServer.GracefulStop() here as we are actually already handling shutdown gracefully.

2. Catch panic

Another approach if we just catch that panic here and recover properly.

example

defer func() {
	if r := recover(); r != nil {
		// handle panic
	}
}()
if grpcServer != nil {
	logger.Warn("stopping grpc server...")
	grpcServer.GracefulStop()
}