Jyoti Mahapatra

This is different from https://github.com/kubernetes-sigs/aws-iam-authenticator/issues/421 This one looks like happening with `WithLabelValues` ..which flow are you hitting the nil pointer in?

i didnt merge the fix i thought could fix it. Please do the flow where you hit the issue

Ideally the `upstream.Recv()` calls in xds-relay will be guided by [retry policies](https://github.com/grpc/proposal/blob/master/A6-client-retries.md#maximum-number-of-retries) set up like this https://github.com/grpc/grpc-go/blob/master/examples/features/retry/client/main.go#L51. Since `Recv` calls are idempotent, this the retry policy settings should help us...

That's a great discussion. Even the [grpc-middleware retry](https://github.com/grpc-ecosystem/go-grpc-middleware/blob/master/retry/retry.go#L75-L77) does not allow retry on clientstreams. From the upstreamclient perspective, an error which could not be transparently handled ([here](https://github.com/envoyproxy/xds-relay/blob/master/internal/app/upstream/client.go#L204), [here](https://github.com/envoyproxy/xds-relay/blob/master/internal/app/upstream/client.go#L177)) always causes...

We should explore the option. I'm not sure if we will have to build an application level buffer to save messages, or whether an interceptor introduces any complexity in case...

From my reading of [grpc-go](https://github.com/grpc/grpc-go/blob/master/clientconn.go#L1247-L1263), the connection failures are transparently handled by grpc. We should be ok with just implementing a retry interceptor for bidi rpc.

@jessicayuen We need some kind of monitor to make sure we alarm when cache drifts. We can make such a mechanism inhouse or think about this feature in the open...

Found the reason for this. The aggregator rule in our private repo specific to Lyft had a bug due to which eds requests were cached on service name. So when...

After adding rules to add resource name for eds, the cache is happy now. ``` - rules: - match: request_type_match: types: - "type.googleapis.com/envoy.api.v2.RouteConfiguration" - "type.googleapis.com/envoy.config.route.v3.RouteConfiguration" - "type.googleapis.com/envoy.api.v2.ClusterLoadAssignment" - "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment" result:...

An important aspect here is that these rules will possibly apply to all users of the project.