ExitCode:45 Status:RemoteErrorResultStatus Err:context deadline exceeded retry budget exhausted (6 attempts)

[DarkMatterV]

the reproxy.WARNING has error log like:

W0117 17:33:34.051482   24671 server.go:347] a62xxxxx-9b08-4ca6-b59c-70xxxxxe5: Remote execution failed with &{ExitCode:45 Status:RemoteErrorResultStatus Err:rpc error: code = DeadlineExceeded desc = rpc error: code = DeadlineExceeded desc = retry budget exhausted (6 attempts): stream terminated by RST_STREAM with error code: CANCEL}, falling back to local.
W0117 17:33:34.051529   24671 server.go:348] a62xxxxx-9b08-4ca6-b59c-70xxxxxe5: stdout:
 stderr:
W0117 17:33:36.412261   24671 server.go:347] 21axxxx-f0dd-4b57-b820-7exxxxxxxe33: Remote execution failed with &{ExitCode:45 Status:RemoteErrorResultStatus Err:context deadline exceeded
retry budget exhausted (6 attempts)
github.com/bazelbuild/remote-apis-sdks/go/pkg/retry.WithPolicy
        external/com_github_bazelbuild_remote_apis_sdks/go/pkg/retry/retry.go:116
github.com/bazelbuild/remote-apis-sdks/go/pkg/client.(*Retrier).Do
        external/com_github_bazelbuild_remote_apis_sdks/go/pkg/client/client.go:753
github.com/bazelbuild/remote-apis-sdks/go/pkg/client.(*Client).GetActionResult
        external/com_github_bazelbuild_remote_apis_sdks/go/pkg/client/client.go:767
github.com/bazelbuild/remote-apis-sdks/go/pkg/client.(*Client).CheckActionCache
        external/com_github_bazelbuild_remote_apis_sdks/go/pkg/client/exec.go:101
github.com/bazelbuild/remote-apis-sdks/go/pkg/rexec.(*Context).GetCachedResult
        external/com_github_bazelbuild_remote_apis_sdks/go/pkg/rexec/rexec.go:213
github.com/bazelbuild/remote-apis-sdks/go/pkg/rexec.(*Client).Run
        external/com_github_bazelbuild_remote_apis_sdks/go/pkg/rexec/rexec.go:456
team/foundry-x/re-client/internal/pkg/reproxy.(*action).runRemote
        internal/pkg/reproxy/action.go:91
team/foundry-x/re-client/internal/pkg/reproxy.(*Server).runRemote
        internal/pkg/reproxy/server.go:559
team/foundry-x/re-client/internal/pkg/reproxy.(*Server).runAction
        internal/pkg/reproxy/server.go:344
team/foundry-x/re-client/internal/pkg/reproxy.(*Server).RunCommand
        internal/pkg/reproxy/server.go:209
team/foundry-x/re-client/api/proxy._Commands_RunCommand_Handler.func1
        bazel-out/k8-opt/bin/api/proxy/proxy_go_proto_/team/foundry-x/re-client/api/proxy/proxy.pb.go:1091
team/foundry-x/re-client/internal/pkg/interceptors.NewTruncInterceptor.func1
        internal/pkg/interceptors/truncate.go:23
google.golang.org/grpc.getChainUnaryHandler.func1
        external/org_golang_google_grpc/server.go:1019
team/foundry-x/re-client/internal/pkg/interceptors.UnaryServerInterceptor
        internal/pkg/interceptors/interceptors.go:31
google.golang.org/grpc.chainUnaryServerInterceptors.func1
        external/org_golang_google_grpc/server.go:1005
team/foundry-x/re-client/api/proxy._Commands_RunCommand_Handler
        bazel-out/k8-opt/bin/api/proxy/proxy_go_proto_/team/foundry-x/re-client/api/proxy/proxy.pb.go:1093
google.golang.org/grpc.(*Server).processUnaryRPC
        external/org_golang_google_grpc/server.go:1180
google.golang.org/grpc.(*Server).handleStream
        external/org_golang_google_grpc/server.go:1503
google.golang.org/grpc.(*Server).serveStreams.func1.2
        external/org_golang_google_grpc/server.go:843
runtime.goexit
        GOROOT/src/runtime/asm_amd64.s:1581
github.com/bazelbuild/remote-apis-sdks/go/pkg/client.statusWrap
        external/com_github_bazelbuild_remote_apis_sdks/go/pkg/client/client.go:980
github.com/bazelbuild/remote-apis-sdks/go/pkg/client.(*Client).GetActionResult
        external/com_github_bazelbuild_remote_apis_sdks/go/pkg/client/client.go:774
github.com/bazelbuild/remote-apis-sdks/go/pkg/client.(*Client).CheckActionCache
        external/com_github_bazelbuild_remote_apis_sdks/go/pkg/client/exec.go:101
github.com/bazelbuild/remote-apis-sdks/go/pkg/rexec.(*Context).GetCachedResult
        external/com_github_bazelbuild_remote_apis_sdks/go/pkg/rexec/rexec.go:213
github.com/bazelbuild/remote-apis-sdks/go/pkg/rexec.(*Client).Run
        external/com_github_bazelbuild_remote_apis_sdks/go/pkg/rexec/rexec.go:456
team/foundry-x/re-client/internal/pkg/reproxy.(*action).runRemote
        internal/pkg/reproxy/action.go:91
team/foundry-x/re-client/internal/pkg/reproxy.(*Server).runRemote
        internal/pkg/reproxy/server.go:559
team/foundry-x/re-client/internal/pkg/reproxy.(*Server).runAction
        internal/pkg/reproxy/server.go:344
team/foundry-x/re-client/internal/pkg/reproxy.(*Server).RunCommand
        internal/pkg/reproxy/server.go:209
team/foundry-x/re-client/api/proxy._Commands_RunCommand_Handler.func1
        bazel-out/k8-opt/bin/api/proxy/proxy_go_proto_/team/foundry-x/re-client/api/proxy/proxy.pb.go:1091
team/foundry-x/re-client/internal/pkg/interceptors.NewTruncInterceptor.func1
        internal/pkg/interceptors/truncate.go:23
google.golang.org/grpc.getChainUnaryHandler.func1
        external/org_golang_google_grpc/server.go:1019
team/foundry-x/re-client/internal/pkg/interceptors.UnaryServerInterceptor
        internal/pkg/interceptors/interceptors.go:31
google.golang.org/grpc.chainUnaryServerInterceptors.func1
        external/org_golang_google_grpc/server.go:1005
team/foundry-x/re-client/api/proxy._Commands_RunCommand_Handler
        bazel-out/k8-opt/bin/api/proxy/proxy_go_proto_/team/foundry-x/re-client/api/proxy/proxy.pb.go:1093
google.golang.org/grpc.(*Server).processUnaryRPC
        external/org_golang_google_grpc/server.go:1180
google.golang.org/grpc.(*Server).handleStream
        external/org_golang_google_grpc/server.go:1503
google.golang.org/grpc.(*Server).serveStreams.func1.2
        external/org_golang_google_grpc/server.go:843
runtime.goexit
        GOROOT/src/runtime/asm_amd64.s:1581
checking the action cache}, falling back to local.
W0117 17:33:36.412456   24671 server.go:348] 21axxxx-f0dd-4b57-b820-7exxxxxxxe33: stdout:
 stderr:

I'm not sure what cause it