actions-runner-controller
actions-runner-controller copied to clipboard
actions
Runner connect error after successful Github authentication
Checks
- [X] I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
- [X] I am using charts that are officially provided
Controller Version
v0.25.2
Deployment Method
Other - kubectl apply
Checks
- [X] This isn't a question or user support case (For Q&A and community support, go to Discussions).
- [X] I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
To Reproduce
1. Create a private EKS cluster running Kubernetes v1.27 and using IPv4 cluster IP address family. I created the cluster with a managed node group of 3 nodes of `t3.medium`.
2. Setup an AWS Client VPN to be able to communicate with the cluster API inside the private subnet.
3. Create a Github App for my organization with appropriate permissions (as detailed in the docs).
3. Connect to the Kubernetes API via VPN.
4. Install `cert-manager` on the cluster.
5. Create the K8s secret `controller-manager` with the relevant fields for the Github app including app_id, app_installation_id, app_private_key.
6. Deploy ARC using `kubectl create -f https://github.com/actions/actions-runner-controller/releases/download/v0.25.2/actions-runner-controller.yaml`
7. Create a RunnerDeployment with the following:
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
name: example-runnerdeploy
spec:
replicas: 1
template:
spec:
repository: chariot-giving/chariot
8. Tail the pod logs. I see the pod successfully authenticates to Github and I even can confirm the runner is registered on Github (I can see it `Idle` state). After 1 minute, the status changes to `Offline` and I see the following in the logs:
Current runner version: '2.310.2' 2023-10-19 18:45:03Z: Listening for Jobs 2023-10-19 18:46:54Z: Runner connect error: The HTTP request timed out after 00:01:00.. Retrying until reconnected.
Describe the bug
The problem is that the runner disconnects from Github after 1 minute after it had previously authenticated and connected to Github.
- I've checked the network connection between the ec2 instance and Github's servers via AWS Reachability Analyzer and it confirms what I see initially - that the network is reachable.
- Every time I delete the pod and new one spins up, it reconnects to Github successfully and then eventually times out after 1 minute.
- I've been able to schedule a Github actions workflow job to the runner (immediately after I restarted it and it was still connected to Github) however the job failed to finish after the runner disconnected ->
offline. - I even exec'd into the runner pod and executed
nslookup github.comandcurl -v https://github.comand what I found was that before the connect time out, both commands returned successfully, however AFTER the connect time out, both commands fail.
Describe the expected behavior
The Github Runner stays connected to Github.
Additional Context
Here is the runner diagnostic logs:
[2023-10-19 19:24:27Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2023-10-19 19:24:47Z WARN GitHubActionsService] Attempt 1 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The operation will be retried in 10.802 seconds.
[2023-10-19 19:25:18Z WARN GitHubActionsService] Attempt 2 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The operation will be retried in 12.757 seconds.
[2023-10-19 19:25:51Z WARN GitHubActionsService] Attempt 3 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The operation will be retried in 17.742 seconds.
[2023-10-19 19:26:29Z ERR GitHubActionsService] Attempt 4 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The maximum number of attempts has been reached.
[2023-10-19 19:26:29Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2023-10-19 19:26:29Z INFO RunnerServer] Catch exception during connect. 4 attempt left.
[2023-10-19 19:26:29Z ERR RunnerServer] System.Net.Http.HttpRequestException: Resource temporarily unavailable (pipelinesghubeus8.actions.githubusercontent.com:443)
---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at GitHub.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at GitHub.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
at GitHub.Services.Location.Client.LocationHttpClient.GetConnectionDataAsync(ConnectOptions connectOptions, Int64 lastChangeId, CancellationToken cancellationToken, Object userState)
at GitHub.Services.WebApi.Location.VssServerDataProvider.GetConnectionDataAsync(ConnectOptions connectOptions, Int32 lastChangeId, CancellationToken cancellationToken)
at GitHub.Services.WebApi.Location.VssServerDataProvider.ConnectAsync(ConnectOptions connectOptions, CancellationToken cancellationToken)
at GitHub.Runner.Common.RunnerService.EstablishVssConnection(Uri serverUrl, VssCredentials credentials, TimeSpan timeout)
[2023-10-19 19:26:29Z ERR RunnerServer] #####################################################
[2023-10-19 19:26:29Z ERR RunnerServer] System.Net.Sockets.SocketException (11): Resource temporarily unavailable
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
[2023-10-19 19:26:29Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2023-10-19 19:26:49Z WARN GitHubActionsService] Attempt 1 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The operation will be retried in 10.825 seconds.
[2023-10-19 19:27:20Z WARN GitHubActionsService] Attempt 2 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The operation will be retried in 12.805 seconds.
[2023-10-19 19:27:52Z WARN GitHubActionsService] Attempt 3 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The operation will be retried in 15.922 seconds.
[2023-10-19 19:28:28Z ERR GitHubActionsService] Attempt 4 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The maximum number of attempts has been reached.
[2023-10-19 19:28:28Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2023-10-19 19:28:28Z INFO RunnerServer] Catch exception during connect. 3 attempt left.
[2023-10-19 19:28:28Z ERR RunnerServer] System.Net.Http.HttpRequestException: Resource temporarily unavailable (pipelinesghubeus8.actions.githubusercontent.com:443)
---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at GitHub.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at GitHub.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
at GitHub.Services.Location.Client.LocationHttpClient.GetConnectionDataAsync(ConnectOptions connectOptions, Int64 lastChangeId, CancellationToken cancellationToken, Object userState)
at GitHub.Services.WebApi.Location.VssServerDataProvider.GetConnectionDataAsync(ConnectOptions connectOptions, Int32 lastChangeId, CancellationToken cancellationToken)
at GitHub.Services.WebApi.Location.VssServerDataProvider.ConnectAsync(ConnectOptions connectOptions, CancellationToken cancellationToken)
at GitHub.Runner.Common.RunnerService.EstablishVssConnection(Uri serverUrl, VssCredentials credentials, TimeSpan timeout)
[2023-10-19 19:28:28Z ERR RunnerServer] #####################################################
[2023-10-19 19:28:28Z ERR RunnerServer] System.Net.Sockets.SocketException (11): Resource temporarily unavailable
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
[2023-10-19 19:28:29Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2023-10-19 19:28:49Z WARN GitHubActionsService] Attempt 1 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The operation will be retried in 10.964 seconds.
[2023-10-19 19:29:20Z WARN GitHubActionsService] Attempt 2 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The operation will be retried in 12.91 seconds.
[2023-10-19 19:29:52Z WARN GitHubActionsService] Attempt 3 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The operation will be retried in 15.782 seconds.
### Controller Logs
```shell
https://gist.github.com/magaldima/8af6b4db0e724ed5c87e9ba7dc558f0c
Runner Pod Logs
Defaulted container "runner" out of: runner, docker
2023-10-19 18:44:56.996 NOTICE --- Runner init started with pid 8
2023-10-19 18:44:57.8 DEBUG --- Github endpoint URL https://github.com/
2023-10-19 18:44:57.878 DEBUG --- Passing --ephemeral to config.sh to enable the ephemeral runner.
2023-10-19 18:44:57.889 DEBUG --- Configuring the runner.
--------------------------------------------------------------------------------
| ____ _ _ _ _ _ _ _ _ |
| / ___(_) |_| | | |_ _| |__ / \ ___| |_(_) ___ _ __ ___ |
| | | _| | __| |_| | | | | '_ \ / _ \ / __| __| |/ _ \| '_ \/ __| |
| | |_| | | |_| _ | |_| | |_) | / ___ \ (__| |_| | (_) | | | \__ \ |
| \____|_|\__|_| |_|\__,_|_.__/ /_/ \_\___|\__|_|\___/|_| |_|___/ |
| |
| Self-hosted runner registration |
| |
--------------------------------------------------------------------------------
# Authentication
√ Connected to GitHub
# Runner Registration
√ Runner successfully added
√ Runner connection is good
# Runner settings
√ Settings Saved.
2023-10-19 18:45:01.895 DEBUG --- Runner successfully configured.
{
"agentId": 30,
"agentName": "example-runnerdeploy-xzx8m-99d4f",
"poolId": 1,
"poolName": "Default",
"ephemeral": true,
"serverUrl": "https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/",
"gitHubUrl": "https://github.com/chariot-giving/chariot",
"workFolder": "/runner/_work"
2023-10-19 18:45:01.901 DEBUG --- Docker enabled runner detected and Docker daemon wait is enabled
2023-10-19 18:45:01.903 DEBUG --- Waiting until Docker is available or the timeout of 120 seconds is reached
}CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2023-10-19 18:45:01.998 NOTICE --- WARNING LATEST TAG HAS BEEN DEPRECATED. SEE GITHUB ISSUE FOR DETAILS:
2023-10-19 18:45:02.5 NOTICE --- https://github.com/actions/actions-runner-controller/issues/2056
√ Connected to GitHub
Current runner version: '2.310.2'
2023-10-19 18:45:03Z: Listening for Jobs
2023-10-19 18:46:54Z: Runner connect error: The HTTP request timed out after 00:01:00.. Retrying until reconnected.
Hello! Thank you for filing an issue.
The maintainers will triage your issue shortly.
In the meantime, please take a look at the troubleshooting guide for bug reports.
If this is a feature request, please review our contribution guidelines.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
@magaldima I am running into the same issue.
Helm chart: actions-runner-controller-0.23.3
App Version: 0.27.4
Were you able to resolve the issue?
Having same issue: 2023-10-19 18:45:03Z: Listening for Jobs 2023-10-19 18:46:54Z: Runner connect error: The HTTP request timed out after 00:01:00.. Retrying until reconnected.
I'm experiencing many issue(s) and it's not from today -- and I'm using ephemeral runners, which in theory should make some of these errors less frequent. Sometimes I experience 2-3 per day while sometimes it can be 10, 20, 30 in a single day.
Here's a collection of them:
Runner connect error: The HTTP request timed out after 00:01:00.. Retrying until reconnected.
Another one (remember that I'm using an ephemeral runner so the following message doesn't even make sense):
Cannot configure the runner because it is already configured. To reconfigure the runner, run 'config.cmd remove' or './config.sh remove' first."}
Or a random error such as:
The signature is not valid.
Or this one where (probably during a deployment) the API for registering a runner returns 404:
Http response code: NotFound from 'POST https://api.github.com/actions/runner-registration' (Request Id: XXXX:XXXXXX:XXXXXXX:XXXXXXX:XXXXXXXX)
{\"message\":\"Not Found\",\"documentation_url\":\"https://docs.github.com/rest\"}
Response status code does not indicate success: 404 (Not Found).
And most recently:
Failed to create session. Access denied. System:ServiceIdentity;DDDDDDDD-DDDD-DDDD-DDDD-DDDDDDDDDDDD needs View permissions to perform the action.
All of these are transient errors but enough to make many build fail. On the 404 error, after explaining in detail what the code was doing and how 99% of the calls were working on the runner startup script, GitHub support just told me it was due to a network connectivity issue on my side blocking GitHub.com from being accessed (!) 🤷
I'm also seeing this issue, both with the "legacy" runners (using summerwind/actions-runner) AND with the new style runner scale sets (using ghcr.io/actions/actions-runner:latest) -- @nikola-jokic. I ONLY see this issue when the dind sidecar is enabled. Our runners that do not use the docker daemon do not present this issue.
On the runner scale set versions the logs are a lot more verbose, but it is still very unclear what's happening.
Show logs
[RUNNER 2024-05-06 19:27:36Z INFO Terminal] WRITE LINE: Current runner version: '2.316.1'
Current runner version: '2.316.1'
[RUNNER 2024-05-06 19:27:36Z INFO Terminal] WRITE LINE: 2024-05-06 19:27:36Z: Listening for Jobs
2024-05-06 19:27:36Z: Listening for Jobs
[RUNNER 2024-05-06 19:27:36Z INFO JobDispatcher] Set runner/worker IPC timeout to 30 seconds.
[RUNNER 2024-05-06 19:29:26Z WARN GitHubActionsService] GET request to https://pipelinesghubeus25.actions.githubusercontent.com/2SWaK2eaVanB8UxNcI8NDUyxlDZQDpe73DUbnlAsAG8rPwLgHI/_apis/distributedtask/pools/14/messages?sessionId=5dd775d0-26e0-4a09-be79-687e567be2dd&status=Online&runnerVersion=2.316.1&os=Linux&architecture=X64&disableUpdate=true timed out after 60 seconds.
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] Catch exception during get next message.
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] System.TimeoutException: The HTTP request timed out after 00:01:00.
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] ---> System.Net.Sockets.SocketException (125): Operation canceled
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] --- End of inner exception stack trace ---
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] --- End of inner exception stack trace ---
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at GitHub.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] --- End of inner exception stack trace ---
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at GitHub.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at GitHub.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at GitHub.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at GitHub.Runner.Listener.MessageListener.GetNextMessageAsync(CancellationToken token)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] #####################################################
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] System.Threading.Tasks.TaskCanceledException: The operation was canceled.
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] ---> System.Net.Sockets.SocketException (125): Operation canceled
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] --- End of inner exception stack trace ---
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] --- End of inner exception stack trace ---
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at GitHub.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] #####################################################
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] ---> System.Net.Sockets.SocketException (125): Operation canceled
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] --- End of inner exception stack trace ---
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] #####################################################
[RUNNER 2024-05-06 19:29:26Z ERR MessageListener] System.Net.Sockets.SocketException (125): Operation canceled
[RUNNER 2024-05-06 19:29:26Z INFO MessageListener] Retriable exception: The HTTP request timed out after 00:01:00.
[RUNNER 2024-05-06 19:29:26Z ERR Terminal] WRITE ERROR: 2024-05-06 19:29:26Z: Runner connect error: The HTTP request timed out after 00:01:00.. Retrying until reconnected.
2024-05-06 19:29:26Z: Runner connect error: The HTTP request timed out after 00:01:00.. Retrying until reconnected.
[RUNNER 2024-05-06 19:29:26Z INFO RunnerServer] Refresh MessageQueue VssConnection to get on a different AFD node.
[RUNNER 2024-05-06 19:29:26Z INFO RunnerServer] EstablishVssConnection
[RUNNER 2024-05-06 19:29:26Z INFO RunnerServer] Establish connection with 60 seconds timeout.
[RUNNER 2024-05-06 19:29:26Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[RUNNER 2024-05-06 19:29:31Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[RUNNER 2024-05-06 19:29:32Z INFO MessageListener] Sleeping for 25.893 seconds before retrying.
And then the cycle keeps repeating itself. Oddly, I can kubectl exec into the runner container and curl that URL without issue, and the runners seem to pick up jobs also, so I don't know why it's complaining about an HTTP timeout.
I never saw this error until 2024-04-07.
Seeing this issue as well. I'm also able to kubectl exec and curl without issues
Encountering a similar issue: in my case, I ran ./config.sh --url {REPO_HERE} --token {TOKEN_HERE} --check, which yielded this output:
*********************************************************************************************************************
** Check: Internet Connection
** Description: Check if the Actions runner has internet access.
**********************************************************************************************************************
** **
** P A S S **
** **
**********************************************************************************************************************
** Log: <LOG_HERE>
**********************************************************************************************************************
**********************************************************************************************************************
** Check: GitHub Actions Connection
** Description: Check if the Actions runner has access to the GitHub Actions service.
**********************************************************************************************************************
** **
** F A I L **
** **
**********************************************************************************************************************
** Log: <LOG_HERE>
** Help Doc: https://github.com/actions/runner/blob/main/docs/checks/actions.md
**********************************************************************************************************************
**********************************************************************************************************************
** Check: Git Certificate/Proxy Validation
** Description: Check if the Git CLI can access GitHub.com or GitHub Enterprise Server.
**********************************************************************************************************************
** **
** P A S S **
** **
**********************************************************************************************************************
** Log: <LOG_HERE>
**********************************************************************************************************************
**********************************************************************************************************************
** Check: Node.js Certificate/Proxy Validation
** Description: Check if Node.js has access to GitHub.com or GitHub Enterprise Server.
**********************************************************************************************************************
** **
** P A S S **
** **
**********************************************************************************************************************
** Log: <LOG_HERE>
**********************************************************************************************************************
In the end, our team was able to resolve this issue by disabling IPV6, using this guide:
https://superuser.com/questions/575684/how-to-disable-ipv6-on-a-specific-interface-in-linux
In my case, it was probably some incompatibility with the operating system... After changing the operating system from Bottlerocket to AmazonLinux or Ubuntu, all the jobs started working again. Previously, it was crashing with timeout
