runner icon indicating copy to clipboard operation
runner copied to clipboard
verified publisher icon actions

Self host crush - is not authorized to access this resource

Open noamgreen opened this issue 1 year ago • 3 comments

Describe the bug I start running a job and i geting this error many times and the job is cancelled

System.UnauthorizedAccessException: Access to the path '/proc/554/oom_score_adj' is denied.

test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper] Starting process:
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]   File name: '/usr/bin/bash'
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]   Arguments: '--noprofile --norc -e -o pipefail /home/runner/_work/_temp/0e164abb-bd20-4274-91b5-72f26c8fae7f.sh'
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]   Working directory: '/home/runner/_work/devops'
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]   Require exit code zero: 'False'
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]   Encoding web name:  ; code page: ''
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]   Force kill process on cancellation: 'False'
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]   Redirected STDIN: 'False'
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]   Persist current code page: 'False'
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]   Keep redirected STDIN open: 'False'
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]   High priority process: 'False'
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper] Failed to update oom_score_adj for PID: 554.
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper] System.UnauthorizedAccessException: Access to the path '/proc/554/oom_score_adj' is denied.
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]  ---> System.IO.IOException: Permission denied
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]    --- End of inner exception stack trace ---
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]    at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
test-t82bf-runner-tzkbq runner [WORKER 2024-07-10 13:30:38Z INFO JobServerQueue] Try to append 1 batches web console lines for record 'd8be3eeb-808b-5cb1-9c71-884a43769e10', success rate: 1/1.
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]    at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]    at System.IO.Strategies.BufferedFileStreamStrategy.Dispose(Boolean disposing)
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]    at System.IO.StreamWriter.CloseStreamFromDispose(Boolean disposing)
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]    at System.IO.StreamWriter.Dispose(Boolean disposing)
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]    at System.IO.File.WriteAllText(String path, String contents)
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]    at GitHub.Runner.Sdk.ProcessInvoker.WriteProcessOomScoreAdj(Int32 processId, Int32 oomScoreAdj)


test-t82bf-runner-bzrls runner [RUNNER 2024-07-10 13:29:47Z WARN GitHubActionsService] GET request to https://pipelinesghubeus26.actions.githubusercontent.com/0TagnS0qaEOWaiMRmM6TEfnPxHtFutTo92gSnylMXf5cdGZrsW/_apis/distributedtask/pools/10/messages?sessionId=7259e804-7eac-4621-ba0e-6bcd61f8a395&lastMessageId=1&status=Busy&runnerVersion=2.317.0&os=Linux&architecture=X64&disableUpdate=true has been cancelled.
test-t82bf-runner-bzrls runner [RUNNER 2024-07-10 13:29:47Z INFO MessageListener] Get next message has been cancelled.
test-t82bf-runner-bzrls runner [RUNNER 2024-07-10 13:29:47Z INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished.
test-t82bf-runner-bzrls runner [RUNNER 2024-07-10 13:29:47Z INFO JobDispatcher] Ensure WorkerDispatcher for job 1cfa91de-17dd-561a-f553-3e6471df7df2 run to finish, cancel any running job.
test-t82bf-runner-bzrls runner [RUNNER 2024-07-10 13:29:47Z INFO JobDispatcher] Send job cancellation message to worker for job 1cfa91de-17dd-561a-f553-3e6471df7df2.
test-t82bf-runner-bzrls runner [RUNNER 2024-07-10 13:29:47Z INFO ProcessChannel] Sending message of length 0, with hash 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO ProcessChannel] Receiving message of length 0, with hash 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO Worker] Cancellation/Shutdown message received.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO HostContext] Runner will be shutdown for UserCancelled
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO JobServerQueue] Try to append 1 batches web console lines for record '1cfa91de-17dd-561a-f553-3e6471df7df2', success rate: 1/1.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO StepsRunner] Cancel current running step.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO ProcessInvokerWrapper] Sending SIGINT to process 175.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO ProcessInvokerWrapper] Successfully send SIGINT to process 175.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO ProcessInvokerWrapper] Waiting for process exit or 7.5 seconds after SIGINT signal fired.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO ProcessInvokerWrapper] Process exit successfully.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO ProcessInvokerWrapper] Process cancelled successfully through Ctrl+C/SIGINT.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO ProcessInvokerWrapper] Process Cancellation finished.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO ProcessInvokerWrapper] Finished process 175 with exit code 130, and elapsed time 00:00:48.9883086.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO CreateStepSummaryCommand] Step Summary file (/home/runner/_work/_temp/_runner_file_commands/step_summary_6931528a-638b-4f31-8479-e73982adfb14) is empty; skipping attachment upload
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z ERR  StepsRunner] Caught cancellation exception from step: System.OperationCanceledException: The operation was canceled.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z ERR  StepsRunner]    at System.Threading.CancellationToken.ThrowOperationCanceledException()
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z ERR  StepsRunner]    at GitHub.Runner.Sdk.ProcessInvoker.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Channel`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, CancellationToken cancellationToken)
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z ERR  StepsRunner]    at GitHub.Runner.Common.ProcessInvokerWrapper.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Channel`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, CancellationToken cancellationToken)
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z ERR  StepsRunner]    at GitHub.Runner.Worker.Handlers.DefaultStepHost.ExecuteAsync(IExecutionContext context, String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Boolean inheritConsoleHandler, String standardInInput, CancellationToken cancellationToken)
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z ERR  StepsRunner]    at GitHub.Runner.Worker.Handlers.ScriptHandler.RunAsync(ActionRunStage stage)
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z ERR  StepsRunner]    at GitHub.Runner.Worker.ActionRunner.RunAsync()
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z ERR  StepsRunner]    at GitHub.Runner.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO StepsRunner] Step result: Canceled```

{ "$id": "1", "innerException": null, "message": "The user 'System:PublicAccess;aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa' is not authorized to access this resource.", "typeName": "Microsoft.TeamFoundation.Framework.Server.UnauthorizedRequestException, Microsoft.TeamFoundation.Framework.Server", "typeKey": "UnauthorizedRequestException", "errorCode": 0, "eventId": 3000 }


 i am not sure why this happens  , working on eks + dind 

**To Reproduce**
Steps to reproduce the behavior:
NA 


## Runner Version and Platform
2.317.0
OS of the machine running the runner? OSX/Windows/Linux/...

## What's not working?
job is not finishing 

## Job Log Output
If applicable, include the relevant part of the job / step log output here.  All sensitive information should already be masked out, but please double-check before pasting here.

noamgreen avatar Jul 10 '24 13:07 noamgreen

So i did full checkup .... i can explain the issues but i cant explain the error . long story short the runner is burst the CPU, MEM to a point of OOM and if you use k8s you will star evection process

i think the runner needs to have some mem + cpu limits ( k8s "resources" will not work ) not sure why i see this CPU and MEM issues

noamgreen avatar Jul 14 '24 06:07 noamgreen

I had the same issue. I also saw some other issues in the same direction here, for conservation and also if someone else has this issue. I had to increase the shm-size of the docker container.

    container:
      image: your/image:latest
      options: --shm-size=4gb # otherwise we get a bus error (core dumped), do not ask me how long it took to debug this :-)

Maybe this helps someone :-)

joennlae avatar Aug 03 '24 16:08 joennlae

@joennlae i try to fix for me its not helped i think its in the runner code its try to update in /proc/ and its cant

noamgreen avatar Aug 18 '24 20:08 noamgreen

+1

mitrevb avatar Oct 02 '24 14:10 mitrevb

+1

noamgreen avatar Oct 16 '24 09:10 noamgreen

After increasing the memory of the runner and adding: - name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS value: "120" to the runner container, the issue got fixed

mitrevb avatar Oct 16 '24 09:10 mitrevb