PowerShell
Add 2 parameters to Start-ThreadJob: timeout seconds, priority
Summary of the new feature / enhancement
Add 2 parameters to Start-ThreadJob: timeout seconds, priority
Proposed technical implementation details (optional)
priority:
A single asynchronous task has no timeout and no priority， It is very original.
The priority range is 1-9, and the default is 5,priority 1 will be executed first.
timeoutseconds:
For that things, I think thread-timeout is a good medicine:
- can not stop Invoke-WebRequest / Invoke-RestMethod by ctrl + c .see: https://github.com/PowerShell/PowerShell/issues/16145
- “Shit-mountain-code”
- HANG ~~The regularity of unpredictable execution time~~
- Regular expression for a long time and unpredictable time.
- Code executed remotely.
- memory leaks
- cpu 100%
like： golang : “context.WithTimeout” python : timeout* in threading lib
It is better to add a Boolean type of parameter : -AutoRemoveJob whether to automatically remove threadjob after timeout, parameter likes “receive-job -AutoRemoveJob” In this way, users can only consider new thread and put ps1 script code.
both "ForEach-Object -Parallel" and "Start-ThreadJob" without timeout, This has seriously affected my development of "kasini3000" at present.
Powershell community do a "engine"，i do a "car". the car is kasini3000 and k4t . I love powershell. I want to give my opinion to the engine. I hope the engine factory can transform a better engine. I think timeoutseconds can make the ps1 script in the thread more robust. I hope powershell will become stronger and stronger.
- “Shit-mountain-code”
How does a timeout solve this? All you end up with is a section the task run and an indeterminate end state.
- HANG
If the thread is hung and unresponsive, you won't be able to cancel it with a cooperative cancellation mechanism (eg cancellation is an unwind performed in the thread). Preemptive termination leaves the process in an indeterminate state.
- The regularity of unpredictable execution time.
PowerShell is not a RTOS
- Code executed remotely.
Is the expectation that cancelling a local thread will also propagate to cancel remote threads, processes or any child local processes? What mechanism should be used to terminate a remote process?
- memory leaks
How does a timeouts solve memory leaks? If a thread is leaking memory and you stop it, it has still leaked memory, the memory won't be recovered. If the memory is recovered by the thread termination then it is not actually a leak, it is just memory usage.
- cpu 100%
100% CPU is not an issue if it is performing the required jobs. Cancelling a thread because you have high CPU just means you aren't performing the jobs you expected it to perform.
@kasini3000 Are you able to implement timed thread cancellation in your own code?
For example when you create a thread also create a CancellationToken and call CancellationTokenSource.CancelAfter to set your timeout. Then each thread has a cancellation token, use a static ThreadLocal<> to associate the thread token with each thread when they start.
Where ever you have a loop, then also check the state of the CancellationToken associated with the current thread and if cancelled throw an exception. Likewise anywhere that you Thread.Sleep(timeout), do cancellationToken.WaitHandle.WaitOne(timeout) and again throw an exception if the cancellation has occurred.
This could be implemented in a small C# class that you use in your PowerShell project.
Likewise, in the same module you can do
Thread current = Thread.CurrentThread;
current.Priority = ThreadPriority.BelowNormal;
To adjust the priority.
The priority range is 1-9, and the default is 5,priority 1 will be executed first.
Suggest use existing ThreadPriority enum
This is a pattern you can use now and also on original non-core PowerShell.exe
$cancellationTokenSource = New-Object -Type System.Threading.CancellationTokenSource
try
{
$cancellationTokenSource.CancelAfter(5000)
$cancellationToken = $cancellationTokenSource.Token
[Threading.Thread]::CurrentThread.Priority = [Threading.ThreadPriority]::BelowNormal
Write-Host $cancellationToken.IsCancellationRequested
$result = $cancellationToken.WaitHandle.WaitOne(10000)
Write-Host $result
}
finally
{
$cancellationTokenSource.Dispose()
}
timeoutseconds can make the ps1 script in the thread more robust. The example you share is very important to the powershell community and USERS !!! thanks @rhubarb-geek-nz !!!
I hope ANYONE can provide PR as soon as possible to realize this function
@rhubarb-geek-nz Could you show a small example how to use ThreadLocal with cancelation token inside of the Start-ThreadJob scriptblock?
The more I look at PowerShell threads the more I think it is better to forget about them actually being threads. I may be completely wrong about this but PowerShell threads are completely isolated from other threads, this is unlike a traditional threading system where threads share memory. Here it appears that threads use their own runspace and are isolated from others. From my tests, even $Global are not global, they seem effectively visible only with that thread/runspace. So within a thread script block you might as well just pass the token around as a variable or put it in $Global. This does mean you can use exactly the same mechanism for jobs that run in other processes or on other machines. Rather than think about it as job cancellation or job timeout, think of it in terms of what are the conditions under which my job will terminate. The token is then just part of the calculation.
That said, if they are threads running in the same process you might be able to pass a cancellation token created in the main thread or any other thread onto another thread via argument list into the parameters of a script block. This would not work for jobs in other process or remote jobs.
@stevenebutler I saw you adding a cancellation token to IWR. Do you have time to address this issue?
news：asp.net8 pr3 add timeout for per request
https://devblogs.microsoft.com/dotnet/asp-net-core-updates-in-dotnet-8-preview-3/
You can set request timeouts for individual endpoints, controllers, or dynamically per request.
To apply request timeouts, first add the request timeout services:
builder.Services.AddRequestTimeouts(); You can then apply request timeouts using middleware by calling :UseRequestTimeouts()
@stevenebutler I saw you adding a cancellation token to IWR. Do you have time to address this issue?
I don't really know much about this side of PowerShell so I don't think I can help here. If I read the issue correctly (and I may not) you are looking for something to cancel jobs that are running after a timeout.
If that's the case I don't think cancellation tokens are going to help you because they rely on the code being cancelled being co-operative in that it must periodically check for cancellation. If you're stuck inside third party code that does not use/check a cancellation token then it will not take any notice of the cancellation of the token and will continue to process until it finishes. The fix I applied was to make the code base in IWR (which we control) more co-operative by checking the cancellation token in places where it was not.
it must periodically check for cancellation.
sounds good. 1 Add an attribute to the job: endtime. The value is the current time plus [timespan] 'sometime' 2 Then regularly reduce the inspection and cancel it.
If you're stuck inside third party code that does not use/check a cancellation token then it will not take any notice of the cancellation of the token and will continue to process until it finishes.
Can the job be placed in a subfunction that enforces the use of a cancel token? There seems to be a saying that forces the insertion of a sleep some millisecond in the powershell job code. Canceling the token will be executed.
The only way I know of to deal with an uncancellable job is to do one of the following
- terminate the process (if running out of process);
- terminate the thread running it (very unreliable and causes major issues with stability and probably not even possible if it's a C# Task based job); or
- run it in a background task and just leave it running but give up waiting for the outcome once it goes past its timeout. This is obviously not ideal because the job will continue to consume resources but you no longer care about the outcome.
If you're trying to cancel arbitrary code, the safest way to do it is to run it in a separate process and terminate that once the timeout is reached. You should be able to do this already using start-job and related cmdlets.
If you're trying to cancel arbitrary code, the safest way to do it is to run it in a separate process and terminate that once the timeout is reached
Even if you don't want to help PR, don't say that. IWR's timeout does not start a new process and can also terminate threads. no new thread in IWR ？i don‘t know.
The thread must be able to be terminated. this issue only want adding a timeout-- If a thread may never be terminated, who dares to use it?
golang : “context.WithTimeout” python : timeout* in threading lib
this issue force me to use Python or Golang
Iwr isn't terminating a thread it is cooperatively checking the cancellation token and throwing an exception of it is cancelled. This is the safe way to do cancellation in c#.
If you control the code you're running in a thread you can pass it a cancellation token and check it in your code whenever doing a long running operation. If you don't control the code you need it to be checking the cancellation token or it won't cancel cooperatively.
The c# APIs all check cancellation token when they're doing io if you use the right overrides (those that accept a token).
You can cancel a thread but tasks run on the thread pool which I would assume makes it hard to know which thread to cancel and they aren't designed to work this way. Thread cancellation is designed to work with known threads but I still think using it is not recommended.
Note I am not speaking for PowerShell team as I have only just started contributing to it as a community member. I could be wrong and may just not understand what you are trying to do. Unfortunately this is not something I can help with.
The thread must be able to be terminated. this issue only want adding a timeout-- If a thread may never be terminated, who dares to use it?
https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-terminatethread
TerminateThread is a dangerous function that should only be used in the most extreme cases. You should call TerminateThread only if you know exactly what the target thread is doing, and you control all of the code that the target thread could possibly be running at the time of the termination.
With C#, PowerShell and arbitrary tasks these conditions are not met.
@rhubarb-geek-nz
1 powershell has the function of terminating （runspace）threads. So don't tell me that it's not possible or not recommended.
Although I don't understand the details, I think ending a thread is easy. If the process is a soldier, can disabled soldiers without arms continue to fight? The answer is: of course it can.
Think: Some of the vegetables you eat every day have rotten leaves, and the good parts of the vegetables leaves are also discarded by you. I suggest you endure rotten leaves and not discard them.
in other words: 2.1 Try to end the thread gracefully. 2.2 If the graceful termination of a thread fails and the timeout is exceeded, the thread will be forcibly terminated. 2.3 Mark the process as deformity. Waiting for user processing. 2.4 Users regularly detect process disabilities and can rotate processes as early as possible. Or do nothing.
Nothing is impossible, only unexpected. The underlying library does not provide functionality, the process can construct its own data structure.
So don't tell me that it's not possible or not recommended.
https://learn.microsoft.com/en-us/dotnet/standard/threading/destroying-threads
The Thread.Abort method is not supported in .NET 5 (including .NET Core) and later versions. If you need to terminate the execution of third-party code forcibly in .NET 5+, run it in the separate process and use Process.Kill.
That is the C# advice, run the code in a separate process and kill the process.
Although I don't understand the details, I think ending a thread is easy.
What we are saying is you can use a cancellation token and regularly check in the thread if it is time to end and then throw an exception to end the thread. This is supported and is cooperative. I have posted example code here of how to do this.
The current problem is that in PowerShell there is no agreed per-thread cancellation token to check or any contract that PowerShell code has to to do this.
What is not recommended is the hard termination of an unresponsive thread.
@kasini3000
1 powershell has the function of terminating （runspace）threads. So don't tell me that it's not possible or not recommended.
Can you tell us what these functions are that you are referring to so that we are talking about the same thing? A URL to the Microsoft documentation would be good.
@rhubarb-geek-nz
Start-ThreadJob {Start-Sleep -Seconds 999 }
1..2 |ForEach-Object -Parallel { start-sleep -Seconds 999 } -AsJob
get-job |Remove-Job -Force
Lose some, even most, and then continue. You should understand that everything is not perfect. Even if one wheel is lost, a car with three wheels can still reach the finish line.
You cannot throw away the entire apple just because there are wormholes on it. The correct approach is to separate the wormhole and continue eating the apple.
@kasini3000 Thanks for that. I wrote a simple test where the thread writes to a file in a simple loop and then is terminated.
#!/usr/bin/env pwsh
$job = Start-ThreadJob -ScriptBlock {
$i = 0
try
{
try
{
while ($i -lt 60)
{
Add-Content -Path 'thread.log' -Value "$i"
Start-Sleep -Seconds 1
$i = $i+1
}
}
finally
{
Add-Content -Path 'thread.log' -Value "finally $i"
}
}
catch
{
Add-Content -Path 'thread.log' -Value "catch $PSItem"
}
}
Start-Sleep -Seconds 5
$job | Remove-Job -Force
Start-Sleep -Seconds 5
Get-Content -Path 'thread.log'
The output was
0
1
2
3
4
finally 4
What that shows is that the thread was stopped in a controlled manner because the finally was called while $i was still 4.
What was interesting was finally was called, but the catch was not.
If you remove the -Force option then you get
Line |
34 | $job | Remove-Job
| ~~~~~~~~~~
| The command cannot remove the job with the job ID 1 because the job is not finished. To remove the job, first stop the job, or use the Force parameter. (Parameter 'Job')
So it cannot stop a job that is still running, however if the thread is still running when the main thread ends then it is presumably stopped with with Remove-Job -Force because the finally is called before the process terminates.
If that is enough for you then you have your building blocks.
What is you conclusion? can soldiers continue fight without 1 arm?
Nobody is loosing any limbs or having any threads terminated!
I wrote a custom PSCmdlet to determine what was going on. What is happening is that when the jobs are being "forcibly" removed it is still being done very politely.
I wrote a PSCmdlet that did an endless loop within the ProcessRecord. When the job was forcibly removed what happened was the PSCmdlet.StopProcessing() was called. The idea being you use this like an interrupt handler to stop your long running task. But if you don't implement StopProcessing then nothing bad happens (yet) and the system continues to wait for the ProcessRecord to complete. As the processing was still continuing no finally handlers ran.
So far it confirms our understanding that no threads are terminated, all the job removal and attempts to stop long running PSCmdlet are are being done cooperatively.
Finally, if you leave the Cmdlets still running in the threads that it can't stop then when you exit pwsh it then hangs while it waits for cmdlets that never end. You then have to terminate pwsh with pkill or similar.
I have published the source to my simple PSCmdlet so you can also play with it to see how the the Remove-Job -Force behaves.
no threads are terminated .
;( Threads cannot be forcibly stopped or automatically stopped by timeout, which is foolish. I will continue to find the way out
Refer to Golang's approach ： WithTimeout https://github.com/golang/go/blob/master/src/context/context.go
package main
import (
"context"
"fmt"
"time"
)
func longRunningTask(ctx context.Context) {
select {
case <-time.After(99 * time.Second):
fmt.Println("Task complete")
case <-ctx.Done():
fmt.Println("Task cancelled")
}
}
func main() {
ctx := context.Background()
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
go longRunningTask(ctx)
fmt.Println("Task start")
select {
case <-time.After(8 * time.Second):
fmt.Println("Program end")
}
}
I have found many stop thread powershell codes, which indicates that this is a common requirement. I will build my own application based on this. Coming soon.
https://github.com/proxb/PInvoke/blob/master/Stop-Thread.ps1
https://github.com/hlldz/Phant0m/blob/master/old/Invoke-Phant0m.ps1#L1054
https://github.com/das-lab/mpsd/blob/main/powershell_benign_dataset/2760.ps1
https://github.com/vector-sec/ACE/blob/master/ACE-Management/PS-ACE/Scripts/ACE_Get-InjectedThread.ps1
All your links use the Win32 TerminateThread API so are neither recommended or portable.
The Microsoft recommendation for Cancellation in Managed Threads is to use CancellationTokens exactly as we have been proposing here.
I recommend to use threads for code that you trust and that use CancellationTokens or have implemented the Cmdlet.StopProcessing correctly.
For all other code that you may need to cancel; run it in a separate process as recommended by Microsoft.
Note that all external programs that you run, eg "ssh", "nslookup", "ping" etc are already running in separate processes so should not be a problem.
The process can hang, so someone invented the kill command. Your computer can hang, so someone invented the reset key. Next, go to glibc and find a way to stop the thread. I eat every day, cook often, and it's a common thing to remove rotten leaves from a vegetable. (I allow the loss of some good leaves, just like allowing thread termination to allow a small amount of data loss.) Well, leave foolishness and find a way out.
The process can hang, so someone invented the kill command.
First computers could only run on program at a time. Later computers supported processes to varying levels of degree to allow multiple programs to run on the expensive hardware.
If you do not have pre-emptive multi-tasking and memory protection then all other processes are at risk from the worst program. Compare Windows 3.1 to OS/2, or Windows 95 to Windows NT, the system with process and kernel isolation and strong memory protection provides the most reliable system. Consider Windows 3.1, Windows 95 and any Macintosh System below 10 were absolutely terrible in terms of process isolation. Programming errors could cause memory corruption in any process.
The game changer in reliability of systems was the introduction of virtual memory, preemptive multitasking and memory protection. So as you mentioned the kill command. The killing of processes is incredibly efficient because there is no cleanup of the process to be done. All that happens is the open file handles are closed, the memory for the entire process marked as available and a signal raised to indicate the process is finished.
Now if you look at how UNIX shells operate, they don't need to use threads to achieve efficient pipelines and parallelism. They work using fork and copy-on-write, so rather than creating a thread the entire process is cloned and two versions of the same program run using the same memory but with memory protection between them. Only on memory-write is new memory allocated to manage the difference in memory state on a page by page basis. Either could terminate and the other would be unaffected, except for the SIGCHLD signal going to the parent.
Threading in a single process is much harder. It is hard to write robust preemptive multithreading code. It is hard when you write the entire program, it is even harder when you then include 3rd party code into your process which does not abide by the rules.
So I will keep mentioning why TerminateThread is so bad, it completely trashes your ability to maintain a consistent memory image. Let's ask Raymond what happens when a thread is terminated and it releases a mutex. Spoiler, you’re in big trouble.
The TerminateThread does not care if you are running PowerShell script, C# code or native C or C++ code. It will terminate it all without giving the code the opportunity to either protect itself or clean up.
If you want to put this in perspective, this is an ideal way to corrupt an SQLite database. The updates to the database were taking too long so terminate the thread! You risk corrupting the structures used to manage the data and transaction control.
In the POSIX world pthread_cancel is used terminate a thread. The way that code protects itself from this is using pthread_setcancelstate to protect critical sections. In this case the value of PTHREAD_CANCEL_DISABLE will prevent the cancellation from occurring. This is again a cooperative model, pthread_cancel simply marks that a thread should terminate but requires the thread to perform the termination itself by exiting.