SQLiteDistributedLock doesn't play right with async
The following snippet shows a (valid?) usage of SQLiteDistributedLock which leaves/can leave the lock alive forever:
public async Task Method(PerformContext context)
{
using (context.Connection.AcquireDistributedLock(..., TimeSpan.FromMinutes(1)))
{
// await calls here
}
}
Expected: since the lock is in a using block, it should be released when the using loses scope.
Actual: the lock is held indefinitely.
I believe the problem resides in the use of ThreadLocal in SQLiteDistributedLock.
https://github.com/raisedapp/Hangfire.Storage.SQLite/blob/251983b6c194b0098393e60b4936fd16d0e1671f/src/main/Hangfire.Storage.SQLite/SQLiteDistributedLock.cs#L17-L18
When using async, there's no guarantee that you resume in the same thread you were before - even less in ASP.NET Core where there's no sync context.
I'll try to add a test case and a fix for this issue.
If I see that correctly, the distributed locking mechanism isn't event correct for single node servers
It should also block the current thread from re-entering a lock (no re-entrency) but it doesn't.
By making use of database commands the whole code could be simplified to...
- a simple Insert statement, on collision by key retry until timeout
- heartbeat: update where key and owner is ourself
- remove: remove where key and owner is ourself
That way the collision checks and updates work atomically on the database level.
I could code something up in the upcoming days
@kirides I did compare the sourcecode of other database plugins, they look to follow what you wrote, so that's probably the way to go. Unfortunately I don't currently have time to work on this.