Hangfire.Storage.SQLite icon indicating copy to clipboard operation
Hangfire.Storage.SQLite copied to clipboard

SQLiteDistributedLock doesn't play right with async

Open fnajera-rac-de opened this issue 1 year ago • 2 comments

The following snippet shows a (valid?) usage of SQLiteDistributedLock which leaves/can leave the lock alive forever:

public async Task Method(PerformContext context)
{
    using (context.Connection.AcquireDistributedLock(..., TimeSpan.FromMinutes(1)))
    {
        // await calls here
    }
}

Expected: since the lock is in a using block, it should be released when the using loses scope.

Actual: the lock is held indefinitely.

I believe the problem resides in the use of ThreadLocal in SQLiteDistributedLock.

https://github.com/raisedapp/Hangfire.Storage.SQLite/blob/251983b6c194b0098393e60b4936fd16d0e1671f/src/main/Hangfire.Storage.SQLite/SQLiteDistributedLock.cs#L17-L18

When using async, there's no guarantee that you resume in the same thread you were before - even less in ASP.NET Core where there's no sync context.

I'll try to add a test case and a fix for this issue.

fnajera-rac-de avatar Mar 03 '24 16:03 fnajera-rac-de

If I see that correctly, the distributed locking mechanism isn't event correct for single node servers

It should also block the current thread from re-entering a lock (no re-entrency) but it doesn't.

By making use of database commands the whole code could be simplified to...

  • a simple Insert statement, on collision by key retry until timeout
  • heartbeat: update where key and owner is ourself
  • remove: remove where key and owner is ourself

That way the collision checks and updates work atomically on the database level.

I could code something up in the upcoming days

kirides avatar Apr 01 '24 16:04 kirides

@kirides I did compare the sourcecode of other database plugins, they look to follow what you wrote, so that's probably the way to go. Unfortunately I don't currently have time to work on this.

fnajera-rac-de avatar Apr 02 '24 08:04 fnajera-rac-de