Lucene.Net.Store.Azure icon indicating copy to clipboard operation
Lucene.Net.Store.Azure copied to clipboard

Performance drops when using versions >= 4.8.0-beta015

Open ankrause opened this issue 1 year ago • 0 comments

We've noticed some performance issues since upgrading to the versions of this library which incorporate Azure.Storage.Blobs. This either manifests as significant drops in performance or a straight up hang/block. This only occurs when under load (or is only obvious under load).

From my investigation, this appears to be the result of the GetProperties calls that were added to replace FetchAttributes. There's 2 cases:

  1. The GetProperties call in AzureIndexInput occurs within the Mutex critical section here. GetProperties is a service call to the blob service. Previously, FetchAttributes was only executed prior to calling AzureIndexInput and the accessor for ContentLength within the mutex section was in-memory-only.
  2. The GetProperties call in AzureDirectory.OpenInput is now redundant. This doubles the number of requests made to the blob service during normal operation.

Simple fix for the core issue is to remove the redundant call in AzureDirectory and move the GetProperties call above GrabMutex in AzureIndexInput.

Secondarily, the response format from blob service has changed and the response object is a nullable response. HasValue should be checked according to the documentation, though a null response would be unexpected in most/all scenarios used by this repository. Generally, I would expect that there's either a value in Value or that the call threw RequestFailedException, so this is minor and potentially just cosmetic.

I may eventually be able to come back and submit a PR for this, documenting the issue until then.

ankrause avatar May 17 '24 23:05 ankrause