dotnet
distributedcache using SQL - Execution Timeout Expired.
We have been using SQL server distributed cache for few days and it does regularly stop working generating the exception reported at the bottom of this issue description.
Which is produced by the simple SetAsync call:
await SetAsync(q.id, bytes,
new DistributedCacheEntryOptions()
.SetAbsoluteExpiration(TimeSpan.FromMinutes(120))
);
I have logged the size of the bytes being saved and are in the range 30/50 Mbytes. The server is lab machine I am currently using for this purpose only and network and server load are out of question. It's in the same rack, same switch, and has no other load. I switched to ncache installed on the same machine to check if the problem was the sql or not, and with ncache I have no issue and the SetAsync takes no longer than a couple of seconds. I then switched to a different SQL server machine, and after a little while I started getting back the exceptions.
I would have liked to try extending the connection time out to check if it's taking much much longer than expected or just crashing/looping/something, but I can't figure out where command timeout can be set in services.AddDistributedSqlServerCache.
I don't know how to investigate that further.
And since it's working fine with ncache without changing anything in the code, and it's not working with either a lab SQL or a production one, I am lead to think there's something wrong in the SQL distributed cache.
Can please anyone help?
Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action
1 wrapCloseInAction) at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at Microsoft.Data.SqlClient.SqlCommand.InternalEndExecuteNonQuery(IAsyncResult asyncResult, Boolean isInternal, String endMethod) at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryInternal(IAsyncResult asyncResult) at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryAsync(IAsyncResult asyncResult) at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at Microsoft.Extensions.Caching.SqlServer.DatabaseOperations.SetCacheItemAsync(String key, Byte[] value, DistributedCacheEntryOptions options, CancellationToken token) at Microsoft.Extensions.Caching.SqlServer.SqlServerCache.SetAsync(String key, Byte[] value, DistributedCacheEntryOptions options, CancellationToken token) at umusa.DisposableCache.SetAsync(String key, Byte[] value, DistributedCacheEntryOptions options, CancellationToken token) in C:\websites-repo\gommecore\umusa\_Code\DisposableCache.cs:line 51 at umusa.DisposableCache.Refresh(List1 queue, gommeautoContext _db) in C:\websites-repo\gommecore\umusa_Code\DisposableCache.cs:line 333 at feeds.Controllers.importerController.AttemptToDoJob(FeedsUtility fu, FeedStep feed, Nullable1 distributorid, Boolean forcestart) in C:\websites-repo\gommecore\feeds\Controllers\importerController.cs:line 540 at feeds.Controllers.importerController.Trigger(String nonce, Int32 distributorid, String destination, String path, Boolean fromlocalfile, Boolean test, Boolean forcestart) in C:\websites-repo\gommecore\feeds\Controllers\importerController.cs:line 244 at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask1 actionResultValueTask) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext) at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at umusa.SerilogMiddleware.Invoke(HttpContext httpContext, IHostingEnvironment _env, IConfiguration _configuration, gommeautoContext dbcontext) in C:\websites-repo\gommecore\umusa\Serilog\CustomEnricher.cs:line 119
Hi,
I have a similar issue with Timeout, I could not figure out how to set the CommandTimeOut for services.AddDistributedSqlServerCache(options => )
In my case, I am getting this exception message at GetStringAsync instead
Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
---> System.ComponentModel.Win32Exception (258): Unknown error 258
at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)
at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at Microsoft.Data.SqlClient.SqlCommand.InternalEndExecuteReader(IAsyncResult asyncResult, Boolean isInternal, String endMethod)
at Microsoft.Data.SqlClient.SqlCommand.EndExecuteReaderInternal(IAsyncResult asyncResult)
at Microsoft.Data.SqlClient.SqlCommand.EndExecuteReaderAsync(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location ---
at Microsoft.Extensions.Caching.SqlServer.DatabaseOperations.GetCacheItemAsync(String key, Boolean includeValue, CancellationToken token)
at Microsoft.Extensions.Caching.SqlServer.DatabaseOperations.GetCacheItemAsync(String key, CancellationToken token)
at Microsoft.Extensions.Caching.SqlServer.SqlServerCache.GetAsync(String key, CancellationToken token)
at Microsoft.Extensions.Caching.Distributed.DistributedCacheExtensions.GetStringAsync(IDistributedCache cache, String key, CancellationToken token)
Any suggestions? and workaround
I also still experience this issue. The only "solution" I was able to come up with was to compress the bytes passed in to the Set methods. That took care of 99% of the timeout expired issues I was having with larger data values.
public static async System.Threading.Tasks.Task<byte[]> CompressByteArray(byte[] Data)
{
if (Data != null)
{
try
{
using (var objOutputData = new MemoryStream())
{
using (var objCompressor = new BufferedStream(new System.IO.Compression.GZipStream(objOutputData, System.IO.Compression.CompressionLevel.Fastest), 4096))
await objCompressor.WriteAsync(Data, 0, Data.Length).ConfigureAwait(false);
return objOutputData.ToArray();
}
}
catch (Exception ex) { System.Diagnostics.Debug.Print($"ERROR: {ex}"); }
}
return Data;
}
