Azure-Samples
[CSS] Throttling during provisioning
This issue is for a: (mark with an x)
- [x] bug report -> please search issues before submitting
- [ ] feature request
- [X] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)
Minimal steps to reproduce
Run the following command docker run -it -e "IotHubConnectionString=[redacted]" -e DeviceCount=1000 mcr.microsoft.com/oss/azure-samples/azureiot-simulatordeviceprovisioning
Provisioning crashes with a ThrottlingBacklogTimeout that causes the script to stop running.
Any log messages given by the failure
PS C:\Users\kyichii>
Starting device provisioning
2021-10-12T07:17:21.6511837Z: 100 devices have been created @ 63.20/sec
2021-10-12T07:17:21.8319529Z: 200 devices have been created @ 553.18/sec
Microsoft.Azure.Devices.Common.Exceptions.ThrottlingException: {"Message":"ErrorCode:ThrottlingBacklogTimeout;The request has been throttled. Wait 10 seconds and try again. Operation type: CRUD","ExceptionMessage":"Tracking ID:55f0784980964ecf83ec9bf4cce2bc7c-G:0-TimeStamp:10/12/2021 07:18:02"}
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func2 isMappedToException, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken) at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T1,T2](Uri requestUri, T1 entity, IDictionary2 errorMappingOverrides, IDictionary2 customHeaders, CancellationToken cancellationToken) at IotSimulatorDeviceProvisioning.Program.BulkCreateDevicesAsync(List1 devices, RegistryManager registryManager, DeviceProvisionStats stats) in /src/src/IotSimulatorDeviceProvisioning/Program.cs:line 163
Microsoft.Azure.Devices.Common.Exceptions.ThrottlingException: {"Message":"ErrorCode:ThrottlingBacklogTimeout;The request has been throttled. Wait 10 seconds and try again. Operation type: CRUD","ExceptionMessage":"Tracking ID:3ea3022e253e4d7b95ae629f77c7c403-G:0-TimeStamp:10/12/2021 07:18:02"}
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func2 isMappedToException, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken) at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T1,T2](Uri requestUri, T1 entity, IDictionary2 errorMappingOverrides, IDictionary2 customHeaders, CancellationToken cancellationToken) at IotSimulatorDeviceProvisioning.Program.BulkCreateDevicesAsync(List1 devices, RegistryManager registryManager, DeviceProvisionStats stats) in /src/src/IotSimulatorDeviceProvisioning/Program.cs:line 163
Microsoft.Azure.Devices.Common.Exceptions.ThrottlingException: {"Message":"ErrorCode:ThrottlingBacklogTimeout;The request has been throttled. Wait 10 seconds and try again. Operation type: CRUD","ExceptionMessage":"Tracking ID:474cf77e1c214b409f7429e4ed439825-G:0-TimeStamp:10/12/2021 07:18:02"}
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func2 isMappedToException, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken) at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T1,T2](Uri requestUri, T1 entity, IDictionary2 errorMappingOverrides, IDictionary2 customHeaders, CancellationToken cancellationToken) at IotSimulatorDeviceProvisioning.Program.BulkCreateDevicesAsync(List1 devices, RegistryManager registryManager, DeviceProvisionStats stats) in /src/src/IotSimulatorDeviceProvisioning/Program.cs:line 163
Microsoft.Azure.Devices.Common.Exceptions.ThrottlingException: {"Message":"ErrorCode:ThrottlingBacklogTimeout;The request has been throttled. Wait 10 seconds and try again. Operation type: CRUD","ExceptionMessage":"Tracking ID:a6b0b0fd21ab40c0a341fbe65a366d54-G:0-TimeStamp:10/12/2021 07:18:02"}
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func2 isMappedToException, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken) at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T1,T2](Uri requestUri, T1 entity, IDictionary2 errorMappingOverrides, IDictionary2 customHeaders, CancellationToken cancellationToken) at IotSimulatorDeviceProvisioning.Program.BulkCreateDevicesAsync(List1 devices, RegistryManager registryManager, DeviceProvisionStats stats) in /src/src/IotSimulatorDeviceProvisioning/Program.cs:line 163
Expected/desired behavior
This should not crash when throttling. Ideally it would allow us to set the SKU rate for the IoT Hub to avoid the throttling all together and test provisioning at the maximum limits of a specific iot hub
OS and Version?
Windows 10
Versions
Latest
Mention any other details that might be useful
This is behavior that inconsistently crashes but consistently throttles on lower end IoT Hub due to the registry operations. I am also attempting to get some additional logs from customers that are encountering the same issuje.
