[SCIM] Notification payload size too big

[gabrielemilan]

Hello guys,

we have an issue with group notification.

We use Azure Service Bus, and it has (in standard tier) only 256kb payload limitation.

In our system we have a group with multiple users and when we add new user the notification of group payload exceeds 256kb.

Do you have a "workaround" for this issue? Or do we need to use Premium Azure Service Bus? That supports 100mb payload size?

Thanks

[simpleidserver]

Hello,

There are two possible solutions to address your issue.

The first solution involves storing the content of the message in Blob storage and adding a reference to it within the message. If you agree with this approach, we can make some modifications to the SCIM project to store the message content in Blob storage. This will result in a smaller message that contains only a reference to the Blob.

The second solution is to split the message into chunks. Unfortunately, the MassTransit NuGet package doesn’t support this feature, nor is it mentioned on their website. If you feel more comfortable with this option, you could ask in their GitHub project whether message chunking is possible or if it might be supported in the future.

KR, SID

[gabrielemilan]

Ok, after few investigation on our side, I think that the best approach is to implement first solution, I mean store message to blob storage, if possible with a setting to enable or disable it.

Can you please make this change?

Thanks a lot.

[simpleidserver]

We will add this feature in the next release.

KR,

SID

[simpleidserver]

I made some modifications to the Release503 branch to support your scenario. Two new parameters have been added to the appsettings.json file. It should look something like this to store messages in Azure Blob Storage:

"MassTransitStorageConfiguration": {
  "IsEnabled": true,
  "Type": "AZURESTORAGE",
  "ConnectionString": "https://{accountName}.blob.core.windows.net"
}

Changes : https://github.com/simpleidserver/SimpleIdServer/commit/54084dfb045a76173169fe7d393fdaf7f469e2e4

[gabrielemilan]

Nice, I will try it soon

[polybogdan]

Hello ,

We tested the changes and all seems fine. Do you already have the plan for the release of the new version ?

Regards

[simpleidserver]

A pre-release package, version 5.0.3-rc1, has been published and is available for use.

[gabrielemilan]

Hello @simpleidserver , we noticed that the payload has data-ref attribute, but it also contains the "data" attribute in base64.

This can cause the exception related to the size bigger than 256kb, is it possible to remove the "data" attribute if data-ref is present?

"message": { "name": "RepresentationAddedEvent", "payload": { "data-ref": "https://---------.blob.core.windows.net/scim/y1374nrjredjceoyhknynyyyyy", "text": null, "data": "eyJUb2tlbiI6IiIsIklkIjoiZTNkYjQ5Y2MtYTVkZi00NWZiLTlmMzYtYjEzMzM3MzE4N2JlIiwiVmVyc2lvbiI6MCwiUmVzb3VyY2VUeXBlIjoiU2NpbVVzZXIiLCJTZXJpYWxpemVkUmVwcmVzZW50YXRpb24iOiJ7XG4gIFwiaWRcIjogXCJlM2RiNDljYy1hNWRmLTQ1ZmItOWYzNi1iMTMzMzczMTg3YmVcIixcbiAgXCJ1c2VyTmFtZVwiOiBcInRlc3RNYWJ1MThAdGVzdC5jb21cIixcbiAgXCJuYW1lXCI6IHtcbiAgICBcImZhbWlseU5hbWVcIjogXCJF0sXG4gIFwiZ3JvdXBzXCI6IFtdLFxuICBcImVudGl0bGVtZW50c1wiOiBbXSxcbiAgXCJtZXRhXCI6IHtcbiAgICBcInJlc291cmNlVHlwZVwiOiBcIlVzZXJcIixcbiAgICBcImNyZWF0ZWRcIjogXCIyMDI1LTA2LTI0VDA5OjQ3OjI4LjA5MDEzNTZaXCIsXG4gICAgXCJsYXN0TW9kaWZpZWRcIjogXCIyMDI1LTA2LTI0VDA5OjQ3OjI4LjA5MDE3NTNaXCIsXG4gICAgXCJ2ZXJzaW9uXCI6IDAsXG4gICAgXCJsb2NhdGlvblwiOiBcImh0dHA6Ly90cmF2ZWwtYXBpLXNjaW0uZGV2LmVudGVycHJpc2UudWVmYS5jb20vdjIvVXNlcnMvZTNkYjQ5Y2MtYTVkZi00NWZiLTlmMzYtYjEzMzM3MzE4N2JlXCJcbiAgfSxcbiAgXCJleHRlcm5hbElkXCI6IFwiMjAwMjRcIixcbiAgXCJzY2hlbWFzXCI6IFtcbiAgICBcInVybjppZXRmOnBhcmFtczpzY2ltOnNjaGVtYXM6Y29yZToyLjA6VXNlclwiXG4gIF1cbn0ifQ==" }

[simpleidserver]

Hello @gabrielemilan !

In your Program.cs file, can you add the following instruction and try again? MessageDataDefaults.AlwaysWriteToRepository = true; According to the documentation, when this property is set to true, the data will not be present in the JSON.