opensearch-project
[BUG] Tenant Calculation(?) quite expensive
What is the bug? After Upgrading from 2.18 to 2.19 we tried to drop our custom build security plugin which would return true as soon as it could determine that a user has all_access.
I'm currently unsure if this is a bottleneck of the optimized priviledge evaluation or a different root cause which occurs as we have arround 250 Tenants.
How can one reproduce the bug? Steps to reproduce the behavior:
- Create a high amount of tenants
- Try to Index a huge amount of data
- Observe slow ingestion and coordination nodes hot_threads showing attached thread stack
What is the expected behavior? In a write requests tenants should not be evaluated.
22.0% (109.9ms out of 500ms) cpu usage by thread 'opensearch[coordination-node][transport_worker][T#5]'
10/10 snapshots sharing following 130 elements
org.opensearch.security.securityconf.impl.SecurityDynamicConfiguration.getCEntries(SecurityDynamicConfiguration.java:288)
org.opensearch.security.securityconf.ConfigModelV7$TenantHolder.lambda$mapTenants$2(ConfigModelV7.java:1149)
org.opensearch.security.securityconf.ConfigModelV7$TenantHolder$$Lambda/0x00007fd2e9413748.accept(Unknown Source)
[email protected]/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
[email protected]/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
[email protected]/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
com.google.common.collect.CollectSpliterators$1.lambda$forEachRemaining$1(CollectSpliterators.java:128)
com.google.common.collect.CollectSpliterators$1$$Lambda/0x00007fd2e9416d88.accept(Unknown Source)
[email protected]/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1715)
com.google.common.collect.CollectSpliterators$1.forEachRemaining(CollectSpliterators.java:128)
com.google.common.collect.CollectSpliterators$FlatMapSpliterator.lambda$forEachRemaining$1(CollectSpliterators.java:377)
com.google.common.collect.CollectSpliterators$FlatMapSpliterator$$Lambda/0x00007fd2e9413dc0.accept(Unknown Source)
[email protected]/java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1858)
com.google.common.collect.CollectSpliterators$FlatMapSpliterator.forEachRemaining(CollectSpliterators.java:373)
[email protected]/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
[email protected]/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
[email protected]/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
[email protected]/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
[email protected]/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
[email protected]/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
org.opensearch.security.securityconf.ConfigModelV7$TenantHolder.mapTenants(ConfigModelV7.java:1135)
org.opensearch.security.securityconf.ConfigModelV7.mapTenants(ConfigModelV7.java:1285)
org.opensearch.security.privileges.PrivilegesEvaluatorImpl.mapTenants(PrivilegesEvaluatorImpl.java:598)
org.opensearch.security.privileges.PrivilegesEvaluatorImpl.evaluate(PrivilegesEvaluatorImpl.java:520)
org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:377)
org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:166)
app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:81)
app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:190)
app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:109)
Have the performance issues been present since the first time you have used OpenSearch, or was it a regression at some point?
Do you have figures regarding the slowdown comparing your custom plugin and the performance observed with OpenSearch 2.19?
Hi,
maybe I opened the issue too early. Sorry for disturbance.
We optimized the role assigned to our bulk index user to not have tenant_permissions '*' . This omits the mapTenants call as the user has no access to any tenant.
Your optimizations work quite fine - thanks a lot for this!
I'm unsure if this issue should stay open, as a user with tenant_permissions '*' in a environment with many tenants till has a slow(er) permission check or if we should close this as inperformant configuration for a bulk index user.
best regards and thanks again! Henry
maybe I opened the issue too early. Sorry for disturbance.
No worries, I do think that the hot thread dump indicates an issue. However, at the moment I believe it is a separate issue. That's why I am curious about the actual impact and whether it was a regression at some point.
@HenryTheSir FYI 2.19 has a separate optimization for all_access as well using the legacy authz code path: https://github.com/opensearch-project/security/pull/4926
@HenryTheSir thank you for filing this issue. It sounds like possibly optimizing the privileges evaluation flow has uncovered other issues in terms of tenancy calculations. Someone will take a look.
@HenryTheSir There will be an optimization around multi-tenancy in the 2.19.3 releae: https://github.com/opensearch-project/security/pull/5368