excessive CPU usage when access control is enabled

[ViliusS]

When security access control is enabled SBA consumes 10x times more CPU than without it. For example on a very simple Spring Boot Admin application without security enabled and like 12 services connected I see 0.004 CPU usage, but with security enabled and with the same amount of services CPU consumption is 0.18 CPU It also looks like every additional service connected to Spring Boot Admin adds even more CPU usage considerably. I'm using reactive security sample from spring-boot-admin github repository, if that makes a difference.

[SteKoe]

Hi @ViliusS,

does the issue occur in general meaning, when SBA is just running (without accessing it via browser), or only when on a specific page (like details) in the browser? How did you measure the CPU usage? Did you just start up the example and called htop or similar to compare the CPU usage when running profile secure vs. insecure?

[ViliusS]

Issue occurs in general, even if I don't log in via browser or connect any other external monitoring.

I'm measuring CPU usage using Google GKE Monitoring of the service. Attached is a screenshot of CPU usage for SBA running with 12 services connected.

0.12 of CPU is a lot considering our Kubernetes cluster runs on Cascade Lake Xeon platform.

[erikpetzold]

Hi @ViliusS we will need some more information to reproduce the issue.

• Which version of Spring Boot Admin do you use (with which Spring Boot Version)?
• Where is security configured? For the user accessing the SBA UI or for the services that get monitored by SBA?
• Do you have ssl enabled?
• What kind of Security do you have configured? What authentication mechanism do you use (basic auth, ldap, ...)
• Did you override the polling interval which SBA Server uses to get the status of the apps spring.boot.admin.monitor.status-interval? This polling also happens when the UI is not opened, default is 10s. Maybe you can test with an increased interval to see if this is the cause.
• You wrote there is no external monitoring, but kubernetes is doing health checks (default every 3 seconds), right? Is the Health check secured? Is there something like prometheus scraping SBA?

Thank you

[ViliusS]

Hi @ViliusS we will need some more information to reproduce the issue.

• Which version of Spring Boot Admin do you use (with which Spring Boot Version)?

I'm running SBA 2.6.7 with Spring Boot 2.7.0 (there was no SBA 2.7.x at that time). If you think update could solve this issue I could try.

• Where is security configured? For the user accessing the SBA UI or for the services that get monitored by SBA?

Security is configured for accessing SBA UI which I suppose includes API:

spring.security.user.name=aaa
spring.security.user.password=bbb

Also on client side on services so they can access SBA API:

spring.boot.admin.client.url=http://pathtosba:8887/admin/
spring.boot.admin.client.instance.prefer-ip=true
spring.boot.admin.client.username=aaa
spring.boot.admin.client.password=bbb 

• Do you have ssl enabled?

No.

• What kind of Security do you have configured? What authentication mechanism do you use (basic auth, ldap, ...)

I'm using very simple basic auth with this reactive sample https://github.com/codecentric/spring-boot-admin/tree/master/spring-boot-admin-samples/spring-boot-admin-sample-reactive , minus LoggerNotification part.

• Did you override the polling interval which SBA Server uses to get the status of the apps spring.boot.admin.monitor.status-interval? This polling also happens when the UI is not opened, default is 10s. Maybe you can test with an increased interval to see if this is the cause.

I'm not overriding it.

• You wrote there is no external monitoring, but kubernetes is doing health checks (default every 3 seconds), right? Is the Health check secured? Is there something like prometheus scraping SBA?

Kubernetes checks /actuator/health/readiness every 5 seconds. Actuator endpoint is excluded from authentication as per sample code above. No prometheus scrapping or anything else exist. The only change I made to observe higher CPU usage is enable security. And I can reproduce lower CPU usage by disabling security.

Thank you

Thank you too!

[SteKoe]

Hi @ViliusS,

we have run some tests and figured out that every incoming request targeting SBA issues a CPU spike due to Spring Security. The first image shows a running SBA instance with security enabled using default settings (polling services, registration of services, health check of SBA). The CPU usage is higher than using no security as you already stated. We then lowered requests like registration requests from services (only applicable when self-registration is used) and saw a magnificent decrease in CPU usage (shown in figure 2).

Hence, please check what kind of requests in your setting are sent to SBA and check if the interval of requests can either be lowered or security can be disabled for these requests.

Regards, Stekoe

[erikpetzold]

One more thing: from the properties you posted we can see that you are using self registration from the clients.

This means the clients are telling SBA every 10 seconds they are there. You can try to set the property spring.boot.admin.client.period to e.g. 10 minutes and check if cpu usage is lowered. Of course it is not a good idea to have this high value permanently. These are the requests @SteKoe mentioned.

In a kubernetes setup you could also use the kubernetes discovery mechanism, without self registration. No client lib needed in your apps.

[ViliusS]

@SteKoe just to confirm I understood it right. It is possible to have SBA UI protected with password, but at the same time service registration and pooling without a password? It should be great for our use case, but I didn't find anything in the documentation about that.

@erikpetzold our developer machines do not run Kubernetes, so currently we are stuck with configuration which works for all environments (i.e. self-registration), but we have this task to solve for the mid-term future. I will experiment with intervals a little bit later, and I suspect it will help to alleviate the issue, but that's probably not the real cause of the issue.

Anyway, thank you both for your help so far.

[SteKoe]

@ViliusS that should be possible by adding /register endpoint to the list of permitAll() configurations in your spring security setup. Technically, self-registration is performed by a POST request every 10 seconds calling the API path /register. When removing security for this path, UI should still be protected.

[ViliusS]

Today I have upgraded to SBA 2.7.5.

I have also tried to add .pathMatchers(this.adminServer.path("/register")).permitAll() to my reactive server but this didn't help at all. Changing it to /instances/register doesn't help either. I also see that when I go to /register endpoint it returns 404 and only /instances/register responds, but I cannot verify if that's correct endpoint since /instances/anyurl always returns the same response.

Am I using the right endpoint? I didn't find any SBA API documentation.

Rising spring.boot.admin.client.period to 60 seconds alleviates the issue, however this doesn't solve it at the core. I would prefer just disable authentication for registration.

Also, I'm wondering what are production configuration recommendation for security enabled SBA deployments? I suppose in some cases allowing API access without authorization could be considered a security risk, and rising registration period to 60 seconds or more could mean no so quick monitoring notifications. Is Kubernetes Discovery, in such cases the only option, or maybe there is a way to pass some customization to Spring Security so it won't use so CPU expensive authentication?

[SteKoe]

Hi @ViliusS,

I am sorry but I mentioned the wrong endpoint. There is no /register endpoint. Please disable security for POST requests on /instances endpoint (see https://github.com/codecentric/spring-boot-admin/blob/master/spring-boot-admin-server/src/main/java/de/codecentric/boot/admin/server/web/InstancesController.java#L76).

As we know that this is not a perfect solution, we are currently checking the possibility to reduce costs of registration / availability requests for clients. Instead of registering over and over again, we strive to implement a heartbeat endpoint that works without the need to have security enabled. The heartbeat request will constantly check the availability of SBA-Server and trigger a reregistration of an instance if necessary.

[ViliusS]

@SteKoe thank you for the correction. An incoming feature you are describing looks like a really good solution. I will definitely wait for it.

In the meantime, I will leave spring.boot.admin.client.period at 60 seconds so that I can keep /instances endpoint protected still.