UAA ratelimit is not handled and blocks integrationtests

[Lokowandtg]

If UAA is configured with ratelimiting, the integration tests do not work and productive code will fail in unpredictable ways when http 429 errors are returned. Fixed in #1313

[Kehrlann]

I'm getting into the details of this, and I have a few questions:

• What's the current rate limit config for your UAA instance?
• When you say "HTTP 429 are returned", you mean that CF-java-client gets HTTP 429 from the UAA, correct?
• If that's the case, which requests exactly are being throttled?

I'm surprised that the tests make many requests to the UAA in parallel, most of the time CFJC should have a cached token that can be reused across many requests ; and with more recent additions to our tests, we have only 11 classes with @CleanupCloudFoundryAfterClass which trigger the creation of a new client.

I'm provisioning to test this out myself.

[Kehrlann]

Initial testing suggests that the userId bean in the IntegrationTestConfiguration is a heavy user:

19:03:51.732    POST   /oauth/clients
19:03:52.460    POST   /Users
19:03:53.116    GET    /Groups?filter=displayName%20eq%20%22zones.read%22
19:03:53.287    GET    /Groups?filter=displayName%20eq%20%22routing.routes.read%22
19:03:53.303    GET    /Groups?filter=displayName%20eq%20%22clients.admin%22
19:03:53.306    GET    /Groups?filter=displayName%20eq%20%22routing.routes.write%22
19:03:53.310    GET    /Groups?filter=displayName%20eq%20%22scim.userids%22
19:03:53.313    GET    /Groups?filter=displayName%20eq%20%22network.admin%22
19:03:53.315    GET    /Groups?filter=displayName%20eq%20%22routing.router_groups.read%22
19:03:53.319    GET    /Groups?filter=displayName%20eq%20%22scim.zones%22
19:03:53.322    GET    /Groups?filter=displayName%20eq%20%22scim.create%22
19:03:53.324    GET    /Groups?filter=displayName%20eq%20%22scim.read%22
19:03:53.327    GET    /Groups?filter=displayName%20eq%20%22scim.invite%22
19:03:53.329    GET    /Groups?filter=displayName%20eq%20%22routing.router_groups.write%22
19:03:53.332    GET    /Groups?filter=displayName%20eq%20%22uaa.admin%22
19:03:53.334    GET    /Groups?filter=displayName%20eq%20%22cloud_controller.admin%22
19:03:53.336    GET    /Groups?filter=displayName%20eq%20%22zones.write%22
19:03:53.338    GET    /Groups?filter=displayName%20eq%20%22idps.write%22
19:03:53.341    GET    /Groups?filter=displayName%20eq%20%22clients.secret%22
19:03:53.351    GET    /Groups?filter=displayName%20eq%20%22scim.write%22
19:03:53.649    POST   /Groups/d351ac13-496c-4d63-a9d7-492c6d480f9f/members
19:03:53.797    POST   /Groups/61e301c3-395f-4ef6-b1c7-83f6192b40a4/members
19:03:53.811    POST   /Groups/fabf4bc4-53f8-464e-871d-e20951e1a5be/members
19:03:53.980    POST   /Groups/907adadd-1cc2-490e-948e-9f64656a30da/members
19:03:54.007    POST   /Groups/3dc60024-abb2-4a38-81a1-4e0bb81fe26e/members
19:03:54.010    POST   /Groups/200407fe-054e-4bf6-8eb0-8624058f0c37/members
19:03:54.016    POST   /Groups/12ff350f-83ae-47b0-82c1-989e632d1831/members
19:03:54.021    POST   /Groups/9068507f-d84a-4bff-bd24-84faefc9472b/members
19:03:54.026    POST   /Groups/bf7e75ad-e5cc-4d27-8c58-bb29fce264d6/members
19:03:54.032    POST   /Groups/c27f86a1-e6a9-47c5-af5a-7504fd0b8f52/members
19:03:54.033    POST   /Groups/4a817717-9450-4218-9d87-fe85ec005b00/members
19:03:54.048    POST   /Groups/6b9ad6f1-c049-490e-ba2a-3b3ea6f3892b/members
19:03:54.051    POST   /Groups/c0f57873-7489-46c9-8736-ee65a2fbf878/members
19:03:54.058    POST   /Groups/8f489e93-7cb7-41d8-80cd-6300ef02c2de/members
19:03:54.071    POST   /Groups/ad9d528c-7c0a-47cc-84b5-ceaa4ca9ce59/members
19:03:54.075    POST   /Groups/9fb7762b-861a-46d9-9305-0fddba342fd0/members
19:03:54.076    POST   /Groups/996bd947-6a19-40ba-a797-9a52ab6965af/members
19:03:54.081    POST   /Groups/b481bac2-475d-48fd-ae60-95f567024033/members

I'm going to run the full test suite and see if there are other problematic tests.

[Lokowandtg]

I think the configuration used in my instance is this: uaa_rate_limiter: credentialID: 'JWTjsonField:1:zid' limiterMappings: - name: AuthToken withCallerRemoteAddressID: 20r/s global: 60r/s pathSelectors: [ "equals:/oauth/token" ] - name: AuthAuthorize withCallerRemoteAddressID: 20r/s global: 60r/s pathSelectors: [ "equals:/oauth/authorize" ] - name: SCIM withCallerCredentialsID: 5r/s pathSelectors: - "startsWith:/Users" - "startsWith:/Groups" So any call to /Users and /Groups is limited to 5 requests per second. Other limits do not create problems in my case. My code checks all requests to "uaa" and does not interfere with requests to "api". In the integration tests, I only evaluate one variable 'UAA_API_REQUEST_LIMIT' and use that value for anything requesting uaa. That's a bit of "overblocking", but I did not find a way to access the configuration in a programatically way.

I will rebase and undo the changes that are related to bean @Qualifier.

[Kehrlann]

Thanks. For the record, here's the formatted version:

uaa_rate_limiter:
  credentialID: 'JWTjsonField:1:zid'
  limiterMappings:
    - name: AuthToken
      withCallerRemoteAddressID: 20r/s
      global: 60r/s
      pathSelectors:
        - "equals:/oauth/token"
    - name: AuthAuthorize
      withCallerRemoteAddressID: 20r/s
      global: 60r/s
      pathSelectors:
        - "equals:/oauth/authorize"
    - name: SCIM
      withCallerCredentialsID: 5r/s
      pathSelectors:
        - "startsWith:/Users"
        - "startsWith:/Groups"