graylog2-server Event definitions using many streams can cause HTTP 413 error on Elasticsearch

Expected Behavior

Event definitions involving many / all streams should work.

Current Behavior

We noticed the following error when using events involving many streams:

Caused by: org.graylog.shaded.elasticsearch7.org.elasticsearch.client.ResponseException: method [POST], host [https://<REMOVED>], URI [/gl_windows_security_49,gl_windows_common_4,gl_windows_common_3,gl_okta_7,gl_okta_6,gl_okta_5,gl_okta_4,gl_okta_9,gl_okta_8,gl_okta_3,gl_okta_2,gl_okta_1,gl_okta_0,gl_cisco_36,gl_cisco_35,gl_cisco_34,gl_cisco_33,gl_cisco_39,gl_cisco_38,gl_cisco_37,gl_cisco_43,gl_cisco_42,gl_cisco_41,gl_cisco_40,gl_cisco_47,gl_cisco_46,gl_cisco_45,gl_cisco_44,gl_cisco_49,gl_cisco_48,gl_cisco_50,gl_cisco_19,gl_cisco_14,gl_cisco_13,gl_cisco_12,gl_cisco_11,gl_cisco_18,gl_cisco_17,gl_cisco_16,gl_cisco_15,gl_cisco_21,gl_sysmon_280,gl_cisco_20,gl_sysmon_281,gl_sysmon_282,gl_sysmon_283,gl_sysmon_284,gl_sysmon_285,gl_sysmon_286,gl_sysmon_287,gl_sysmon_288,gl_sysmon_289,gl_windows_security_13,gl_windows_security_10,gl_windows_security_11,gl_cisco_25,gl_cisco_24,gl_cisco_23,gl_cisco_22,gl_cisco_29,gl_cisco_28,gl_cisco_27,gl_cisco_26,gl_cisco_32,gl_cisco_31,gl_sysmon_270,gl_cisco_30,gl_sysmon_271,gl_sysmon_272,gl_sysmon_273,gl_sysmon_274,gl_sysmon_275,gl_sysmon_276,gl_sysmon_277,gl_sysmon_278,gl_sysmon_279,gl-system-events_315,gl-system-events_314,gl-system-events_313,gl-system-events_312,gl-system-events_311,gl-system-events_310,gl-system-events_319,gl-system-events_318,gl-system-events_317,gl-system-events_316,gl-system-events_304,gl-system-events_303,gl-system-events_302,gl-system-events_301,gl_sysmon_290,gl-system-events_300,gl_cisco_10,gl_sysmon_291,gl_sysmon_292,gl_sysmon_293,gl_sysmon_294,gl_sysmon_295,gl_sysmon_296,gl_sysmon_297,gl-system-events_309,gl_sysmon_298,gl-system-events_308,gl_sysmon_299,gl-system-events_307,gl-system-events_306,gl-system-events_305,gl_linux_auditbeat_47,gl_windowsevent_296,gl_linux_auditbeat_48,gl_windowsevent_295,gl_linux_auditbeat_45,gl_windowsevent_298,gl_linux_auditbeat_46,gl_windowsevent_297,gl_linux_auditbeat_43,gl_linux_auditbeat_44,gl_windowsevent_299,gl_linux_auditbeat_41,gl_linux_auditbeat_42,gl-system-events_340,gl_windowssecurity_308,gl_windowssecurity_307,gl-events_298,gl_windowssecurity_309,gl_windowssecurity_304,gl_windowssecurity_303,gl_linux_auditbeat_49,gl_windowssecurity_306,gl_windowssecurity_305,gl-system-events_337,gl_windowssecurity_311,gl-system-events_336,gl_windowssecurity_310,gl-system-events_335,gl_windowssecurity_313,gl-system-events_334,gl_windowssecurity_312,gl-events_291,gl-system-events_333,gl-system-events_332,gl-system-events_331,gl-system-events_330,gl_linux_auditbeat_40,gl-system-events_339,gl-system-events_338,gl_linux_auditbeat_36,gl_windowsevent_285,gl_linux_auditbeat_37,gl_windowsevent_284,gl_linux_auditbeat_34,gl_windowsevent_287,gl_linux_auditbeat_35,gl_windowsevent_286,gl_linux_auditbeat_32,gl_windowsevent_289,gl_linux_auditbeat_33,gl_windowsevent_288,gl_linux_auditbeat_30,gl_linux_auditbeat_31,gl_linux_auditbeat_38,gl_linux_auditbeat_39,gl-system-events_326,gl_windowssecurity_300,gl-system-events_325,gl-system-events_324,gl_windowssecurity_302,gl-system-events_323,gl_windowssecurity_301,gl-system-events_322,gl-system-events_321,gl-system-events_320,gl_windowsevent_290,gl_windowsevent_292,gl-system-events_329,gl_windowsevent_291,gl-system-events_328,gl_windowsevent_294,gl-system-events_327,gl_windowsevent_293,gl_linux_auditbeat_25,gl_linux_auditbeat_26,gl_linux_auditbeat_23,gl_linux_auditbeat_24,gl_linux_auditbeat_21,gl_linux_auditbeat_22,gl_linux_auditbeat_20,gl_windowssecurity_329,gl_linux_auditbeat_29,gl_windowssecurity_326,gl_windowssecurity_325,gl_linux_auditbeat_27,gl_windowssecurity_328,gl_linux_auditbeat_28,gl_windowssecurity_327,gl_windowssecurity_333,gl_windowssecurity_332,gl_windowssecurity_335,gl_windowssecurity_334,gl_sysmon_260,gl_windowssecurity_331,gl_sysmon_261,gl_windowssecurity_330,gl_sysmon_262,gl_sysmon_263,gl_sysmon_264,gl_sysmon_265,gl_sysmon_266,gl_sysmon_267,gl_sysmon_268,gl_sysmon_269,gl_linux_auditbeat_14,gl_sysmon_259,gl_linux_auditbeat_15,gl_linux_auditbeat_12,gl_linux_auditbeat_13,gl_linux_auditbeat_10,gl_linux_auditbeat_11,gl-system-events_351,gl_windowssecurity_319,gl-system-events_350,gl_windowssecurity_318,gl_linux_auditbeat_18,gl_windowssecurity_315,gl_linux_auditbeat_19,gl_windowssecurity_314,gl_linux_auditbeat_16,gl_windowssecurity_317,gl_linux_auditbeat_17,gl_windowssecurity_316,gl-system-events_348,gl_windowssecurity_322,gl-system-events_347,gl_windowssecurity_321,gl-system-events_346,gl_windowssecurity_324,gl-system-events_345,gl_windowssecurity_323,gl-system-events_344,gl-system-events_343,gl-system-events_342,gl_windowssecurity_320,gl-system-events_341,gl_sysmon_257,gl-system-events_349,gl_sysmon_258,gl_windowsevent_258,gl_windowsevent_257,gl_windowsevent_259,gl_o365_261,gl_o365_260,gl_o365_263,gl_o365_262,gl_o365_265,gl_o365_264,gl_o365_267,gl_o365_266,gl_o365_269,gl_o365_268,gl_windowsevent_261,gl_windowsevent_260,gl-events_263,gl_o365_258,gl_o365_257,gl_o365_259,gl_windowsevent_274,gl_windowsevent_273,gl_windowsevent_276,gl_windowsevent_275,gl_windowsevent_278,gl_windowsevent_277,gl_windowsevent_279,gl_o365_281,gl_o365_280,gl_o365_283,gl_o365_282,gl-events_277,gl_o365_285,gl_o365_284,gl_o365_287,gl_o365_286,gl_o365_289,gl_o365_288,gl_windowsevent_281,gl_windowsevent_280,gl_windowsevent_283,gl_windowsevent_282,gl_windows_sysmon_7,gl_windowsevent_263,gl_windows_sysmon_6,gl_windowsevent_262,gl_windows_sysmon_9,gl_windowsevent_265,gl_windows_sysmon_8,gl_windowsevent_264,gl_windows_sysmon_3,gl_windowsevent_267,gl_windows_sysmon_2,gl_windowsevent_266,gl_windows_sysmon_5,gl_windowsevent_269,gl_windows_sysmon_4,gl_windowsevent_268,gl_o365_270,gl_o365_272,gl_windows_sysmon_1,gl_o365_271,gl_windows_sysmon_0,gl_o365_274,gl_o365_273,gl_o365_276,gl_o365_275,gl_o365_278,gl_o365_277,gl_o365_279,gl-events_283,gl_windowsevent_270,gl_windowsevent_272,gl_windowsevent_271,gl_windowsevent_331,gl_windowsevent_330,gl_windowsevent_333,gl_windowsevent_332,graylog_320,gl_windowsevent_335,gl_windowsevent_334,gl_fortinet_8,gl_windowsevent_337,gl_fortinet_9,gl_windowsevent_336,gl_fortinet_6,gl_windowsevent_339,gl-events_331,gl_fortinet_7,gl_windowsevent_338,gl-events_332,gl-events_333,gl-events_334,gl_cbdefense_16,gl_fortinet_0,gl_windowssecurity_278,gl_cbdefense_15,gl_fortinet_1,gl_windowssecurity_277,gl_cbdefense_18,gl_cbdefense_17,gl_windowssecurity_279,gl_cbdefense_12,gl_fortinet_4,gl_windowssecurity_274,gl_cbdefense_11,gl_fortinet_5,gl_windowssecurity_273,gl_cbdefense_14,gl_fortinet_2,gl_windowssecurity_276,gl_cbdefense_13,gl_fortinet_3,gl_windowssecurity_275,gl_windowssecurity_270,gl_cbdefense_10,gl_windowssecurity_272,gl_windowssecurity_271,gl-events_347,graylog_312,graylog_313,gl-events_349,gl_windowsevent_320,graylog_310,gl_windowsevent_322,gl_windowsevent_321,gl_o365_290,gl_windowsevent_324,gl_windowsevent_323,gl_o365_292,gl_windowsevent_326,gl-events_340,gl_o365_291,gl_windowsevent_325,gl_o365_294,gl_windowsevent_328,gl_o365_293,gl_windowsevent_327,gl_o365_296,graylog_316,gl_o365_295,gl_windowsevent_329,gl_windowssecurity_259,graylog_317,gl_o365_298,graylog_314,gl_o365_297,graylog_315,gl_windowssecurity_267,gl_o365_299,gl_windowssecurity_266,gl_windowssecurity_269,gl_windowssecurity_268,gl_windowssecurity_263,gl_windowssecurity_262,gl_windowssecurity_265,gl_windowssecurity_264,gl_windowssecurity_261,gl_windowssecurity_260,gl-events_351,graylog_307,graylog_308,graylog_305,graylog_306,graylog_304,gl_cbdefense_38,gl_cbdefense_37,gl_windowssecurity_299,gl_cbdefense_39,gl_cbdefense_34,gl_windowssecurity_296,gl_cbdefense_33,gl_windowssecurity_295,gl_cbdefense_36,gl_windowssecurity_298,gl_cbdefense_35,gl_windowssecurity_297,gl_cbdefense_30,gl_windowssecurity_292,gl_windowssecurity_291,gl_cbdefense_32,gl_windowssecurity_294,gl_cbdefense_31,gl_windowssecurity_293,gl_windowssecurity_290,gl_windowsevent_340,gl_windowsevent_342,gl_windowsevent_341,gl_windowsevent_344,gl_windowsevent_343,gl_windowsevent_346,gl_windowsevent_345,gl_symantec_0,gl_symantec_2,gl_symantec_1,gl_symantec_4,gl_cbdefense_19,gl_symantec_3,gl_symantec_6,gl_symantec_5,gl_cbdefense_27,gl_symantec_8,gl_windowssecurity_289,gl_cbdefense_26,gl_symantec_7,gl_windowssecurity_288,gl_cbdefense_29,gl_cbdefense_28,gl_symantec_9,gl_cbdefense_23,gl_windowssecurity_285,gl_cbdefense_22,gl_windowssecurity_284,gl_cbdefense_25,gl_windowssecurity_287,gl_cbdefense_24,gl_windowssecurity_286,gl_windowssecurity_281,gl_windowssecurity_280,gl_cbdefense_21,gl_windowssecurity_283,gl_cbdefense_20,gl_windowssecurity_282,gl-events_309,graylog_351,gl-events_302,gl_cbdefense_49,gl_windows_security_3,gl_cbdefense_48,gl_windows_security_4,gl_windows_security_2,gl_cbdefense_45,gl_cbdefense_44,gl_windows_security_8,gl_cbdefense_47,gl_cbdefense_46,gl_windows_security_6,gl_cbdefense_41,gl_cbdefense_40,gl_cbdefense_43,gl_cbdefense_42,gl-events_318,gl_windowsevent_311,gl-events_319,gl_windowsevent_310,gl_windowsevent_313,gl_windowsevent_312,gl_windowsevent_315,gl_windowsevent_314,gl_windowsevent_317,gl_windowsevent_316,gl-events_310,gl_windowsevent_319,gl_windowsevent_318,gl-events_312,gl-events_313,gl_linux_auditbeat_8,gl_linux_auditbeat_7,gl_windowssecurity_258,gl_linux_auditbeat_9,gl_windowssecurity_257,gl_linux_auditbeat_4,gl_linux_auditbeat_3,gl_linux_auditbeat_6,gl_linux_auditbeat_5,gl-events_325,graylog_334,gl-events_326,gl-events_327,gl-events_329,gl_windowsevent_300,gl_windowsevent_302,gl_windowsevent_301,gl_linux_auditbeat_0,gl_windowsevent_304,gl_windowsevent_303,gl_linux_auditbeat_2,gl_windowsevent_306,gl-events_320,gl_linux_auditbeat_1,gl_windowsevent_305,gl_windowsevent_308,gl_windowsevent_307,graylog_336,gl_windowsevent_309,gl_paloalto_3,gl_paloalto_4,gl_paloalto_5,gl_paloalto_6,gl_paloalto_7,gl_paloalto_8,gl_paloalto_9,gl_paloalto_0,gl_paloalto_1,gl_paloalto_2,gl-failures_204,gl-failures_205,gl-failures_206,gl-failures_207,gl-failures_208,gl-failures_209,gl-failures_200,gl-failures_201,gl-failures_202,gl-failures_203,gl-failures_210,gl-failures_211,gl-failures_212,gl-failures_213,gl_windows_common_50,gl_cbdefense_8,gl_cbdefense_9,gl_cbdefense_0,gl_cbdefense_1,gl_cbdefense_2,gl_cbdefense_3,gl_cbdefense_4,gl_cbdefense_5,gl_cbdefense_6,gl_cbdefense_7,gl_okta_49,gl_okta_48,gl_okta_45,gl_okta_44,gl_okta_47,gl_okta_46,gl_okta_41,gl_okta_40,gl_okta_43,gl_okta_42,gl_okta_38,gl_okta_37,gl_okta_39,gl_okta_34,gl_okta_33,gl_okta_36,gl_okta_35,gl_okta_30,gl_okta_32,gl_okta_31,gl_symantec_41,gl_symantec_42,gl_symantec_43,gl_symantec_44,gl_okta_27,gl_okta_26,gl_okta_29,gl_okta_28,gl_symantec_40,gl_symantec_49,gl_symantec_45,gl_symantec_46,gl_symantec_47,gl_symantec_48,gl_okta_23,gl_okta_22,gl_okta_25,gl_okta_24,gl_okta_21,gl_okta_20,gl_symantec_30,gl_okta_19,gl_symantec_31,gl_symantec_32,gl_symantec_33,gl_okta_16,gl_okta_15,gl_okta_18,gl_okta_17,gl_symantec_38,gl_symantec_39,gl_symantec_34,gl_symantec_35,gl_symantec_36,gl_symantec_37,gl_paloalto_49,gl_okta_12,gl_okta_11,gl_okta_14,gl_okta_13,gl_okta_10,gl_o365_300,gl_symantec_20,gl_o365_302,gl_symantec_21,gl_o365_301,gl_symantec_22,gl_o365_304,gl_o365_303,gl_o365_306,gl_o365_305,gl_paloalto_40,gl_o365_308,gl_paloalto_41,gl_symantec_27,gl_o365_307,gl_paloalto_42,gl_symantec_28,gl_paloalto_43,gl_symantec_29,gl_o365_309,gl_paloalto_44,gl_paloalto_45,gl_symantec_23,gl_paloalto_46,gl_symantec_24,gl_paloalto_47,gl_symantec_25,gl_paloalto_48,gl_symantec_26,gl_paloalto_38,gl_paloalto_39,gl_symantec_10,gl_symantec_11,gl_paloalto_30,gl_symantec_16,gl_paloalto_31,gl_symantec_17,gl_paloalto_32,gl_symantec_18,gl_paloalto_33,gl_symantec_19,gl_paloalto_34,gl_symantec_12,gl_paloalto_35,gl_symantec_13,gl_paloalto_36,gl_symantec_14,gl_paloalto_37,gl_symantec_15,gl_paloalto_27,gl_paloalto_28,gl_paloalto_29,gl_o365_320,gl_o365_322,gl_o365_321,gl_o365_324,gl_o365_323,gl_o365_326,gl_o365_325,gl_o365_328,gl_o365_327,gl_o365_329,gl_paloalto_20,gl_paloalto_21,gl_paloalto_22,gl_paloalto_23,gl_paloalto_24,gl_paloalto_25,gl_paloalto_26,gl_paloalto_16,gl_paloalto_17,gl_paloalto_18,gl_paloalto_19,gl_o365_311,gl_o365_310,gl_o365_313,gl_o365_312,gl_o365_315,gl_o365_314,gl_o365_317,gl_o365_316,gl_o365_319,gl_o365_318,gl_paloalto_10,gl_paloalto_11,gl_paloalto_12,gl_paloalto_13,gl_paloalto_14,gl_paloalto_15,gl_o365_340,gl_o365_342,gl_o365_341,gl_o365_344,gl_windows_sysmon_49,gl_o365_343,gl_windows_sysmon_48,gl_o365_346,gl_windows_sysmon_47,gl_o365_345,gl_windows_sysmon_46,gl_windows_sysmon_45,gl_windows_sysmon_44,gl_windows_sysmon_43,gl_windows_sysmon_42,gl_windows_sysmon_41,gl_windows_sysmon_40,gl_o365_331,gl_o365_330,gl_o365_333,gl_o365_332,gl_o365_335,gl_o365_334,gl_o365_337,gl_o365_336,gl_o365_339,gl_o365_338,gl_sysmon_325,gl_sysmon_326,gl_sysmon_327,gl_sysmon_328,gl_sysmon_329,gl-system-events_263,gl-system-events_262,gl_sysmon_320,gl_sysmon_321,gl_sysmon_322,gl_sysmon_323,gl_sysmon_324,gl_sysmon_314,gl_sysmon_315,gl_sysmon_316,gl_sysmon_317,gl_sysmon_318,gl_sysmon_319,gl_sysmon_310,gl_sysmon_311,gl_sysmon_312,gl_sysmon_313,gl-system-events_285,gl-system-events_284,gl-system-events_283,gl-system-events_282,gl-system-events_281,gl-system-events_280,gl-system-events_279,gl-system-events_278,gl-system-events_277,gl-system-events_276,gl-system-events_275,gl_sysmon_340,gl_sysmon_341,gl_sysmon_342,gl_sysmon_343,gl_sysmon_344,gl_sysmon_345,gl_sysmon_346,gl_sysmon_336,gl_sysmon_337,gl_sysmon_338,gl_sysmon_339,gl-system-events_274,gl-system-events_273,gl-system-events_272,gl-system-events_271,gl-system-events_270,gl-system-events_269,gl-system-events_268,gl-system-events_267,gl-system-events_266,gl-system-events_265,gl-system-events_264,gl_sysmon_330,gl_sysmon_331,gl_sysmon_332,gl_sysmon_333,gl_sysmon_334,gl_sysmon_335,gl-system-events_299,gl-system-events_298,gl-system-events_297,gl-failures_184,gl-system-events_296,gl-system-events_295,gl-system-events_294,gl-system-events_293,gl-system-events_292,gl-system-events_291,gl-system-events_290,gl-system-events_289,gl-system-events_288,gl-system-events_287,gl-system-events_286,gl_sysmon_303,gl_sysmon_304,gl-failures_190,gl_sysmon_305,gl-failures_191,gl_sysmon_306,gl-failures_192,gl_sysmon_307,gl-failures_193,gl_sysmon_308,gl-failures_194,gl_sysmon_309,gl-failures_195,gl-failures_185,gl-failures_186,gl-failures_187,gl-failures_188,gl-failures_189,gl_sysmon_300,gl_sysmon_301,gl_sysmon_302,gl-failures_196,gl-failures_197,gl-failures_198,gl-failures_199,gl_windowssecurity_337,gl_windows_sysmon_19,gl_windowssecurity_336,gl_windows_sysmon_18,gl_windowssecurity_339,gl_windows_sysmon_17,gl_windowssecurity_338,gl_windows_sysmon_16,gl_windowssecurity_344,gl_windows_sysmon_15,gl_windowssecurity_343,gl_windows_sysmon_14,gl_windowssecurity_346,gl_windows_sysmon_13,gl_windowssecurity_345,gl_windows_sysmon_12,gl_windowssecurity_340,gl_windows_sysmon_11,gl_windows_sysmon_10,gl_windowssecurity_342,gl_windowssecurity_341,gl_windows_sysmon_29,gl_windows_sysmon_28,gl_windows_sysmon_27,gl_windows_sysmon_26,gl_windows_sysmon_25,gl_windows_sysmon_24,gl_windows_sysmon_23,gl_windows_sysmon_22,gl_windows_sysmon_21,gl_windows_sysmon_20,gl_windows_sysmon_39,gl_windows_sysmon_38,gl_windows_sysmon_37,gl_windows_sysmon_36,gl_windows_sysmon_35,gl_windows_sysmon_34,gl_windows_sysmon_33,gl_windows_sysmon_32,gl_windows_sysmon_31,gl_windows_sysmon_30,gl_fortinet_41,gl_fortinet_40,gl_fortinet_43,gl_fortinet_42,gl_fortinet_45,gl_fortinet_44,gl_fortinet_47,gl_fortinet_46,gl_fortinet_49,gl_fortinet_48,gl_fortinet_21,gl_fortinet_20,gl_fortinet_23,gl_fortinet_22,gl_fortinet_25,gl_fortinet_24,gl_fortinet_27,gl_fortinet_26,gl_fortinet_29,gl_fortinet_28,gl_cisco_3,gl_cisco_2,gl_cisco_1,gl_cisco_0,gl_cisco_7,gl_cisco_6,gl_cisco_5,gl_cisco_4,gl_cisco_9,gl_cisco_8,gl_fortinet_30,gl_fortinet_32,gl_fortinet_31,gl_fortinet_34,gl_fortinet_33,gl_fortinet_36,gl_fortinet_35,gl_fortinet_38,gl_fortinet_37,gl_fortinet_39,graylog_295,gl_fortinet_10,gl_fortinet_12,gl_fortinet_11,gl_fortinet_14,gl_fortinet_13,graylog_294,gl_fortinet_16,gl_fortinet_15,gl_fortinet_18,gl_fortinet_17,gl_fortinet_19/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&scroll=1m&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 413 FULL head]

	at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302) ~[?:?]
	at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:272) ~[?:?]
	at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) ~[?:?]
	at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613) ~[?:?]
	at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583) ~[?:?]
	at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553) ~[?:?]
	at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1069) ~[?:?]
	at org.graylog.storage.elasticsearch7.ElasticsearchClient.lambda$singleSearch$1(ElasticsearchClient.java:77) ~[?:?]
	at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:109) ~[?:?]
	... 19 more

Possible Solution

It looks to me like this could be solved by moving the stream names from the URI to the request body.

Context

@yfoelling and @craig-thomas noticed this in Graylog Cloud.

Jul 04 '22 15:07 boosty

Comment from @craig-thomas:

This does appear to cause the events to fail silently for the customer, so they won't know there is an issue after enabling the event.

Jul 05 '22 08:07 boosty

It looks to me like this could be solved by moving the stream names from the URI to the request body.

Only if the Elasticsearch API allows that, of course. If not we have to think about a different solution.

Jul 05 '22 08:07 boosty

@boosty could we get a full not truncated stack trace?

Jul 05 '22 08:07 mpfz0r

@yfoelling @craig-thomas Could you help @mpfz0r with his request for a full stack trace?

Please remove any internals before posting it here (e.g. API endpints), as this is our public repo.

Jul 05 '22 08:07 boosty

Since our index names follow a pattern, maybe the wildcard syntax could be used: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/multi-index.html

In multi-target syntax, you can use a comma-separated list to run a request on multiple resources, such as data streams, indices, or index aliases: test1,test2,test3. You can also use glob-like wildcard (*) expressions to target resources that match a pattern: test* or *test or te*t or *test*.

Jul 05 '22 08:07 boosty

Since our index names follow a pattern, maybe the wildcard syntax could be used:

@boosty We probably have to do that if the API we use (I guess scroll API) only offers GET endpoints.

The drawback with using the wildcard syntax is that Elasticserach/OpenSearch might load metadata for indices into memory that don't even contain the data for the query time range. This can have performance and resource utilization implications. (e.g., on AWS UltraWarm setups)

Jul 05 '22 09:07 bernd

We probably have to do that if the API we use (I guess scroll API) only offers GET endpoints.

@bernd The example error in the description shows that it was using a POST request against the <target>/_search API endpoint.

I checked its documentation, but cannot find any hint that it would accept the target indexes via the request body (which would have been an easy fix).

The drawback with using the wildcard syntax is that Elasticserach/OpenSearch might load metadata for indices into memory that don't even contain the data for the query time range. This can have performance and resource utilization implications. (e.g., on AWS UltraWarm setups)

I see 😐

I will try to get the search team involved.

Jul 05 '22 11:07 boosty

FYI: the POST error is: 413 FULL head which hints that we exceeded the HTTP Header buffer. Typically limited to 8kB. Does the search request submit a header with all index names?

Jul 05 '22 12:07 mpfz0r

Idea from @bernd:

I thought about another possible optimization:

We are storing index metadata in the index_ranges collection. We could extend the collected data with the number of documents for each index. The Searches#determineAffectedIndices() method could use that information to remove indices from the list that don't have any documents.

The extension of the index ranges data requires a recalculation of all index ranges to become effective. It could either be done manually or we trigger it automatically with a migration or something similar.

Jul 12 '22 07:07 boosty

FYI: the POST error is: 413 FULL head which hints that we exceeded the HTTP Header buffer. Typically limited to 8kB. Does the search request submit a header with all index names?

@mpfz0r Yes, the search POST request adds all index names to the URL for scroll requests. (see the original description with the error message) The path with all the indices is already 16 kB.

As far as I can see, there is no way of adding the index list to the request body for scroll requests. We might be able to switch to search_after queries for the alerting use case, but I am not sure if there any drawbacks with that. That needs research and tests. I think we are already using search_after for the message export.

Jul 12 '22 08:07 bernd

Unassigned the Core Team so that it appears in the next triage. We should discuss if this might be something for the Search team. /cc @dennisoelkers

Aug 11 '22 09:08 boosty