elastic
[POC] Using ES|QL as a general rule engine
Summary
Initially, we decided to separate the cloud resources collection and the rule engine for CSP. We planned to use Cloudbeat to collect the cloud resources and then send them to ES/Kibana. After that, we would use EQL as the rule engine to assess these resources and generate findings. However, we encountered some limitations while using EQL, which made us switch to using OPA on the agent side as the rule engine. With the recent introduction of ES|QL in version 8.11, we are hoping it will provide a suitable solution for a rule engine in Kibana.
As we have already implemented all the relevant cloud provider benchmarks in OPA, this POC is not about testing the feasibility of migrating them to ES|QL. Instead, we aim to determine if ES|QL rules platform could be utilized as a Custom Rules Platform for CSPM/KSPM.
To verify if ES|QL can provide this functionality, we plan to implement one of our already existing CIS benchmarks with it.
Definition of done
- [ ] Review the EQL ticket to understand why we decided not to use it.
- [ ] Attempt to implement an existing benchmark and identify any limitations in ES|QL.
Out of scope
- [ ] Anything related to migrating the existing benchmarks to ES|QL.
Resources
- https://www.elastic.co/blog/introduction-to-esql-new-query-language-flexible-iterative-analytics
- https://github.com/elastic/security-team/issues/1814
- https://github.com/elastic/security-team/issues/7658
From what I remember, we've decided not to use the current DE rules because a rule creates an alert. Which isn't a finding. It completely behaves differently.
In case we wish to a create a findings out of DE, the query language is less relevant. A findings is always being created, so that when the rule evaluates, and result of the finding is Failed and when it passes, it evaluates Passed.
Currently, there's no such logic in the DE for such rules. I'm not sure how ES|QL is relevant to this POC
From what I remember, we've decided not to use the current DE rules because a rule creates an alert. Which isn't a finding. It completely behaves differently.
In case we wish to a create a findings out of DE, the query language is less relevant. A findings is always being created, so that when the rule evaluates, and result of the finding is Failed and when it passes, it evaluates Passed.
Currently, there's no such logic in the DE for such rules. I'm not sure how ES|QL is relevant to this POC
I think the new ENRICH command might be useful here:
The ES|QL ENRICH processing command combines, at query-time, data from one or more source indexes with field-value combinations found in Elasticsearch enrich indexes.
This could be helpful to make queries on non-mapped json fields https://github.com/elastic/elasticsearch/issues/104934