Request for comments : parsers & scenarios customization in the CrowdSec agent

[buixor]

(note: this is a cross-post with https://discourse.crowdsec.net/t/request-for-comments-parsers-scenarios-customization-in-the-crowdsec-agent/129)

Preface

Currently, if a CrowdSec user wants to modify a parser or a scenario, he has no choice but to edit the downloaded version, or create a complete copy. None of this options are actually convenient: items are considered tainted, and hence can't be updated automatically.

Users have requested a better way to allow customization of one or more properties of their scenarios and parsers. This is useful in a lot of cases, for instance when you have to deal with custom web server log formats, or because you want to apply stricter/laxer policy on any given scenario etc.

The purpose of this post is to describe the approach we have in mind to add this feature to CrowdSec and collect feedback from the community.

Proposal overview

The idea is to use JavaScript Object Notation (JSON) Patch to offer a generic enough interface to allow any kind of modification to existing objects, and to wrap it into cscli tooling to make it accessible.

The proposal would allow to add a /etc/crowdsec/patches/ directory that can contain one or more yaml files. Each of these files can contain one or more patches as per below:

#I have a stricter ssh bruteforce policy
scenario: crowdsecurity/ssh-bf
patches:
  - op: replace
    path: /leakspeed
    value: 1s
---
#I'm using custom apache2 logs
parser: crowdsecurity/apache2
patches:
  - op: replace
    #/nodes/1/... targets the 2nd element of the nodes array
    path: /nodes/1/grok/pattern
    value: 'CUSTOM_PREFIX %{HTTPD_ERRORLOG}'

At startup, CrowdSec would apply all available patches on existing parsers and scenarios.

Patch creation and maintenance

To make patch creation and maintenance easier, some tooling would be included in cscli

Patch creation

cscli hub diff <file1> <file2> [-o <output>] [-a|--auto]: given that 2 files (ie. upstream scenario and locally patched copy), generate a yaml/json patch file. Displays it by default on stdout, but can write it to a given (-o) file, or to an appropriate generated file directly in /etc/crowdsec/patches/

$ cscli hub diff /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml ./my-ssh-logs.yaml

$ cscli parsers diff crowdsecurity/sshd-logs ./my-sshd-logs.yaml

Patch maintenance and preview

cscli hub patch file <patch1> <patch2> ...: given the path to an existing scenario/parser, output its final state after all patches have been applied. If paths to specific patches are given, showz the result only after these patches have been applied.

$ cscli hub patch /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml

$ cscli parsers patch crowdsecurity/sshd-logs ./mypatch1.yaml ./mypatch2.yaml

Your opinion matters to us

• Would you prefer the patch/diff feature to be all located in cscli hub or directly/as well in cscli parsers|scenarios|...?
  • Locating them in cscli hub would limit the amount of new commands
  • Locating them in cscli parsers|scenarios|... would allow to specify items by name rather than by path only (ie. cscli parsers diff crowdsecurity/sshd-logs ./myfile.yaml )

Please feel free to comment and provide general feedback,

[reinerj]

Preference would be to use "cscli parsers|scenarios|"

[pierrehenrymuller]

I approuve this method. We need to modify all variables in a scenario without fork them.

Some standard configuration are too permissive and many real scan for ssh or ports for example are not blocked.

[arjenvri]

This would be really useful! Is there any roadmap at this time to implement this feature?

[NoelProf]

I agree with @pierrehenrymuller also the cscli parsers|scenarios| seems like the most straightforward option to me.

[sabban]

Keep here for reference.