crowdsec icon indicating copy to clipboard operation
crowdsec copied to clipboard
verified publisher icon crowdsecurity

CrowdSec AppSec can't import SecLang data files when using `PmFromFile` operator

Open GNU-Plus-Windows-User opened this issue 11 months ago • 9 comments

What happened?

When importing CRS via SecLang AppSec isn't able to correctly parse the data files, it assumes that it's SecLang rules when it's actually a data file. FATAL crowdsec init: while loading acquisition config: while configuring datasource of type appsec from /etc/crowdsec/acquis.d/appsec.yaml (position 0): unable to initialize runner: unable to initialize inband engine : invalid WAF config from string: failed to compile the directive "secrule": readfile /var/lib/crowdsec/data/coreruleset/scanners-user-agents.data: invalid argument

What did you expect to happen?

I should be able to make use of the pmFromFile operator

How can we reproduce it (as minimally and precisely as possible)?

Import a SecLang rules file that makes use of the pmFromFile operator or try to import CRS via SecLang.

Anything else we need to know?

N/A

Crowdsec version

1.6.5

OS version

Ubuntu 24.04

Enabled collections and parsers

N/A

Acquisition config

N/A

Config show

Out of the box defaults

Prometheus metrics

N/A

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

N/A

GNU-Plus-Windows-User avatar Feb 08 '25 03:02 GNU-Plus-Windows-User

@GNU-Plus-Windows-User: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

  1. Check Crowdsec Documentation to see if your issue can be self resolved.
  2. You can also join our Discord.
  3. Check Releases to make sure your agent is on the latest version.
Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

github-actions[bot] avatar Feb 08 '25 03:02 github-actions[bot]

Hey 👋🏻

So I tested 1.6.5 with our CRS which I know is outdated and worked fine no error here the pmFromFile snippets

/var/lib/crowdsec/data/REQUEST-913-SCANNER-DETECTION.conf:SecRule REQUEST_HEADERS:User-Agent "@pmFromFile scanners-user-agents.data"     "id:913100,    phase:1,    block,    capture,    t:none,    msg:'Found User-Agent associated with security scanner',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',    tag:'application-multi',    tag:'language-multi',    tag:'platform-multi',    tag:'attack-reputation-scanner',    tag:'paranoia-level/1',    tag:'OWASP_CRS',    tag:'capec/1000/118/224/541/310',    tag:'PCI/6.5.10',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'CRITICAL',    chain"
/var/lib/crowdsec/data/REQUEST-913-SCANNER-DETECTION.conf:SecRule REQUEST_HEADERS_NAMES|REQUEST_HEADERS "@pmFromFile scanners-headers.data"     "id:913110,    phase:1,    block,    capture,    t:none,    msg:'Found request header associated with security scanner',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',    tag:'application-multi',    tag:'language-multi',    tag:'platform-multi',    tag:'attack-reputation-scanner',    tag:'paranoia-level/1',    tag:'OWASP_CRS',    tag:'capec/1000/118/224/541/310',    tag:'PCI/6.5.10',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'CRITICAL',    setvar:'tx.inbound_anomaly_score_pl1=+%{tx.critical_anomaly_score}'"
/var/lib/crowdsec/data/REQUEST-913-SCANNER-DETECTION.conf:SecRule REQUEST_FILENAME|ARGS "@pmFromFile scanners-urls.data"     "id:913120,    phase:2,    block,    capture,    t:none,    msg:'Found request filename/argument associated with security scanner',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',    tag:'application-multi',    tag:'language-multi',    tag:'platform-multi',    tag:'attack-reputation-scanner',    tag:'paranoia-level/1',    tag:'OWASP_CRS',    tag:'capec/1000/118/224/541/310',    tag:'PCI/6.5.10',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'CRITICAL',    setvar:'tx.inbound_anomaly_score_pl1=+%{tx.critical_anomaly_score}'"
/var/lib/crowdsec/data/REQUEST-913-SCANNER-DETECTION.conf:SecRule REQUEST_HEADERS:User-Agent "@pmFromFile scripting-user-agents.data"     "id:913101,    phase:1,    block,    capture,    t:none,    msg:'Found User-Agent associated with scripting/generic HTTP client',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',    tag:'application-multi',    tag:'language-multi',    tag:'platform-multi',    tag:'attack-reputation-scripting',    tag:'OWASP_CRS',    tag:'capec/1000/118/224/541/310',    tag:'PCI/6.5.10',    tag:'paranoia-level/2',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'CRITICAL',    setvar:'tx.inbound_anomaly_score_pl2=+%{tx.critical_anomaly_score}'"
/var/lib/crowdsec/data/REQUEST-913-SCANNER-DETECTION.conf:SecRule REQUEST_HEADERS:User-Agent "@pmFromFile crawlers-user-agents.data"     "id:913102,    phase:1,    block,    capture,    t:none,    msg:'Found User-Agent associated with web crawler/bot',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',    tag:'application-multi',    tag:'language-multi',    tag:'platform-multi',    tag:'attack-reputation-crawler',    tag:'OWASP_CRS',    tag:'capec/1000/118/116/150',    tag:'PCI/6.5.10',    tag:'paranoia-level/2',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'CRITICAL',    setvar:'tx.inbound_anomaly_score_pl2=+%{tx.critical_anomaly_score}'"
/var/lib/crowdsec/data/REQUEST-930-APPLICATION-ATTACK-LFI.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/* "@pmFromFile lfi-os-files.data"     "id:930120,    phase:2,    block,    capture,    t:none,t:utf8toUnicode,t:urlDecodeUni,t:normalizePathWin,    msg:'OS File Access Attempt',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',    tag:'application-multi',    tag:'language-multi',    tag:'platform-multi',    tag:'attack-lfi',    tag:'paranoia-level/1',    tag:'OWASP_CRS',    tag:'capec/1000/255/153/126',    tag:'PCI/6.5.4',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'CRITICAL',    setvar:'tx.lfi_score=+%{tx.critical_anomaly_score}',    setvar:'tx.inbound_anomaly_score_pl1=+%{tx.critical_anomaly_score}'"
/var/lib/crowdsec/data/REQUEST-930-APPLICATION-ATTACK-LFI.conf:SecRule REQUEST_FILENAME "@pmFromFile restricted-files.data"     "id:930130,    phase:1,    block,    capture,    t:none,t:utf8toUnicode,t:urlDecodeUni,t:normalizePathWin,    msg:'Restricted File Access Attempt',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',    tag:'application-multi',    tag:'language-multi',    tag:'platform-multi',    tag:'attack-lfi',    tag:'paranoia-level/1',    tag:'OWASP_CRS',    tag:'capec/1000/255/153/126',    tag:'PCI/6.5.4',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'CRITICAL',    setvar:'tx.lfi_score=+%{tx.critical_anomaly_score}',    setvar:'tx.inbound_anomaly_score_pl1=+%{tx.critical_anomaly_score}'"
/var/lib/crowdsec/data/REQUEST-930-APPLICATION-ATTACK-LFI.conf:SecRule REQUEST_HEADERS:Referer|REQUEST_HEADERS:User-Agent "@pmFromFile lfi-os-files.data"     "id:930121,    phase:1,    block,    capture,    t:none,t:utf8toUnicode,t:urlDecodeUni,t:normalizePathWin,    msg:'OS File Access Attempt in REQUEST_HEADERS',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',    tag:'application-multi',    tag:'language-multi',    tag:'platform-multi',    tag:'attack-lfi',    tag:'paranoia-level/2',    tag:'OWASP_CRS',    tag:'capec/1000/255/153/126',    tag:'PCI/6.5.4',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'CRITICAL',    setvar:'tx.lfi_score=+%{tx.critical_anomaly_score}',    setvar:'tx.inbound_anomaly_score_pl2=+%{tx.critical_anomaly_score}'"
/var/lib/crowdsec/data/REQUEST-932-APPLICATION-ATTACK-RCE.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/* "@pmFromFile windows-powershell-commands.data"     "id:932120,    phase:2,    block,    capture,    t:none,t:cmdLine,    msg:'Remote Command Execution: Windows PowerShell Command Found',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',    tag:'application-multi',    tag:'language-shell',    tag:'language-powershell',    tag:'platform-windows',    tag:'attack-rce',    tag:'paranoia-level/1',    tag:'OWASP_CRS',    tag:'capec/1000/152/248/88',    tag:'PCI/6.5.2',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'CRITICAL',    setvar:'tx.rce_score=+%{tx.critical_anomaly_score}',    setvar:'tx.inbound_anomaly_score_pl1=+%{tx.critical_anomaly_score}'"
/var/lib/crowdsec/data/REQUEST-932-APPLICATION-ATTACK-RCE.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/* "@pmFromFile unix-shell.data"     "id:932160,    phase:2,    block,    capture,    t:none,t:cmdLine,t:normalizePath,    msg:'Remote Command Execution: Unix Shell Code Found',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',    tag:'application-multi',    tag:'language-shell',    tag:'platform-unix',    tag:'attack-rce',    tag:'paranoia-level/1',    tag:'OWASP_CRS',    tag:'capec/1000/152/248/88',    tag:'PCI/6.5.2',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'CRITICAL',    setvar:'tx.rce_score=+%{tx.critical_anomaly_score}',    setvar:'tx.inbound_anomaly_score_pl1=+%{tx.critical_anomaly_score}'"
/var/lib/crowdsec/data/REQUEST-932-APPLICATION-ATTACK-RCE.conf:SecRule FILES|REQUEST_HEADERS:X-Filename|REQUEST_HEADERS:X_Filename|REQUEST_HEADERS:X-File-Name     "@pmFromFile restricted-upload.data"     "id:932180,    phase:2,    block,    capture,    t:none,    msg:'Restricted File Upload Attempt',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',    tag:'application-multi',    tag:'language-multi',    tag:'platform-multi',    tag:'attack-rce',    tag:'paranoia-level/1',    tag:'OWASP_CRS',    tag:'capec/1000/152/248/88',    tag:'PCI/6.5.2',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'CRITICAL',    setvar:'tx.rce_score=+%{tx.critical_anomaly_score}',    setvar:'tx.inbound_anomaly_score_pl1=+%{tx.critical_anomaly_score}'"
/var/lib/crowdsec/data/REQUEST-932-APPLICATION-ATTACK-RCE.conf:SecRule REQUEST_HEADERS:User-Agent|REQUEST_HEADERS:Referer "@pmFromFile unix-shell.data"     "id:932161,    phase:2,    block,    capture,    t:none,t:cmdLine,t:normalizePath,    msg:'Remote Command Execution: Unix Shell Code Found in REQUEST_HEADERS',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',    tag:'application-multi',    tag:'language-shell',    tag:'platform-unix',    tag:'attack-rce',    tag:'paranoia-level/2',    tag:'OWASP_CRS',    tag:'capec/1000/152/248/88',    tag:'PCI/6.5.2',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'CRITICAL',    setvar:'tx.rce_score=+%{tx.critical_anomaly_score}',    setvar:'tx.inbound_anomaly_score_pl2=+%{tx.critical_anomaly_score}'"
/var/lib/crowdsec/data/REQUEST-933-APPLICATION-ATTACK-PHP.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/* "@pmFromFile php-config-directives.data"     "id:933120,    phase:2,    block,    capture,    t:none,t:normalisePath,    msg:'PHP Injection Attack: Configuration Directive Found',    logdata:'Matched Data: %{TX.933120_TX_0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',    tag:'application-multi',    tag:'language-php',    tag:'platform-multi',    tag:'attack-injection-php',    tag:'paranoia-level/1',    tag:'OWASP_CRS',    tag:'capec/1000/152/242',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'CRITICAL',    setvar:'tx.933120_tx_0=%{tx.0}',    chain"
/var/lib/crowdsec/data/REQUEST-933-APPLICATION-ATTACK-PHP.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/* "@pmFromFile php-variables.data"     "id:933130,    phase:2,    block,    capture,    t:none,t:normalisePath,t:urlDecodeUni,    msg:'PHP Injection Attack: Variables Found',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',    tag:'application-multi',    tag:'language-php',    tag:'platform-multi',    tag:'attack-injection-php',    tag:'paranoia-level/1',    tag:'OWASP_CRS',    tag:'capec/1000/152/242',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'CRITICAL',    setvar:'tx.php_injection_score=+%{tx.critical_anomaly_score}',    setvar:'tx.inbound_anomaly_score_pl1=+%{tx.critical_anomaly_score}'"
/var/lib/crowdsec/data/REQUEST-933-APPLICATION-ATTACK-PHP.conf:#                These words are detected as a match directly using @pmFromFile.
/var/lib/crowdsec/data/REQUEST-933-APPLICATION-ATTACK-PHP.conf:#                For performance reasons, the @pmFromFile operator is used, and many functions from lesser
/var/lib/crowdsec/data/REQUEST-933-APPLICATION-ATTACK-PHP.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|REQUEST_FILENAME|ARGS_NAMES|ARGS|XML:/* "@pmFromFile php-function-names-933150.data"     "id:933150,    phase:2,    block,    capture,    t:none,    msg:'PHP Injection Attack: High-Risk PHP Function Name Found',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',    tag:'application-multi',    tag:'language-php',    tag:'platform-multi',    tag:'attack-injection-php',    tag:'paranoia-level/1',    tag:'OWASP_CRS',    tag:'capec/1000/152/242',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'CRITICAL',    setvar:'tx.php_injection_score=+%{tx.critical_anomaly_score}',    setvar:'tx.inbound_anomaly_score_pl1=+%{tx.critical_anomaly_score}'"
/var/lib/crowdsec/data/REQUEST-933-APPLICATION-ATTACK-PHP.conf:# but uses a phrase file (@pmFromFile), and additionally looks for an '(' character
/var/lib/crowdsec/data/REQUEST-933-APPLICATION-ATTACK-PHP.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|REQUEST_FILENAME|ARGS_NAMES|ARGS|XML:/* "@pmFromFile php-function-names-933151.data"     "id:933151,    phase:2,    block,    capture,    t:none,    msg:'PHP Injection Attack: Medium-Risk PHP Function Name Found',    logdata:'Matched Data: %{TX.933151_TX_0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',    tag:'application-multi',    tag:'language-php',    tag:'platform-multi',    tag:'attack-injection-php',    tag:'OWASP_CRS',    tag:'capec/1000/152/242',    tag:'paranoia-level/2',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'CRITICAL',    setvar:'tx.933151_tx_0=%{tx.0}',    chain"
/var/lib/crowdsec/data/REQUEST-933-APPLICATION-ATTACK-PHP.conf:# @pmFromFile for flexibility and performance.
/var/lib/crowdsec/data/REQUEST-934-APPLICATION-ATTACK-GENERIC.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|REQUEST_FILENAME|ARGS_NAMES|ARGS|XML:/* "@pmFromFile ssrf.data"     "id:934110,    phase:2,    block,    capture,    t:none,    msg:'Possible Server Side Request Forgery (SSRF) Attack: Cloud provider metadata URL in Parameter',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',    tag:'application-multi',    tag:'language-multi',    tag:'platform-multi',    tag:'attack-ssrf',    tag:'paranoia-level/1',    tag:'OWASP_CRS',    tag:'capec/1000/225/664',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'CRITICAL',    setvar:'tx.rce_score=+%{tx.critical_anomaly_score}',    setvar:'tx.inbound_anomaly_score_pl1=+%{tx.critical_anomaly_score}'"
/var/lib/crowdsec/data/REQUEST-944-APPLICATION-ATTACK-JAVA.conf:SecRule ARGS|ARGS_NAMES|REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|REQUEST_BODY|REQUEST_FILENAME|REQUEST_HEADERS|XML:/*|XML://@*     "@pmFromFile java-classes.data"     "id:944130,    phase:2,    block,    t:none,    msg:'Suspicious Java class detected',    logdata:'Matched Data: %{MATCHED_VAR} found within %{MATCHED_VAR_NAME}',    tag:'application-multi',    tag:'language-java',    tag:'platform-multi',    tag:'attack-rce',    tag:'OWASP_CRS',    tag:'capec/1000/152/248',    tag:'PCI/6.5.2',    tag:'paranoia-level/1',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'CRITICAL',    setvar:'tx.rce_score=+%{tx.critical_anomaly_score}',    setvar:'tx.inbound_anomaly_score_pl1=+%{tx.critical_anomaly_score}'"
/var/lib/crowdsec/data/RESPONSE-951-DATA-LEAKAGES-SQL.conf:SecRule RESPONSE_BODY "!@pmFromFile sql-errors.data"     "id:951100,    phase:4,    pass,    t:none,    nolog,    tag:'application-multi',    tag:'language-multi',    tag:'platform-multi',    tag:'attack-disclosure',    tag:'OWASP_CRS',    tag:'capec/1000/118/116/54',    ver:'OWASP_CRS/4.0.0-rc1',    skipAfter:END-SQL-ERROR-MATCH-PL1"
/var/lib/crowdsec/data/RESPONSE-952-DATA-LEAKAGES-JAVA.conf:SecRule RESPONSE_BODY "@pmFromFile java-code-leakages.data"     "id:952100,    phase:4,    block,    capture,    t:none,    msg:'Java Source Code Leakage',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}',    tag:'application-multi',    tag:'language-java',    tag:'platform-multi',    tag:'attack-disclosure',    tag:'paranoia-level/1',    tag:'OWASP_CRS',    tag:'capec/1000/118/116',    tag:'PCI/6.5.6',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'ERROR',    setvar:'tx.outbound_anomaly_score_pl1=+%{tx.error_anomaly_score}'"
/var/lib/crowdsec/data/RESPONSE-952-DATA-LEAKAGES-JAVA.conf:SecRule RESPONSE_BODY "@pmFromFile java-errors.data"     "id:952110,    phase:4,    block,    capture,    t:none,    msg:'Java Errors',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}',    tag:'application-multi',    tag:'language-java',    tag:'platform-multi',    tag:'attack-disclosure',    tag:'paranoia-level/1',    tag:'OWASP_CRS',    tag:'capec/1000/118/116',    tag:'PCI/6.5.6',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'ERROR',    setvar:'tx.outbound_anomaly_score_pl1=+%{tx.error_anomaly_score}'"
/var/lib/crowdsec/data/RESPONSE-953-DATA-LEAKAGES-PHP.conf:SecRule RESPONSE_BODY "@pmFromFile php-errors.data"     "id:953100,    phase:4,    block,    capture,    t:none,    msg:'PHP Information Leakage',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}',    tag:'application-multi',    tag:'language-php',    tag:'platform-multi',    tag:'attack-disclosure',    tag:'paranoia-level/1',    tag:'OWASP_CRS',    tag:'capec/1000/118/116',    tag:'PCI/6.5.6',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'ERROR',    setvar:'tx.outbound_anomaly_score_pl1=+%{tx.error_anomaly_score}'"
/var/lib/crowdsec/data/RESPONSE-953-DATA-LEAKAGES-PHP.conf:SecRule RESPONSE_BODY "@pmFromFile php-errors-pl2.data"     "id:953101,    phase:4,    block,    capture,    t:none,    msg:'PHP Information Leakage',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}',    tag:'application-multi',    tag:'language-php',    tag:'platform-multi',    tag:'attack-disclosure',    tag:'paranoia-level/2',    tag:'OWASP_CRS',    tag:'capec/1000/118/116',    tag:'PCI/6.5.6',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'ERROR',    setvar:'tx.outbound_anomaly_score_pl2=+%{tx.error_anomaly_score}'"
/var/lib/crowdsec/data/RESPONSE-954-DATA-LEAKAGES-IIS.conf:SecRule RESPONSE_BODY "@pmFromFile iis-errors.data"     "id:954120,    phase:4,    block,    capture,    t:none,    msg:'IIS Information Leakage',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}',    tag:'application-multi',    tag:'language-multi',    tag:'platform-iis',    tag:'platform-windows',    tag:'attack-disclosure',    tag:'paranoia-level/1',    tag:'OWASP_CRS',    tag:'capec/1000/118/116',    tag:'PCI/6.5.6',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'ERROR',    setvar:'tx.outbound_anomaly_score_pl1=+%{tx.error_anomaly_score}'"
/var/lib/crowdsec/data/RESPONSE-955-WEB-SHELLS.conf:SecRule RESPONSE_BODY "@pmFromFile web-shells-php.data"     "id:955100,    phase:4,    block,    capture,    t:none,    msg:'Web shell detected',    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}',    tag:'language-php',    tag:'platform-multi',    tag:'attack-rce',    tag:'paranoia-level/1',    tag:'OWASP_CRS',    tag:'capec/1000/225/122/17/650',    ver:'OWASP_CRS/4.0.0-rc1',    severity:'CRITICAL',    setvar:'tx.outbound_anomaly_score_pl1=+%{tx.critical_anomaly_score}'"

Updating CRS to latest is also fine:

REQUEST-913-SCANNER-DETECTION.conf:SecRule REQUEST_HEADERS:User-Agent "@pmFromFile scanners-user-agents.data" \
REQUEST-930-APPLICATION-ATTACK-LFI.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/* "@pmFromFile lfi-os-files.data" \
REQUEST-930-APPLICATION-ATTACK-LFI.conf:SecRule REQUEST_FILENAME "@pmFromFile restricted-files.data" \
REQUEST-930-APPLICATION-ATTACK-LFI.conf:SecRule REQUEST_HEADERS:Referer|REQUEST_HEADERS:User-Agent "@pmFromFile lfi-os-files.data" \
REQUEST-932-APPLICATION-ATTACK-RCE.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/* "@pmFromFile windows-powershell-commands.data" \
REQUEST-932-APPLICATION-ATTACK-RCE.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/* "@pmFromFile unix-shell.data" \
REQUEST-932-APPLICATION-ATTACK-RCE.conf:SecRule FILES|REQUEST_HEADERS:X-Filename|REQUEST_HEADERS:X_Filename|REQUEST_HEADERS:X-File-Name "@pmFromFile restricted-upload.data" \
REQUEST-932-APPLICATION-ATTACK-RCE.conf:SecRule REQUEST_HEADERS:User-Agent|REQUEST_HEADERS:Referer "@pmFromFile unix-shell.data" \
REQUEST-933-APPLICATION-ATTACK-PHP.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/* "@pmFromFile php-config-directives.data" \
REQUEST-933-APPLICATION-ATTACK-PHP.conf:        SecRule TX:1 "@pmFromFile php-config-directives.data" \
REQUEST-933-APPLICATION-ATTACK-PHP.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/* "@pmFromFile php-variables.data" \
REQUEST-933-APPLICATION-ATTACK-PHP.conf:#               These words are detected as a match directly using @pmFromFile.
REQUEST-933-APPLICATION-ATTACK-PHP.conf:#               For performance reasons, the @pmFromFile operator is used, and many functions from lesser
REQUEST-933-APPLICATION-ATTACK-PHP.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|REQUEST_FILENAME|ARGS_NAMES|ARGS|XML:/* "@pmFromFile php-function-names-933150.data" \
REQUEST-933-APPLICATION-ATTACK-PHP.conf:# but uses a phrase file (@pmFromFile), and additionally looks for an '(' character
REQUEST-933-APPLICATION-ATTACK-PHP.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|REQUEST_FILENAME|ARGS_NAMES|ARGS|XML:/* "@pmFromFile php-function-names-933151.data" \
REQUEST-933-APPLICATION-ATTACK-PHP.conf:        SecRule TX:1 "@pmFromFile php-function-names-933151.data" \
REQUEST-933-APPLICATION-ATTACK-PHP.conf:# @pmFromFile for flexibility and performance.
REQUEST-934-APPLICATION-ATTACK-GENERIC.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|REQUEST_FILENAME|ARGS_NAMES|ARGS|XML:/* "@pmFromFile ssrf.data" \
REQUEST-944-APPLICATION-ATTACK-JAVA.conf:    "@pmFromFile java-classes.data" \
RESPONSE-951-DATA-LEAKAGES-SQL.conf:SecRule RESPONSE_BODY "!@pmFromFile sql-errors.data" \
RESPONSE-952-DATA-LEAKAGES-JAVA.conf:SecRule RESPONSE_BODY "@pmFromFile java-code-leakages.data" \
RESPONSE-952-DATA-LEAKAGES-JAVA.conf:SecRule RESPONSE_BODY "@pmFromFile java-errors.data" \
RESPONSE-953-DATA-LEAKAGES-PHP.conf:SecRule RESPONSE_BODY "@pmFromFile php-errors.data" \
RESPONSE-953-DATA-LEAKAGES-PHP.conf:SecRule RESPONSE_BODY "@pmFromFile php-errors-pl2.data" \
RESPONSE-954-DATA-LEAKAGES-IIS.conf:SecRule RESPONSE_BODY "@pmFromFile iis-errors.data" \
RESPONSE-955-WEB-SHELLS.conf:SecRule RESPONSE_BODY "@pmFromFile web-shells-php.data" \

I dont use CRS much so let me know if there any additional thing I have to do other than downloading and configuring appsec that need to be done to the CRS confs themselves.

Could you ensure you only have one crowdsec binary (beware which may should two if /bin/ is symlinked):

which -a crowdsec

Just so it complete:

appsec-config:

root@bookworm:/var/lib/crowdsec/data# cat /etc/crowdsec/appsec-configs/crs.yaml
name: crowdsecurity/crs
default_remediation: ban
#log_level: debug
inband_rules:
 - crowdsecurity/crs

appsec-rules:

root@bookworm:/var/lib/crowdsec/data# cat /etc/crowdsec/appsec-rules/crs.yaml
name: crowdsecurity/crs
seclang_rules:
 - SecRuleEngine On
 - SecRequestBodyAccess On
seclang_files_rules:
 - crs-setup.conf
 - REQUEST-901-INITIALIZATION.conf
 - REQUEST-905-COMMON-EXCEPTIONS.conf
 - REQUEST-911-METHOD-ENFORCEMENT.conf
 - REQUEST-913-SCANNER-DETECTION.conf
 - REQUEST-920-PROTOCOL-ENFORCEMENT.conf
 - REQUEST-921-PROTOCOL-ATTACK.conf
 - REQUEST-922-MULTIPART-ATTACK.conf
 - REQUEST-930-APPLICATION-ATTACK-LFI.conf
 - REQUEST-931-APPLICATION-ATTACK-RFI.conf
 - REQUEST-932-APPLICATION-ATTACK-RCE.conf
 - REQUEST-933-APPLICATION-ATTACK-PHP.conf
 - REQUEST-934-APPLICATION-ATTACK-GENERIC.conf
 - REQUEST-941-APPLICATION-ATTACK-XSS.conf
 - REQUEST-942-APPLICATION-ATTACK-SQLI.conf
 - REQUEST-943-APPLICATION-ATTACK-SESSION-FIXATION.conf
 - REQUEST-944-APPLICATION-ATTACK-JAVA.conf
 - REQUEST-949-BLOCKING-EVALUATION.conf
 - RESPONSE-950-DATA-LEAKAGES.conf
 - RESPONSE-951-DATA-LEAKAGES-SQL.conf
 - RESPONSE-952-DATA-LEAKAGES-JAVA.conf
 - RESPONSE-953-DATA-LEAKAGES-PHP.conf
 - RESPONSE-954-DATA-LEAKAGES-IIS.conf
 - RESPONSE-955-WEB-SHELLS.conf
 - RESPONSE-959-BLOCKING-EVALUATION.conf
 - RESPONSE-980-CORRELATION.conf

data:
  - source_url: https://hub-data.crowdsec.net/appsec/crs/crs-setup.conf
    dest_file: crs-setup.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/REQUEST-901-INITIALIZATION.conf
    dest_file: REQUEST-901-INITIALIZATION.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/REQUEST-905-COMMON-EXCEPTIONS.conf
    dest_file: REQUEST-905-COMMON-EXCEPTIONS.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/REQUEST-911-METHOD-ENFORCEMENT.conf
    dest_file: REQUEST-911-METHOD-ENFORCEMENT.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/REQUEST-913-SCANNER-DETECTION.conf
    dest_file: REQUEST-913-SCANNER-DETECTION.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/REQUEST-920-PROTOCOL-ENFORCEMENT.conf
    dest_file: REQUEST-920-PROTOCOL-ENFORCEMENT.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/REQUEST-921-PROTOCOL-ATTACK.conf
    dest_file: REQUEST-921-PROTOCOL-ATTACK.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/REQUEST-922-MULTIPART-ATTACK.conf
    dest_file: REQUEST-922-MULTIPART-ATTACK.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/REQUEST-930-APPLICATION-ATTACK-LFI.conf
    dest_file: REQUEST-930-APPLICATION-ATTACK-LFI.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/REQUEST-931-APPLICATION-ATTACK-RFI.conf
    dest_file: REQUEST-931-APPLICATION-ATTACK-RFI.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/REQUEST-932-APPLICATION-ATTACK-RCE.conf
    dest_file: REQUEST-932-APPLICATION-ATTACK-RCE.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/REQUEST-933-APPLICATION-ATTACK-PHP.conf
    dest_file: REQUEST-933-APPLICATION-ATTACK-PHP.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/REQUEST-934-APPLICATION-ATTACK-GENERIC.conf
    dest_file: REQUEST-934-APPLICATION-ATTACK-GENERIC.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/REQUEST-941-APPLICATION-ATTACK-XSS.conf
    dest_file: REQUEST-941-APPLICATION-ATTACK-XSS.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/REQUEST-942-APPLICATION-ATTACK-SQLI.conf
    dest_file: REQUEST-942-APPLICATION-ATTACK-SQLI.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/REQUEST-943-APPLICATION-ATTACK-SESSION-FIXATION.conf
    dest_file: REQUEST-943-APPLICATION-ATTACK-SESSION-FIXATION.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/REQUEST-944-APPLICATION-ATTACK-JAVA.conf
    dest_file: REQUEST-944-APPLICATION-ATTACK-JAVA.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/REQUEST-949-BLOCKING-EVALUATION.conf
    dest_file: REQUEST-949-BLOCKING-EVALUATION.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/RESPONSE-950-DATA-LEAKAGES.conf
    dest_file: RESPONSE-950-DATA-LEAKAGES.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/RESPONSE-951-DATA-LEAKAGES-SQL.conf
    dest_file: RESPONSE-951-DATA-LEAKAGES-SQL.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/RESPONSE-952-DATA-LEAKAGES-JAVA.conf
    dest_file: RESPONSE-952-DATA-LEAKAGES-JAVA.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/RESPONSE-953-DATA-LEAKAGES-PHP.conf
    dest_file: RESPONSE-953-DATA-LEAKAGES-PHP.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/RESPONSE-954-DATA-LEAKAGES-IIS.conf
    dest_file: RESPONSE-954-DATA-LEAKAGES-IIS.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/RESPONSE-955-WEB-SHELLS.conf
    dest_file: RESPONSE-955-WEB-SHELLS.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/RESPONSE-959-BLOCKING-EVALUATION.conf
    dest_file: RESPONSE-959-BLOCKING-EVALUATION.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/RESPONSE-980-CORRELATION.conf
    dest_file: RESPONSE-980-CORRELATION.conf
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/crawlers-user-agents.data
    dest_file: crawlers-user-agents.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/iis-errors.data
    dest_file: iis-errors.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/java-classes.data
    dest_file: java-classes.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/java-code-leakages.data
    dest_file: java-code-leakages.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/java-errors.data
    dest_file: java-errors.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/lfi-os-files.data
    dest_file: lfi-os-files.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/php-config-directives.data
    dest_file: php-config-directives.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/php-errors.data
    dest_file: php-errors.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/php-errors-pl2.data
    dest_file: php-errors-pl2.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/php-function-names-933150.data
    dest_file: php-function-names-933150.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/php-function-names-933151.data
    dest_file: php-function-names-933151.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/php-variables.data
    dest_file: php-variables.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/restricted-files.data
    dest_file: restricted-files.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/restricted-upload.data
    dest_file: restricted-upload.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/scanners-headers.data
    dest_file: scanners-headers.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/scanners-urls.data
    dest_file: scanners-urls.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/scanners-user-agents.data
    dest_file: scanners-user-agents.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/scripting-user-agents.data
    dest_file: scripting-user-agents.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/sql-errors.data
    dest_file: sql-errors.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/ssrf.data
    dest_file: ssrf.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/unix-shell.data
    dest_file: unix-shell.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/web-shells-php.data
    dest_file: web-shells-php.data
    type: modsec
  - source_url: https://hub-data.crowdsec.net/appsec/crs/windows-powershell-commands.data
    dest_file: windows-powershell-commands.data
    type: modsec

The key thing in the rules is seclang_files_rules never import a .data file only .conf files which pmFromFile will load it later.

LaurenceJJones avatar Feb 08 '25 10:02 LaurenceJJones

@LaurenceJJones I have 2 binaries but they are not symlinked:

$ which -a crowdsec
/usr/bin/crowdsec
/bin/crowdsec

appsec-rules:

I'm doing it slightly differently, if I import via URL it's fine but if I import via a local file it's not:

name: crowdsecurity/seclang-custom
seclang_rules:
 - SecRuleEngine On
 - SecRequestBodyAccess On
seclang_files_rules:
 - coreruleset/crs-setup.conf
 - coreruleset/rules/REQUEST-901-INITIALIZATION.conf
 - coreruleset/rules/REQUEST-905-COMMON-EXCEPTIONS.conf
 - coreruleset/rules/REQUEST-911-METHOD-ENFORCEMENT.conf
 - coreruleset/rules/REQUEST-913-SCANNER-DETECTION.conf
 - coreruleset/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf
 - coreruleset/rules/REQUEST-921-PROTOCOL-ATTACK.conf
 - coreruleset/rules/REQUEST-922-MULTIPART-ATTACK.conf
 - coreruleset/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf
 - coreruleset/rules/REQUEST-931-APPLICATION-ATTACK-RFI.conf
 - coreruleset/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf
 - coreruleset/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf
 - coreruleset/rules/REQUEST-934-APPLICATION-ATTACK-GENERIC.conf
 - coreruleset/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf
 - coreruleset/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf
 - coreruleset/rules/REQUEST-943-APPLICATION-ATTACK-SESSION-FIXATION.conf
 - coreruleset/rules/REQUEST-944-APPLICATION-ATTACK-JAVA.conf
 - coreruleset/rules/REQUEST-949-BLOCKING-EVALUATION.conf
 - coreruleset/rules/RESPONSE-950-DATA-LEAKAGES.conf
 - coreruleset/rules/RESPONSE-951-DATA-LEAKAGES-SQL.conf
 - coreruleset/rules/RESPONSE-952-DATA-LEAKAGES-JAVA.conf
 - coreruleset/rules/RESPONSE-953-DATA-LEAKAGES-PHP.conf
 - coreruleset/rules/RESPONSE-954-DATA-LEAKAGES-IIS.conf
 - coreruleset/rules/RESPONSE-955-WEB-SHELLS.conf
 - coreruleset/rules/RESPONSE-959-BLOCKING-EVALUATION.conf
 - coreruleset/rules/RESPONSE-980-CORRELATION.conf

doesn't matter what CRS version you use, it's the same for all of them.

GNU-Plus-Windows-User avatar Feb 08 '25 11:02 GNU-Plus-Windows-User

So I managed to get it working, there two issues one coraza does not allow full paths so you cannot use /var/lib/crowdsec/data/.... you have to use relative paths, two you cannot control the "base_dir" as pmFromFile does not expand variables. So I managed to create the patch bash script which basically grabs all .data files and goes through each conf file and patches the .data name with coreruleset/rules/<name>.data

#!/bin/bash

# Directory to search, can be passed as an argument or defaults to current directory
SEARCH_DIR="${1:-.}"

# Find all .data files in the directory
find "$SEARCH_DIR" -type f -name "*.data" | while read -r data_file; do
  # Get the basename of the data file
  data_filename="$(basename "$data_file")"

  # Find all .conf files in the directory
  find "$SEARCH_DIR" -type f -name "*.conf" | while read -r conf_file; do
    # Use sed to replace example.data with coreruleset/rules/example.data
    sed -i "s|$data_filename|coreruleset/rules/$data_filename|g" "$conf_file"
    echo "Updated $conf_file with coreruleset/rules/$data_filename"
  done
done

also was generated by mr gpt, I used it and it worked on patching the files.

data/coreruleset/rules/REQUEST-913-SCANNER-DETECTION.conf:SecRule REQUEST_HEADERS:User-Agent "@pmFromFile coreruleset/rules/scanners-user-agents.data" \
data/coreruleset/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/* "@pmFromFile coreruleset/rules/lfi-os-files.data" \
data/coreruleset/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf:SecRule REQUEST_FILENAME "@pmFromFile coreruleset/rules/restricted-files.data" \
data/coreruleset/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf:SecRule REQUEST_HEADERS:Referer|REQUEST_HEADERS:User-Agent "@pmFromFile coreruleset/rules/lfi-os-files.data" \
data/coreruleset/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/* "@pmFromFile coreruleset/rules/windows-powershell-commands.data" \
data/coreruleset/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/* "@pmFromFile coreruleset/rules/unix-shell.data" \
data/coreruleset/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf:SecRule FILES|REQUEST_HEADERS:X-Filename|REQUEST_HEADERS:X_Filename|REQUEST_HEADERS:X-File-Name "@pmFromFile coreruleset/rules/restricted-upload.data" \
data/coreruleset/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf:SecRule REQUEST_HEADERS:User-Agent|REQUEST_HEADERS:Referer "@pmFromFile coreruleset/rules/unix-shell.data" \
data/coreruleset/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/* "@pmFromFile coreruleset/rules/php-config-directives.data" \
data/coreruleset/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf:        SecRule TX:1 "@pmFromFile coreruleset/rules/php-config-directives.data" \
data/coreruleset/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/* "@pmFromFile coreruleset/rules/php-variables.data" \
data/coreruleset/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf:#                These words are detected as a match directly using @pmFromFile.
data/coreruleset/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf:#                For performance reasons, the @pmFromFile operator is used, and many functions from lesser
data/coreruleset/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|REQUEST_FILENAME|ARGS_NAMES|ARGS|XML:/* "@pmFromFile coreruleset/rules/php-function-names-933150.data" \
data/coreruleset/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf:# but uses a phrase file (@pmFromFile), and additionally looks for an '(' character
data/coreruleset/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|REQUEST_FILENAME|ARGS_NAMES|ARGS|XML:/* "@pmFromFile coreruleset/rules/php-function-names-933151.data" \
data/coreruleset/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf:        SecRule TX:1 "@pmFromFile coreruleset/rules/php-function-names-933151.data" \
data/coreruleset/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf:# @pmFromFile for flexibility and performance.
data/coreruleset/rules/REQUEST-934-APPLICATION-ATTACK-GENERIC.conf:SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|REQUEST_FILENAME|ARGS_NAMES|ARGS|XML:/* "@pmFromFile coreruleset/rules/ssrf.data" \
data/coreruleset/rules/REQUEST-944-APPLICATION-ATTACK-JAVA.conf:    "@pmFromFile coreruleset/rules/java-classes.data" \
data/coreruleset/rules/RESPONSE-951-DATA-LEAKAGES-SQL.conf:SecRule RESPONSE_BODY "!@pmFromFile coreruleset/rules/sql-errors.data" \
data/coreruleset/rules/RESPONSE-952-DATA-LEAKAGES-JAVA.conf:SecRule RESPONSE_BODY "@pmFromFile coreruleset/rules/java-code-leakages.data" \
data/coreruleset/rules/RESPONSE-952-DATA-LEAKAGES-JAVA.conf:SecRule RESPONSE_BODY "@pmFromFile coreruleset/rules/java-errors.data" \
data/coreruleset/rules/RESPONSE-953-DATA-LEAKAGES-PHP.conf:SecRule RESPONSE_BODY "@pmFromFile coreruleset/rules/php-errors.data" \
data/coreruleset/rules/RESPONSE-953-DATA-LEAKAGES-PHP.conf:SecRule RESPONSE_BODY "@pmFromFile coreruleset/rules/php-errors-pl2.data" \
data/coreruleset/rules/RESPONSE-954-DATA-LEAKAGES-IIS.conf:SecRule RESPONSE_BODY "@pmFromFile coreruleset/rules/iis-errors.data" \
data/coreruleset/rules/RESPONSE-955-WEB-SHELLS.conf:SecRule RESPONSE_BODY "@pmFromFile coreruleset/rules/web-shells-php.data" \

LaurenceJJones avatar Feb 08 '25 20:02 LaurenceJJones

@LaurenceJJones I just tested the workaround and now it works fine, although that confusing error message should be fixed. Do you want me to open up a separate issue about the error messages?

GNU-Plus-Windows-User avatar Feb 09 '25 08:02 GNU-Plus-Windows-User

@LaurenceJJones I just tested the workaround and now it works fine, although that confusing error message should be fixed. Do you want me to open up a separate issue about the error messages?

Yeah you can, however, the error message is return from coraza about "invalid argument". I only found out about the full path error when I search the "invalid argument" in coraza issues.

LaurenceJJones avatar Feb 09 '25 15:02 LaurenceJJones

I just got this when I updated CRS to 4.14.0. It has this commit that added web-shells-asp.data. It's interesting, that it never complained about any other files, though

Simbiat avatar May 04 '25 08:05 Simbiat

Another interesting thing: I changed the way I download the CRS files from cloning from Git to letting CrowdSec download those by setting data directive (this commit), and on my test this resolved things completely, but on PROD - it still fails with same error complaining specifically about web-shells-asp.data and nothing else. In both cases I am using containers with same setup, but test uses Windows, while PROD is on Ubuntu.

Simbiat avatar May 04 '25 09:05 Simbiat

Oh, wait. I think I know why it works on test. Because crowdsec data folder has web-shells-asp.data which got there when I was trying to resolve this. But it was not working for me with initial solution because I had crowdsecurity/appsec-crs AppSec rule, which has previously downloaded the CRS files, and thus they were not downloading the ones that I needed. I removed crowdsecurity/appsec-crs and started downloading everything to root of crowdsec data folder and now it works in both places.

Not sure if that was a clear explanation, but I think there are 2 problems, that need solution on CrowdSec side:

  1. We need a way to force redownload of files set in source_url even if destination exists. There can be a duration attribute like refresh. If not set, we use current behavior, if set to a duration string - checks if modification time of the destination file is less than current_time - duration and if it is - redownloads the file. If set to something like 0 or a string always - always redownload on service start.
  2. The batch script workaround should be used inside memory. That is the content should be changed during parsing of files somehow. Not sure what would be the best approach, but perhaps the type field can be utilized (BTW, data section is not described anywhere in context of AppSec, only here, and it's not obvious that they are about the same thing or not). Like, if type is set to modsec or modsecurity all contents of the respective files a loaded into memory, and then the logic similar to the batch script is applied internally using the dest_file value (search for filename only and replace it with full path)

Alternative is to keep crowdsecurity/appsec-crs AppSec rule up-to-date, that is sync it with CRS repo in some way. Although that may still require the first point, since I am not sure if the files get redownloaded.

Simbiat avatar May 04 '25 13:05 Simbiat