[Rule Tuning] Web Server Potential Spike in Error Response Codes

[vx-sec]

Link to Rule

https://github.com/elastic/detection-rules/blob/main/rules/cross-platform/reconnaissance_web_server_unusual_spike_in_error_response_codes.toml

Rule Tuning Type

Data Quality - Ensuring integrity and quality of data used by detection rules.

Description

The rule mentioned above, as well as these rules below fail for Nginx integration (possibly other's too that use uri_parts in ingest pipeline to dissect the url), with the following error.

verification_exception
	Root causes:
		verification_exception: Found 1 problem
line 3:34: Unknown column [url.full]

Issue is that url.full is not a field that gets defined and parsed by uri_parts ingest processor and so the is not null checks on it fail.

Web Server Potential Command Injection Request Web Server Suspicious User Agent Requests Web Server Discovery or Fuzzing Activity

Example Data

This is what the nginx integration for example indexes.

{
    "@timestamp": "2022-12-09T10:39:23.000Z",
    "_tmp": {},
    "agent": {
        "ephemeral_id": "<redacted>",
        "id": "<redacted>",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.5.0"
    },
    "data_stream": {
        "dataset": "nginx.access",
        "namespace": "ep",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "<redacted>",
        "snapshot": false,
        "version": "8.5.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "web"
        ],
        "created": "2022-12-09T10:39:38.896Z",
        "dataset": "nginx.access",
        "ingested": "2022-12-09T10:39:40Z",
        "kind": "event",
        "outcome": "success",
        "timezone": "+00:00",
        "type": [
            "access"
        ]
    },
    "host": {
        "architecture": "x86_64",
        "containerized": false,
        "hostname": "docker-fleet-agent",
        "id": "<redacted>",
        "ip": [
            "<redacted>"
        ],
        "mac": [
            "02-42-AC-12-00-07"
        ],
        "name": "docker-fleet-agent",
        "os": {
            "codename": "focal",
            "family": "debian",
            "kernel": "5.15.49-linuxkit",
            "name": "Ubuntu",
            "platform": "ubuntu",
            "type": "linux",
            "version": "20.04.5 LTS (Focal Fossa)"
        }
    },
    "http": {
        "request": {
            "method": "GET"
        },
        "response": {
            "body": {
                "bytes": 97
            },
            "status_code": 200
        },
        "version": "1.1"
    },
    "input": {
        "type": "log"
    },
    "log": {
        "file": {
            "path": "/tmp/service_logs/access.log"
        },
        "offset": 0
    },
    "nginx": {
        "access": {
            "remote_ip_list": [
                "127.0.0.1"
            ]
        }
    },
    "related": {
        "ip": [
            "127.0.0.1"
        ]
    },
    "source": {
        "address": "127.0.0.1",
        "ip": "127.0.0.1"
    },
    "tags": [
        "nginx-access"
    ],
    "url": {
        "original": "/server-status",
        "path": "/server-status"
    },
    "user_agent": {
        "device": {
            "name": "Other"
        },
        "name": "curl",
        "original": "curl/7.64.0",
        "version": "7.64.0"
    }
}

[Aegrah]

Hi @vx-sec thank you for bringing this to our attention! It indeed looks like the issue is related to not having data from both access logs and network packet capture logs available, thus the is not null check on a non-existent field fails. I will see what the best solution is for these rules. I will get back to you with the outcome + tuning PR once it's created.

[Aegrah]

@vx-sec, again, thank you for bringing this to our attention. Elasticsearch ES|QL validates all referenced fields against the combined mappings of every index in a multi-index FROM clause. Because some indices do not define url.full, any ES|QL expression that references this field (even inside CASE, COALESCE, or FORK) fails at compile time with Unknown column, making it impossible to implement query logic that conditionally selects between fields from two different integration categories in a single query. As a result, the query cannot be fixed without schema changes or separate rules, and we will explore raising this limitation with the ES|QL development team to determine whether it can be improved.

For now, I added a tuning PR to fully remove NPC compatibility to these web rules, which should fix the issue for now. We will revisit to see if we can make a change in the future to be able to combine the different integration categories.

Tuning PR:

• [ ] https://github.com/elastic/detection-rules/pull/5384

Feel free to close the issue if this helps! You can already grab the updated rules from the PRs if you want.