fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard
verified publisher icon fluent

Python multiline parser is not working with systemd input

Open scottwn opened this issue 1 year ago • 2 comments

Bug Report

My setup is somewhat similar to https://github.com/fluent/fluent-bit/issues/8787

I have several containers running on podman on RHEL8 EC2.

I use fluent bit to forward the container logs to cloudwatch.

I run my container stack using docker compose.

This is the fluent bit service definition.

  logs:
    image: fluent/fluent-bit:3.0.6
    restart: always
    user: root
    network_mode: host
    command: /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf
    volumes:
      - /run/log/journal:/run/log/journal:z
      - /etc/machine-id:/etc/machine-id:ro
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:z
      - ./parsers.conf:/fluent-bit/etc/parsers.conf:z

fluent-bit.conf

[SERVICE]
  Parsers_File /fluent-bit/etc/parsers.conf

[INPUT]
  Name systemd
  Tag dd-service
  Path /run/log/journal
  Read_From_Tail On
  Systemd_Filter _COMM=conmon
  Systemd_Filter CONTAINER_TAG=dd-service
  Systemd_Filter_Type And

[INPUT]
  Name systemd
  Tag yuma
  Path /run/log/journal
  Read_From_Tail On
  Systemd_Filter _COMM=conmon
  Systemd_Filter CONTAINER_TAG=yuma
  Systemd_Filter_Type And

[INPUT]
  Name systemd
  Tag db
  Path /run/log/journal
  Read_From_Tail On
  Systemd_Filter _COMM=conmon
  Systemd_Filter CONTAINER_TAG=db
  Systemd_Filter_Type And

[INPUT]
  Name systemd
  Tag app
  Path /run/log/journal
  Read_From_Tail On
  Systemd_Filter _COMM=conmon
  Systemd_Filter CONTAINER_TAG=app
  Systemd_Filter_Type And

[INPUT]
  Name systemd
  Tag cron
  Path /run/log/journal
  Read_From_Tail On
  Systemd_Filter _COMM=conmon
  Systemd_Filter CONTAINER_TAG=cron
  Systemd_Filter_Type And

[INPUT]
  Name systemd
  Tag server
  Path /run/log/journal
  Read_From_Tail On
  Systemd_Filter _COMM=conmon
  Systemd_Filter CONTAINER_TAG=server
  Systemd_Filter_Type And

[FILTER]
  Name multiline
  Match_Regex (app|cron|dd-service)
  multiline.key_content MESSAGE
  Buffer On
  multiline.parser python

[FILTER]
  Name parser
  Match dd-service
  Key_Name MESSAGE
  Parser dd-service

[FILTER]
  Name parser
  Match db
  Key_Name MESSAGE
  Parser db

[FILTER]
  Name parser
  Match_Regex (app|cron)
  Key_Name MESSAGE
  Parser app

[FILTER]
  Name parser
  Match server
  Key_Name MESSAGE
  Parser nginx

[OUTPUT]
  Name cloudwatch_logs 
  Match *
  region us-east-1
  log_group_name soak
  log_stream_prefix ec2-

parsers.conf

[PARSER]
  Name dd-service
  Format regex
  Regex (?<level>DEBUG|INFO|WARNING|ERROR|CRITICAL)?:? *(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})?(?: -- )?((?<remote>[\d+\.]+):\d+ )?(- "(?<method>[A-Z]+) (?<path>\/[a-z\/-]*) HTTP.*(?<code>\d{3}))?(?<message>.+)
  Time_Key time
  Time_Format %Y-%m-%d %H:%M:%S,%L

[PARSER]
  Name db
  Format regex
  Regex (?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3} [A-Z]{3}) .* LOG: (?<message>.*)
  Time_Key time
  Time_Format %Y-%m-%d %H:%M:%S.%L %Z

[PARSER]
  Name app
  Format regex
  Regex (?<level>DEBUG|INFO|WARNING|ERROR|CRITICAL):(?<module>[\w\.]+):(User: (?<user>.+?), )?(Customer: (?<customer>.+?), )?(from (?<remote>[\d+\.]+) )?(with body (?<body>.+?) )?(requested (?<method>[A-Z]+))?(?<message>(?!\/).+?)?((?<path>\/[a-z\/-]*)|[^\/]$).*?(?: at (?<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{6})\.? We responded (?<code>\d{3})?)?
  Time_Key time
  Time_Format %Y-%m-%dT%H:%M:%S.%L

[PARSER]
    Name   nginx
    Format regex
    Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

Multiline Python stacktraces are not parsed as single log messages. They are split into multiple messages. Consider this example. In the journal, each line is a separate entry. These entries are the start of one traceback, that should be captured by the multiline parser as one log. $ sudo journalctl -r -o json-pretty

{
	"__CURSOR" : "s=c53f7edb08d144628eec6d6c258c10e7;i=afe207;b=5d379cbb22d94db78503e2ce48ba675b;m=7c51847b918;t=620ae76477d78;x=ca9012edf835c0b7",
	"__REALTIME_TIMESTAMP" : "1724783541058936",
	"__MONOTONIC_TIMESTAMP" : "8543097305368",
	"_BOOT_ID" : "5d379cbb22d94db78503e2ce48ba675b",
	"PRIORITY" : "3",
	"CODE_FILE" : "src/ctr_logging.c",
	"CODE_LINE" : "320",
	"CODE_FUNC" : "write_journald",
	"_TRANSPORT" : "journal",
	"_UID" : "0",
	"_GID" : "0",
	"_COMM" : "conmon",
	"_EXE" : "/usr/bin/conmon",
	"_CAP_EFFECTIVE" : "1ffffffffff",
	"_SELINUX_CONTEXT" : "system_u:system_r:container_runtime_t:s0",
	"_SYSTEMD_SLICE" : "machine.slice",
	"_MACHINE_ID" : "593a407f0bf44f8da32824920ad62d25",
	"_HOSTNAME" : "ip-172-31-92-238.ec2.internal",
	"CONTAINER_ID_FULL" : "37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d",
	"CONTAINER_ID" : "37a3e93ed098",
	"CONTAINER_TAG" : "app",
	"CONTAINER_NAME" : "aliro-orchestrator-app",
	"SYSLOG_IDENTIFIER" : "app",
	"_PID" : "3879832",
	"_CMDLINE" : "/usr/bin/conmon --api-version 1 -c 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d -u 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata -p /run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/pidfile -n aliro-orchestrator-app --exit-dir /run/libpod/exits --full-attach -s -l journald --log-level info --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/oci-log --log-tag app --conmon-pidfile /run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg info --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg boltdb --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d",
	"_SYSTEMD_CGROUP" : "/machine.slice/libpod-conmon-37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d.scope",
	"_SYSTEMD_UNIT" : "libpod-conmon-37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d.scope",
	"_SYSTEMD_INVOCATION_ID" : "724b21586ffe4e008e0ed099352ae284",
	"MESSAGE" : "  File \"/usr/local/lib/python3.12/site-packages/django/core/handlers/base.py\", line 197, in _get_response\n",
	"_SOURCE_REALTIME_TIMESTAMP" : "1724783541055306"
}
{
	"__CURSOR" : "s=c53f7edb08d144628eec6d6c258c10e7;i=afe206;b=5d379cbb22d94db78503e2ce48ba675b;m=7c51847b8da;t=620ae76477d3a;x=a9774aa57f2d382d",
	"__REALTIME_TIMESTAMP" : "1724783541058874",
	"__MONOTONIC_TIMESTAMP" : "8543097305306",
	"_BOOT_ID" : "5d379cbb22d94db78503e2ce48ba675b",
	"PRIORITY" : "3",
	"CODE_FILE" : "src/ctr_logging.c",
	"CODE_LINE" : "320",
	"CODE_FUNC" : "write_journald",
	"_TRANSPORT" : "journal",
	"_UID" : "0",
	"_GID" : "0",
	"_COMM" : "conmon",
	"_EXE" : "/usr/bin/conmon",
	"_CAP_EFFECTIVE" : "1ffffffffff",
	"_SELINUX_CONTEXT" : "system_u:system_r:container_runtime_t:s0",
	"_SYSTEMD_SLICE" : "machine.slice",
	"_MACHINE_ID" : "593a407f0bf44f8da32824920ad62d25",
	"_HOSTNAME" : "ip-172-31-92-238.ec2.internal",
	"CONTAINER_ID_FULL" : "37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d",
	"CONTAINER_ID" : "37a3e93ed098",
	"CONTAINER_TAG" : "app",
	"CONTAINER_NAME" : "aliro-orchestrator-app",
	"SYSLOG_IDENTIFIER" : "app",
	"_PID" : "3879832",
	"_CMDLINE" : "/usr/bin/conmon --api-version 1 -c 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d -u 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata -p /run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/pidfile -n aliro-orchestrator-app --exit-dir /run/libpod/exits --full-attach -s -l journald --log-level info --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/oci-log --log-tag app --conmon-pidfile /run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg info --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg boltdb --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d",
	"_SYSTEMD_CGROUP" : "/machine.slice/libpod-conmon-37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d.scope",
	"_SYSTEMD_UNIT" : "libpod-conmon-37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d.scope",
	"_SYSTEMD_INVOCATION_ID" : "724b21586ffe4e008e0ed099352ae284",
	"MESSAGE" : "               ^^^^^^^^^^^^^^^^^^^^^\n",
	"_SOURCE_REALTIME_TIMESTAMP" : "1724783541055302"
}
{
	"__CURSOR" : "s=c53f7edb08d144628eec6d6c258c10e7;i=afe205;b=5d379cbb22d94db78503e2ce48ba675b;m=7c51847b89f;t=620ae76477cff;x=2059de01a3eb14c0",
	"__REALTIME_TIMESTAMP" : "1724783541058815",
	"__MONOTONIC_TIMESTAMP" : "8543097305247",
	"_BOOT_ID" : "5d379cbb22d94db78503e2ce48ba675b",
	"PRIORITY" : "3",
	"CODE_FILE" : "src/ctr_logging.c",
	"CODE_LINE" : "320",
	"CODE_FUNC" : "write_journald",
	"_TRANSPORT" : "journal",
	"_UID" : "0",
	"_GID" : "0",
	"_COMM" : "conmon",
	"_EXE" : "/usr/bin/conmon",
	"_CAP_EFFECTIVE" : "1ffffffffff",
	"_SELINUX_CONTEXT" : "system_u:system_r:container_runtime_t:s0",
	"_SYSTEMD_SLICE" : "machine.slice",
	"_MACHINE_ID" : "593a407f0bf44f8da32824920ad62d25",
	"_HOSTNAME" : "ip-172-31-92-238.ec2.internal",
	"CONTAINER_ID_FULL" : "37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d",
	"CONTAINER_ID" : "37a3e93ed098",
	"CONTAINER_TAG" : "app",
	"CONTAINER_NAME" : "aliro-orchestrator-app",
	"SYSLOG_IDENTIFIER" : "app",
	"_PID" : "3879832",
	"_CMDLINE" : "/usr/bin/conmon --api-version 1 -c 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d -u 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata -p /run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/pidfile -n aliro-orchestrator-app --exit-dir /run/libpod/exits --full-attach -s -l journald --log-level info --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/oci-log --log-tag app --conmon-pidfile /run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg info --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg boltdb --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d",
	"_SYSTEMD_CGROUP" : "/machine.slice/libpod-conmon-37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d.scope",
	"_SYSTEMD_UNIT" : "libpod-conmon-37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d.scope",
	"_SYSTEMD_INVOCATION_ID" : "724b21586ffe4e008e0ed099352ae284",
	"MESSAGE" : "    response = get_response(request)\n",
	"_SOURCE_REALTIME_TIMESTAMP" : "1724783541055298"
}
{
	"__CURSOR" : "s=c53f7edb08d144628eec6d6c258c10e7;i=afe204;b=5d379cbb22d94db78503e2ce48ba675b;m=7c51847b866;t=620ae76477cc6;x=8bb5004ae31c2137",
	"__REALTIME_TIMESTAMP" : "1724783541058758",
	"__MONOTONIC_TIMESTAMP" : "8543097305190",
	"_BOOT_ID" : "5d379cbb22d94db78503e2ce48ba675b",
	"PRIORITY" : "3",
	"CODE_FILE" : "src/ctr_logging.c",
	"CODE_LINE" : "320",
	"CODE_FUNC" : "write_journald",
	"_TRANSPORT" : "journal",
	"_UID" : "0",
	"_GID" : "0",
	"_COMM" : "conmon",
	"_EXE" : "/usr/bin/conmon",
	"_CAP_EFFECTIVE" : "1ffffffffff",
	"_SELINUX_CONTEXT" : "system_u:system_r:container_runtime_t:s0",
	"_SYSTEMD_SLICE" : "machine.slice",
	"_MACHINE_ID" : "593a407f0bf44f8da32824920ad62d25",
	"_HOSTNAME" : "ip-172-31-92-238.ec2.internal",
	"CONTAINER_ID_FULL" : "37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d",
	"CONTAINER_ID" : "37a3e93ed098",
	"CONTAINER_TAG" : "app",
	"CONTAINER_NAME" : "aliro-orchestrator-app",
	"SYSLOG_IDENTIFIER" : "app",
	"_PID" : "3879832",
	"_CMDLINE" : "/usr/bin/conmon --api-version 1 -c 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d -u 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata -p /run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/pidfile -n aliro-orchestrator-app --exit-dir /run/libpod/exits --full-attach -s -l journald --log-level info --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/oci-log --log-tag app --conmon-pidfile /run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg info --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg boltdb --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d",
	"_SYSTEMD_CGROUP" : "/machine.slice/libpod-conmon-37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d.scope",
	"_SYSTEMD_UNIT" : "libpod-conmon-37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d.scope",
	"_SYSTEMD_INVOCATION_ID" : "724b21586ffe4e008e0ed099352ae284",
	"MESSAGE" : "  File \"/usr/local/lib/python3.12/site-packages/django/core/handlers/exception.py\", line 55, in inner\n",
	"_SOURCE_REALTIME_TIMESTAMP" : "1724783541055294"
}
{
	"__CURSOR" : "s=c53f7edb08d144628eec6d6c258c10e7;i=afe203;b=5d379cbb22d94db78503e2ce48ba675b;m=7c51847b82d;t=620ae76477c8d;x=6e2ecbfda72b37a",
	"__REALTIME_TIMESTAMP" : "1724783541058701",
	"__MONOTONIC_TIMESTAMP" : "8543097305133",
	"_BOOT_ID" : "5d379cbb22d94db78503e2ce48ba675b",
	"PRIORITY" : "3",
	"CODE_FILE" : "src/ctr_logging.c",
	"CODE_LINE" : "320",
	"CODE_FUNC" : "write_journald",
	"_TRANSPORT" : "journal",
	"_UID" : "0",
	"_GID" : "0",
	"_COMM" : "conmon",
	"_EXE" : "/usr/bin/conmon",
	"_CAP_EFFECTIVE" : "1ffffffffff",
	"_SELINUX_CONTEXT" : "system_u:system_r:container_runtime_t:s0",
	"_SYSTEMD_SLICE" : "machine.slice",
	"_MACHINE_ID" : "593a407f0bf44f8da32824920ad62d25",
	"_HOSTNAME" : "ip-172-31-92-238.ec2.internal",
	"CONTAINER_ID_FULL" : "37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d",
	"CONTAINER_ID" : "37a3e93ed098",
	"CONTAINER_TAG" : "app",
	"CONTAINER_NAME" : "aliro-orchestrator-app",
	"SYSLOG_IDENTIFIER" : "app",
	"_PID" : "3879832",
	"_CMDLINE" : "/usr/bin/conmon --api-version 1 -c 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d -u 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata -p /run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/pidfile -n aliro-orchestrator-app --exit-dir /run/libpod/exits --full-attach -s -l journald --log-level info --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/oci-log --log-tag app --conmon-pidfile /run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg info --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg boltdb --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d",
	"_SYSTEMD_CGROUP" : "/machine.slice/libpod-conmon-37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d.scope",
	"_SYSTEMD_UNIT" : "libpod-conmon-37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d.scope",
	"_SYSTEMD_INVOCATION_ID" : "724b21586ffe4e008e0ed099352ae284",
	"MESSAGE" : "Traceback (most recent call last):\n",
	"_SOURCE_REALTIME_TIMESTAMP" : "1724783541055290"
}

In Cloudwatch, the first 3 lines of the traceback are parsed into one message as expected

{
    "PRIORITY": "3",
    "CODE_FILE": "src/ctr_logging.c",
    "CODE_LINE": "320",
    "CODE_FUNC": "write_journald",
    "_TRANSPORT": "journal",
    "_UID": "0",
    "_GID": "0",
    "_COMM": "conmon",
    "_EXE": "/usr/bin/conmon",
    "_CAP_EFFECTIVE": "1ffffffffff",
    "_SELINUX_CONTEXT": "system_u:system_r:container_runtime_t:s0",
    "_SYSTEMD_SLICE": "machine.slice",
    "_BOOT_ID": "5d379cbb22d94db78503e2ce48ba675b",
    "_MACHINE_ID": "593a407f0bf44f8da32824920ad62d25",
    "_HOSTNAME": "ip-172-31-92-238.ec2.internal",
    "CONTAINER_ID_FULL": "37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d",
    "CONTAINER_ID": "37a3e93ed098",
    "CONTAINER_TAG": "app",
    "CONTAINER_NAME": "aliro-orchestrator-app",
    "SYSLOG_IDENTIFIER": "app",
    "_PID": "3879832",
    "_CMDLINE": "/usr/bin/conmon --api-version 1 -c 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d -u 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata -p /run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/pidfile -n aliro-orchestrator-app --exit-dir /run/libpod/exits --full-attach -s -l journald --log-level info --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/oci-log --log-tag app --conmon-pidfile /run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg info --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg boltdb --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d",
    "_SYSTEMD_CGROUP": "/machine.slice/libpod-conmon-37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d.scope",
    "_SYSTEMD_UNIT": "libpod-conmon-37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d.scope",
    "_SYSTEMD_INVOCATION_ID": "724b21586ffe4e008e0ed099352ae284",
    "MESSAGE": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.12/site-packages/django/core/handlers/exception.py\", line 55, in inner\n    response = get_response(request)\n",
    "_SOURCE_REALTIME_TIMESTAMP": "1724783541055290"
}

but then it breaks into another message. I expect the entire traceback to be one message.

{
    "PRIORITY": "3",
    "CODE_FILE": "src/ctr_logging.c",
    "CODE_LINE": "320",
    "CODE_FUNC": "write_journald",
    "_TRANSPORT": "journal",
    "_UID": "0",
    "_GID": "0",
    "_COMM": "conmon",
    "_EXE": "/usr/bin/conmon",
    "_CAP_EFFECTIVE": "1ffffffffff",
    "_SELINUX_CONTEXT": "system_u:system_r:container_runtime_t:s0",
    "_SYSTEMD_SLICE": "machine.slice",
    "_BOOT_ID": "5d379cbb22d94db78503e2ce48ba675b",
    "_MACHINE_ID": "593a407f0bf44f8da32824920ad62d25",
    "_HOSTNAME": "ip-172-31-92-238.ec2.internal",
    "CONTAINER_ID_FULL": "37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d",
    "CONTAINER_ID": "37a3e93ed098",
    "CONTAINER_TAG": "app",
    "CONTAINER_NAME": "aliro-orchestrator-app",
    "SYSLOG_IDENTIFIER": "app",
    "_PID": "3879832",
    "_CMDLINE": "/usr/bin/conmon --api-version 1 -c 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d -u 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata -p /run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/pidfile -n aliro-orchestrator-app --exit-dir /run/libpod/exits --full-attach -s -l journald --log-level info --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/oci-log --log-tag app --conmon-pidfile /run/containers/storage/overlay-containers/37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg info --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg boltdb --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d",
    "_SYSTEMD_CGROUP": "/machine.slice/libpod-conmon-37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d.scope",
    "_SYSTEMD_UNIT": "libpod-conmon-37a3e93ed098539ccb1c57f61288c4732d26b45776e2039cf1686f6a784df10d.scope",
    "_SYSTEMD_INVOCATION_ID": "724b21586ffe4e008e0ed099352ae284",
    "MESSAGE": "               ^^^^^^^^^^^^^^^^^^^^^\n",
    "_SOURCE_REALTIME_TIMESTAMP": "1724783541055302"
}

scottwn avatar Aug 27 '24 20:08 scottwn

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

github-actions[bot] avatar Dec 15 '24 02:12 github-actions[bot]

This is unresolved, I would like it triaged and addressed instead of closed

scottwn avatar Dec 15 '24 04:12 scottwn

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

github-actions[bot] avatar Mar 25 '25 02:03 github-actions[bot]

This is still unresolved, please triage it

scottwn avatar Mar 25 '25 14:03 scottwn

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

github-actions[bot] avatar Jun 24 '25 02:06 github-actions[bot]

this is still unresolved

scottwn avatar Jun 24 '25 17:06 scottwn

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

github-actions[bot] avatar Oct 24 '25 02:10 github-actions[bot]

@edsiper @cosmo0920 would anyone be able to take a look at this for @scottwn?

I don't have the access to adjust labels or assign anything here...

eschabell avatar Oct 28 '25 20:10 eschabell