fluent
Custom-Fluentbit not working as expected for multiline parsing
Problem statement:
-
I have deployed custom-fluent-deployment to achieve multiline parsing, but Its not working as expected but facing issue is Some traces are appearing in a single log entry, while others are still being displayed across multiple lines. (Attched sample log)
-
Please find the below fluentbit-configurationa and deamon set ,let me know what are the changes needs to done to enable multiline parsing .
Note: And i want to exclude all health check logs using fluentbit deployment, i tried with below filter but doesn't working as expected, please suggest what is configuration required to exclude all health health check logs.
[FILTER] Name grep Match * Exclude message healthcheck
Custom-fluentbit-config.yaml:
apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-config namespace: fluentbit-custom data: fluent-bit.conf: |- [SERVICE] Flush 1 Grace 120 Log_Level debug Log_File /var/log/fluentbit-custom.log Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_PORT 2030
[INPUT]
Name tail
Alias kube_containers
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*.log
Exclude_Path /var/log/containers/*_kube-system_*.log,/var/log/containers/*_istio-system_*.log,/var/log/containers/*_knative-serving_*.log,/var/log/containers/*_gke-system_*.log,/var/log/containers/*_config-management-system_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube.db
Mem_Buf_Limit 100MB
Skip_Long_Lines On
Refresh_Interval 1
multiline.parser multiline-regex-test
[INPUT]
Name tail
Alias kube_containers_kube-system
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_kube-system_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_kube-system.db
#Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
#Read_from_Head True
[INPUT]
Name tail
Alias kube_containers_istio-system
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_istio-system_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_istio-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias kube_containers_knative-serving
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_knative-serving_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_knative-serving.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias kube_containers_gke-system
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_gke-system_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_gke-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias kube_containers_config-management-system
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_config-management-system_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_config-management-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
[INPUT]
Name tail
Alias kube_containers_gmp-system
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_gmp-system_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_gmp-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias kube_containers_gke-managed-cim
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_gke-managed-cim_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_gke-managed-cim.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias knative
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex \/var\/lib\/kubelet\/pods\/.+\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/knative-internal/**/*/**/*
DB /var/run/custom-fluentbit/pos-files/knative.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_kube-system
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/kube-system_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_kube-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_istio-system
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/istio-system_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_istio-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_knative-serving
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/knative-serving_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_knative-serving.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_gke-system
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/gke-system_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_gke-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_config-management-system
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/config-management-system_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_config-management-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_gmp-system
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/gmp-system_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_gmp-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_gke-managed-cim
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/gke-managed-cim_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_gke-managed-cim.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Exclude_Path /var/log/pods/*/*_panic.log,/var/log/pods/kube-system_*/*,/var/log/pods/istio-system_*/*,/var/log/pods/knative-serving_*/*,/var/log/pods/gke-system_*/*,/var/log/pods/config-management-system_*/*,/var/log/pods/gmp-system_*/*,/var/log/pods/gke-managed-cim_*/*
Path /var/log/pods/*/*.log
DB /var/run/custom-fluentbit/pos-files/gvisor.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_panic
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>panic
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)_panic\.log
Exclude_Path /var/log/pods/kube-system_*/*,/var/log/pods/istio-system_*/*,/var/log/pods/knative-serving_*/*,/var/log/pods/gke-system_*/*,/var/log/pods/config-management-system_*/*,/var/log/pods/gmp-system_*/*,/var/log/pods/gke-managed-cim_*/*
Path /var/log/pods/*/*_panic.log
DB /var/run/custom-fluentbit/pos-files/gvisor_panic.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Parser syslog
Path /var/log/startupscript.log
DB /var/run/custom-fluentbit/pos-files/startupscript.db
Alias startupscript
Tag startupscript
[INPUT]
Name tail
Parser network-log
Alias policy-action
Tag policy-action
Path /var/log/network/policy_action.log
DB /var/run/custom-fluentbit/pos-files/policy-action.db
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
# Example:
# I1118 21:26:53.9757896 proxier.go:1096] Port "nodePort for kube-system/default-http-backend:http" (:31429/tcp) was open before and is still needed
[INPUT]
Name tail
Alias kube-proxy
Tag kube-proxy
Path /var/log/kube-proxy.log
DB /var/run/custom-fluentbit/pos-files/kube-proxy.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Parser glog
Read_from_Head True
# Logs from systemd-journal for interesting services.
[INPUT]
Name systemd
Alias docker
Tag docker
Systemd_Filter _SYSTEMD_UNIT=docker.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/docker.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias kubelet
Tag kubelet
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/kubelet.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias kube-node-installation
Tag kube-node-installation
Systemd_Filter _SYSTEMD_UNIT=kube-node-installation.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/kube-node-installation.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias kube-node-configuration
Tag kube-node-configuration
Systemd_Filter _SYSTEMD_UNIT=kube-node-configuration.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/kube-node-configuration.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias kube-logrotate
Tag kube-logrotate
Systemd_Filter _SYSTEMD_UNIT=kube-logrotate.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/kube-logrotate.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias node-problem-detector
Tag node-problem-detector
Systemd_Filter _SYSTEMD_UNIT=node-problem-detector.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/node-problem-detector.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias kube-container-runtime-monitor
Tag kube-container-runtime-monitor
Systemd_Filter _SYSTEMD_UNIT=kube-container-runtime-monitor.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/kube-container-runtime-monitor.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias kubelet-monitor
Tag kubelet-monitor
Systemd_Filter _SYSTEMD_UNIT=kubelet-monitor.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/kubelet-monitor.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias gcfsd
Tag gcfsd
Systemd_Filter _SYSTEMD_UNIT=gcfsd.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/gcfsd.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias gcfs-snapshotter
Tag gcfs-snapshotter
Systemd_Filter _SYSTEMD_UNIT=gcfs-snapshotter.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/gcfs-snapshotter.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias container-runtime
Tag container-runtime
Systemd_Filter _SYSTEMD_UNIT=containerd.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/container-runtime.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[FILTER]
Name parser
Match kube_*
Key_Name log
Reserve_Data True
Kube_Tag_Prefix kube.var.log.containers.
Parser docker
Parser containerd
[FILTER]
Name modify
Match *
Hard_rename log message
[FILTER]
Name parser
Match kube_*
Key_Name message
Reserve_Data True
Parser glog
Parser json
[FILTER]
Name grep
Match *
Exclude healthcheck
Key_Name message
[OUTPUT]
Name http
Match *
Host 127.0.0.1
Port 2031
URI /logs
header_tag FLUENT-TAG
Format msgpack
Retry_Limit 2
parsers.conf: |- [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L%z
[MULTILINE_PARSER]
name multiline-regex-test
type regex
flush_timeout 1000
#
# Regex rules for multiline parsing
# ---------------------------------
#
# configuration hints:
#
# - first state always has the name: start_state
# - every field in the rule must be inside double quotes
#
# rules | state name | regex pattern | next state
# ------|---------------|--------------------------------------------
#rule "start_state" "/([a-zA-Z]+(.*)+\s+\S+(.*))/" "cont"
#rule "cont" "/^\s+at.*/" "cont"
rule "start_state" "/(([a-zA-Z]+ \d+ \d+\:\d+\:\d+)|(([a-zA-Z]+ \[){0,1}+\d+\-\d+\-\d+))(.*)/" "cont"
rule "cont" "/^\s+.*/" "cont"
[PARSER]
Name containerd
Format regex
Regex ^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
[PARSER]
Name json
Format json
[PARSER]
Name syslog
Format regex
Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
Time_Key time
Time_Format %b %d %H:%M:%S
[PARSER]
Name glog
Format regex
Regex ^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source_file>[^ \]]+)\:(?<source_line>\d+)\]\s(?<message>.*)$
Time_Key time
Time_Format %m%d %H:%M:%S.%L
[PARSER]
Name network-log
Format json
Time_Key timestamp
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
Deamon-set.yaml:
---
apiVersion: apps/v1 kind: DaemonSet metadata: labels: k8s-app: fluent-bit-logging kubernetes.io/cluster-service: "true" name: fluentbit-custom namespace: fluentbit-custom spec: selector: matchLabels: k8s-app: fluent-bit-logging template: metadata: labels: k8s-app: fluent-bit-logging kubernetes.io/cluster-service: "true" spec: containers: - image: gke.gcr.io/fluent-bit:v1.8.12-gke.19 imagePullPolicy: IfNotPresent name: fluentbit-custom ports: - containerPort: 2030 hostPort: 2030 name: metrics protocol: TCP resources: limits: memory: 512Mi requests: cpu: 50m memory: 100Mi volumeMounts: - mountPath: /var/run/custom-fluentbit/pos-files name: varrun - mountPath: /var/log name: varlog - mountPath: /var/lib/kubelet/pods name: varlibkubeletpods - mountPath: /var/lib/docker/containers name: varlibdockercontainers readOnly: true - mountPath: /fluent-bit/etc/ name: config-volume - command: - /fluent-bit-gke-exporter - --port=2031 - --kubernetes-separator=_ - --stackdriver-resource-model=k8s - --enable-pod-label-discovery - --pod-label-dot-replacement=_ - --split-stdout-stderr - --logtostderr - --pool-size=100 image: gke.gcr.io/fluent-bit-gke-exporter:v0.11.0-gke.0 imagePullPolicy: IfNotPresent name: fluentbit-gke-custom ports: - containerPort: 2031 hostPort: 2031 name: metrics protocol: TCP resources: limits: memory: 250Mi requests: cpu: 50m memory: 100Mi dnsPolicy: Default hostNetwork: true serviceAccount: fluentbit-access-sa serviceAccountName: fluentbit-access-sa terminationGracePeriodSeconds: 120 volumes: - hostPath: path: /var/run/custom-fluentbit/pos-files type: "" name: varrun - hostPath: path: /var/log type: "" name: varlog - hostPath: path: /var/lib/kubelet/pods type: "" name: varlibkubeletpods - hostPath: path: /var/lib/docker/containers type: "" name: varlibdockercontainers - configMap: defaultMode: 420 name: fluent-bit-config name: config-volume updateStrategy: rollingUpdate: maxUnavailable: 1 type: RollingUpdate
Thanks Waiting for your valuable inputs.
@patrick-stephens/ @Team Could you please check and give your valuable inputs
If your application is reading logs from a container, first you should parse the runtime(cri or docker) and then use a multiline filter with your custom multiline parser.
Change multiline.parser multiline-regex-test in the elow INPUT section by multiline.parser cri, docker
[INPUT]
Name tail
Alias kube_containers
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*.log
Exclude_Path /var/log/containers/*_kube-system_*.log,/var/log/containers/*_istio-system_*.log,/var/log/containers/*_knative-serving_*.log,/var/log/containers/*_gke-system_*.log,/var/log/containers/*_config-management-system_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube.db
Mem_Buf_Limit 100MB
Skip_Long_Lines On
Refresh_Interval 1
multiline.parser multiline-regex-test
Then add a [FILTER] section where you can use the below built-in multiline parser if your java stack trace is standard.
[FILTER]
name multiline
match *
multiline.key_content log
multiline.parser java
Hi Richardo,
Thanks for your valuable feedback, so it means i dont need to use custom multipline parser right if i parse the runtime(cri or docker) ?
[INPUT] Name tail Alias kube_containers Tag kube_<namespace_name><pod_name><container_name> Tag_Regex (?<pod_name>a-z0-9?(.a-z0-9?))(?<namespace_name>[^]+)_(?<container_name>.+)- Path /var/log/containers/.log Exclude_Path /var/log/containers/kube-system.log,/var/log/containers/istio-system.log,/var/log/containers/knative-serving.log,/var/log/containers/gke-system.log,/var/log/containers/config-management-system.log DB /var/run/custom-fluentbit/pos-files/flb_kube.db Mem_Buf_Limit 100MB Skip_Long_Lines On Refresh_Interval 1 multiline.parser cri, docker
Then add a [FILTER] section where you can use the below built-in multiline parser if your java stack trace is standard.
[FILTER] name multiline match * multiline.key_content log multiline.parser java
Can i use the same syntax the above?
And i am seeing one more issue, I need to exclude all health check logs for that i have added a filter, but it doesnot working as expected.Please suggest and share your inputs.
[FILTER] Name grep Match * Exclude message /.healthcheck./
Attaching the configuration here please check and share your valuable feedback.
On Tue, Jan 30, 2024 at 9:16 PM Ricardo Ahumada @.***> wrote:
If your application is reading logs from a container, first you should parse the runtime(cri or docker) and then use a multiline filter with your custom multiline parser.
Change multiline.parser multiline-regex-test in the elow INPUT section by multiline.parser cri, docker
[INPUT] Name tail Alias kube_containers Tag kube_<namespace_name><pod_name><container_name> Tag_Regex (?<pod_name>a-z0-9?(.a-z0-9?))(?<namespace_name>[^]+)_(?<container_name>.+)- Path /var/log/containers/.log Exclude_Path /var/log/containers/kube-system.log,/var/log/containers/istio-system.log,/var/log/containers/knative-serving.log,/var/log/containers/gke-system.log,/var/log/containers/config-management-system.log DB /var/run/custom-fluentbit/pos-files/flb_kube.db Mem_Buf_Limit 100MB Skip_Long_Lines On Refresh_Interval 1 multiline.parser multiline-regex-test
Then add a [FILTER] section where you can use the below built-in multiline parser if your java stack trace is standard.
[FILTER] name multiline match * multiline.key_content log multiline.parser java
— Reply to this email directly, view it on GitHub https://github.com/fluent/fluent-bit/issues/8397#issuecomment-1917293418, or unsubscribe https://github.com/notifications/unsubscribe-auth/BBGDCX2C5D4OBKDUQQGFAJTYREITLAVCNFSM6AAAAABCC5DLI6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJXGI4TGNBRHA . You are receiving this because you authored the thread.Message ID: @.***>
--
-----------------------------------------------------------------------------------------
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.****
****
Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.****
****
Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information provided, unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
-----------------------------------------------------------------------------------------
Hi @RicardoAAD
As i mentioned in the above, we have deployed custom fluent bit,with existing configuration all deployment logs are coming but that time multiline parsing didnt work for that i made configuration changes and tried , but the new deployment changes logs are not being ingested in gcp logging,After i rollback logs are coming, but now i want to know what could be the issue my updated configuration,
- The main purpose is enable multiline parsing and also want to exclude all health check logs.
Older version Configuration: Multiline not working , but all logs are being ingested to gcp cloud logging.
apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-config namespace: fluentbit-custom data: fluent-bit.conf: |- [SERVICE] Flush 1 Grace 120 Log_Level debug Log_File /var/log/fluentbit-custom.log Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_PORT 2030
[INPUT]
Name tail
Alias kube_containers
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*.log
Exclude_Path /var/log/containers/*_kube-system_*.log,/var/log/containers/*_istio-system_*.log,/var/log/containers/*_knative-serving_*.log,/var/log/containers/*_gke-system_*.log,/var/log/containers/*_config-management-system_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube.db
Mem_Buf_Limit 100MB
Skip_Long_Lines On
Refresh_Interval 1
multiline.parser multiline-regex-test
[INPUT]
Name tail
Alias kube_containers_kube-system
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_kube-system_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_kube-system.db
#Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
#Read_from_Head True
[INPUT]
Name tail
Alias kube_containers_istio-system
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_istio-system_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_istio-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias kube_containers_knative-serving
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_knative-serving_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_knative-serving.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias kube_containers_gke-system
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_gke-system_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_gke-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias kube_containers_config-management-system
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_config-management-system_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_config-management-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
[INPUT]
Name tail
Alias kube_containers_gmp-system
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_gmp-system_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_gmp-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias kube_containers_gke-managed-cim
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_gke-managed-cim_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_gke-managed-cim.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias knative
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex \/var\/lib\/kubelet\/pods\/.+\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/knative-internal/**/*/**/*
DB /var/run/custom-fluentbit/pos-files/knative.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_kube-system
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/kube-system_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_kube-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_istio-system
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/istio-system_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_istio-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_knative-serving
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/knative-serving_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_knative-serving.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_gke-system
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/gke-system_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_gke-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_config-management-system
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/config-management-system_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_config-management-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_gmp-system
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/gmp-system_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_gmp-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_gke-managed-cim
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/gke-managed-cim_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_gke-managed-cim.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Exclude_Path /var/log/pods/*/*_panic.log,/var/log/pods/kube-system_*/*,/var/log/pods/istio-system_*/*,/var/log/pods/knative-serving_*/*,/var/log/pods/gke-system_*/*,/var/log/pods/config-management-system_*/*,/var/log/pods/gmp-system_*/*,/var/log/pods/gke-managed-cim_*/*
Path /var/log/pods/*/*.log
DB /var/run/custom-fluentbit/pos-files/gvisor.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_panic
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>panic
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)_panic\.log
Exclude_Path /var/log/pods/kube-system_*/*,/var/log/pods/istio-system_*/*,/var/log/pods/knative-serving_*/*,/var/log/pods/gke-system_*/*,/var/log/pods/config-management-system_*/*,/var/log/pods/gmp-system_*/*,/var/log/pods/gke-managed-cim_*/*
Path /var/log/pods/*/*_panic.log
DB /var/run/custom-fluentbit/pos-files/gvisor_panic.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Parser syslog
Path /var/log/startupscript.log
DB /var/run/custom-fluentbit/pos-files/startupscript.db
Alias startupscript
Tag startupscript
[INPUT]
Name tail
Parser network-log
Alias policy-action
Tag policy-action
Path /var/log/network/policy_action.log
DB /var/run/custom-fluentbit/pos-files/policy-action.db
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
# Example:
# I1118 21:26:53.9757896 proxier.go:1096] Port "nodePort for kube-system/default-http-backend:http" (:31429/tcp) was open before and is still needed
[INPUT]
Name tail
Alias kube-proxy
Tag kube-proxy
Path /var/log/kube-proxy.log
DB /var/run/custom-fluentbit/pos-files/kube-proxy.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Parser glog
Read_from_Head True
# Logs from systemd-journal for interesting services.
[INPUT]
Name systemd
Alias docker
Tag docker
Systemd_Filter _SYSTEMD_UNIT=docker.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/docker.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias kubelet
Tag kubelet
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/kubelet.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias kube-node-installation
Tag kube-node-installation
Systemd_Filter _SYSTEMD_UNIT=kube-node-installation.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/kube-node-installation.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias kube-node-configuration
Tag kube-node-configuration
Systemd_Filter _SYSTEMD_UNIT=kube-node-configuration.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/kube-node-configuration.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias kube-logrotate
Tag kube-logrotate
Systemd_Filter _SYSTEMD_UNIT=kube-logrotate.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/kube-logrotate.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias node-problem-detector
Tag node-problem-detector
Systemd_Filter _SYSTEMD_UNIT=node-problem-detector.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/node-problem-detector.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias kube-container-runtime-monitor
Tag kube-container-runtime-monitor
Systemd_Filter _SYSTEMD_UNIT=kube-container-runtime-monitor.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/kube-container-runtime-monitor.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias kubelet-monitor
Tag kubelet-monitor
Systemd_Filter _SYSTEMD_UNIT=kubelet-monitor.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/kubelet-monitor.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias gcfsd
Tag gcfsd
Systemd_Filter _SYSTEMD_UNIT=gcfsd.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/gcfsd.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias gcfs-snapshotter
Tag gcfs-snapshotter
Systemd_Filter _SYSTEMD_UNIT=gcfs-snapshotter.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/gcfs-snapshotter.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias container-runtime
Tag container-runtime
Systemd_Filter _SYSTEMD_UNIT=containerd.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/container-runtime.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[FILTER]
Name parser
Match kube_*
Key_Name log
Reserve_Data True
Kube_Tag_Prefix kube.var.log.containers.
Parser docker
Parser containerd
[FILTER]
Name modify
Match *
Hard_rename log message
[FILTER]
Name parser
Match kube_*
Key_Name message
Reserve_Data True
Parser glog
Parser json
[OUTPUT]
Name http
Match *
Host 127.0.0.1
Port 2031
URI /logs
header_tag FLUENT-TAG
Format msgpack
Retry_Limit 2
parsers.conf: |- [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L%z
[MULTILINE_PARSER]
name multiline-regex-test
type regex
flush_timeout 1000
#
# Regex rules for multiline parsing
# ---------------------------------
#
# configuration hints:
#
# - first state always has the name: start_state
# - every field in the rule must be inside double quotes
#
# rules | state name | regex pattern | next state
# ------|---------------|--------------------------------------------
#rule "start_state" "/([a-zA-Z]+(.*)+\s+\S+(.*))/" "cont"
#rule "cont" "/^\s+at.*/" "cont"
rule "start_state" "/(([a-zA-Z]+ \d+ \d+\:\d+\:\d+)|(([a-zA-Z]+ \[){0,1}+\d+\-\d+\-\d+))(.*)/" "cont"
rule "cont" "/^\s+.*/" "cont"
[PARSER]
Name containerd
Format regex
Regex ^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
[PARSER]
Name json
Format json
[PARSER]
Name syslog
Format regex
Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
Time_Key time
Time_Format %b %d %H:%M:%S
[PARSER]
Name glog
Format regex
Regex ^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source_file>[^ \]]+)\:(?<source_line>\d+)\]\s(?<message>.*)$
Time_Key time
Time_Format %m%d %H:%M:%S.%L
[PARSER]
Name network-log
Format json
Time_Key timestamp
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
New version configuration: When i deployed this configuration multiline parsing working for few cases but logs are not being ingested to gcp logging.
apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-config namespace: fluentbit-custom data: fluent-bit.conf: |- [SERVICE] Flush 1 Grace 120 Log_Level info Log_File /var/log/fluentbit-custom.log Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_PORT 2030
[INPUT]
Name tail
Alias kube_containers
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*.log
Exclude_Path /var/log/containers/*_kube-system_*.log,/var/log/containers/*_istio-system_*.log,/var/log/containers/*_knative-serving_*.log,/var/log/containers/*_gke-system_*.log,/var/log/containers/*_config-management-system_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube.db
Mem_Buf_Limit 100MB
Skip_Long_Lines On
Refresh_Interval 1
multiline.parser multiline-regex-test
[INPUT]
Name tail
Alias kube_containers_kube-system
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_kube-system_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_kube-system.db
#Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
#Read_from_Head True
[INPUT]
Name tail
Alias kube_containers_istio-system
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_istio-system_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_istio-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias kube_containers_knative-serving
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_knative-serving_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_knative-serving.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias kube_containers_gke-system
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_gke-system_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_gke-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias kube_containers_config-management-system
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_config-management-system_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_config-management-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
[INPUT]
Name tail
Alias kube_containers_gmp-system
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_gmp-system_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_gmp-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias kube_containers_gke-managed-cim
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Path /var/log/containers/*_gke-managed-cim_*.log
DB /var/run/custom-fluentbit/pos-files/flb_kube_gke-managed-cim.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias knative
Tag kube_<namespace_name>_<pod_name>_<container_name>
Tag_Regex \/var\/lib\/kubelet\/pods\/.+\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/knative-internal/**/*/**/*
DB /var/run/custom-fluentbit/pos-files/knative.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_kube-system
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/kube-system_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_kube-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_istio-system
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/istio-system_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_istio-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_knative-serving
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/knative-serving_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_knative-serving.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_gke-system
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/gke-system_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_gke-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_config-management-system
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/config-management-system_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_config-management-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_gmp-system
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/gmp-system_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_gmp-system.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_gke-managed-cim
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Path /var/log/pods/gke-managed-cim_*/*
DB /var/run/custom-fluentbit/pos-files/gvisor_gke-managed-cim.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)\.log
Exclude_Path /var/log/pods/*/*_panic.log,/var/log/pods/kube-system_*/*,/var/log/pods/istio-system_*/*,/var/log/pods/knative-serving_*/*,/var/log/pods/gke-system_*/*,/var/log/pods/config-management-system_*/*,/var/log/pods/gmp-system_*/*,/var/log/pods/gke-managed-cim_*/*
Path /var/log/pods/*/*.log
DB /var/run/custom-fluentbit/pos-files/gvisor.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Alias gvisor_panic
Tag kube-pod_<namespace_name>_<pod_name>_<runtime>panic
Tag_Regex \/var\/log\/pods\/(?<namespace_name>[^_]+)_(?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<id>[\-a-z0-9]+)\/(?<runtime>[a-z]+)_panic\.log
Exclude_Path /var/log/pods/kube-system_*/*,/var/log/pods/istio-system_*/*,/var/log/pods/knative-serving_*/*,/var/log/pods/gke-system_*/*,/var/log/pods/config-management-system_*/*,/var/log/pods/gmp-system_*/*,/var/log/pods/gke-managed-cim_*/*
Path /var/log/pods/*/*_panic.log
DB /var/run/custom-fluentbit/pos-files/gvisor_panic.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
[INPUT]
Name tail
Parser syslog
Path /var/log/startupscript.log
DB /var/run/custom-fluentbit/pos-files/startupscript.db
Alias startupscript
Tag startupscript
[INPUT]
Name tail
Parser network-log
Alias policy-action
Tag policy-action
Path /var/log/network/policy_action.log
DB /var/run/custom-fluentbit/pos-files/policy-action.db
Skip_Long_Lines On
Refresh_Interval 5
Read_from_Head True
# Example:
# I1118 21:26:53.9757896 proxier.go:1096] Port "nodePort for kube-system/default-http-backend:http" (:31429/tcp) was open before and is still needed
[INPUT]
Name tail
Alias kube-proxy
Tag kube-proxy
Path /var/log/kube-proxy.log
DB /var/run/custom-fluentbit/pos-files/kube-proxy.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
Parser glog
Read_from_Head True
# Logs from systemd-journal for interesting services.
[INPUT]
Name systemd
Alias docker
Tag docker
Systemd_Filter _SYSTEMD_UNIT=docker.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/docker.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias kubelet
Tag kubelet
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/kubelet.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias kube-node-installation
Tag kube-node-installation
Systemd_Filter _SYSTEMD_UNIT=kube-node-installation.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/kube-node-installation.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias kube-node-configuration
Tag kube-node-configuration
Systemd_Filter _SYSTEMD_UNIT=kube-node-configuration.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/kube-node-configuration.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias kube-logrotate
Tag kube-logrotate
Systemd_Filter _SYSTEMD_UNIT=kube-logrotate.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/kube-logrotate.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias node-problem-detector
Tag node-problem-detector
Systemd_Filter _SYSTEMD_UNIT=node-problem-detector.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/node-problem-detector.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias kube-container-runtime-monitor
Tag kube-container-runtime-monitor
Systemd_Filter _SYSTEMD_UNIT=kube-container-runtime-monitor.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/kube-container-runtime-monitor.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias kubelet-monitor
Tag kubelet-monitor
Systemd_Filter _SYSTEMD_UNIT=kubelet-monitor.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/kubelet-monitor.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias gcfsd
Tag gcfsd
Systemd_Filter _SYSTEMD_UNIT=gcfsd.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/gcfsd.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias gcfs-snapshotter
Tag gcfs-snapshotter
Systemd_Filter _SYSTEMD_UNIT=gcfs-snapshotter.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/gcfs-snapshotter.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[INPUT]
Name systemd
Alias container-runtime
Tag container-runtime
Systemd_Filter _SYSTEMD_UNIT=containerd.service
Path /var/log/journal
DB /var/run/custom-fluentbit/pos-files/container-runtime.db
Buffer_Max_Size 1MB
Mem_Buf_Limit 1MB
[FILTER]
Name parser
Match kube_*
Key_Name log
Reserve_Data True
Kube_Tag_Prefix kube.var.log.containers.
Parser docker
Parser containerd
[FILTER]
Name modify
Match *
Hard_rename log message
[FILTER]
Name parser
Match kube_*
Key_Name message
Reserve_Data True
Parser glog
Parser json
[FILTER]
Name grep
Match *
Exclude message /.*healthcheck.*/
#[FILTER]
# name multiline
# match *
# multiline.key_content log
# multiline.parser multiline-regex-test
[OUTPUT]
Name http
Match *
Host 127.0.0.1
Port 2031
URI /logs
header_tag FLUENT-TAG
Format msgpack
Retry_Limit 2
parsers.conf: |- [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L%z
[MULTILINE_PARSER]
name multiline-regex-test
type regex
parser java,python
flush_timeout 1000
#
# Regex rules for multiline parsing
# ---------------------------------
#
# configuration hints:
#
# - first state always has the name: start_state
# - every field in the rule must be inside double quotes
#
# rules | state name | regex pattern | next state
# ------|---------------|--------------------------------------------
#rule "start_state" "/(([a-zA-Z]+ \d+ \d+\:\d+\:\d+)|(([a-zA-Z]+ \[){0,1}+\d+\-\d+\-\d+))(.*)/" "cont"
#rule "cont" "/^\s+.*/" "cont"
rule "start_state" "/([a-zA-Z]+(.*)+\s+\S+(.*))/" "cont"
rule "cont" "/^\s+.*/" "cont"
[PARSER]
Name containerd
Format regex
Regex ^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
[PARSER]
Name json
Format json
[PARSER]
Name syslog
Format regex
Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
Time_Key time
Time_Format %b %d %H:%M:%S
[PARSER]
Name glog
Format regex
Regex ^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source_file>[^ \]]+)\:(?<source_line>\d+)\]\s(?<message>.*)$
Time_Key time
Time_Format %m%d %H:%M:%S.%L
[PARSER]
Name network-log
Format json
Time_Key timestamp
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
Not able to find what could be the root cause to stop all the log ingestion when i use newer version.This is the major block in our environment.
Waiting for your valuable inputs
Hi @RicardoAAD could you please check and share your valuable inputs.
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue was closed because it has been stalled for 5 days with no activity.