[FR] Proposal: Loki exporter

[jiekun]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Description

Hi. I'm proposing a Loki exporter based on the current exporter interface. We can transform flow_log into formatted logs and display them on any log platform.

To make my proposal more intuitive, I am providing some examples of output logs and configuration here.

Logs:

time=2023-08-29 18:13:46.022201 +0800 CST, service_name=deepflow-server-querier, log_level=INFO, trace_id=5363c6bb2a62d0033d6cd6b20ca39326, span_id=38ff552004e1aa32, method=POST, name=deepflow-server:20416, path=/v1/query/?debug=false, status_code=826012680696, response_exception=

configuration:

ingester:
  exporters:
    - name: example loki exporter
      exporter_type: loki-exporter
      loki-exporter:
        url: my-loki-url
        tenant-id: default-tenant-id # this could be replaced by some field in flow_log
        queue-count: 16
        queue-size: 1024
        max-message-wait-second: 1
        max-message-bytes: 16384
        timeout-second: 5
        min-backoff-second: 1
        max-backoff-second: 5
        max-retries: 3
        static-labels:
          foo: bar
        export-datas:
          - ebpf-sys-span
          - ...
        export-data-types:
          - service_info
          - tracing_info
          - network_layer
          - ...
        export-only-with-traceid: false
        log-format:  # provide costom log format support
          mapping:
            service_name: app
            ...

I'm looking foward to some feedback of the design. Feel free to comment with your requirement and expectation.

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[Nick-0314]

LGTM

[jiekun]

I'm going to postpone this issue and related pull request due to the changes of exporter folder structure. and work on prometheus exporter first.

[sharang]

@jiekun, how is the progress on this task? However, transforming data from structured to unstructured (log) might not be a good idea :)

[jiekun]

1. We developed an internal exporter that exports data in both types: log and metric. Since the customized exporter is not universal, we did not merge it back to the upstream. I think setting up a 3rd party repo that holds those exporter would be better, which require a more flexible exporter interface. (e.g. git submodule)
2. And yes the log format is quite variable, it could be hard to provide a good user interface to customize it. So there are a lot of hardcoded things in our current exporter, which also stops us from merging it back to the upstream.

[sharang]

@jiekun, I've initialized a new repo: https://github.com/deepflowio/deepflow-exporter-contrib. I was wondering if you would be interested in becoming a maintainer of this repo?

[jiekun]

@sharang I'm happy to be a reviewer for some new exporters in contrib repo. However, before we can proceed, the exporter in the main repository may need some changes to adopt the submodule way (or any other extension form). @lzf575

[sharang]

@sharang I'm happy to be a reviewer for some new exporters in contrib repo. However, before we can proceed, the exporter in the main repository may need some changes to adopt the submodule way (or any other extension form). @lzf575

@jiekun, I have set you as a maintainer of deepflow-exporter-contrib, and also, any modifications related to the 'submodule' are welcome to be submitted as PRs.