fluentd icon indicating copy to clipboard operation
fluentd copied to clipboard

Multiple store sections cannot be described in yaml format

Open soukichi opened this issue 3 years ago • 2 comments

Describe the bug

When using the copy plugin to write multiple output sections in yaml, the store section is not expanded correctly. For example, if I load the following configuration, the store keyword is appended. If I use the config keyword under the store keyword, I get an error message "Missing '@type' parameter in section".

e.g.:

system:
  enable_jit: true
config:
  - source:
      $type: sample
      sample: '{"hello":"world"}'
      auto_increment_key: foo_key
      tag: sample
  - match:
      $type: copy
      $tag: "**"
      store:
        - store:
          $type: stdout
        - store:
          $type: stdout

result:

<ROOT>
  <system>
    enable_jit true
  </system>
  <source>
    @type sample
    sample {"hello":"world"}
    auto_increment_key "foo_key"
    tag "sample"
  </source>
  <match **>
    @type copy
    <store>
      @type "stdout"
      @id stdout1
      store 
    </store>
    <store>
      @type "stdout"
      @id stdout2
      store 
    </store>
  </match>
</ROOT>

To Reproduce

see above

Expected behavior

I expect the configuration expanded as below

<ROOT>
  <system>
    enable_jit true
  </system>
  <source>
    @type sample
    sample {"hello":"world"}
    auto_increment_key "foo_key"
    tag "sample"
  </source>
  <match **>
    @type copy
    <store>
      @type "stdout"
      @id stdout1
    </store>
    <store>
      @type "stdout"
      @id stdout2
    </store>
  </match>
</ROOT>

Your Environment

- Fluentd version:1.15.2
- Operating system:Ubuntu22.04
- Kernel version:5.15.0-40

Your Configuration

system:
  enable_jit: true
config:
  - source:
      $type: sample
      sample: '{"hello":"world"}'
      auto_increment_key: foo_key
      tag: sample
  - match:
      $type: copy
      $tag: "**"
      store:
        - store:
          $type: stdout
        - store:
          $type: stdout

Your Error Log

if use the config keyword, it says below:

2022-10-24 13:54:42 +0900 [error]: config error file="fluentd.yml" error_class=Fluent::ConfigError 
error="Missing '@type' parameter in <store> section"

config:
system:
  enable_jit: true
config:
  - source:
      $type: sample
      sample: '{"hello":"world"}'
      auto_increment_key: foo_key
      tag: sample
  - match:
      $type: copy
      $tag: "**"
      store:
        config:
          - store:
              $type: stdout
          - store:
              $type: stdout

Additional context

No response

soukichi avatar Oct 24 '22 05:10 soukichi

I can confirm this is bug. Needs a fix.

fujimotos avatar Oct 26 '22 06:10 fujimotos

Like this:

config:
  - source:
      $type: forward
      port: 24224

  - source:
      $type: http
      port: 9880

  - match:
      $tag: fluentd-*.**
      $type: copy
      store:
        - $type: elasticsearch
          host: elasticsearch
          port: 9200
          logstash_format: true
          include_tag_key: true
          tag_key: $log_name
        - $type: stdout

tabuyos avatar Jan 05 '23 09:01 tabuyos