high receive queue buffer and udp drop issue

[Seylul]

"Hi, When i run this appliation with high traffic , i see udp errors above 14M written line count. I added the ring buffer parameter and run again but nothing change. I don't see interface drop , i dont see high memory usage , i see disj i/o only 200 MBs and its normal for my server. How can i decrease this udp error. I use json plugin for output like this.

<ipfixcol2>
  <!-- Input plugins -->
  <inputPlugins>
    <input>
      <verbosity>debug</verbosity>
      <name>UDP input - 1234</name>
      <plugin>udp</plugin>
      <params>
        <localPort>1234</localPort>
        <localIPAddress></localIPAddress>
        <!-- Optional parameters -->
        <connectionTimeout>600</connectionTimeout>
        <templateLifeTime>600</templateLifeTime>
        <optionsTemplateLifeTime>600</optionsTemplateLifeTime>
      </params>
    </input>
  </inputPlugins>

  <!-- Output plugins -->
  <outputPlugins>
    <output>
      <name>JSON output</name>
      <plugin>json</plugin>
      <params>
        <tcpFlags>formatted</tcpFlags>
        <timestamp>unix</timestamp>
        <protocol>formatted</protocol>
        <ignoreUnknown>false</ignoreUnknown>
        <ignoreOptions>false</ignoreOptions>
        <nonPrintableChar>true</nonPrintableChar>
        <octetArrayAsUint>true</octetArrayAsUint>
        <numericNames>false</numericNames>
        <splitBiflow>false</splitBiflow>
        <detailedInfo>true</detailedInfo>
        <templateInfo>false</templateInfo>

        <outputs>
          <!-- Choose one or more of the following outputs -->
          <file>
            <name>Store to files</name>
            <path>local_path/</path>
            <prefix>path_1234_</prefix>
            <timeWindow>60</timeWindow>
            <timeAlignment>yes</timeAlignment>
            <compression>none</compression>
          </file>
        </outputs>
      </params>
    </output>
  </outputPlugins>
</ipfixcol2>

[sedmicha]

Hello, just to confirm, by "ring buffer parameter" do you mean that you tried increasing "net.core.rmem_max" as mentioned in the README? If so, what value have you tried? And how many flow records per second are you trying to process?

[Seylul]

i mean as below on start command: ipfixcol2 -d -r 163840 -c /udp2json.xml -v 0 also my rmem_max parameter is ; rmem_default = 943718400 rmem_max = 943718400. Also os: redhat7.9 receiving udp packets are normally per second 22.6K , if it goes over 22.5K udp packet drops starts

[sedmicha]

Could you run the collector with the dummy output plugin with stats enabled for a set number of time, so we can get an idea of how much flow records per second that is?

For example:

$ cat config.xml
<ipfixcol2>
  <inputPlugins>
    <input>
      <name>UDP input - 1234</name>
      <plugin>udp</plugin>
      <params>
        <localPort>1234</localPort>
        <localIPAddress></localIPAddress>
      </params>
    </input>
  </inputPlugins>
  <outputPlugins>
    <output>
      <name>Dummy</name>
      <plugin>dummy</plugin>
      <params>
        <delay>0</delay>
        <stats>true</stats>
      </params>
    </output>
  </outputPlugins>
</ipfixcol2>

$ timeout 60s ipfixcol2 -c config.xml
Stats:
- data records:     894402536
- options records:          0
- total bytes:     11014571329116
- total packets:   12743947426