Regular expressions in include_units for the journald source

[biomack]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Can you add regex for the list include_units: ? It should be look like this

include_units:
  - ^ceph.*.service$

or

include_units:
  - ceph*

We use journald and when we want to collect logs only from ceph, we need to look what services need to add on each server because all have different names.

include_units:
  - [email protected]
  - ceph-dd1cb5bc-a397-11ed-8a26-7278ce097749@haproxy.rgw.group0.ceph-n6.ctnfnj.service
  - ceph-dd1cb5bc-a397-11ed-8a26-7278ce097749@keepalived.rgw.group0.ceph-n6.gbqilx.service
  - ceph-dd1cb5bc-a397-11ed-8a26-7278ce097749@mds.group0.ceph-n6.nkwhnt.service
  - ceph-dd1cb5bc-a397-11ed-8a26-7278ce097749@mds.group0.ceph-n6.xcxzau.service
  - ceph-dd1cb5bc-a397-11ed-8a26-7278ce097749@mgr.ceph-n6.ctwclv.service
  - [email protected]
  - [email protected]
  - [email protected]
  - [email protected]
  - [email protected]
  - [email protected]

Attempted Solutions

No response

Proposal

No response

References

No response

Version

vector 0.26.0

[jszwedko]

Thanks for opening this @biomack !

Agreed, that would be useful. It seems like it'd be pretty straightforward to add too since we do the matching in Vector itself:

https://github.com/vectordotdev/vector/blob/11cb5f7463eb86d8ee064784c1741853f58484fe/src/sources/journald.rs#L852-L860

[jszwedko]

It looks like journalctl also supports filtering by units including patterns:

-u, --unit=UNIT|PATTERN
Show messages for the specified systemd unit UNIT (such as a service unit), or for any of the units matched by PATTERN. If a pattern is specified, a list of unit names found in the journal is compared with the specified pattern and all that match are used. For each unit name, a match is added for messages from the unit ("_SYSTEMD_UNIT=UNIT"), along with additional matches for messages from systemd and messages about coredumps for the specified unit. A match is also added for "_SYSTEMD_SLICE=UNIT", such that if the provided UNIT is a [systemd.slice(5)](https://www.freedesktop.org/software/systemd/man/systemd.slice.html#) unit, all logs of children of the slice will be shown.

With --user, all --unit arguments will be converted to match user messages as if specified with --user-unit.

This parameter can be specified multiple times.

https://www.freedesktop.org/software/systemd/man/journalctl.html

[inatale]

We are also interested in this feature (at least to include wildcard). We use journald source to monitor OpenNebula and it would be execllent to use:

[sources.one_journald]
type = "journald"
include_units = [ "one-*" ]

instead of including each service.

[jiping-s]

We've been troubled by this for a long time. But journalctl's filtering looks like it expands wildcard to static unit names at startup, and probably won't work for newly-added services unless the process is restarted.

The underlying library calls (man sd_journal_add_match) offer no such functions.

[RegreTTO]

I'd like to revive this topic since I have also faced with such a problem, which can be solved by implementing regexp matching in include_units