[prometheus] Consider adding native support for `cs_lapi_decision`

[DuvelCorp]

What happened?

Just installed Crowdsec for Caddy. Everything seems to work fine I triggered myself from a VPN IP an alert by scanning manually several sensitive files, and I got banned as expected :

tom@cerbere:/opt/crowdsec/config$ docker exec crowdsec cscli decisions list
+-------+----------+------------------+------------------------------------+--------+---------+---------------------------------------+--------+------------+----------+
|   ID  |  Source  |    Scope:Value   |               Reason               | Action | Country |                   AS                  | Events | expiration | Alert ID |
+-------+----------+------------------+------------------------------------+--------+---------+---------------------------------------+--------+------------+----------+
| 84721 | crowdsec | Ip:45.134.79.139 | crowdsecurity/http-sensitive-files | ban    | FR      | 39486 HostRoyale Technologies Pvt Ltd | 6      | 2h49m26s   | 8        |
+-------+----------+------------------+------------------------------------+--------+---------+---------------------------------------+--------+------------+----------+

tom@cerbere:/opt/crowdsec/config$ docker exec crowdsec cscli alerts list
+----+------------------+------------------------------------+---------+---------------------------------------+-----------+-----------------------------------------+
| ID |       value      |               reason               | country |                   as                  | decisions |                created_at               |
+----+------------------+------------------------------------+---------+---------------------------------------+-----------+-----------------------------------------+
| 8  | Ip:45.134.79.139 | crowdsecurity/http-sensitive-files | FR      | 39486 HostRoyale Technologies Pvt Ltd | ban:1     | 2024-10-19 16:04:34.205253523 +0000 UTC |
+----+------------------+------------------------------------+---------+---------------------------------------+-----------+-----------------------------------------+

Prometheus seems to retrieve correctly most of crowdsec metrics :

However I dont get any metric cs_lapi_decision According to me I should have received it as there is an active decision on-going. This prevent me to add a list of banned IPs and their Geomap in Grafana, which probably the most important information.

Is there something I missed?

What did you expect to happen?

Get metric data cs_lapi_decision

How can we reproduce it (as minimally and precisely as possible)?

I dont know

Anything else we need to know?

No response

Crowdsec version

version: v1.6.3-4851945a
Codename: alphaga
BuildDate: 2024-09-12_09:39:08
GoVersion: 1.22.6
Platform: docker
libre2: C++
User-Agent: crowdsec/v1.6.3-4851945a-docker
Constraint_parser: >= 1.0, <= 3.0
Constraint_scenario: >= 1.0, <= 3.0
Constraint_api: v1
Constraint_acquis: >= 1.0, < 2.0

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Enabled collections and parsers

$ cscli hub list -o raw
# paste output here

Acquisition config

```console # On Linux: $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* # paste output here

On Windows:

C:> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml

paste output here

Config show

$ cscli config show
# paste output here

Prometheus metrics

$ cscli metrics
# paste output here

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

[github-actions[bot]]

@DuvelCorp: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[DuvelCorp]

Hello

After a few days, my lapi and caddy/crowdsec seems to work fine. I've got real alerts :

Some of my grafana dashboards works perfectly:

But, still : other dashboard that are supposed to report the IPs banned and their geoloc are not working because I am missing some cs_lapi metrics.

This all what I have :

And thus I am missing a part of those listed in this doc : https://docs.crowdsec.net/docs/next/observability/prometheus/

Local API cs_lapi_route_requests_total : number of calls to each route per method cs_lapi_machine_requests_total : number of calls to each route per method grouped by machines cs_lapi_bouncer_requests_total : number of calls to each route per method grouped by bouncers cs_lapi_decisions_ko_total : number of unsuccessfully responses when bouncers ask for an IP. cs_lapi_decisions_ok_total : number of successfully responses when bouncers ask for an

Any hint please

[LaurenceJJones]

Hey 👋🏻

We dont provide the actual decisions (ip address and geo location data) via the Prometheus endpoint we only provide metrics about counts (EG: scenarios has trigger X times).

Thing to note is Prometheus will only expose metrics it has counts for so the list you provided has some:

cs_lapi_route_requests_total : number of calls to each route per method
cs_lapi_machine_requests_total : number of calls to each route per method grouped by machines

However I dont see the ones below:

cs_lapi_bouncer_requests_total : number of calls to each route per method grouped by bouncers
cs_lapi_decisions_ko_total : number of unsuccessfully responses when bouncers ask for an IP.
cs_lapi_decisions_ok_total : number of successfully responses when bouncers ask for an IP.

So the question is do you have a remediation component (bouncers) interfacing with this machine LAPI?

[DuvelCorp]

@LaurenceJJones Thank you for your answer.

If that was not clear I obviously have a bouncer embedded in Caddy using custom built image this https://github.com/hslatman/caddy-crowdsec-bouncer

But now that you state that you dont expose this metric I have understood my mistake

I imported this dashboard in Grafana : https://grafana.com/grafana/dashboards/21689-crowdsec-cyber-threat-insights/ And I didnt read the doc correctly. They use a VictoriaMetrics component between Crowdsec and prometheus to enrich crowdsec metrics with notably Geoip, and they label it "cs_lapi_decision" : https://freefd.github.io/articles/8_cyber_threat_insights_with_crowdsec_victoriametrics_and_grafana/

 {
    "metric": {
        "__name__": "cs_lapi_decision",
        "instance": "host00.domain.tld",
        "country": "SG",
        "asname": "Tencent Building, Kejizhongyi Avenue",
        "asnumber": "132203",
        "latitude": "1.2868",
        "longitude": "103.8503",
        "iprange": "129.226.0.0/16",
        "scenario": "crowdsecurity/ssh-slow-bf",
        "type": "ban",
        "duration": "4h",
        "scope": "Ip",
        "ip": "129.226.84.230"
    },
    "values": [1],
    "timestamps": [1723013301000]
}

My mistake was that seing the name, I had expected that this was coming directly from Crowdsec metrics. Sorry for the useless bug report :-)

On a side note, its a bit over killing to setup an additional metric component like Victoria to enrich and serve that information, considering that all data needed, including Maxmind GeoIP, is already available in Crowdsec LAPI. So I hope you will consider to add that kind of metrics at some point, as it is extremely helpful... and you can turn my bug report into a feature request !

[FlorentLM]

Hah, I was looking for this too!

Following this issue

[joshoram80]

I am also following this. I have everything setup exactly.

cs_lapi_decision{instance=~".*"}[86400s:1m] WARNING: No match! This query hasn't selected any time series from database. Either the requested metrics are missing in the database, or there is a typo in series selector

It seems like

type: http
name: http_default
log_level: info
format: >
  {{- range $Alert := . -}}
  {{- range .Decisions -}}
  {"metric":{"__name__":"cs_lapi_decision","instance":"HTPC","country":"{{$Alert.Source.Cn}}","asname":"{{$Alert.Source.AsName}}","asnumber":"{{$Alert.Source.AsNumber}}","latitude":"{{$Alert.Source.Latitude}}","longitude":"{{$Alert.Source.Longitude}}","iprange":"{{$Alert.Source.Range}}","scenario":"{{.Scenario}}","type":"{{.Type}}","duration":"{{.Duration}}","scope":"{{.Scope}}","ip":"{{.Value}}"},"values": [1],"timestamps":[{{now|unixEpoch}}000]}
  {{- end }}
  {{- end -}}
url: http://victoria-metrics:8428/api/v1/import
method: POST
headers:
  Content-Type: application/json

is not sending any data to the VictoriaMetrics instance

[LaurenceJJones]

I am also following this. I have everything setup exactly.

cs_lapi_decision{instance=~".*"}[86400s:1m] WARNING: No match! This query hasn't selected any time series from database. Either the requested metrics are missing in the database, or there is a typo in series selector

It seems like

type: http
name: http_default
log_level: info
format: >
  {{- range $Alert := . -}}
  {{- range .Decisions -}}
  {"metric":{"__name__":"cs_lapi_decision","instance":"HTPC","country":"{{$Alert.Source.Cn}}","asname":"{{$Alert.Source.AsName}}","asnumber":"{{$Alert.Source.AsNumber}}","latitude":"{{$Alert.Source.Latitude}}","longitude":"{{$Alert.Source.Longitude}}","iprange":"{{$Alert.Source.Range}}","scenario":"{{.Scenario}}","type":"{{.Type}}","duration":"{{.Duration}}","scope":"{{.Scope}}","ip":"{{.Value}}"},"values": [1],"timestamps":[{{now|unixEpoch}}000]}
  {{- end }}
  {{- end -}}
url: http://victoria-metrics:8428/api/v1/import
method: POST
headers:
  Content-Type: application/json

is not sending any data to the VictoriaMetrics instance

Did you add http_default under notifications to the profiles.yaml? and if so did you restart crowdsec after the update?

[joshoram80]

Yes to both

On Fri, Jan 24, 2025, 9:16 AM Laurence Jones @.***> wrote:

I am also following this. I have everything setup exactly.

cs_lapi_decision{instance=~".*"}[86400s:1m] WARNING: No match! This query hasn't selected any time series from database. Either the requested metrics are missing in the database, or there is a typo in series selector

It seems like

type: http name: http_default log_level: info format: > {{- range $Alert := . -}} {{- range .Decisions -}} {"metric":{"name":"cs_lapi_decision","instance":"HTPC","country":"{{$Alert.Source.Cn}}","asname":"{{$Alert.Source.AsName}}","asnumber":"{{$Alert.Source.AsNumber}}","latitude":"{{$Alert.Source.Latitude}}","longitude":"{{$Alert.Source.Longitude}}","iprange":"{{$Alert.Source.Range}}","scenario":"{{.Scenario}}","type":"{{.Type}}","duration":"{{.Duration}}","scope":"{{.Scope}}","ip":"{{.Value}}"},"values": [1],"timestamps":[{{now|unixEpoch}}000]} {{- end }} {{- end -}} url: http://victoria-metrics:8428/api/v1/import method: POST headers: Content-Type: application/json

is not sending any data to the VictoriaMetrics instance

Did you add http_default under notifications to the profiles.yaml? and if so did you restart crowdsec after the update?

— Reply to this email directly, view it on GitHub https://github.com/crowdsecurity/crowdsec/issues/3290#issuecomment-2611127983, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAP7FG4MPMLBNEIMO5LWCFD2MFS3FAVCNFSM6AAAAABQHVD3SCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJRGEZDOOJYGM . You are receiving this because you commented.Message ID: @.***>

[DuvelCorp]

here's some guide to get cs_lapi_desicion from victoria metrics https://freefd.github.io/articles/8_cyber_threat_insights_with_crowdsec_victoriametrics_and_grafana/

There's something right to the point on this page:

Weakness in Crowdsec OpenMetrics statistics

Out of the box, each Crowdsec instance can provide the statistics in the Prometheus 4 or OpenMetrics 5 format about its engine status, such as:

decisions
alerts
etc.

All possible data slices are explained in the Crowdec metrics documentation 6. There are official Grafana dashboards for Crowdsec 7, but these metrics do not contain the information that most operators are interested in, such as cyber threat statistics. For example, active decisions and alerts contain only the scenario metrics

For me at least, this whole statement is true and obvious : The most interesting data isn't provided by crowsec metrics. I dont really care to see buckets and decisions, what I want to see is who are the bad guys that were blocked by my crowdec component. And it makes no sense to have to check the online crowdesc dashboard to see them.

I thus hope that those will be implemented natively in crowdec prometheus metrics any time soon.

Lets keep this feature request alive.

[ymrsmns]

here's some guide to get cs_lapi_desicion from prometheus ( used pushgateway )

  notifications:
    # yamllint disable rule:line-length
    http.yaml: |
      type: http
      name: http_default
      log_level: info
      format: |
        {{- range $Alert := . -}}
        {{- range .Decisions -}}
        cs_lapi_decision{instance="HTPC",country="{{$Alert.Source.Cn}}",asname="{{$Alert.Source.AsName}}",asnumber="{{$Alert.Source.AsNumber}}",latitude="{{$Alert.Source.Latitude}}",longitude="{{$Alert.Source.Longitude}}",iprange="{{$Alert.Source.Range}}",scenario="{{.Scenario}}",type="{{.Type}}",duration="{{.Duration}}",scope="{{.Scope}}",ip="{{.Value}}"} 1
        {{- end }}
        {{- end }}
      url: http://pushgateway.prometheus.svc.cluster.local.:9091/metrics/job/crowdsec
      method: POST
      headers:
        Content-Type: text/plain

maybe it will save someone's time

[FlorentLM]

Thanks! Did not know about pushgateway... :)

[pittbull]

here's some guide to get cs_lapi_desicion from prometheus ( used pushgateway )

  notifications:
    # yamllint disable rule:line-length
    http.yaml: |
      type: http
      name: http_default
      log_level: info
      format: |
        {{- range $Alert := . -}}
        {{- range .Decisions -}}
        cs_lapi_decision{instance="HTPC",country="{{$Alert.Source.Cn}}",asname="{{$Alert.Source.AsName}}",asnumber="{{$Alert.Source.AsNumber}}",latitude="{{$Alert.Source.Latitude}}",longitude="{{$Alert.Source.Longitude}}",iprange="{{$Alert.Source.Range}}",scenario="{{.Scenario}}",type="{{.Type}}",duration="{{.Duration}}",scope="{{.Scope}}",ip="{{.Value}}"} 1
        {{- end }}
        {{- end }}
      url: http://pushgateway.prometheus.svc.cluster.local.:9091/metrics/job/crowdsec
      method: POST
      headers:
        Content-Type: text/plain

maybe it will save someone's time

Trying to get this dashboard to work as well.

@ymrsmns - does your change replace VictoriaMetrics with Pushgateway?

[kolesaev]

Following here. Would be nice to expose country and AS-name to existing metrics