Jigsaw-Code
Clustered server and external Prometheus
Hi guys!
I'm setting up OutlineVPN in my kubernetes cluster. During work I figured out, that each shadowbox container has its own prometheus. And this makes mess, when you scale shadowbox more than 1 replica. Is it possible to configure shadowbox to use existing cluster prometheus and disable prom in shadowbox containers? This will help users to scale horizontally shadowbox containers and balance the load between them. In current situation, to run multiple replicas tou have to setup RWX storage between them (such as nfs or cephfs), but prom generates huge IO and it can overload nfs and make queues.
Probably that should need to create ServiceMonitor file, but it is not very big problem.
P.S. After inspecting sources, I've found such line in main.ts
const prometheusLocation = `127.0.0.1:${prometheusPort}`;
in https://github.com/Jigsaw-Code/outline-server/blob/ae8f378cb7bfc549a5aaff16567da2d9603ddbf4/src/shadowbox/server/main.ts#L126
Am I right, if I change it to url of my prometheus, than shadowbox will communicate with my own external prometheus?
Really want to export metrics to my own Prometheus directly from 9091 and 9092 ports. We could probably add Prometheus listen ip variable.
+1 on this. I really want to scrape metrics from external prometheus. I understand that it's not possible for security reasons, but there must be an option for advanced users (that can take care of the firewall) to expose metrics
We already have that feature. Perhaps it needs to be better documented. Or turned into a feature request to more easily access from outside.
You can already run your own centralized Prometheus scraper and point to the metrics endpoint of each of your servers. The Prometheus endpoint runs on localhost:9091 of the server. Note that it's not exposed in the public network. To access from a remote server, you will need some sort of port forwarding. I've seen people do that. Perhaps the issue here is how to make it easier.
If you want to not run each individual scraper, or expose the metrics on externally (dangerous) you should use outline-ss-server directly, not the docker image. Here is the command-line: https://github.com/Jigsaw-Code/outline-ss-server#how-to-run-it
The -metrics flag lets you specify the location of the worker Prometheus metrics endpoint you need to scrape.
If you expose the metrics, prefer to use a local network IP. I'd also set up a firewall to allow access from your scraper only.
@fortuna thanks, I've already set up port forwarding and firewall for external Prometheus and it works flawlessly. For anyone who found this post in Google - iptables NAT is the key.
@dev0nizer do you have detailed actions? I haven't been able to do this since 3 months
@nedome something like this (these are iptables rules)
-A PREROUTING -p tcp --dport 9091 -j DNAT --to-destination 127.0.0.1:9091 --comment "redirect outline main metrics"
-A PREROUTING -p tcp --dport 9092 -j DNAT --to-destination 127.0.0.1:9092 --comment "redirect shadowsocks metrics"
Find a basic manual in google if you are not familiar with iptables. It's relatively simple.