Enhance Security for SysVMs Behind Reverse Proxy

[daviftorres]

The required feature described as a wish

I’d like to propose extending the Reverse Proxy support introduced in Management servers in (https://github.com/apache/cloudstack/commit/67e2061f4b87934c863cbafc5031821ad7ef6d02) to the SysVMs as well.

As shown in the screenshot below, Management servers can now recognize (and trust) Reverse Proxy servers:

This brings both security, observability, and control, by preserving the real client IP address, which makes logs significantly more useful.

Today we leverage Reverse Proxies for: WAF, Rate limiting for abusive users, DoS/DDoS protection, and SSL/TLS termination. Each of these alone is a valid reason to support Reverse Proxies in SysVMs.

The current workaround: we had to modify the SysVM template manually:

1. Update Apache vhost config:

RemoteIPHeader X-Forwarded-For
RemoteIPTrustedProxy 10.7.1.0/24

2. Enable the module and restart Apache:

a2enmod remoteip
systemctl restart apache2

Feel free to ask questions and share your thoughts. I can provide details and perform tests as needed.

Keep up with the good work!!

[weizhouapache]

@daviftorres do you still need to add the static route inside the SSVM ?

[daviftorres]

@weizhouapache , no. The route (discussed here https://github.com/apache/cloudstack/pull/7288) breaks my setup.

[DaanHoogland]

@daviftorres , do you mean that we should revert the behaviour of #7288 in addtion to adding automation you describe as a workaround?

[daviftorres]

@weizhouapache , based on my limited understanding of the purpose of the added route, I would say:

• Yes, I am proposing to remove the route and add the automation for the workaround. Unless the route is required.

[weizhouapache]

@weizhouapache , based on my limited understanding of the purpose of the added route, I would say:

• Yes, I am proposing to remove the route and add the automation for the workaround. Unless the route is required.

I do not know which route should be removed. Can you clarify ?

#7288 did not add any route from my understanding.

the routes to internal sites via private network is expected, from many years ago.

[daviftorres]

@weizhouapache , you are probably right about when the route was added. I might have made the assumption based on added line 32 plus my limited understanding of how things work under the rood:

What I can say is that the route is created and that I couldn't find out the purpose of it yet.

[daviftorres]

Moreover,

In our current setup, we rotate the edge certificate without restarting the SSVM, CPVM, or Management service to avoid downtime.

Since NGINX doesn’t validate backend certificates by default (though it can), clients aren’t affected when a new certificate is applied (aside from possible cert caching or possible re-login, which haven’t been reported.

Ideally, SysVMs should be updated using a RollingUpdate strategy (starting a new one before stopping the old) to avoid the ~1 minute of downtime across all zones.

For Management, a multi-instance setup with a reverse proxy makes staggered updates easy.

In short, this proposed feature would bring more benefits than expected.

[daviftorres]

As a workaround the access control of who can "copy" can be done on the edge (Reverse Proxy), not inside the SSVM.

    location /copy {
        allow 100.100.100.100;        # Satellite Zone A
        allow 200.200.200.200;        # Satellite Zone B
        # add more as needed
        deny all;
    }

[daviftorres]

This issue is being discussed at https://github.com/apache/cloudstack/discussions/11562#discussioncomment-14305414