Invalid server certificate error for `internet-up` endpoint

[lawrence-forooghian]

@sacOO7 noticed that the acceptance tests are failing (e.g. here) with the following error:

     OpenSSL::SSL::SSLError:
       unable to verify the server certificate for "internet-up.ably-realtime.com"

Not sure how long this has been happening for.

@lmars mentioned this might be to do with SNI; when the client doesn’t provide SNI then the server responds with a certificate that doesn’t include internet-up.ably-realtime.com as an allowed subject name. It’s quite possible that we are not supplying SNI; need to look into that. As for why this used to work, perhaps we changed the way that we generate certs so that they now only work when SNI is supplied, not sure (flagged this to infrastructure support).

Internal Slack thread.

┆Issue is synchronized with this Jira Task by Unito

[lawrence-forooghian]

I might well be misunderstanding something, but running locally on macOS it appears that the client is sending SNI, and yet I get the same test failure:

Will investigate further.

[lawrence-forooghian]

OK, on my machine this error is occurring because em-http-request is failing to verify the ISRG Root X1 certificate (cross-signed by DST Root CA X3). The error is coming from this line. The certificate trust store in use comes from telling OpenSSL to use its "default store". On my machine that seems to be some set of certificates that come with the version of OpenSSL that rbenv installed 🤷.

Need to look into it more.

[lawrence-forooghian]

Interestingly, our WebSocket transport uses some very similar-looking code for TLS certificate verification. I think that the best route forwards might be to try and understand why that’s (presumably) working and this isn’t. Will do that tomorrow.

[lawrence-forooghian]

I believe the issue here is the served certificate chain’s reference to the expired DST Root CA X3 certificate. I need to understand better how the client should be behaving (my understanding is that basically, it should be ignored because the ISRG Root X1 certificate is a trusted root these days) and how this logic is implemented in EventMachine / em-http-request / our WebSocket code.

As for my question about why this isn’t affecting our WebSocket code (https://github.com/ably/ably-ruby/issues/396#issuecomment-2073276482), I think that’s just because the Realtime endpoints don’t use Let’s Encrypt and hence don’t receive this expired certificate.

[lawrence-forooghian]

@sacOO7 shared this relevant EventMachine issue: https://github.com/eventmachine/eventmachine/issues/954

[lawrence-forooghian]

Not sure why this got closed; shouldn't have done.