sozu-proxy
Invalid Backend Token on BackendConnectAction::Replace
When Sōzu reads the Host of an incoming request, it tries to find a suitable backend to redirect to.
https://github.com/sozu-proxy/sozu/blob/cfde50b0f2294c655a1b9fa90648c2b7ed505f76/lib/src/http.rs#L1061-L1064
Suppose the session is already connected to a backend, meaning it's not the first request the client sent and the session was reset on the last response. In that case, Sōzu checks if it can reuse this already established connection by checking if the current cluster id and the request's cluster id are equal.
https://github.com/sozu-proxy/sozu/blob/cfde50b0f2294c655a1b9fa90648c2b7ed505f76/lib/src/http.rs#L1073-L1075
If it can reuse the connection, it checks if the connection is hanging up.
https://github.com/sozu-proxy/sozu/blob/cfde50b0f2294c655a1b9fa90648c2b7ed505f76/lib/src/http.rs#L1089
If the old backend has disconnected, Sōzu closes this connection and runs the standard connection code.
https://github.com/sozu-proxy/sozu/blob/cfde50b0f2294c655a1b9fa90648c2b7ed505f76/lib/src/http.rs#L1091-L1097
The corner case comes after Sōzu found a new backend and connected to it.
https://github.com/sozu-proxy/sozu/blob/cfde50b0f2294c655a1b9fa90648c2b7ed505f76/lib/src/http.rs#L1124
The session has an "old back token" so Sōzu reuses it to register the new socket in the SessionManager.
https://github.com/sozu-proxy/sozu/blob/cfde50b0f2294c655a1b9fa90648c2b7ed505f76/lib/src/http.rs#L1148-L1165
But the entry for that token in the SessionManager was deleted when Sōzu closed the connection.
https://github.com/sozu-proxy/sozu/blob/cfde50b0f2294c655a1b9fa90648c2b7ed505f76/lib/src/http.rs#L1092
So when the backend tries to talk to Sōzu, the SessionManager won't recognize its Token.
https://github.com/sozu-proxy/sozu/blob/cfde50b0f2294c655a1b9fa90648c2b7ed505f76/lib/src/server.rs#L1555
This is a really rare case as the old backend has to hang while the request is handled. If it hangs before the request arrives, the disconnection is a bit different and the "old back token" is set to None before entering connect_to_backend. If it hangs after the request was sent then it's a normal timeout.
I'm almost sure we should also set the backend Token to None when Sōzu closes the connection to the old backend, but close_backend already does this. It is explicitly set back to its original value by this code:
https://github.com/sozu-proxy/sozu/blob/cfde50b0f2294c655a1b9fa90648c2b7ed505f76/lib/src/http.rs#L1094-L1096
In our case backend_from_request doesn't fail. I don't understand the comments as nothing seems to be done in case backend_from_request and connect_to_backend fail. Moreover it is already removed from the slab by close_backend:
https://github.com/sozu-proxy/sozu/blob/cfde50b0f2294c655a1b9fa90648c2b7ed505f76/lib/src/http.rs#L822
The end-to-end PR #805 does test this issue, however the current test is satisfied with an Undecided outcome. To be bettered.
