HTTP2 Trailers not being sent

[adambudziak]

Hi, I know that it's probably a stretch since it's likely covered by a bunch of tests, but I tried everything and I can't get it to work. So I'm playing with connecting gRPC to ASGI using hypercorn, and I made this very simple app:

import asyncio
from hypercorn.config import Config
from hypercorn.asyncio import serve

async def grpc_echo(scope, receive, send):
    if scope['type'] == 'http' and scope['method'] == 'POST':
        message = await receive()
        request_body = message.get('body', b'')

        # Send headers
        await send({
            'type': 'http.response.start',
            'status': 200,
            'headers': [
                (b'content-type', b'application/grpc'),
            ],
            'trailers': True
        })

        await send({
            'type': 'http.response.body',
            'body': request_body,
        })

        await send({
            'type': 'http.response.trailers',
            'headers': [
                (b'grpc-status', b'0'),
                (b'grpc-message', b'OK'),
            ],
        })
    else:
        raise NotImplementedError


async def main():
    config = Config()
    config.bind = ["localhost:50051"]
    await serve(grpc_echo, config)

if __name__ == "__main__":
    asyncio.run(main())

here's the proto (though I doubt it matters)

syntax = "proto3";

package helloworld;

service Greeter {
  rpc SayHello (HelloRequest) returns (HelloRequest) {}
}

message HelloRequest {
  string name = 1;
}

I run the client by grpcurl -proto helloworld.proto -plaintext -d '{"name": "World"}' localhost:50051 helloworld.Greeter/SayHello

and the error I get is

  Code: Internal
  Message: server closed the stream without sending trailers

This area is completely new to me so there might be some rookie mistake (maybe not related to trailers at all), but I checked the ASGI spec, and my trailers look ok, I ran it through debugger and it looks like it's being sent, however I don't see them in Wireshark and the clients (I tried Postman and grpcurl) both fail.

In the debugger I saw that the response body is sent (I see the "World" string), I also see some stuff being sent afterwards, but nothing looking like the trailers that I set.

Any help is appreciated, and congrats for an amazing lib, it's been a long time since I saw one that's so readable inside.

[adambudziak]

Ok, so I did more digging, and it looks like some issue with end_stream value in H2Stream.send_headers.

https://github.com/python-hyper/h2/blob/master/src/h2/stream.py#L863the function taken from the map here sets trailers_sent in the state machine to true.

Then this check https://github.com/python-hyper/h2/blob/master/src/h2/stream.py#L877 always raises an exception because trailers_sent is True, but end_stream is always False (hypercorn doesn't set it).

(it looks correct since according to the spec "The HEADERS frame starting the trailers header block has the END_STREAM flag set.")

I tried setting end_stream to True, but then I get the following error in grpcurl.

Error invoking method "helloworld.Greeter/SayHello": grpc call for "helloworld.Greeter.SayHello" failed: EOF

however it works better in Postman (I see the status code, but no response body).

I'm gonna keep investigating, it's likely that it's a different issue.

[adambudziak]

It looks like because trailers now set end_stream=True, the response body isn't sent. I made a "fix" (quotes because I really have no idea what I'm doing). I'll check if tox passes and if so, I'll make a PR.

[drevoborod]

It looks like because trailers now set end_stream=True, the response body isn't sent. I made a "fix" (quotes because I really have no idea what I'm doing). I'll check if tox passes and if so, I'll make a PR.

Tried your fix, and it worked for me! Although I don't understand the intention of other lines except one with end_stream=True. Also, unfortunately your MR remains open for half of a year, so it can be applied only locally yet.

[adambudziak]

TBH I just saw them in the other places where end_stream=True was used. I tried to understand more if I need to call these things, but it was too deep for me. But it worked!

[XChikuX]

Tried your fix, and it worked for me! Although I don't understand the intention of other lines except one with end_stream=True. Also, unfortunately your MR remains open for half of a year, so it can be applied only locally yet.

@pgjones Hypercorn needs some due inputs from you.

[JRial95]

I'm facing the same issue and I can confirm that the proposed modification also works in my use case.

[oktaydavid]

I have the same problem, the modification also works