Receiving chunks from stream are being cut / incomplete

[gmile]

I'm connecting to Twitter Stream API, and seeing a weird behaviour when receiving data from Twitter service.

It looks like hackney does not respect the size of chunk, or it cuts it half-way sometimes. Or likely I'm doing something wrong.

In my process that waits for chunk messages from hackney, while some responses are fine, I'm seeing this a lot:

FWIW, this is the line I'm using to start hackney process (the code is in Elixir):

:hackney.get("https://userstream.twitter.com/1.1/user.json", [header], "", [:async, {:recv_timeout, 30000}])

To cross-check this, I issued the same request using plain curl in terminal and I see no problems: all responses streamed are "complete".

[ludwikbukowski]

we are experiencing the same problem. Is that resolved?

[benoitc]

@ludwikbukowski can you provide me a way to reproduce it?

[dextercd]

From the Twitter documentation:

https://developer.twitter.com/en/docs/tutorials/consuming-streaming-data

Chunked encoding Streaming API connections will be encoded using chunked transfer encoding, as indicated by the presence of a Transfer-Encoding: chunked HTTP header in the response. Because most HTTP libraries deal with chunked transfer encoding transparently, this document will assume that your code has access to the reassembled HTTP stream and does not have to deal with this encoding.

If this is not the case, note that Tweets and other streamed messages will not necessarily fall on HTTP chunk boundaries – be sure to use the delimiter defined above to determine activity boundaries when reassembling the stream

The delimiter is \r\n. This doesn't look like an issue with hackney to me.