Behavior when chunk type check not passed

[johnd0e]

Consider such (real) api response:

data: {"id":"chatcmpl-7ZZCd6bd5DwGT9UdoK9Ph4","model":"llama2-70b","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}],"usage":null}

data: {"id":"chatcmpl-7ZZCd6bd5DwGT9UdoK9Ph4","model":"llama2-70b","choices":[{"index":0,"delta":{"content":" "},"finish_reason":null}]}

data: {"id":"chatcmpl-7ZZCd6bd5DwGT9UdoK9Ph4","model":"llama2-70b","choices":[{"index":0,"delta":{"content":" Hello!"},"finish_reason":null}]}

data: {"id":"chatcmpl-7ZZCd6bd5DwGT9UdoK9Ph4","model":"llama2-70b","choices":[],"usage":{"prompt_tokens":11,"total_tokens":15,"completion_tokens":4}}

data: [DONE]

The response objects are missing "object": "chat.completion.chunk", so they aren't passing the check here:

https://github.com/leafo/lua-openai/blob/08128b93bc546e04681ec244929a901105e2983b/openai/init.moon#L253-L254

This leads to to some consequences:

1. Json is valid and decodes without problems.
2. However parse_completion_chunk doesn't accept it.
3. As a result, it's simply ignored.
4. And chunk_callback is never invoked.

It might be more sensible to throw an error instead of just disregarding it. Or alternatively, invoke chunk_callback regardless, but with an additional isInvalid argument.

The latter is more flexible.

[johnd0e]

Upon closer inspection, I understand that parse_completion_chunk does more than just verification. It extracts the content and index fields from the entire structure.

I find this problematic because other fields are equally significant! For instance, I require the usage and finish_reason fields.

Moreover, the absence of "object": "chat.completion.chunk" is not the sole reason for parse_completion_chunk failing to capture a chunk; sometimes, the content is missing as well.

Therefore, my proposed solution is not to utilise parse_completion_chunk at all, but to relay the decoded object to the callback as is: https://github.com/leafo/lua-openai/compare/main...johnd0e:lua-openai:no-parse

parse_completion_chunk can be provided also, in the API object, to use when needed.