awslabs
response.text() does not support content-encoding
When the following code is executed, the result of response.text() is garbled.
The same result is obtained when accept-encoding is set to something else(such as br, deflate, zstd).
And although I have not tried it, I suspect that response.json() will produce the same result.
// reproduction.js
const main = async () => {
try {
const response = await fetch('https://google.com', {headers: {'accept-encoding': 'gzip'}});
console.log(response.headers.get('content-encoding'));
const responseText = await response.text();
console.log('vvv');
console.log(responseText.substring(0, 100));
console.log('^^^');
} catch(e) {
console.log(e);
}
}
main();
result of llrt:
shinya@MBA2022M2 llrt-test % ./llrt reproduction.js
gzip
vvv
// This part is actually garbled.
^^^
result of bun:
shinya@MBA2022M2 llrt-test % bun reproduction.js
gzip
vvv
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="ja"><head><meta content
^^^
This is a must have. We already heavily rely on zstd for reading LLRT bytecode format. Gzip and deflate (zip) decompression can be added using flate2 crate with zlib-ng backend. Compression & decompression are also requirements of WinterCG compression stream with said algorithms. For now (since streaming is not yet in place), decompression can be done when reading from response here: https://github.com/awslabs/llrt/blob/8fa91565bd7446eca010bb93aac336a0fe3ba0ac/llrt_core/src/modules/http/response.rs#L135
I know that many sites return content-encoding as br when accept-encoding is "gzip, deflate, br, zstd", but do you plan to support brotli?
I know that many sites return
content-encodingasbrwhenaccept-encodingis"gzip, deflate, br, zstd", but do you plan to support brotli?
Yes brotli as well
After this implementation is complete, We also need to decide on the default accept-encoding.
For example,
- In general:
gzip, deflate, br, zstd - If you want to make a difference:
zstd, br, gzip, deflate
Personally, I think that within the scope of WinterCG, challenging settings that lead to improved performance could be incorporated.
Hi @richarddavison, are you already working on this issue? If you haven't already, would you give me the opportunity to work on it?
Hi @richarddavison, are you already working on this issue? If you haven't already, would you give me the opportunity to work on it?
Hi @nabetti1720! I'm not, please go ahead 👌