GZipDecoder only decodes the first member in gzip data

[lizeyan]

httpx version:

0.27.0

Current Behavior:

GZipDecoder only decodes the first member in gzip data.

Expected Behavior:

Be capable of decompressing multi-member gzip data (multiple gzip blocks concatenated together) just like gzip.decompress

Steps To Reproduce:

In [1]: import httpx

In [2]: raw_bytes = b'\x1f\x8b\x08\x00\x00\tn\x88\x00\xff\x00\x15\x00\xea\xff{"status": "success",\x03\x00\xeb\xdb\xa3\xb0\x15\x00\x00\x00\x1f\x8
   ...: b\x08\x00\x00\tn\x88\x00\xff\x00\x08\x00\xf7\xff"data": \x03\x00\x1d\xb4\xe6\xc8\x08\x00\x00\x00\x1f\x8b\x08\x00\x00\tn\x88\x00\xff\x00#\
   ...: x00\xdc\xff{"resultType":"matrix","result":[]}\x03\x00\x12\xb7\x95\x1b#\x00\x00\x00\x1f\x8b\x08\x00\x00\tn\x88\x00\xff\x00\x01\x00\xfe\xf
   ...: f}\x03\x00\x0c\xe2\xb6\xfc\x01\x00\x00\x00'

In [3]: from httpx._decoders import GZipDecoder

In [4]: GZipDecoder().decode(raw_bytes)
Out[4]: b'{"status": "success",'

In [5]: import gzip

In [6]: gzip.decompress(raw_bytes)
Out[6]: b'{"status": "success","data": {"resultType":"matrix","result":[]}}'

In [7]:

(The raw_bytes are from a Prometheus query request)

Anything else:

A possible implementation:

class GZipDecoder(ContentDecoder):
    ...

    def decode(self, data: bytes) -> bytes:
        decompressed_data = b""
        try:
            length = len(data)
            offset = 0
            while offset < length:
                chunk = self.decompressor.decompress(data[offset:])
                decompressed_data += chunk
                # Update the offset to the next member
                offset += len(data[offset:]) - len(self.decompressor.unused_data)
                if not self.decompressor.unused_data:
                    break
                else:
                    self.decompressor = zlib.decompressobj(zlib.MAX_WBITS | 16)
            return decompressed_data
        except zlib.error as exc:
            raise DecodingError(str(exc)) from exc

I have test it and it can handle this case.

[lizeyan]

I have opened a pull request: #3270
Seems that this issue has been discovered in 2022? #2171

[lovelydinosaur]

Okay, I don't think we've sufficiently demo'ed that this is an issue yet. (tho it may be.)

Let's ground this against the ecosystem, so it's less abstract...

• Could you demo a public URL that exhibits this behavior?
• Could you check what behavior a couple of browsers have vs. this?
• Could you compare/contrast existing behavior of other Python clients?

[lizeyan]

from flask import Flask, Response, make_response
import io
import gzip
import requests
import httpx
import threading

app = Flask(__name__)

@app.route('/')
def gzip_response():
    # 创建一个BytesIO对象来存储gzip数据
    buffer = io.BytesIO()

    # 创建一个GzipFile对象,写入多个成员
    with gzip.GzipFile(fileobj=buffer, mode='w') as gz:
        gz.write(b'First member\n')
        gz.close()  # 关闭当前成员

        # 开始新的gzip成员
        gz = gzip.GzipFile(fileobj=buffer, mode='w')
        gz.write(b'Second member\n')
        gz.close()

        # 可以继续添加更多成员...

    # 获取压缩后的数据
    compressed_data = buffer.getvalue()

    # 创建一个Flask响应对象
    response = Response(compressed_data)

    # 设置正确的头部
    response.headers['Content-Type'] = 'text/plain'
    response.headers['Content-Encoding'] = 'gzip'
    response.headers['Content-Length'] = len(compressed_data)

    return response

if __name__ == '__main__':
    threading.Thread(target=lambda: app.run(port=8080), daemon=True).start()
    print(f"requests: {requests.get('http://localhost:8080').text}")
    print(f"httpx   : {httpx.get('http://localhost:8080').text}")
    while True:
        pass

It outputs:

requests: First member
Second member

httpx   : First member

@tomchristie

[lovelydinosaur]

Okay yep that replicates.

[lovelydinosaur]

What behaviours do Chrome/Firefox/Safari have with the example shown above?

[lizeyan]

What behaviours do Chrome/Firefox/Safari have with the example shown above?

Chrome and Safari only decode the first gzip member.

[lovelydinosaur]

Ok, so let's stick with existing behaviour.

(Thanks for taking the time to work through it)

[lizeyan]

It is better to maintain consistent behavior with requests. People won't use httpx to replace browsers, but they will want to use httpx to replace requests.