What is the problem?
In all the objcli we currently set ObjectAttrs.Size based on the response's Content-Length header. Although all the blob stores do send this header correctly we have seen a few cases where third party S3 compatible blob stores miss out the header. As an example, they may decide to transfer using chunk encoding where the header should not be supplied.
In the HTTP/1.1 spec there are two relevant sections:
- A recipient MUST be able to parse the chunked transfer coding
- The message body length is determined by the number of octets received prior to the server closing the connection (i.e. Content-Length MAY be passed, it does not have to be).
It's arguable whether we should consider S3's observed behaviour as the contract for object stores or whether we should consider being a good HTTP citizen more important. Given a few customers have hit this problem it doesn't seem unreasonable to be more accepting of responses here.
What is the solution?
We should check whether or not we use ObjectAttrs.Size from GetObject - from a brief search we do not though. We probably then want to change GetObject's API to not return a full ObjectAttrs. Ideally we would make the field optional or return a different struct.