Counting actual input size [was: svn commit: r1088435]

On 8 April 2011 07:45, Milamber <[email protected]> wrote:
[...]

>>> With my last submission (r1088748), I try to respond to your feedback.
>>> Please say me if another thing to improve.
>>>
>> The problem of chunked responses still exists - such responses don't
>> have a Content-Length header.
>>
>> One way round this would be to wrap the input Stream with a
>> org.apache.commons.io.input.CountingInputStream.
>> I don't think this will affect performance adversely.
>>
>> Does that make sense?
>>
>
> Yes may be a good idea.
> Since your last commit on HC4Impl, entity.getContentLength() return -1
> (unknown size) (but http response have a content-length define)
> I thinks the ResponseContentEncoding class which decompress stream is
> the cause.

That seems likely.

> On HC4, I try to use a CountingInputStream on instream, but the return
> size is uncompressed.
> //                InputStream instream = entity.getContent();
>                InputStream instream = new
> CountingInputStream(entity.getContent());
>                res.setResponseData(readResponse(res, instream, (int)
> entity.getContentLength()));
>                int cnt = ((CountingInputStream) instream).getCount();
>                log.debug("CNT=" + cnt);
>
>
> I thinks that CountingInputStream must be in more deep in code, directly
> in HttpClient, or inside the Gzip/deflate input stream?

Yes, you're right - the streams we currently use are somewhat removed
from the actual input .

For HC3 and Java, we decompress the inputstream directly, so could
wrap that with a CountingInputStream first.
However, the stream contains the de-chunked data, so the chunking
overhead would not be seen.
But it would be closer to the true size, and might be acceptable.

Ideally, one would like to intercept the input stream before
de-chunking, but I'm not sure that's possible with HC3 and Java.

However with HC3 and HC4 one can provide custom sockets, so it would
be possible to count the actual input.

One could even detect the end of the header by looking for CRLF CRLF -
but that might add an unacceptable overhead, in which case we could
use the current header calculation which would be reasonably close.

It's not possible to provide a custom socket implementation for Java
HTTP, only Java HTTPS, so this approach would not work there, so we
would have to use the CountingInputStream.

I suggest we use the simple approach of CountingInputStream (CIS) for
Java and HC3; it's easy to do and fairly accurate. No point spending
lots of time on those implementations as HC4 is better.

I'll have a look at HC4 to see what can be done - would you like to
see if CIS works OK for HC3 and Java?