Re: mod_perl and Transfer-Encoding: chunked

On 03/07/2013 23:26, Jim Schueler wrote:

> 
>>             Second, if there's no Content-Length header then how
>>             does one know how much
>>             data to read using $r->read?   
>>
>>             One answer is until $r->read returns zero bytes, of
>>             course.  But, is
>>             that guaranteed to always be the case, even for,
>>             say, pipelined requests?  
>>             My guess is yes because whatever is de-chunking the
> 
> read() is blocking.  So it never returns 0, even in a pipeline request
> (if no data is available, it simply waits).  I don't wish to discuss the
> merits here, but there is no technical imperative for a content-length
> request in the request header.
> 
>  -Jim

Probably.  If you, for some reason, were doing the chunking work
yourself, each chunk says how many bytes are in it (or in the next one
perhaps; I forget offhand), so you'd know what size read to do.

> 
> 
> 
> 
> 
> On Wed, 3 Jul 2013, Bill Moseley wrote:
> 
>> Hi Jim,
>> This is the Transfer-Encoding: chunked I was writing about:
>>
>> http://tools.ietf.org/html/rfc2616#section-3.6.1
>>
>>
>>
>> On Wed, Jul 3, 2013 at 11:34 AM, Jim Schueler <[email protected]>
>> wrote:
>>       I played around with chunking recently in the context of media
>>       streaming: The client is only requesting a "chunk" of data.
>>        "Chunking" is how media players perform a "seek".  It was
>>       originally implemented for FTP transfers:  E.g, to transfer a
>>       large file in (say 10K) chunks.  In the case that you describe
>>       below, if no Content-Length is specified, that indicates "send
>>       the remainder".
>>
>>       >From what I know, a "chunk" request header is used this way to
>>       specify the server response.  It does not reflect anything about
>>       the data included in the body of the request.  So first, I would
>>       ask if you're confused about this request information.
>>
>>       Hypothetically, some browsers might try to upload large files in
>>       small chunks and the "chunk" header might reflect a push
>>       transfer.  I don't know if "chunk" is ever used for this
>>       purpose.  But it would require the following characteristics:
>>
>>         1.  The browser would need to originally inquire if the server
>>       is
>>             capable of this type of request.
>>         2.  Each chunk of data will arrive in a separate and
>>       independent HTTP
>>             request.  Not necessarily in the order they were sent.
>>         3.  Two or more requests may be handled by separate processes
>>             simultaneously that can't be written into a single
>>       destination.
>>         4.  Somehow the server needs to request a resend if a chunk is
>>       missing.
>>             Solving this problem requires an imaginitive use of HTTP.
>>
>>       Sounds messy.  But might be appropriate for 100M+ sized uploads.
>>        This *may* reflect your situation.  Can you please confirm?
>>
>>       For a single process, the incoming content-length is
>>       unnecessary. Buffered I/O automatically knows when transmission
>>       is complete.  The read() argument is the buffer size, not the
>>       content length.  Whether you spool the buffer to disk or simply
>>       enlarge the buffer should be determined by your hardware
>>       capabilities.  This is standard IO behavior that has nothing to
>>       do with HTTP chunk.  Without a "Content-Length" header, after
>>       looping your read() operation, determine the length of the
>>       aggregate data and pass that to Catalyst.
>>
>>       But if you're confident that the complete request spans several
>>       smaller (chunked) HTTP requests, you'll need to address all the
>>       problems I've described above, plus the problem of re-assembling
>>       the whole thing for Catalyst.  I don't know anything about
>>       Plack, maybe it can perform all this required magic.
>>
>>       Otherwise, if the whole purpose of the Plack temporary file is
>>       to pass a file handle, you can pass a buffer as a file handle.
>>        Used to be IO::String, but now that functionality is built into
>>       the core.
>>
>>       By your last paragraph, I'm really lost.  Since you're already
>>       passing the request as a file handle, I'm guessing that Catalyst
>>       creates the tempororary file for the *response* body.  Can you
>>       please clarify?  Also, what do you mean by "de-chunking"?  Is
>     >       that the same think as re-assembling?
>>
>>       Wish I could give a better answer.  Let me know if this helps.
>>
>>       -Jim
>>
>>
>>       On Tue, 2 Jul 2013, Bill Moseley wrote:
>>
>>             For requests that are chunked (Transfer-Encoding:
>>             chunked and no
>>             Content-Length header) calling $r->read returns
>>             unchunked data from the
>>             socket.
>>             That's indeed handy.  Is that mod_perl doing that
>>             un-chunking or is it
>>             Apache?
>>
>>             But, it leads to some questions.   
>>
>>             First, if $r->read reads unchunked data then why is
>>             there a
>>             Transfer-Encoding header saying that the content is
>>             chunked?   Shouldn't
>>             that header be removed?   How does one know if the
>>             content is chunked or
>>             not, otherwise?
>>
>>             Second, if there's no Content-Length header then how
>>             does one know how much
>>             data to read using $r->read?   
>>
>>             One answer is until $r->read returns zero bytes, of
>>             course.  But, is
>>             that guaranteed to always be the case, even for,
>>             say, pipelined requests?  
>>             My guess is yes because whatever is de-chunking the
>>             request knows to stop
>>             after reading the last chunk, trailer and empty
>>             line.   Can anyone elaborate
>>             on how Apache/mod_perl is doing this? 
>>
>>
>>             Perhaps I'm approaching this incorrectly, but this
>>             is all a bit untidy.
>>
>>             I'm using Catalyst and Catalyst needs a
>>             Content-Length.  So, I have a Plack
>>             Middleware component that creates a temporary file
>>             writing the buffer from
>>             $r->read( my $buffer, 64 * 1024 ) until that returns
>>             zero bytes.  I pass
>>             this file handle onto Catalyst.
>>
>>             Then, for some content-types, Catalyst (via
>>             HTTP::Body) writes the body to
>>             another temp file.    I don't know how
>>             Apache/mod_perl does its de-chunking,
>>             but I can call $r->read with a huge buffer length
>>             and Apache returns that.
>>              So, maybe Apache is buffering to disk, too.
>>
>>             In other words, for each tiny chunked JSON POST or
>>             PUT I'm creating two (or
>>             three?) temp files which doesn't seem ideal.
>>
>>
>>             --
>>             Bill Moseley
>>             [email protected]
>>
>>
>>
>>
>> -- 
>> Bill Moseley
>> [email protected]
>>
>>