Re: form-date with newline in filename
[email protected] (Bill Moseley) Wed, 05 Jul 2017 16:54:01 +0000
| Newsgroups | perl.libwww |
|---|---|
| Message-ID | <CAKhN_m7Mb24YVJXE3nC=1Gxwrwknja03v-2FqPJXw66UQ=btfw@mail.gmail.com> |
--94eb2c199f6070dfb9055394d935 Content-Type: text/plain; charset="UTF-8" Oh, how did I start talking about HTTP headers? Holiday daze. It's about form-data. I'll see what browsers do. On Wed, Jul 5, 2017 at 09:20 Bill Moseley <[email protected]> wrote: > On Tue, Jul 4, 2017 at 8:36 PM, Olaf Alders <[email protected]> > wrote: > >> >> > On Jul 4, 2017, at 4:41 PM, Bill Moseley <[email protected]> wrote: >> > >> > I'm trying to understand if the Perl code is doing the right thing by >> placing a newline directly in the double-quoted filename in the >> Content-Disposition header. >> > >> > Even though it's probably a bad thing to do, POSIX allows newlines in >> filenames. Receiving systems, of course, must be careful how that filename >> is used -- but in this case it's never actually used in a filesystem (i.e. >> it's just considered metadata). >> > >> > The code below does a full round-trip successfully (meaning the newline >> in the filename is preserved), but that's all within Perl. >> > >> > I'm POSTing to a service written in Golang and that library is >> complaining about malformed headers. >> > >> > My question: Is HTTP::Request not escaping correctly or is Golang >> library not parsing correctly? >> > >> > use strict; >> > use warnings; >> > use HTTP::Request::Common; >> > use HTTP::Response; >> > use HTTP::Body; >> > use Data::Dumper; >> > >> > my $filename = "name with\na newline"; >> > >> > my $req = POST( >> > 'http://example.com/post', >> > content_type => 'form-data', >> > Content => [ >> > file => [ >> > $0, >> > $filename, >> > ], >> > one => 1, >> > two => 2, >> > ], >> > ); >> > >> > my $res = HTTP::Response->parse( $req->as_string ); >> > my $body = HTTP::Body->new( join( ' ' ,$res->content_type), >> $res->content_length ); >> > $body->add( $res->decoded_content ); >> > print Dumper $body->upload; >> > >> > Above returns: >> > >> > $VAR1 = { >> > 'file' => { >> > 'filename' => 'name with >> > a newline', >> > 'tempname' => >> '/var/folders/sz/w4rntlpx76vcy5xrp441m0qw0000gn/T/6QNv98gd5B', >> > 'size' => 561, >> > 'headers' => { >> > 'Content-Disposition' => >> 'form-data; name="file"; filename="name with >> > a newline"', >> > 'Content-Type' => 'text/plain' >> > }, >> > 'name' => 'file' >> > } >> > }; >> > >> > It seems like header folding is no longer allowed, but I'm not clear >> that this is a case of header-folding: >> > >> > >> https://stackoverflow.com/questions/521275/how-to-escape-a-line-break-literal-in-the-http-header >> > >> > Or asked another way, is Perl or Golang breaking Postel's law? >> >> Hi Bill, >> >> Is it possible for you to print the actual outgoing headers? With the >> newlines being involved, the order of the headers could make a difference >> here. >> >> Best, >> >> Olaf >> >> > Sure, but I think this is more of a question about how the header is > created. I thought maybe an issue with header-folding, but maybe not. > > https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html say a header value > ("field-content") can include a "quoted-string", which is what > HTTP::Request (or HTTP::Headers) is doing. > > But https://www.w3.org/Protocols/rfc2616/rfc2616-sec2.html says a > quoted-string is: > > A string of text is parsed as a single word if it is quoted using > double-quote marks. > > quoted-string = ( <"> *(qdtext | quoted-pair ) <"> ) > > qdtext = <any TEXT except <">> > > > And "TEXT" is: > > TEXT = <any OCTET except CTLs, > but including LWS> > > And "CTL" is: > > CTL = <any US-ASCII control character > (octets 0 - 31) and DEL (127)> > > So, that looks like newlines cannot be used. > > Is that the correct reading? > > I'm also not sure there is a standard way to escape the newlines -- i.e. > might have to be an agreement between the sender and receiver. > > If the sender permits newlines in filenames it should be able to represent > those in the header in a standard way -- even if not a good idea to use > newlines in filenames. > > What is expected with HTTP::Request::Common? Should the HTTP::* methods > handle escaping or it expected that all escaping must be done be the caller? > > > > > > > -- > Bill Moseley > [email protected] > -- Bill Moseley [email protected] --94eb2c199f6070dfb9055394d935 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div><div dir=3D"auto">Oh, how did I start talking about HTTP headers?=C2= =A0 Holiday daze. It's about form-data.=C2=A0 I'll see what browser= s do.=C2=A0</div><br><div class=3D"gmail_quote"><div>On Wed, Jul 5, 2017 at= 09:20 Bill Moseley <<a href=3D"mailto:[email protected]">[email protected]= g</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin= :0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class=3D= "gmail_extra"><div class=3D"gmail_quote">On Tue, Jul 4, 2017 at 8:36 PM, Ol= af Alders <span><<a href=3D"mailto:[email protected]" target=3D"_= blank">[email protected]</a>></span> wrote:<br><blockquote class= =3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg= b(204,204,204);padding-left:1ex"><div class=3D"m_5713435937883206901gmail-H= OEnZb"><div class=3D"m_5713435937883206901gmail-h5"><br> > On Jul 4, 2017, at 4:41 PM, Bill Moseley <<a href=3D"mailto:moseley= @hank.org" target=3D"_blank">[email protected]</a>> wrote:<br> ><br> > I'm trying to understand if the Perl code is doing the right thing= by placing a newline directly in the double-quoted filename in the Content= -Disposition header.<br> ><br> > Even though it's probably a bad thing to do, POSIX allows newlines= in filenames. Receiving systems, of course, must be careful how that filen= ame is used -- but in this case it's never actually used in a filesyste= m (i.e. it's just considered metadata).<br> ><br> > The code below does a full round-trip successfully (meaning the newlin= e in the filename is preserved), but that's all within Perl.<br> ><br> > I'm POSTing to a service written in Golang and that library is com= plaining about malformed headers.<br> ><br> > My question: Is HTTP::Request not escaping correctly or is Golang libr= ary not parsing correctly?<br> ><br> > use strict;<br> > use warnings;<br> > use HTTP::Request::Common;<br> > use HTTP::Response;<br> > use HTTP::Body;<br> > use Data::Dumper;<br> ><br> > my $filename =3D "name with\na newline";<br> ><br> > my $req =3D POST(<br> >=C2=A0 =C2=A0 =C2=A0'<a href=3D"http://example.com/post" rel=3D"nor= eferrer" target=3D"_blank">http://example.com/post</a>',<br> >=C2=A0 =C2=A0 =C2=A0content_type =3D> 'form-data',<br> >=C2=A0 =C2=A0 =C2=A0Content =3D> [<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0file =3D> [<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0$0,<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0$filename,<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0],<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0one =3D> 1,<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0two =3D> 2,<br> >=C2=A0 =C2=A0 =C2=A0],<br> > );<br> ><br> > my $res =3D HTTP::Response->parse( $req->as_string );<br> > my $body =3D HTTP::Body->new( join( ' ' ,$res->content_t= ype), $res->content_length );<br> > $body->add( $res->decoded_content );<br> > print Dumper $body->upload;<br> ><br> > Above returns:<br> ><br> > $VAR1 =3D {<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0'file' =3D> {<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0'filename' =3D> 'name with<br> > a newline',<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0'tempname' =3D> '/var/folders/sz/w4rntlpx76vcy5= xrp441m0qw0000gn/T/6QNv98gd5B',<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0'size' =3D> 561,<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0'headers' =3D> {<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 'Content= -Disposition' =3D> 'form-data; name=3D"file"; filename= =3D"name with<br> > a newline"',<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 'Content= -Type' =3D> 'text/plain'<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 },<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0'name' =3D> 'file'<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0}<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0};<br> ><br> > It seems like header folding is no longer allowed, but I'm not cle= ar that this is a case of header-folding:<br> ><br> > <a href=3D"https://stackoverflow.com/questions/521275/how-to-escape-a-= line-break-literal-in-the-http-header" rel=3D"noreferrer" target=3D"_blank"= >https://stackoverflow.com/questions/521275/how-to-escape-a-line-break-lite= ral-in-the-http-header</a><br> ><br> > Or asked another way, is Perl or Golang breaking Postel's law?<br> <br> </div></div>Hi Bill,<br> <br> Is it possible for you to print the actual outgoing headers?=C2=A0 With the= newlines being involved, the order of the headers could make a difference = here.<br> <br> Best,<br> <br> Olaf<br> <br></blockquote><div><br></div></div></div></div><div><div class=3D"gmail_= extra"><div class=3D"gmail_quote"><div>Sure, but I think this is more of a = question about how the header is created. I thought maybe an issue with hea= der-folding, but maybe not.</div><div><br></div><div><a href=3D"https://www= .w3.org/Protocols/rfc2616/rfc2616-sec4.html" target=3D"_blank">https://www.= w3.org/Protocols/rfc2616/rfc2616-sec4.html</a> say a header value ("fi= eld-content") can include a "<span style=3D"color:rgb(0,0,0)">quo= ted-string", which is what HTTP::Request (or HTTP::Headers) is doing.<= /span><br></div><div><span style=3D"color:rgb(0,0,0)"><br></span></div><div= ><span style=3D"color:rgb(0,0,0)">But=C2=A0</span><font color=3D"#000000"><= a href=3D"https://www.w3.org/Protocols/rfc2616/rfc2616-sec2.html" target=3D= "_blank">https://www.w3.org/Protocols/rfc2616/rfc2616-sec2.html</a> says a = quoted-string is:</font></div><div><font color=3D"#000000"><br></font></div= ></div></div><blockquote style=3D"margin:0px 0px 0px 40px;border:none;paddi= ng:0px"><div class=3D"gmail_extra"><div class=3D"gmail_quote"><div><p style= =3D"color:rgb(0,0,0);font-family:Times;font-size:medium">A string of text i= s parsed as a single word if it is quoted using double-quote marks.</p></di= v></div></div><div class=3D"gmail_extra"><div class=3D"gmail_quote"><div><p= re style=3D"color:rgb(0,0,0)"> quoted-string =3D ( <"> *(= qdtext | quoted-pair ) <"> )</pre></div></div></div><div class= =3D"gmail_extra"><div class=3D"gmail_quote"><div><pre style=3D"color:rgb(0,= 0,0)"> qdtext =3D <any TEXT except <">></pre= ></div></div></div></blockquote><div class=3D"gmail_extra"><div class=3D"gm= ail_quote"><div><pre style=3D"color:rgb(0,0,0)"><br></pre><pre style=3D"col= or:rgb(0,0,0)"><font face=3D"arial, helvetica, sans-serif">And "TEXT&q= uot; is:</font></pre></div></div></div><blockquote style=3D"margin:0px 0px = 0px 40px;border:none;padding:0px"><div class=3D"gmail_extra"><div class=3D"= gmail_quote"><div><pre style=3D"color:rgb(0,0,0)"><pre> TEXT = =3D <any OCTET except CTLs, but including LWS></pre></pre></div></div></div>= </blockquote><font color=3D"#000000"><span style=3D"white-space:pre-wrap"><= font face=3D"arial, helvetica, sans-serif">And "CTL" is:</font></= span></font><div><font color=3D"#000000"><span style=3D"white-space:pre-wra= p"><font face=3D"arial, helvetica, sans-serif"><br></font></span></font></d= iv><blockquote style=3D"margin:0px 0px 0px 40px;border:none;padding:0px"><d= iv><pre style=3D"color:rgb(0,0,0)"> CTL =3D <any US-ASC= II control character (octets 0 - 31) and DEL (127)></pre></div></bloc= kquote><div><div class=3D"gmail_extra"><div class=3D"gmail_quote"><div>So, = that looks like newlines cannot be used.</div><div><br></div><div>Is that t= he correct reading?</div><div><br></div><div>I'm also not sure there is= a standard way to escape the newlines -- i.e. might have to be an agreemen= t between the sender and receiver.</div><div><br></div><div>If the sender p= ermits newlines in filenames it should be able to represent those in the he= ader in a standard way -- even if not a good idea to use newlines in filena= mes.</div><div><br></div><div>What is expected with HTTP::Request::Common?= =C2=A0 Should the HTTP::* methods handle escaping or it expected that all e= scaping must be done be the caller?</div><div><br></div><div><br></div><div= >=C2=A0</div></div></div></div></div><div><div><div class=3D"gmail_extra"><= br><br clear=3D"all"><div><br></div>-- <br><div class=3D"m_5713435937883206= 901gmail_signature">Bill Moseley<br><a href=3D"mailto:[email protected]" tar= get=3D"_blank">[email protected]</a></div> </div></div></div></blockquote></div></div><div dir=3D"ltr">-- <br></div><d= iv data-smartmail=3D"gmail_signature">Bill Moseley<br><a href=3D"mailto:mos= [email protected]" target=3D"_blank">[email protected]</a></div> --94eb2c199f6070dfb9055394d935--