Re: form-date with newline in filename
[email protected] (Bill Moseley) Wed, 5 Jul 2017 09:20:50 -0700
| Newsgroups | perl.libwww |
|---|---|
| Message-ID | <CAKhN_m6-P3f6_kBXJPmys0mtJiiuODmSf3vnEwdrcpNkjk40wg@mail.gmail.com> |
--f403045c0cb4f1654a0553946358 Content-Type: text/plain; charset="UTF-8" On Tue, Jul 4, 2017 at 8:36 PM, Olaf Alders <[email protected]> wrote: > > > On Jul 4, 2017, at 4:41 PM, Bill Moseley <[email protected]> wrote: > > > > I'm trying to understand if the Perl code is doing the right thing by > placing a newline directly in the double-quoted filename in the > Content-Disposition header. > > > > Even though it's probably a bad thing to do, POSIX allows newlines in > filenames. Receiving systems, of course, must be careful how that filename > is used -- but in this case it's never actually used in a filesystem (i.e. > it's just considered metadata). > > > > The code below does a full round-trip successfully (meaning the newline > in the filename is preserved), but that's all within Perl. > > > > I'm POSTing to a service written in Golang and that library is > complaining about malformed headers. > > > > My question: Is HTTP::Request not escaping correctly or is Golang > library not parsing correctly? > > > > use strict; > > use warnings; > > use HTTP::Request::Common; > > use HTTP::Response; > > use HTTP::Body; > > use Data::Dumper; > > > > my $filename = "name with\na newline"; > > > > my $req = POST( > > 'http://example.com/post', > > content_type => 'form-data', > > Content => [ > > file => [ > > $0, > > $filename, > > ], > > one => 1, > > two => 2, > > ], > > ); > > > > my $res = HTTP::Response->parse( $req->as_string ); > > my $body = HTTP::Body->new( join( ' ' ,$res->content_type), > $res->content_length ); > > $body->add( $res->decoded_content ); > > print Dumper $body->upload; > > > > Above returns: > > > > $VAR1 = { > > 'file' => { > > 'filename' => 'name with > > a newline', > > 'tempname' => '/var/folders/sz/ > w4rntlpx76vcy5xrp441m0qw0000gn/T/6QNv98gd5B', > > 'size' => 561, > > 'headers' => { > > 'Content-Disposition' => > 'form-data; name="file"; filename="name with > > a newline"', > > 'Content-Type' => 'text/plain' > > }, > > 'name' => 'file' > > } > > }; > > > > It seems like header folding is no longer allowed, but I'm not clear > that this is a case of header-folding: > > > > https://stackoverflow.com/questions/521275/how-to- > escape-a-line-break-literal-in-the-http-header > > > > Or asked another way, is Perl or Golang breaking Postel's law? > > Hi Bill, > > Is it possible for you to print the actual outgoing headers? With the > newlines being involved, the order of the headers could make a difference > here. > > Best, > > Olaf > > Sure, but I think this is more of a question about how the header is created. I thought maybe an issue with header-folding, but maybe not. https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html say a header value ("field-content") can include a "quoted-string", which is what HTTP::Request (or HTTP::Headers) is doing. But https://www.w3.org/Protocols/rfc2616/rfc2616-sec2.html says a quoted-string is: A string of text is parsed as a single word if it is quoted using double-quote marks. quoted-string = ( <"> *(qdtext | quoted-pair ) <"> ) qdtext = <any TEXT except <">> And "TEXT" is: TEXT = <any OCTET except CTLs, but including LWS> And "CTL" is: CTL = <any US-ASCII control character (octets 0 - 31) and DEL (127)> So, that looks like newlines cannot be used. Is that the correct reading? I'm also not sure there is a standard way to escape the newlines -- i.e. might have to be an agreement between the sender and receiver. If the sender permits newlines in filenames it should be able to represent those in the header in a standard way -- even if not a good idea to use newlines in filenames. What is expected with HTTP::Request::Common? Should the HTTP::* methods handle escaping or it expected that all escaping must be done be the caller? -- Bill Moseley [email protected] --f403045c0cb4f1654a0553946358 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><br><div class=3D"gmail_extra"><br><div class=3D"gmail_quo= te">On Tue, Jul 4, 2017 at 8:36 PM, Olaf Alders <span dir=3D"ltr"><<a hr= ef=3D"mailto:[email protected]" target=3D"_blank">olaf@wundersolutio= ns.com</a>></span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"= margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-lef= t:1ex"><div class=3D"gmail-HOEnZb"><div class=3D"gmail-h5"><br> > On Jul 4, 2017, at 4:41 PM, Bill Moseley <<a href=3D"mailto:moseley= @hank.org">[email protected]</a>> wrote:<br> ><br> > I'm trying to understand if the Perl code is doing the right thing= by placing a newline directly in the double-quoted filename in the Content= -Disposition header.<br> ><br> > Even though it's probably a bad thing to do, POSIX allows newlines= in filenames. Receiving systems, of course, must be careful how that filen= ame is used -- but in this case it's never actually used in a filesyste= m (i.e. it's just considered metadata).<br> ><br> > The code below does a full round-trip successfully (meaning the newlin= e in the filename is preserved), but that's all within Perl.<br> ><br> > I'm POSTing to a service written in Golang and that library is com= plaining about malformed headers.<br> ><br> > My question: Is HTTP::Request not escaping correctly or is Golang libr= ary not parsing correctly?<br> ><br> > use strict;<br> > use warnings;<br> > use HTTP::Request::Common;<br> > use HTTP::Response;<br> > use HTTP::Body;<br> > use Data::Dumper;<br> ><br> > my $filename =3D "name with\na newline";<br> ><br> > my $req =3D POST(<br> >=C2=A0 =C2=A0 =C2=A0'<a href=3D"http://example.com/post" rel=3D"nor= eferrer" target=3D"_blank">http://example.com/post</a>',<br> >=C2=A0 =C2=A0 =C2=A0content_type =3D> 'form-data',<br> >=C2=A0 =C2=A0 =C2=A0Content =3D> [<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0file =3D> [<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0$0,<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0$filename,<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0],<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0one =3D> 1,<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0two =3D> 2,<br> >=C2=A0 =C2=A0 =C2=A0],<br> > );<br> ><br> > my $res =3D HTTP::Response->parse( $req->as_string );<br> > my $body =3D HTTP::Body->new( join( ' ' ,$res->content_t= ype), $res->content_length );<br> > $body->add( $res->decoded_content );<br> > print Dumper $body->upload;<br> ><br> > Above returns:<br> ><br> > $VAR1 =3D {<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0'file' =3D> {<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0'filename' =3D> 'name with<br> > a newline',<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0'tempname' =3D> '/var/folders/sz/<wbr>w4rntlpx7= 6vcy5xrp441m0qw0000gn<wbr>/T/6QNv98gd5B',<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0'size' =3D> 561,<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0'headers' =3D> {<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 'Content= -Disposition' =3D> 'form-data; name=3D"file"; filename= =3D"name with<br> > a newline"',<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 'Content= -Type' =3D> 'text/plain'<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 },<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0'name' =3D> 'file'<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0}<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0};<br> ><br> > It seems like header folding is no longer allowed, but I'm not cle= ar that this is a case of header-folding:<br> ><br> > <a href=3D"https://stackoverflow.com/questions/521275/how-to-escape-a-= line-break-literal-in-the-http-header" rel=3D"noreferrer" target=3D"_blank"= >https://stackoverflow.com/<wbr>questions/521275/how-to-<wbr>escape-a-line-= break-literal-<wbr>in-the-http-header</a><br> ><br> > Or asked another way, is Perl or Golang breaking Postel's law?<br> <br> </div></div>Hi Bill,<br> <br> Is it possible for you to print the actual outgoing headers?=C2=A0 With the= newlines being involved, the order of the headers could make a difference = here.<br> <br> Best,<br> <br> Olaf<br> <br></blockquote><div><br></div><div>Sure, but I think this is more of a qu= estion about how the header is created. I thought maybe an issue with heade= r-folding, but maybe not.</div><div><br></div><div><a href=3D"https://www.w= 3.org/Protocols/rfc2616/rfc2616-sec4.html">https://www.w3.org/Protocols/rfc= 2616/rfc2616-sec4.html</a> say a header value ("field-content") c= an include a "<span style=3D"color:rgb(0,0,0)">quoted-string", wh= ich is what HTTP::Request (or HTTP::Headers) is doing.</span><br></div><div= ><span style=3D"color:rgb(0,0,0)"><br></span></div><div><span style=3D"colo= r:rgb(0,0,0)">But=C2=A0</span><font color=3D"#000000"><a href=3D"https://ww= w.w3.org/Protocols/rfc2616/rfc2616-sec2.html">https://www.w3.org/Protocols/= rfc2616/rfc2616-sec2.html</a> says a quoted-string is:</font></div><div><fo= nt color=3D"#000000"><br></font></div></div></div><blockquote style=3D"marg= in:0px 0px 0px 40px;border:none;padding:0px"><div class=3D"gmail_extra"><di= v class=3D"gmail_quote"><div><p style=3D"color:rgb(0,0,0);font-family:Times= ;font-size:medium">A string of text is parsed as a single word if it is quo= ted using double-quote marks.</p></div></div></div><div class=3D"gmail_extr= a"><div class=3D"gmail_quote"><div><pre style=3D"color:rgb(0,0,0)"> q= uoted-string =3D ( <"> *(qdtext | quoted-pair ) <"> = )</pre></div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_quo= te"><div><pre style=3D"color:rgb(0,0,0)"> qdtext =3D <any = TEXT except <">></pre></div></div></div></blockquote><div cla= ss=3D"gmail_extra"><div class=3D"gmail_quote"><div><pre style=3D"color:rgb(= 0,0,0)"><br></pre><pre style=3D"color:rgb(0,0,0)"><font face=3D"arial, helv= etica, sans-serif">And "TEXT" is:</font></pre></div></div></div><= blockquote style=3D"margin:0px 0px 0px 40px;border:none;padding:0px"><div c= lass=3D"gmail_extra"><div class=3D"gmail_quote"><div><pre style=3D"color:rg= b(0,0,0)"><pre> TEXT =3D <any OCTET except CTLs, but including LWS></pre></pre></div></div></div>= </blockquote><font color=3D"#000000"><span style=3D"white-space:pre"><font = face=3D"arial, helvetica, sans-serif">And "CTL" is:</font></span>= </font><div><font color=3D"#000000"><span style=3D"white-space:pre"><font f= ace=3D"arial, helvetica, sans-serif"><br></font></span></font></div><blockq= uote style=3D"margin:0px 0px 0px 40px;border:none;padding:0px"><div><pre st= yle=3D"color:rgb(0,0,0)"> CTL =3D <any US-ASCII control= character (octets 0 - 31) and DEL (127)></pre></div></bloc= kquote><div><div class=3D"gmail_extra"><div class=3D"gmail_quote"><div>So, = that looks like newlines cannot be used.</div><div><br></div><div>Is that t= he correct reading?</div><div><br></div><div>I'm also not sure there is= a standard way to escape the newlines -- i.e. might have to be an agreemen= t between the sender and receiver.</div><div><br></div><div>If the sender p= ermits newlines in filenames it should be able to represent those in the he= ader in a standard way -- even if not a good idea to use newlines in filena= mes.</div><div><br></div><div>What is expected with HTTP::Request::Common?= =C2=A0 Should the HTTP::* methods handle escaping or it expected that all e= scaping must be done be the caller?</div><div><br></div><div><br></div><div= >=C2=A0</div></div><br><br clear=3D"all"><div><br></div>-- <br><div class= =3D"gmail_signature">Bill Moseley<br><a href=3D"mailto:[email protected]" ta= rget=3D"_blank">[email protected]</a></div> </div></div></div> --f403045c0cb4f1654a0553946358--