Re: "Error 400" error on some servers for "security" reasons

Max Kirillov <max630-JGs/[email protected]> writes:

> Go, for example, at http://psi-im.org/wiki/. You must see
> the "http error 400" page:
> 
> There is link to
> http://www.ioerror.us/bb2-support-key?key=52c8-41be-582e-c5e4
> where something abous viruses written
> 
> Such behavior happens on many blogs. If I disable the
> wwwoffle they work.

The cause of the problem is a script called bad-behaviour that checks
for slightly non-compliant HTTP headers and totally blocks access if
they are found.  There is no reason why the behaviour that they are
checking for would stop the website working.  Most sites that use it
probably don't realise exactly what it does but they believe that it
is a good indicator of bad clients (automated downloaders,
blog-comment spam posters etc.).  You could try reporting the problem
to the admin of psi-im.org and telling him that bad-behaviour is
blocking access for a trivial problem.


> I wrote to admins of the http://www.ioerror.us/, and they
> responded:
> 
> > This is caused by a bug in wwwoffle, which the author is
> > aware of but says on his web site that he does not plan to
> > fix it. When you stop using it, you will be able to access
> > these sites again.
> 
> I could not find anything at the wwwoffle site. Well, I
> found nothing in the google.

There is nothing on the WWWOFFLE webpages and I don't remember ever
saying that I would not fix it.  Another WWWOFFLE user reported the
problem to me earlier in the year and had a conversation with the
admin of www.ioerror.us about the problem.  At that time the fix that
should of worked didn't so we stopped the discussion.

The explanation that we got from www.ioerror.us was:

:  Yes, the correct Connection: header that wwwoffle should be sending
:  is "Connection: TE, close" (or "Connection: close, TE") when
:  requesting chunked data. This would be a bug in wwwoffle. Reference:
:  RFC 2616, section 14.39:
: 
:     The TE header field only applies to the immediate connection.
:     Therefore, the keyword MUST be supplied within a Connection header
:     field (section 14.10) whenever TE is present in an HTTP/1.1 message.
: 
:  The test is in Bad Behaviour because several malicious proxy servers
:  manage to screw this up, while browsers get it right. :)

This is a stupid part of the HTTP specification because the TE header
can only apply to the immediate connection.  There is no reason that
there needs to be another header that contains a list of headers that
only apply to the immediate connection.  Adding the extra header does
nothing useful.

I did fix this bug in WWWOFFLE for version 2.9a, but then I removed
the fix again.  The ChangeLog for version 2.9a shows the two changes.

2006-06-12  Andrew M. Bishop  <amb-Uxr6IM1mbv2TY6FTCsQk+9Bc4/[email protected]>

	* src/wwwoffles.c:
	Add a new header to keep the 'bad-behaviour' PHP script happy.

...

2006-07-16  Andrew M. Bishop  <amb-Uxr6IM1mbv2TY6FTCsQk+9Bc4/[email protected]>

	* src/wwwoffles.c:
	Remove the latest new header to keep IIS servers happy.


The problem is that if I make WWWOFFLE HTTP compliant and add in "TE"
to the end of the "Connection" header then servers that use IIS
(Microsoft's web server) sometimes don't close the connection.  It
seems that they ignore the "close" part of the "Connection" header.

I haven't reported this information to the author of bad-behaviour
because I am sure that he wouldn't care and say that IIS is non
compliant and that it isn't his problem.  The whole purpose of the
bad-behaviour script is to say that if you are not doing things
exactly as the HTTP specification describes then you must be wrong.
The bad-behaviour script is designed to deliberately not follow the
advice of RFC1123:

-------------------- RFC 1123 --------------------
      1.2.2  Robustness Principle

         At every layer of the protocols, there is a general rule whose
         application can lead to enormous benefits in robustness and
         interoperability:

                "Be liberal in what you accept, and
                 conservative in what you send"

-------------------- RFC 1123 --------------------


If you want a work-around then you need to disable chunked requests to
the servers that have the problem.

<http://psi-im.org/*> request-chunked-data = no


I had the problem myself with some servers, but on one of them they
stopped using the bad-behaviour script after a very short time.  At
the moment I have no exceptions in my configuration file.

-- 
Andrew.
----------------------------------------------------------------------
Andrew M. Bishop                             amb-Uxr6IM1mbv2TY6FTCsQk+9Bc4/[email protected]
                                      http://www.gedanken.demon.co.uk/

WWWOFFLE users page:
        http://www.gedanken.demon.co.uk/wwwoffle/version-2.9/user.html