Re: [Fwd] The Spider of Doom (RISKS 24.22)

Dan Jacobson <[email protected]> writes:

> news:comp.risks Risks Digest 24.22
>   "As it turns out, Google's spider doesn't use cookies, which means that it
>   can easily bypass a check for the "isLoggedOn" cookie to be "false". It also
>   doesn't pay attention to Javascript, which would normally prompt and
>   redirect users who are not logged on. It does, however, follow every
>   hyperlink on every page it finds, including those with "Delete Page" in the
>   title. Whoops."
> Glad to hear the Delete item in the AddCacheInfo page is being phase
> out.

I don't know where you heard that, but it isn't being phased out.

> Now all that remains is if Google or some other spider ever
> happened to get into one of those WWWOFFLE index pages with their
> delete options... Or at least where one has unset the password...
> Wait, /etc/wwwoffle/robots.txt (says version 2.7 here) will surely block it...

There are two things that should stop automated spiders from deleting
all of the pages in the WWWOFFLE cache:

1) The robots.txt file blocks access to the index pages and the delete
   page by default.

2) There is the option to require authentication (password) before
   deleting pages.

If you leave your WWWOFFLE cache accessible by untrusted users (people
or URL spiders) and you don't have a password then you are asking for
trouble.


Juliusz Chroboczek <[email protected]> writes:

> Why is anyone using GET for deleting pages in the first place?  It
> should be triggered by a POST so that automated user agents and
> upstream proxies can know that it has a side effect.

Using a POST would not stop a malicious spider from following the
link.  There is no reason why a spider could not create the
appropriate request.

The problem in the original comp.risks message was about using cookies
and javascript to authenticate a request for a URL but with a default
of allowing access if cookies and javascript are not used.


In WWWOFFLE either a POST or a GET request can be used for the page
that deletes a URL (the code accepts either).  I chose to have a GET
request for deleting pages from the cache.  There are lots of things
in WWWOFFLE that have side effects; any time that a page is requested
there is a change to the cache.  These cannot all have POST requests.

-- 
Andrew.
----------------------------------------------------------------------
Andrew M. Bishop                             amb-Uxr6IM1mbv2TY6FTCsQk+9Bc4/[email protected]
                                      http://www.gedanken.demon.co.uk/

WWWOFFLE users page:
        http://www.gedanken.demon.co.uk/wwwoffle/version-2.9/user.html