Bug#217243: Real problem, not wishlist

[Forwarding to curiosa, as this is just too good a short story to miss, 
considering how dry bug reports usually are.]

On 30.12.2013 23:16, Gwern Branwen wrote:
> This bug was a very serious, almost fatal, bug for me recently, and I
> thought I would share my story to emphasize that for me, this was not
> a 'wishlist' severity bug.
>
> I research Tor black-markets (see http://www.gwern.net/Silk%20Road )
> because I am interested in them from economic, historical, and
> statistical perspectives. Black-markets are dangerous risky
> enterprises, even when run as Tor hidden-services, and so people like
> me or Nicolas Christin often download or spider them so as to have
> copies to analyze later.
>
> In October 2013, Silk Road was famously busted (to everyone's complete
> surprise). Fortunately, the FBI seizure left the SR forums alone, and
> it became a top priority for me to grab a copy of the forums while I
> still could since they would be invaluable in the post-mortem of SR
> and the wave of arrests everyone expected to follow the bust. The wget
> spider of the public forum went fine. But even more importantly, I
> needed to get a copy of the members-only subforum, the Vendor
> Roundtable, where all the Silk Road drug dealers talked shop, and more
> importantly, turned out to have discovered some of the early bits and
> pieces of how Silk Road/Ross Ulbricht was busted.
>
> I'm not a drug dealer, but I know a few of the SR ones and was able to
> get login credentials. I logged in, checked that I had access to the
> Roundtable, exported my cookies, and read the wget man page for
> guidance:
>
>         -R rejlist --reject rejlist
>             Specify comma-separated lists of file name suffixes or patterns to
>             accept or reject. Note that if any of the wildcard characters, *,
>             ?, [ or ], appear in an element of acclist or rejlist, it will be
>             treated as a pattern, rather than a suffix.
>
> Perfect. Exactly what I needed to avoid being logged out. I threw in a
> `--reject '*logout*'` to cover all possible logout links, and kicked
> the spider off. I watched for a few minutes, everything looked like it
> was going fine with no suspicious 'index.php?logout' files showing up
> or anything, and I went off to deal with other aspects of breaking
> news.
>
> 2 days later, the spider was still running (it's a very big forum and
> Tor has high latency), and I needed to check a particular claim about
> a Roundtable thread. No problem, I had a copy of the Roundtable - I'd
> just check that. NOPE. The thread wasn't there at all. In fact, almost
> *nothing* in the Roundtable had been downloaded at all!
>
> I panicked. No one knew why the FBI hadn't shut down the forums, who
> was running them, or when they would disappear into the digital ether.
> Christin wasn't spidering the Roundtable, and I was it. If I didn't
> have a copy, then likely, no one did. It would be gone permanently.
> Luckily, the forums were still up... but for how long? Minutes, hours,
> or days? What had gone wrong and how could I fix it?
>
> I logged in again, exported cookies, restarted, checked in a few
> hours. No Roundtable. WTF?! I logged in, exported, restarted, watched
> closely... I spotted in the stream a mention of 'index.php?logout'.
> But why? I went back to the `--reject` documentation. Had I called it
> wrong? Made a syntax error? Did `--reject` not do what it was supposed
> to do? But the documentation is perfectly clear: --reject rejects URLs
> from being downloaded. It doesn't do something remotely as absurd as
> download a URL and then delete it! There is no usecase for that in
> combination with rejecting URLs, it's trivially broken for many
> use-cases, and it would *definitely* be documented in the manpage.
>
> I went back, logged in... Repeat 5 or 10 times with various
> invocations of `--reject` and regexps and escalating blood pressure,
> until I checked the downloaded pages and resigned myself that somehow,
> somehow or other, I couldn't begin to explain it, neither the how nor
> the why, wget was logging itself out of the forums. As absurd as it
> sounded, nothing else fit the evidence.
>
> I started googling 'wget reject'.
>
> To discover this bug report, among others.
>
> Oh how I raged that night. 'principle of least surprise', 'betrayal',
> 'crime against posterity', 'moronic', 'deliberately malicious', 'what
> the hell', and more indelicate phrases were uttered.
>
> I was also not pleased to discover that, `--reject` aside, there was
> apparently no way whatsoever to genuinely reject URLs inside wget.
>
> Eventually, I rigged up a hack where I pointed wget to Privoxy, and
> wrote Privoxy rules to block certain URLs including the logout links.
> It's ugly, it's not easy to modify, I'm not really familiar with
> Privoxy syntax, but at least it does, in fact, work. And I was able to
> get a good chunk of the Roundtable before the forums went down. (Not
> all of it, but that's another story which is not wget's but the forum
> software's fault - I think.)
>
> Summary: `--reject` is a problem. It can't be *that* hard to fix,
> there are short patches floating around. Please fix it.
>