Re: Using the DHT to find download sources

> So what is see is: 
> 
> Credence can: 
>  * stop advertising spam and
>  * stop popular file masking spam. 
> 
> it can't: 
>  * Stop direct attacks against rare file searches. 
> 
> But these direct attacks against rare file searches aren't what
troubles me 
> most, because the damage to rare file searches currently mostly
comes from 
> advertising spam, and that could be stopped, destroying the
"advertising 
> spam" business model and that way also vastly reducing spam which
hinders 
> rare file searches. 
> 

That makes sense! So if we can stop advertising spammers' spams on
popular files, then spams on rare files will disappear too because
advertising spammers will go out of business.  

> > Are there any ideas we can borrow from anti-spam email schemes?
> 
> I just browsed a bit on that. 
> Results: 
> 
> - They do statistical analysis of the content. 
> One example is Bogofilter: 
> http://bogofilter.sourceforge.net/faq.shtml#bogo-how
> It doesn't look like it will work for the short query replies we get in 
> Gnutella, though. 
> Training might work, if the search message is taken into account. 
> Also it could weed out the obnoxious long filenames of porn-spam. 
> - description: http://www.linuxjournal.com/article/6467
> 
> - They have spam-tracking servers. One example is pyzor: 
> http://pyzor.sourceforge.net/
> "It works by computing signatures on the body of the content and
> checking these signatures against a database of known spam."
> ~ from the Razor page whoose protocol pyzor implemented and improved. 
> more detailed: 
> - http://pyzor.sourceforge.net/blocking.html
> Also I read that they check, if multiples of messages are floating
around. 
> Doesn't look like it could be used efficiently, either. The spammers
already 
> change the query replies dynamically. 
> 
> - Additional stuff: http://wiki.apache.org/spamassassin/TrustNetNotes
> 
> Long story short: They don't seem very applyable to p2p spam. 
> 

Good research!

> 
> > It works and we can find rare files. But it doesn't satisfy the first
> > requirement in our requirement list: 1) Fast search and low overhead.
> > Having a requirement list up front is useful. ;-)
> 
> Low overhead: Mostly, since the requests only travel where they
might find 
> results (QRP). 
> 

For rare files, there is big chance that none of the files are within
3 hops from you, then QRP wont help, right? If so, the requests cannot
"only travel where they might find results". 


> Fast search sadly not, but that wouldn't work with searching again
for rare 
> searches, either. 
> 
> To realize "fast search", you'd have to search on both networks 
> simultaneously, which would violate "low overhead". 
> 
> So, why do you need fast search for rare files? Wouldn't a "Only few
results 
> found on fast query, begin rare files query" message to the user be
more 
> useful? 
> 
> The user knows "I see it's scarce, but my program takes care of it"
and so he 
> won't have problems with waiting a bit longer. For me this approach
even 
> seems more satisfying to users. 
> 
> That message could also be shown as soon as Dynamic Querying goes
into "ask 
> everyone" mode. 
> 
> It should be phrased carefully, though, to ensure that it states
clearly that 
> rare file queries aren't efficient when they search for popular
files (it 
> doesn't need to say "not good for the network", just "not good for
popular 
> files" - I don't know if my message really attained that goal). 
> 
> So I'd rephrase that first requirement to 
> 1) A good user experience and low overhead. 
> 

Fast is better than slow, as long as we still have low overhead. :-)
The question is again "how to figure out popularity" 

I agree. "A good user experience" is a better requirement than "fast".

> Could you post an updated version of the requirements list? 
> 

Will do.

Best,
Peng