Re: How Freenet Works.
Greg Wooledge <[email protected]> Wed, 6 Aug 2003 20:56:16 -0400
| Newsgroups | gmane.network.freenet.web |
|---|---|
| Message-ID | <[email protected]> |
--8R/sMIKPJVD0CT2D Content-Type: multipart/mixed; boundary="MUnXZt0Uv08c1hBe" Content-Disposition: inline --MUnXZt0Uv08c1hBe Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Tom Kaitchuck ([email protected]) wrote: >=20 Here's your file back, proofread. I think there are some grave content errors in it, though, especially the last paragraph. But I won't attempt to address those. I didn't send a "diff", because the diff was larger than the original file. This is mostly because you have lines that are incredibly long (no internal newlines within paragraphs). This makes maintenance of the document by multiple people a bit harder. Anyway, this is a pretty good effort despite what I perceive to be some substantial flaws. --=20 Greg Wooledge | "Truth belongs to everybody." [email protected] | - The Red Hot Chili Peppers http://wooledge.org/~greg/ | --MUnXZt0Uv08c1hBe Content-Type: text/html; charset=us-ascii Content-Disposition: attachment; filename="works.html" Content-Transfer-Encoding: quoted-printable <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/htm= l4/strict.dtd"> <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DUTF-8"> <title>How Freenet Works</title> </head> <body> <div align=3D"center"><font size=3D"+3"><b><p>Overview</p></b></font></div> <hr> <p> On the broadest level Freenet works by having each computer that connec= ts to the network allocate a portion of its hard disk. Then when another co= mputer asks your computer for some data that you have stored, your computer= will return the file. If your computer does not have the data it will look= at all the computers it connects to, and ask the one that it thinks is mos= t likely to have the data for it. Inserting data works the same way. Each n= ode looks at the data and passes it along to the next computer that is the = best match for the data. Because all the data is broken up into small encry= pted segments (each is routed separately) all the intermediate computers do= n't know what the data is. The computers on the network decide where the da= ta should be stored biased on its hash. This means no computer can tell wha= t data it is storing or fetching, nor can anyone tell where a particular pi= ece of data is located on the network. No computer on the network knows whe= re the data originally came from or where it went. This is because if you r= equest a file and another computer gets the request, it has no way of telli= ng if you are the originator of that request, or if you are just forwarding= it from another computer. Then when the data is returned you have no way o= f telling if that computer had the data in its data store, or it got it fro= m another computer. Another positive side effect is if two people independe= ntly insert the same data, they collide so the network only needs to store = one copy.</p> <p> Of course it's all much more complex than that. However this has a few = implications that should be addressed. First it means that there is no cent= ral server that the network depends on. It also means that every permanent = node on the network contributes to it by sotring data. As a result of this,= anyone can upload content and never have to worry about the bandwidth, as = Freenet handles both the storage and distribution of content. So even if yo= u don't particularly need anonymity, you should publish your site on Freene= t because it provides free hosting regardless of how big or popular your si= te is. So, you never have to resort to putting ads on your page. </p> <p> The way Freenet's architecture is designed it is inherently good at som= e things, and inherently bad at others. For example, because all inserts an= d requests for data go through 5 - 25 computers before they reach their des= tination, Freenet has inherently high latency. This means it will never be = suitable for playing Real Time games or instant messaging. However, because= when you request a file it downloads it in many chunks, each of which will= likely come from a different computer, it can allow very fast downloads.</= p> <p> The reason that it is able to do this, is that because Freenet has a fe= w inherent advantages over most networks. Because the data is inserted, it = will be spread over a large number of hosts, even if it is unpopular. Becau= se each node has a data store, all nodes are contributing to the speed of d= ownloads even if they are not publishing any content. Freenet also requires= nodes to have higher up-times than most networks, so data is more likely t= o be available. This means that you can download from many computers even i= f the file is unpopular, and the total bandwidth is not directly limited by= the number of people downloading it or sharing it. Once Freenet is further= optimized it will very likely outperform the venerable BitTorrent.</p> <p> Freenet is very good at making information available to everyone anonym= ously. However, when it comes to sending information to a specific person, = or reliably archiving unpopular data for a long time, it is not very good. = If your application requires lower latency than Freenet can provide, and do= es not need lots of bandwidth you might look at Freenet's sister project th= e <a href=3D"http://www.invisible.net/">invisible IRC project</a> (IIP). Al= though IIP has a much lower standard of anonymity, the two complement each = other very nicely. Freenet can be used for storage or publishing data, and = IIP can be used for small fast communications, as well as e-mail.</p> <p> If you are considering uploading your own Freesite, you should look at = some of the guides on Freenet for advice, as good web design is not necessa= rily good Freesite design. </p> <div align=3D"center"><font size=3D"+3"><b><p>How Freenet Works</p></b></fo= nt></div> <hr> <div align=3D"center"><font size=3D"+1"><p><b>How data is located</b> = </p></font></div> <p> To request data a computer sends out a request which contains the hash = of the content that they want. When a node gets a request for data that it = does not have, it forwards the request to the node in its routing table tha= t it thinks will return the data the fastest based on previous requests wit= h similar hashes. If that node fails or times out then it forwards the requ= est to the next closest match. Those nodes will in turn forward it to the c= losest node in their routing table. Inserts works in a similar way: the ins= ert requests follow the path closest to their hash until the HTL (hops to l= ive) expires, then they are stored on that node. This means in the best cas= e the request will travel along the shortest path to the data that each of = the involved nodes know about. Also the worst case (very rare) involves ask= ing all the nodes within the specified range. As time goes on each of the n= odes will learn about more and more of its peers. So over time the typical = case approaches the best case. </p> <p> This system compares favorably to other peer-to-peer networks. For exam= ple Freenet's worst case scenario is the same as what Gnutella uses for all= searches. (All the nodes within a given radius will be asked for the data.= ) However in the best case scenario any data can be located in Log base M o= f N where N is the number of nodes in the network and M is the average numb= er of peers each node has. This means that Freenet scales much better than = Gnutella and similar networks, and uses much less bandwidth for overhead. <= /p> <div align=3D"center"><font size=3D"+1"><p><b>How keys work.</b> </p><= /font></div> <p> All content in Freenet is indexed by its hash. So in order to find the = data you are looking for you need to know its hash. This means that there i= s no simple keyword search. So to find information, you use your web browse= r. </p> <p>When you load Freenet you will see a main page that has hyperlinks to so= me of the more popular index sites in Freenet. When you click a link on one= of these sites you will see a URI. This is what represents the key you wan= t to fetch. There are three main types of keys. </p> <p>The most basic is a Content Hash Key (CHK). It looks something like this= :</p> <p>CHK@hdXaxkwZ9rA8-SidT0AN-bniQlgPAwI,XdCDmBuGsd-ulqbLnZ8v~w</p> <p>A CHK is a 160 bit key. It is represented in base 64 using a-z, A-Z, 0-9= , - and ~. It is divided into two parts separated by a comma. The first par= t is the hash of the encrypted file. It is what the file will be indexed un= der. The second part is the key it was encrypted with (This comes from the = hash of the original file). The second part of the key is not revealed to t= he other nodes when a request is made. </p> <p>A CHK can store one chunk of data (that can be any size up to 1MB). If y= ou want to store more than one MB, the data gets split up. You will use a k= ey just like you would if it was a single chunk. However when you download = that first chunk, rather than having data, you get what is called a manifes= t. It will contain the keys for all the other chunks of data in the file. S= o CHKs can allow you to store any amount of data, and because they are inse= rted under a key that is derived form their content, we can safely assume t= hat if two chunks of data have the same key, that they are the same data. A= s a result if two people insert the exact same data, it only needs to be st= ored on the network once.</p> <p>If you want to have a site that can be updated, or verified as to its or= igin, you could use a Signed Subspace Key (SSK). An SSK look like this:</p> <p>SSK@rBjVda8pC-Kq04jUurIAb8IzAGcPAgM/TFE//thelist.html</p> <p>Like the CHK, it consists of two parts. The first part is a public encry= ption key, this allows you to decrypt the content stored in the SSK. The se= cond part is a plain text description (this may contain slashes to emulate = a directory structure). A single SSK can contain many files, and even many = sites. They are distinguished by the plaintext portion. The data is indexed= under the the hash of the combined string of the encryption key, and the p= lain text description. </p> <p>To have an updateable site there are two options. The first is to have a= n edition site. This is usually done by posting a site in SSK@_key_PAgM/_Na= me_/1/ and then in the HTML for the site, link to SSK@_key_PAgM/_Name_/2/. = This way, when you publish a new edition, you can insert it in the new loca= tion, and people will be able to follow the link to get the new version.</p> <p>A more automated way of doing this is to have a Date Based Redirect (DBR= ). This consists of inserting a piece of metadata at SSK@_key_PAgM/_Name_/ = that tells Freenet clients how often the site is updated and where it will = be located. This is usually done by inserting at regular intervals at SSK@_= key_PAgM/_Date_-_Name_/.</p> <p>There is a third, optional component to an SSK. That is a manifest. Thi= s is represented in the key by the presence of a "//". This place= s a file at the location specified by the URI up until that point. That fil= e is called a manifest. It contains a list of files and their respective ke= ys (usually a CHK). Having a manifest means that all the data for the site = can be stored in CHKs, so when the site has to be updated, because of a new= edition, or a date rollover, one only needs to insert the new manifest, an= d the files that changed.</p> <div align=3D"center"><font size=3D"+1"><p><b>How attacks are prevented</b>= </p></font></div> <p>There are two main attacks that could be used to try to compromise your = anonymity on Freenet. The first involves a local eavesdropper listening in = on all your communications over Freenet. Ordinary encryption is not suffici= ent for protection from this sort of attack. The reason for this is because= if you want to talk to a new person, they have to know how to decrypt the = message you have sent them, so if someone is monitoring all communication b= etween the two of you, there is no way to tell them without the eavesdroppe= r hearing too. However there is a solution to this problem. It is called as= ymmetric cryptography. It works as follows: your computer uses a one-way ma= thematical function to generate two numbers called keys. When a message is = encrypted using one of the keys, it can be decrypted with the other and vic= e versa. Given one of these keys there is no way to find out the other one,= short of trying all possible combinations. One of these keys is designated= the public key, and the other, the private key. When you find out about a = new node, (e.g. through your node references file, or through another node)= , you receive its public key. Then you can send it a a message encrypted wi= th its public key. Because it is the only one that knows its private key, n= obody but it can read your message. Then after it gets your public key, it = can communicate with you, and as long as nobody knows your private key, nob= ody can tell what it sent you.</p> <p>The problem with this is that asymmetric cryptography is very slow. (It = requires a LOT of CPU power.) However Freenet works around this as follows:= when you contact a new node, you send it a "Hello" message that = contains your public key and is encrypted with their public key. Then that = node can reply with a "Hello" message that contains an ordinary e= ncryption key that is encrypted using your public key. This means, because = only you hold your private key, you are the only one who can read the messa= ge. So now you can communicate safely encoding messages both ways using ord= inary encryption encoded with the session key. This effectively thwarts any= attempts to monitor your traffic locally and means that the CPU intensive = asymmetric encryption only needs to be used for the first message you send = (or whenever you want to send a new session key). Even if the node you cont= acted revealed the session key it used to a local attacker, it would not re= veal anything other than what that node already knew.</p> <p>The second type of attack consists of having a large group of nodes that= are connected to you pool their knowledge to try to develop a picture of w= hat is going on. There are two major areas where this could be a problem. T= he first is handling a request with a HTL of 1. A malicious node could gene= rate lots of requests for things with a very small HTL in order to try to p= robe the data stores of its peers. Freenet handles this by forwarding reque= sts with an HTL of one 30% of the time at random. Also to prevent nodes fro= m knowing exactly how far they are from the originator of the request, abou= t 5% of the time nodes forward requests without decrementing the HTL. This = system insures a degree of safety, but it will likely soon be replaced by s= omething more sophisticated.</p> <p>The second area where conspiring nodes could be a problem is when you ar= e downloading a splitfile. A group of nodes could find the manifest of a sp= litfile they did not like by spidering Freenet. Then they would know what a= ll the CHKs that made it contained. Then they could all try to get connecte= d to the same hosts (this is deliberately made difficult). However it would= be extremely suspicious if a single node made requests for multiple CHKs i= n the same splitfile. This is combatted by using something called "The= anonymity filter". This works by sending a message to a node in your = routing table, and requesting that it send message a to another node of you= r choosing. The message that is sent to the other node is a message request= ing the data that is encrypted using its public key. This way, although the= first node you contacted knows you are the originator of the request, it d= oes not know what was being requested. The second node knows what was being= requested, but not who is making the request. So, if ether one of these no= des is trustworthy, the other cannot get any incriminating information. You= can optionally add intermediate nodes in the chain.</p> </body> </html> --MUnXZt0Uv08c1hBe-- --8R/sMIKPJVD0CT2D Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (OpenBSD) iD8DBQE/MaOwkAkqAYpL9t8RAjRNAJ9foUfJkUARikiKcyGCyJ87nWaiqQCggaoB PNyUoXavbN5L5T1Ov6Z4oLs= =WaIX -----END PGP SIGNATURE----- --8R/sMIKPJVD0CT2D--