How Freenet Works.
Tom Kaitchuck <[email protected]> Wed, 6 Aug 2003 19:25:53 -0500
| Newsgroups | gmane.network.freenet.web |
|---|---|
| Message-ID | <[email protected]> |
--Boundary-00=_RyZM/mHrssa8YnP Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Content-Disposition: inline --Boundary-00=_RyZM/mHrssa8YnP Content-Type: text/html; charset="us-ascii"; name="works.html" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline; filename="works.html" <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/htm= l4/strict.dtd"> <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DUTF-8"> <title>How Freenet Works</title> </head> <body> <div align=3D"center"><font size=3D"+3"><b><p>Overview</p></b></font></div> <hr> <p> On the broadest level Freenet works by having each computer that connec= ts to the network allocate a portion of it's hard disk. Then when another c= omputer asks your computer for some data that you have stored, your compute= r will return the file. If your computer does not have the data it will loo= k at all the computers it connects to, and ask the one that it thinks is mo= st likely to have the data for it. Inserting data works the same way. Each = node looks at the data and passes it along to the next computer that is the= best match for the data. Because all the data is broken up into small encr= ypted segments (each is routed separately) all the intermediate computers d= on't know what the data is. The computers on the network decide where the d= ata should be stored biased on it's hash. This means no computer can tell w= hat data it is storing or fetching, nor can anyone tell where a particular = piece of data is located on the network. No computer on the network knows w= here the data originally came from or where it went. This is because if you= request a file and another computer gets the request, it has no way of tel= ling if you are the originator of that request, or if you are just forwardi= ng it from another computer. Then when the data is returned you have no way= of telling if that computer had the data in it's data store, or it got it = from another computer. Another positive side effect is if two people indepe= ndently insert the same data, they collide so the network only needs to sto= re one copy.</p> <p> Of course it's all much more complex than that. However this has a few = implications that should be addressed. First it means that there is no cent= ral server that the network depends on. It also means that every permanent = node on the network contributes to it by sotring data. As a result of this,= anyone can upload content and never have to worry about the bandwidth, as = =46reenet handles both the storage and distribution of content. So even if = you don't particularly need anonymity, you should publish your site on Free= net because it provides free hosting regardless of how big or popular your = site is. So, you never have to resort to putting ads on your page. </p> <p> The way Freenet's architecture is designed it is inherently good at som= e things, and inherently bad at others. For example, because all inserts an= d requests for data go through 5 - 25 computers before they reach their des= tination, Freenet has inherently high latency. This means it will never be = suitable for playing Real Time games or instant messaging. However, because= when you request a file it downloads it in many chunks, each of which will= likely come from a different computer, it can allow very fast downloads.</= p> <p> The reason that it is able to do this, is that because Freenet has a fe= w inherent advantages over most networks. Because the data is inserted, it = will be spread over a large number of hosts, even if it is unpopular. Becau= se each node has a data store, all nodes are contributing to the speed of d= ownloads even if they are not publishing any content. Freenet also requires= nodes to higher up-times than most networks, so data is more likely to be = available. This means that not only can you download from many computers ev= en if the file is unpopular and the total bandwidth is not directly limited= by the number of people downloading it or sharing it. Once Freenet is furt= her optimized it will very likely outpreform the venerable BitTorrent.</p> <p> Freenet is very good at making information available to everyone anonym= ously. However, when it comes to sending information to a specific person, = or relyably archiving unpopular data for a long time, it is not very good. = If your application requires lower latency than Freenet can provide, and do= es not need lots of bandwidth you might look at Freenet's sister project th= e invisible IRC project. www.invisible.net. Although IIRC has a much lower = standard of anonymity, the two complement each other very nicely. Freenet c= an be used for storage or publishing data, and IIRC can be used for small f= ast communications, as well as e-mail.</p> <p> If you are considering uploading your own Freesite, you should look at = some of the guides on Freenet for advise. As good web design is not necessa= rily good Freesite design. </p> <div align=3D"center"><font size=3D"+3"><b><p>How Freenet Works</p></b></fo= nt></div> <hr> <div align=3D"center"><font size=3D"+1"><p><b>How data is located</b> = </p></font></div> <p> To request data a computer sends out a request which contains the hash = of the content that they want. When a node gets a request for data that it = does not have, it forwards the request to the node in it's routing table th= at it thinks will return the data the fastest biased on previous requests w= ith similar hashes. If that node fails or timesout then it forwards the req= uest to the next closest match. Those nodes will inturn forward it to the c= losest node their routing table. Inserts works in a similar way: the insert= requests follow the path closest to their hash until the HTL (hops tell li= ve) expires, then they are stored on that node. This means in the best case= request will travel along the shortest path to the data that each of the i= nvolved nodes know about. Also the worst case (very rare) involves asking a= ll the nodes within the specified range. As time goes on each of the nodes = will learn about more and more of it's peers. So over time the typical case= approaches the best case. </p> <p> This system compairs favorably to other peer-to-peer networks. For exam= ple Freenet's worst case scenario is the same as what Gnutella uses for all= searches. (All the nodes within a given radius will be asked for the data.= ) However in the best case scenario any data can be located in Log base M o= f N where N is the number of nodes in the network and M is the average numb= er of peers each node has. This means that Freenet scales much better than = Gnutella and similar networks, and uses much less bandwidth for overhead. <= /p> <div align=3D"center"><font size=3D"+1"><p><b>How keys work.</b> </p><= /font></div> <p> All content in Freenet is indexed by it's hash. So in order to find the= data you are looking for you need to know it's hash. This means that there= is no simple keyword search. So to find information, you use your webbrous= er. </p> <p>When you load Freenet you will see a main page that has hyperlinks to so= me of the more popular index sites in Freenet. When you click a link on one= of these sites you will see a URI. This is what represents the key you wan= t to fetch. There are three main types of keys. </p> <p>The most basic is a CHK it looks something like this:</p> <p>CHK@hdXaxkwZ9rA8-SidT0AN-bniQlgPAwI,XdCDmBuGsd-ulqbLnZ8v~w</p> <p>A CHK is a 160 bit key. It is represented in base 64 using a-z, A-Z, 0-9= , - and ~. It is divided into two parts separated by a comma. The first par= t is the hash of the encrypted file. It is what the file will be indexed un= der. The second part is the key it was encrypted with (This comes from the = hash of the original file). The second part of the key is not revealed to t= he other nodes when a request is made. </p> <p>A CHK can store one chunk of data (That can be any size up to 1MB.) If y= ou want to store more than one MB, the data gets split up. You will use a k= ey just like you would if it was a single chuck. However when you download = that first chunk, rather than having data, you get what is called a manifes= t. It will contain the keys for all the other chunks of data in the file. S= o CHKs can allow you to store any amount of data, and because they are inse= rted under a key that is derived form their content, we can safely assume t= hat if two chunks of data have the same key, that they are the same data. A= s a result if two people insert the exact same data, it only needs to be st= ored on the network once.</p> <p>If you want to have a site that can be updated, or verified as to it's o= rigin, you could use a SSK. An SSK look like this:</p> <p>SSK@rBjVda8pC-Kq04jUurIAb8IzAGcPAgM/TFE//thelist.html</p> <p>Like the CHK, it consists of two parts. The first part is a public encry= ption key, this allows you to decrypt the content stored in the SSK. The se= cond part is a plain text description (this may contain slashes to emulate = a directory structure). A single SSK can contain many files, and even many = sights. They are distinguished by the plaintext portion. The data is indexe= d under the the hash of the the combined string of the encryption key, and = the plain text description. </p> <p>To have an updateable sight there are two options. The first is to have = an edition site. This is usually done by posting a sight in SSK@_key_PAgM/_= Name_/1/ and then in the HTML for the sight, link to SSK@_key_PAgM/_Name_/2= /. This way, when you publish a new edition, you can insert it in the new l= ocation, and people will be able to follow the link to get the new version.= </p> <p>A more automated way of doing this is to have a Date Biased Redirect (DB= R). This consists of inserting a piece of metadata at SSK@_key_PAgM/_Name_/= that tells Freenet clients how often the sight is updated and where it wil= l be located. This is usually done by inserting at regular intervals at SSK= @_key_PAgM/_Date_-_Name_/.</p> <p>There is a third, optional component to an SSK. That is a manifest. Thi= s is represented in the key by the presence of a "//". This place= s a file at the location specified by the URI up until that point. That fil= e is called a manifest. It contains a list of files and their respective ke= ys (Usually a CHK). Having a manifest means that all the data for the sight= can be stored in CHKs, so when the sight has to be updated, because of a n= ew edition, or a date rollover, one only needs to insert the new manifest, = and the files that changed.</p> <div align=3D"center"><font size=3D"+1"><p><b>How attacks are prevented</b>= </p></font></div> <p>Their are two main attacks that could be used to try to compromise you a= nonymity on Freenet. The first involves a local eavesdropper listening in o= n all your communications over Freenet. Ordinary encryption is not sufficie= nt for protection from this sort of attack. The reason for this is because = if you want to talk to a new person, they have to know how to decrypt the m= essage you have sent them, so if someone is monitoring all communication be= tween the two of you, there is no way to tell them without the eavesdropper= from hearing too. However there is a solution to this problem. It is calle= d asymmetric cryptography. It works as follows; Your computer uses a one-wa= y mathematical function to generate two numbers called keys. When a message= is encrypted using one of the keys, it can be decrypted with the other and= vise-versa. Given one of these keys there is no way to find out the other = one, short of trying all possible combinations. One of these keys is design= ated the public key, and the other, the private key. When you find out abou= t a new node, IE: through your dereferences file, or through another node, = you receive it's public key. Then you can send it a a message encrypted wit= h it's public key. Because it is the only one that knows it's private key, = nobody but it can read your message. Then after it gets your public key, it= can communicate with you, and as long as nobody knows your private key, no= body can tell what it sent you.</p> <p>The problem with this is that Asymmetric cryptography is very slow. (It = requires a LOT of CPU power.) However Freenet works around this as follows:= When you contact a new node, you send it a "Hello" message that = contains your public key and is encrypted with their public key. Then that = node can reply with a "Hello" message that contains an ordinary e= ncryption key that is encrypted using your public key. This means, because = only you hold your private key, you are the only one who can read the messa= ge. So now you can communicate safely encoding messages both ways using ord= inary encryption encoded with the session key. This effectively thwarts any= attempts to monitor your traffic locally and means that the CPU intensive = asymmetric encryption only needs to be used for the first message you send = (or whenever you want to send a new session key). Even if the node you cont= acted revealed the session key it used to a local attacker, it would not re= veal anything other than what that node already knew.</p> <p>The second type of attack consists of having a large group of nodes that= are connected to you pool their knowledge to try to develop a picture of w= hat is going on. There are two major arias where this could be a problem. T= he first is handling a request with a HTL of 1. A malicious node could gene= rate lots of requests for things with a very small HTL in order to try to p= robe the data stores of it's peers. Freenet handles this by forwarding requ= ests with an HTL of one 30% of the time at random. Also to prevent nodes fo= rm knowing exactly how far they are from the originator of the request, abo= ut 5% of the time nodes forward requests without decrementing the HTL. This= system insures a degree of safety, but it will likely soon be replaced by = something more sophisticated.</p> <p>The second aria where conspiring nodes could be a problem is when you ar= e downloading a splitfile. A group of nodes could find the manifest of a sp= litfile they did not like by spidering Freenet. Then they would know all th= e CHKs that made it contained. Then if they could all try to get connected = to the same hosts (this is deliberately made difficult). However it would b= e extremely suspicious if a single node made requests for multiple CHKs in = the same splitfile. This is combatted by using something called "The a= nonymity filter". This works by sending a message to a node in your ro= uting table, and requesting that it send message a to another node of your = choosing. The message that is sent to the other node is a message requestin= g the data that is encrypted using it's public key. This way, although the = first node you contacted knows you are the originator of the request, it do= es not know what was being requested. The second node, knows what was being= requested, but not who is making the request. So, if ether one of these no= des is trustworthy, the other cannot get any incriminating information. You= can optionally add intermediate nodes in the chain.</p> </body> </html> --Boundary-00=_RyZM/mHrssa8YnP--