Re: How Freenet Works.
Tom Kaitchuck <[email protected]> Wed, 6 Aug 2003 21:00:06 -0500
| Newsgroups | gmane.network.freenet.web |
|---|---|
| Message-ID | <[email protected]> |
--Boundary-00=_mKbM/AnMT+Addw+ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline On Wednesday 06 August 2003 07:56 pm, Greg Wooledge wrote: > Here's your file back, proofread. I think there are some grave > content errors in it, though, especially the last paragraph. But > I won't attempt to address those. > > I didn't send a "diff", because the diff was larger than the original > file. This is mostly because you have lines that are incredibly > long (no internal newlines within paragraphs). This makes maintenance > of the document by multiple people a bit harder. > > Anyway, this is a pretty good effort despite what I perceive to be > some substantial flaws. OK, Here is a version with line breaks and a more coherent last paragraph. If you think their content errors point them out, as I have not checked this against the code, just what I have read elsewhere. --Boundary-00=_mKbM/AnMT+Addw+ Content-Type: text/html; charset="iso-8859-1"; name="works.html" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="works.html" <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <title>How Freenet Works</title> </head> <body> <div align="center"><font size="+3"><b><p>Overview</p></b></font></div> <hr> <p> On the broadest level Freenet works by having each computer that connects to the network allocate a portion of its hard disk. Then when another computer asks your computer for some data that you have stored, your computer will return the file. If your computer does not have the data it will look at all the computers it connects to, and ask the one that it thinks is most likely to have the data for it. Inserting data works the same way. Each node looks at the data and passes it along to the next computer that is the best match for the data. Because all the data is broken up into small encrypted segments (each is routed separately) all the intermediate computers don't know what the data is. The computers on the network decide where the data should be stored biased on its hash. This means no computer can tell what data it is storing or fetching, nor can anyone tell where a particular piece of data is located on the network. No computer on the network knows where the data originally came from or where it went. This is because if you request a file and another computer gets the request, it has no way of telling if you are the originator of that request, or if you are just forwarding it from another computer. Then when the data is returned you have no way of telling if that computer had the data in its data store, or it got it from another computer. Another positive side effect is if two people independently insert the same data, they collide so the network only needs to store one copy.</p> <p> Of course it's all much more complex than that. However this has a few implications that should be addressed. First it means that there is no central server that the network depends on. It also means that every permanent node on the network contributes to it by sotring data. As a result of this, anyone can upload content and never have to worry about the bandwidth, as Freenet handles both the storage and distribution of content. So even if you don't particularly need anonymity, you should publish your site on Freenet because it provides free hosting regardless of how big or popular your site is. So, you never have to resort to putting ads on your page. </p> <p> The way Freenet's architecture is designed it is inherently good at some things, and inherently bad at others. For example, because all inserts and requests for data go through 5 - 25 computers before they reach their destination, Freenet has inherently high latency. This means it will never be suitable for playing Real Time games or instant messaging. However, because when you request a file it downloads it in many chunks, each of which will likely come from a different computer, it can allow very fast downloads.</p> <p> The reason that it is able to do this, is that because Freenet has a few inherent advantages over most networks. Because the data is inserted, it will be spread over a large number of hosts, even if it is unpopular. Because each node has a data store, all nodes are contributing to the speed of downloads even if they are not publishing any content. Freenet also requires nodes to have higher up-times than most networks, so data is more likely to be available. This means that you can download from many computers even if the file is unpopular, and the total bandwidth is not directly limited by the number of people downloading it or sharing it. Once Freenet is further optimized it will very likely outperform the venerable BitTorrent.</p> <p> Freenet is very good at making information available to everyone anonymously. However, when it comes to sending information to a specific person, or reliably archiving unpopular data for a long time, it is not very good. If your application requires lower latency than Freenet can provide, and does not need lots of bandwidth you might look at Freenet's sister project the <a href="http://www.invisible.net/">invisible IRC project</a> (IIP). Although IIP has a much lower standard of anonymity, the two complement each other very nicely. Freenet can be used for storage or publishing data, and IIP can be used for small fast communications, as well as e-mail.</p> <p> If you are considering uploading your own Freesite, you should look at some of the guides on Freenet for advice, as good web design is not necessarily good Freesite design. </p> <div align="center"><font size="+3"><b><p>How Freenet Works</p></b></font></div> <hr> <div align="center"><font size="+1"><p><b>How data is located</b> </p></font></div> <p> To request data a computer sends out a request which contains the hash of the content that they want. When a node gets a request for data that it does not have, it forwards the request to the node in its routing table that it thinks will return the data the fastest based on previous requests with similar hashes. If that node fails or times out then it forwards the request to the next closest match. Those nodes will in turn forward it to the closest node in their routing table. Inserts works in a similar way: the insert requests follow the path closest to their hash until the HTL (hops to live) expires, then they are stored on that node. This means in the best case the request will travel along the shortest path to the data that each of the involved nodes know about. Also the worst case (very rare) involves asking all the nodes within the specified range. As time goes on each of the nodes will learn about more and more of its peers. So over time the typical case approaches the best case. </p> <p> This system compares favorably to other peer-to-peer networks. For example Freenet's worst case scenario is the same as what Gnutella uses for all searches. (All the nodes within a given radius will be asked for the data.) However in the best case scenario any data can be located in Log base M of N where N is the number of nodes in the network and M is the average number of peers each node has. This means that Freenet scales much better than Gnutella and similar networks, and uses much less bandwidth for overhead. </p> <div align="center"><font size="+1"><p><b>How keys work.</b> </p></font></div> <p> All content in Freenet is indexed by its hash. So in order to find the data you are looking for you need to know its hash. This means that there is no simple keyword search. So to find information, you use your web browser. </p> <p>When you load Freenet you will see a main page that has hyperlinks to some of the more popular index sites in Freenet. When you click a link on one of these sites you will see a URI. This is what represents the key you want to fetch. There are three main types of keys. </p> <p>The most basic is a Content Hash Key (CHK). It looks something like this:</p> <p>CHK@hdXaxkwZ9rA8-SidT0AN-bniQlgPAwI,XdCDmBuGsd-ulqbLnZ8v~w</p> <p>A CHK is a 160 bit key. It is represented in base 64 using a-z, A-Z, 0-9, - and ~. It is divided into two parts separated by a comma. The first part is the hash of the encrypted file. It is what the file will be indexed under. The second part is the key it was encrypted with (This comes from the hash of the original file). The second part of the key is not revealed to the other nodes when a request is made. </p> <p>A CHK can store one chunk of data (that can be any size up to 1MB). If you want to store more than one MB, the data gets split up. You will use a key just like you would if it was a single chunk. However when you download that first chunk, rather than having data, you get what is called a manifest. It will contain the keys for all the other chunks of data in the file. So CHKs can allow you to store any amount of data, and because they are inserted under a key that is derived form their content, we can safely assume that if two chunks of data have the same key, that they are the same data. As a result if two people insert the exact same data, it only needs to be stored on the network once.</p> <p>If you want to have a site that can be updated, or verified as to its origin, you could use a Signed Subspace Key (SSK). An SSK look like this:</p> <p>SSK@rBjVda8pC-Kq04jUurIAb8IzAGcPAgM/TFE//thelist.html</p> <p>Like the CHK, it consists of two parts. The first part is a public encryption key, this allows you to decrypt the content stored in the SSK. The second part is a plain text description (this may contain slashes to emulate a directory structure). A single SSK can contain many files, and even many sites. They are distinguished by the plaintext portion. The data is indexed under the the hash of the combined string of the encryption key, and the plain text description. </p> <p>To have an updateable site there are two options. The first is to have an edition site. This is usually done by posting a site in SSK@_key_PAgM/_Name_/1/ and then in the HTML for the site, link to SSK@_key_PAgM/_Name_/2/. This way, when you publish a new edition, you can insert it in the new location, and people will be able to follow the link to get the new version.</p> <p>A more automated way of doing this is to have a Date Based Redirect (DBR). This consists of inserting a piece of metadata at SSK@_key_PAgM/_Name_/ that tells Freenet clients how often the site is updated and where it will be located. This is usually done by inserting at regular intervals at SSK@_key_PAgM/_Date_-_Name_/.</p> <p>There is a third, optional component to an SSK. That is a manifest. This is represented in the key by the presence of a "//". This places a file at the location specified by the URI up until that point. That file is called a manifest. It contains a list of files and their respective keys (usually a CHK). Having a manifest means that all the data for the site can be stored in CHKs, so when the site has to be updated, because of a new edition, or a date rollover, one only needs to insert the new manifest, and the files that changed.</p> <div align="center"><font size="+1"><p><b>How attacks are prevented</b></p></font></div> <p>There are two main attacks that could be used to try to compromise your anonymity on Freenet. The first involves a local eavesdropper listening in on all your communications over Freenet. Ordinary encryption is not sufficient for protection from this sort of attack. The reason for this is because if you want to talk to a new person, they have to know how to decrypt the message you have sent them, so if someone is monitoring all communication between the two of you, there is no way to tell them without the eavesdropper hearing too. However there is a solution to this problem. It is called asymmetric cryptography. It works as follows: your computer uses a one-way mathematical function to generate two numbers called keys. When a message is encrypted using one of the keys, it can be decrypted with the other and vice versa. Given one of these keys there is no way to find out the other one, short of trying all possible combinations. One of these keys is designated the public key, and the other, the private key. When you find out about a new node, (e.g. through your node references file, or through another node), you receive its public key. Then you can send it a a message encrypted with its public key. Because it is the only one that knows its private key, nobody but it can read your message. Then after it gets your public key, it can communicate with you, and as long as nobody knows your private key, nobody can tell what it sent you.</p> <p>The problem with this is that asymmetric cryptography is very slow. (It requires a LOT of CPU power.) However Freenet works around this as follows: when you contact a new node, you send it a "Hello" message that contains your public key and is encrypted with their public key. Then that node can reply with a "Hello" message that contains an ordinary encryption key that is encrypted using your public key. This means, because only you hold your private key, you are the only one who can read the message. So now you can communicate safely encoding messages both ways using ordinary encryption encoded with the session key. This effectively thwarts any attempts to monitor your traffic locally and means that the CPU intensive asymmetric encryption only needs to be used for the first message you send (or whenever you want to send a new session key). Even if the node you contacted revealed the session key it used to a local attacker, it would not reveal anything other than what that node already knew.</p> <p>The second type of attack consists of having a large group of nodes that are connected to you pool their knowledge to try to develop a picture of what is going on. There are two major areas where this could be a problem. The first is handling a request with a HTL of 1. A malicious node could generate lots of requests for things with a very small HTL in order to try to probe the data stores of its peers. Freenet handles this by forwarding requests with an HTL of one 30% of the time at random. Also to prevent nodes from knowing exactly how far they are from the originator of the request, about 5% of the time nodes forward requests without decrementing the HTL. This system insures a degree of safety, but it will likely soon be replaced by something more sophisticated.</p> <p>The second area where conspiring nodes could be a problem is when you are downloading a splitfile. A group of nodes could find the manifest of a splitfile they did not like by spidering Freenet. Then they would know the keys for all the data chunks. If they were monitoring network traffic, it would look suspicious if a single node made requests for multiple CHKs in the same splitfile. This is prevented by using something called "The anonymity filter". It works by sending a message to a node, and requesting that it send message a to another node of your choosing. The message that is sent to the other node is a request for the data that is encrypted using its public key. This way, the first node knows the originator of the request but it does not know what was being requested. The second node knows what was being requested, but not who is making the request. So, if ether one of these nodes is trustworthy, you are safe. You can optionally add intermediate nodes in the chain.</p> </body> </html> --Boundary-00=_mKbM/AnMT+Addw+--