Re: Distributed file system using routing inspired by Freenet

2008/4/8 Matthew Toseland <toad-EI5O+8PHWbJeeLb3ft/[email protected]>:
> On Tuesday 08 April 2008 12:36, Matthew Toseland wrote:
>  > On Tuesday 08 April 2008 00:36, Ian Clarke wrote:
>  > > http://video.google.com/videoplay?docid=-2372664863607209585
>  > >
>  > > He mentions Freenet's use of this technique about 10-15 minutes in,
>  > > they also use erasure codes, so it seems they are using a few
>  > > techniques that we also use (unclear about whether we were the direct
>  > > source of this inspiration).
>  >
>  > They use 500% redundancy in their RS codes. Right now we use 150% (including
>  > the original 100%). Maybe we should increase this? A slight increase to say
>  > 200% or 250% may give significantly better performance, despite the
>  increased
>  > overhead...
>
>  In fact, I think I can justify a figure of 200% (the original plus 100%, so
>  128 -> 255 blocks to fit within the 8 bit fast encoding limit). On average,
>  in the long term, a block will be stored on 3 nodes. Obviously a lot of
>  popular data will be stored on more nodes than 3, but in terms of the
>  datastore, this is the approximate figure. On an average node with a 1GB
>  datastore, the 512MB cache has a lifetime of less than a day; stuff lasts a
>  lot longer in the store, and on average data is stored on 3 nodes (by
>  design).

I think the downloader would "heal" a broken file by re-inserting the
missing FEC blocks, right?

If that is the case, I think we can use 300% (or higher) redundancy,
but only insert a random portion of them. When a downloader download
this file, he insert (some other) random blocks of FEC for this file.
Under this scheme, the inserter don't have to pay for a high bandwidth
overhead cost, while increasing the redundancy.

> [snip]

Regards,
Daniel Cheng