Re: Re datastore simulations

On Wednesday 30 April 2008 13:19, Michael Rogers wrote:
> On Apr 30 2008, Matthew Toseland wrote:
> > Keys to block number. Block numbers to keys is handled by the on disk 
> > structure. So we can actually pick a random block number to dump - but at 
> > the cost of having to keep a key index.
> 
> Cool, I see what you mean now - I'll simulate that too.
> 
> > I'm surprised that hashing works so well, it has some big disadvantages 
> > e.g. once the datastore is say half full, half of all new incoming keys 
> > will overwrite old data rather than being added to the end. So we end up 
> > storing less data: it takes a much longer time for the datastore to fill 
> > up.
> 
> Hmm, good point. On the other hand filling the store (or 99% filling it) 
> would typically only take a few days, so maybe it's more important to 
> optimise the steady state behaviour than the startup behaviour?

Depends on how big it is.
> 
> > What is the approximate ratio of store filling rates for the same size 
> > store on LRU versus on a direct hashing implementation? Can you simulate 
> > this?
> 
> So far I've been allowing the simulations to reach a steady state before 
> making any measurements, but it shouldn't be a problem to simulate it.

Ok.
> 
> > IMHO most of it will be filesharing, just as a massive chunk of the total 
> > internet bandwidth is filesharing.
> 
> OK, I'll simulate filesharing two popularity distributions, uniform and 
> Zipf. Each file will contain a lognormally distributed number of blocks, 
> and the downloader will randomly choose 2/3 of them to request. I won't 
> bother with splitfile healing, inserts, churn, congestion, swapping, phase 
> of the moon, etc.
> 
> > SSK polling for messages obviously 
> > will also be huge, right now we have 2.5 SSKs for every CHK (but SSKs are 
> > ~ 10x than CHKs). That should reduce a bit in future with some new 
> > measures such as RecentlyFailed ... but it will increase as FMS is more 
> > widely adopted... So no idea really... I do know that if we spend all our 
> > bandwidth on SSK polling, filesharing will not work well. :| Also, SSKs 
> > are kept in a separate store from CHKs, this is not likely to change.
> 
> I'll stick to simulating CHKs for the moment - RecentlyFailed and ULPRs 
> will affect the way SSKs are cached, but I don't have time to dig into the 
> code to find out how they work (and into Frost and FMS to find out what 
> kind of traffic patterns they produce).

Sensible imho.
> 
> Cheers,
> Michael

_______________________________________________
Tech mailing list
[email protected]
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/tech

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQBIGIYYA9rUluQ9pFARArG0AJ970A7xDZX2PJgRkPKkh49gaOUWnwCgjYhM
s4Hd0vQu0D3lsmKO64GkBbE=
=kdo/
-----END PGP SIGNATURE-----