Re: [FMS] Freenet development...

--===============0569595616==
Content-Type: multipart/signed;
  boundary="nextPart3754812.uRE2kREhEC";
  protocol="application/pgp-signature";
  micalg=pgp-sha1
Content-Transfer-Encoding: 7bit

--nextPart3754812.uRE2kREhEC
Content-Type: text/plain;
  charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

On Friday 24 September 2010 06:10:47 3BUIb3S50i wrote:
> oo@lkXpu0~CDV6dh0Idyw4MBwkUSgn~h~Bs3qqVXYOXSaY wrote :
> > episodique@mTNVoEldfH6zD61fK1BR94hiKBksTf2XSmVQFTS16Lc wrote:
> >
> >> episodique@mTNVoEldfH6zD61fK1BR94hiKBksTf2XSmVQFTS16Lc wrote :
> >>> oo@lkXpu0~CDV6dh0Idyw4MBwkUSgn~h~Bs3qqVXYOXSaY wrote :
> >>>> is dead, right?
> >>>
> >>> last commit to fred-staging: Fri Sep 17 (6 days ago, that is)
> >>> last commit from toad: Sat Sep 11 (12 days ago)
> >>>
> >>> Let's hope no health issue is involved in here.
> >>
> >> that was on master branch, toad is currently working on
> 'new-load-management'
> >> branch (multiple commits today)
> >
> > It will not work.
> >
> > Just like cool down queue.
> > Rewritten but performs as bad as before.
> >
> > Bad backed off rate, bad download rate, bad local download rate, turtles
> > everywhere, bad ping time, bad success rates by HTL, bad pReject for ev=
ery
> > peer, bad payload output, stalled inserts, bad queue management, intern=
al
> > errors for downloads, hash mismatches for downloads,...
> > did I miss any bug that has not been solved for months now?
> >
> > With almost every build situation got worse.
> > 30 broken builds since 1247, absolutely no improvement.
> >
> > Providing log entries, in vaine.
> > Inserting test files, in vaine.
> >
> > Toad doesn't use freenet, he likes to write new code but he doesn't
> > investigate and fix bugs.
> > He's on the run.
> >
> > Development is dead.
>=20
Lets have 0.4 back then. Complete with DataStoreBug, non-working routing, a=
nd all the other fun.

Seriously, I do care about Freenet. And I've had a pretty difficult year on=
e way or another.

And most of the problems you mention are not trivial bugs to be fixed by ch=
anging one line of code. The only line of code that could have caused the l=
oad management issues was changed back in 1277 and surprise surprise everyt=
hing is still borked.

Hash mismatches for downloads and internal errors for downloads are fixed a=
s far as I know. Or at least, are sufficiently rare as to be of much lower =
priority than the main network level problem - slow downloads and high back=
offs.

If you are still getting high pReject's, please tell me what the CAUSE is. =
"Preemptive reject reasons" on the stats page in advanced mode. There are 3=
 common reasons afaics:

Output bandwidth liability - This should be the most common reason. If so, =
the AIMD's still haven't adjusted, too many requests (or inserts?) are stil=
l being issued.

Thread limit - If this is the most common reason, I need to know. If it onl=
y happens on nodes with very high bandwidth then IMHO it is not a critical =
issue. However if it is happening on many average nodes, we *really* need t=
o know, we can reduce the amount of healing further and/or expedite threadl=
ess transfers.

Ping times - This is generally caused by CPU usage or local network problem=
s. If the CPU usage is caused by your node, it might be due to not having e=
nough memory or similar issues. If the node is using a lot of CPU itself, s=
omething interesting might be happening. Various abuses of the network seem=
 to be related to high ping times too. And QoS can also cause this. In any =
case, THIS IS BAD, it makes it difficult for Freenet to manage load properl=
y, and it's generally not caused directly by Freenet itself.

And yes I have heard you complain over and over about your downstream bandw=
idth being lower than your upstream bandwidth. THIS IS NOT A SIMPLE BUG, an=
y more than the high backoffs are a simple bug. The load management system =
is fundamentally very poor, and is neither designed to nor able to without =
significant changes guarantee a specific proportion of capacity is used for=
 your own requests (or for any other peer; it is fundamentally unfair and t=
herefore vulnerable to at least local DoS). Plus, it is not clear that it w=
ould be good for the network if that proportion was too high anyway. But it=
 will be shared fairly, and will be configurable, when the new code lands.

To specifically address the issue of the cooldown queue, one of the more se=
rious usability problems for Freenet has always been (at least since the db=
4o branch landed, and before that we had other problems i.e. no download re=
suming) that it uses a scary amount of disk I/O, especially if you have a l=
ot of downloads queued. The cooldown queue changes were intended to radical=
ly reduce this, and as far as I can see without a detailed investigation th=
ey did. Provided there is enough memory to cache the whole node.db4o file i=
n RAM (turn on defrag on startup to help with this), we should only have di=
sk writes when there is actual progress being made with downloads, and the =
number of reads should be substantially reduced too. If you are looking at =
stats please try to look at the stats for the number of reads/writes that *=
actually reach the disk* (r/s and w/s in iostat -x <number> e.g.), and igno=
re the number of blocks as reading (or writing) 512 bytes or 32KB is almost=
 identical performance wise. Further improvements are planned but can't be =
implemented until after disk crypto is sorted out (either fixed or removed)=
=2E I have spent considerable time debugging the cooldown queue changes and=
 afaics it is working well enough that the load problems are a higher prior=
ity.

It is not true that I don't investigate bugs. I did spend a significant amo=
unt of time investigating various of the current problems, frequently leadi=
ng down wild goose chases. For example with the client layer bugs before th=
e cooldown queue. But when something is fundamentally broken by design and =
will have to be rewritten before release anyway, and where that rewrite is =
likely to substantially improve things, it simply makes sense to do the rew=
rite. It is true that this can sometimes be disruptive in itself. But on th=
e other hand if everything is busted anyway it may be better to solve not o=
nly the existing bugs that everyone is complaining about but also the long =
term issues (such as heavy disk I/O) at the same time, by doing the long-ov=
erdue rewrite - even if that risks creating more, different bugs.

Now, to talk specifically about the load management code: The current load =
management code is broken by design. It is true that it worked for a while,=
 and it is true that it is not immediately clear why it doesn't seem to be =
working well now. However it is also true that by design it involves a rath=
er large amount of misrouting (including routing to the right node and then=
 rejecting because of load), which IMHO is the real reason why data persist=
ence is so poor (with knock on effects for performance on just about everyt=
hing). It works very badly on darknet, it works very badly with fast peers,=
 it can't cope well with lots of new nodes, and it generally just works ver=
y badly. If it works reasonably (NOT WELL) in practice for a while this is =
the product of luck rather than good design.

It is also fairly vulnerable. There is an outside possibility that the curr=
ent problems are malicious in nature; we've seen discussions on #freenet-ch=
at turned into 0day exploits on Frost, and this may have happened again. Ho=
wever it is more likely IMHO that it is just broken - mainly because any fu=
nded attacker wouldn't want to DoS the entire network, they'd rather survei=
l it, which unfortunately is rather easy with even a relatively large openn=
et. But if we make Freenet work well enough to have a much larger number of=
 users, darknet hopefully will solve that problem.

The discussions we've had over the years have brought me to the point where=
 I am fairly confident I can replace it with something which works much bet=
ter.

The current new-load-management branch is still very much a work in progres=
s. During the process I have discovered a number of bugs which are not spec=
ific to load management but which are closely related to it, and these will=
 be deployed first - e.g. in the block transfer and message queue code. Cur=
rently I am working on code related to timeouts: The new load management re=
lies on being able to know how many of our requests are running on the node=
 we are sending them to, so timing out and hoping that the node doesn't con=
tinue the request and thus multiply load (which of course it will, in many =
cases), as we do now, is a really bad idea.

In summary, if you think I am a negative influence and nobody else is contr=
ibuting significant code, you can either go away or you can contribute your=
self. If you think that my dominant position makes any contributions you ma=
ke liable to be sabotaged, and that all the work I've done in the last 8 ye=
ars, starting with eliminating the DataStoreBug (I bet you don't even remem=
ber that one do you?) and getting 0.5 out the door, has been worthless, fee=
l free to fork some ancient pre-September-2002 version of Freenet. It will =
work spectacularly as long as your network is tiny; such forks always have.=
 Or rewrite it in C, O'Caml, Python, ARM assembler language, brainfart or w=
hatever; see you in 8 years. And feel free to believe I am working for the =
NSA to sabotage Freenet, or for Google to do God knows what (that one is tr=
ue, they've donated the bulk of our total funds over the last two years, bu=
t they've never asked for anything of substance; my guess is they are inter=
ested in 1) being a friend to open source, 2) cheaply evaluating some poten=
tially interesting technology, and 3) hostile environments, some of whom th=
ey've had very obvious political issues with recently). Meanwhile I will co=
ntinue to try to improve Freenet based on what I see (with Ian and the comm=
unity) as being its biggest problems. IMHO, despite the problems we have be=
en having lately, over the last two years in particular we have made major =
progress both on implementation and on understanding the nature of the chal=
lenges facing us, thanks particularly to Evan (who sadly has disappeared) b=
ut also to others. We have solved some of them already (e.g. the client lay=
er changes were planned for years and if the network wasn't so messed up cr=
oss-segment FEC e.g. ought to have a big impact) and have a good chance of =
making major progress in the relatively near future IMHO.

--nextPart3754812.uRE2kREhEC
Content-Type: application/pgp-signature; name=signature.asc 
Content-Description: This is a digitally signed message part.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEABEIAAYFAkyc040ACgkQYUNbc3WUHYgcgACdHLqWMNMxRTIllqKpkn8nEzqU
B+EAniQVY+fQXyA44wD80TNdJwVq/e3o
=HdEg
-----END PGP SIGNATURE-----

--nextPart3754812.uRE2kREhEC--

--===============0569595616==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
chat mailing list
[email protected]
Archived: http://news.gmane.org/gmane.network.freenet.general
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/chat
Or mailto:[email protected]?subject=unsubscribe
--===============0569595616==--