Re: ENBD Issues with client crash / weird errors....

"Also sprach Jonathan Carron:"
(something)

I should also remark that it sounds like an issue that was once present
in 2.4 kernels .. namely that filling up memory with requests that are
waiting to go to disk starves tcp of buffers, which means that the
network dies.

This could happen when you wrote fast to an async-mounted filesystem on
top of enbd, because you in fact write only to memory .. many seconds
later the system wakes up and starts trying to flush the data out
across the net - but it can't, because tcp cannot get buffers, them all
being full of data waiting to go to "disk". It's too late - it's in
deadlock.

The cure (apart from "don't do that then"), was in-kernelø to prioritize
tcp usage of ram against all other users of ram.

Maybe that kernel bug is back again.

Anyway, prioritization can't cure all such situations simply because if
there is NO memory left, prioritization does not matter.  I guess
streaming 200GB to a 500MB buffer with an exit hole band-limited to
4MB/s might get you into that kind of situation! No memory and fast.

Change the buffer aging policy on your file system so that buffers go
to disk _immediately_. This used to be done with bdflush, in /proc.

Alternatively, the problem might be at the server end. The situation
can occur there too - if your resource is not a plain partition (and
opened O_DIRECT?).

Anyway, if it is that, it is not happening in enbd - it is happening in
your kernels MM system. One of the simplest cures is NOT to stream
200GB into a 500MB buffer (your ram) with a 4MB/s drainage hole (your
net). 

Another cure is to do that SLOWLY! At 4MB/s. RAID1 takes care of doing
that for you, if you let it! If you write to your raid in degraded mode,
locally, the net is not involved. If you later add the network
component in, raid will update it in the background, throttling the
rate so as not to impact the kernel.

Another technique would be, if you are using cp to do your copying,
change cp to open its targets with O_DIRECT.

That is similar to mounting the fs sync.

I.e. - it is possible that you are experiencing MM issues.


Peter