Re: ENBD Issues with client crash / weird errors....

On 8/22/05, Peter T. Breuer <[email protected]> wrote:
> "Also sprach Jonathan Carron:"
> (something)
> 
> I should also remark that it sounds like an issue that was once present
> in 2.4 kernels .. namely that filling up memory with requests that are
> waiting to go to disk starves tcp of buffers, which means that the
> network dies.
> 

perhaps ? *but* both boxes are identical distributions, with the
exception of specific kernel options for the processor in each. Same
cdrom same packages, same kernel source, is what I mean.

> This could happen when you wrote fast to an async-mounted filesystem on
> top of enbd, because you in fact write only to memory .. many seconds
> later the system wakes up and starts trying to flush the data out
> across the net - but it can't, because tcp cannot get buffers, them all
> being full of data waiting to go to "disk". It's too late - it's in
> deadlock.

neither disk led is on solid? Doesn't seem too badly under stress,
either box that is.

> 
> The cure (apart from "don't do that then"), was in-kernelø to prioritize
> tcp usage of ram against all other users of ram.
> 
> Maybe that kernel bug is back again.

how or what can I do to help verify ?

> 
> Anyway, prioritization can't cure all such situations simply because if
> there is NO memory left, prioritization does not matter.  I guess
> streaming 200GB to a 500MB buffer with an exit hole band-limited to
> 4MB/s might get you into that kind of situation! No memory and fast.
> 



> Change the buffer aging policy on your file system so that buffers go
> to disk _immediately_. This used to be done with bdflush, in /proc.

ok I will google and investigate this


> 
> Alternatively, the problem might be at the server end. The situation
> can occur there too - if your resource is not a plain partition (and
> opened O_DIRECT?).

well the resource is /dev/hda1 , a WD 160 gb hdd, beyond that I'm not
sure what your asking.



> 
> Anyway, if it is that, it is not happening in enbd - it is happening in
> your kernels MM system. One of the simplest cures is NOT to stream
> 200GB into a 500MB buffer (your ram) with a 4MB/s drainage hole (your
> net).

so back to the bdflush and write immdtly. to disk.... no problem ill
look into that for sure, and let you know.

> 
> Another cure is to do that SLOWLY! At 4MB/s. RAID1 takes care of doing
> that for you, if you let it! If you write to your raid in degraded mode,
> locally, the net is not involved. If you later add the network
> component in, raid will update it in the background, throttling the
> rate so as not to impact the kernel.

if I begin to have issues with raid ill flip that on , but for now I'd
like to see it work like this first without that.

> 
> Another technique would be, if you are using cp to do your copying,
> change cp to open its targets with O_DIRECT.

I'll have to google again and see what this is about.

> 
> That is similar to mounting the fs sync.
> 
> I.e. - it is possible that you are experiencing MM issues.

could be... 
> 
> 

Thanks!
Jonathan


> Peter
> 
> _______________________________________________
> ENBD mailing list
> [email protected]
> http://lists.community.tummy.com/mailman/listinfo/enbd
>