Re: interesting kernel discussion

"Also sprach Peter T. Breuer:"
> There's a relevant, even interesting, discussion going on on LKML:
> 
>   Subject: Re: [RFC][PATCH 2/9] deadlock prevention core
> 
>   Peter Zijlstra <[email protected]>
> 
>   Networked Block devices (NBD, iSCSI, AoE) can deadlock in the
>   following manner:
>   deplete normal memory because of memory pressure; deplete reserves by
>   writeout over network (pageout happens under PF_MEMALLOC), little to no
>   memory left for receiving those now crucial ACK packets.
>   A few packets could still fit in memory, but are quickly gobbled up by
>   non-crucial sockets and are left waiting on blocked user-space
>   processes. All memory is depleted and progress stalled forever.
> 
>   (This affects swap and shared mmap)

Another interesting LKML discussion shows a separate RAID1 deadlock on
its own:

    Subject:  Mempool_alloc, bio_alloc_bioset deadlocks 

    Pavel Mironchik <[email protected]>

    A few days ago device mapper raid1 deadlock was discovered.
    Adrew Morton made patch for that bug in mm tree:
    http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.18-rc4/2.6.18-rc4-mm1/broken-out/dm-fix-deadlock-under-high-i-o-load-in-raid1-setup.patch

    However I found that problem is more serious and depends on mempool.
    I reproduced that very same situation on 2.6.17 with device-mapper
    linear target.
    Here my steps:
     - I used 2.6.17 kernel for xscale (arm), boot into initrd image
       (initrd is SYSTEM_BOOTING state I assume) !!!!
     - with the help of evms.sf.net I made md raid1 with device mapper
       volume on top of it.
     -  create xfs volume ; mkfs.xfs /dev/evms/vol ; mount
       /dev/evms/vol /mnt
     - run: cat /dev/zero > /mnt/test &
     - after some period cat, pdflush and raid1d threads went into
       deadlock state, ...

His patch is to have memory allocation return a fail if nothing found
after 5s. I'm unsure of the effect. Seems harmless. Probably can only
help.


   probably, I suppose this could be another solution for  raid1
   deadlock problem described here:
   http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.18-rc4/2.6.18-rc4-mm1/broken-out/dm-fix-deadlock-under-high-i-o-load-in-raid1-setup.patch

Andrew Morton replies:


  dm has a number of problems in this area - the developers are aware of
  them and are working on getting them fixed, but it's apparently fairly
  complex.

He approves the patch, but wants it removed from development kernels so
that the hang is noticed and the underlying causes fixed.

  However I'd want to take it out again in 2.6.19-rc1, because the patch
  will convert a complete hang into
  extreme-suckiness-but-it-appears-to-work.  We'd prefer the hang, so
  the problem gets noticed and fixed for real.

I suspect they will want a 30s-60s hang combined with lots of kernel
messages to remain in the code until what's happening is figured out.

I suppose the problem is due to raid1 not being able to get memory to
duplicate requests with under high load through dm .. curious. I
thought raid1 used to preallocate a pool. Maybe dm introduces another
allocation problem.



  


Peter