Re: How to automatically add a nbd to a raid1?

Hi .. from a hotel in freezing-cold almeria.

"Also sprach Michael Rendell:"
>   Background info: both machines are P4's at 2.4GHz with 512M memory
> running 2.6.18-92.1.22.el5; mkfs is using 4k blocks;  the second mkfs

OK .. I think there was a MD bug with 512MB ram some time back.
Probably not significant, but you might want to grep the archives for
it.

> that was mentioned in the last email finished after about 68 minutes
> with no errors reported in /proc/nbdinfo.

68 mins!

>   Had a look at the strace output - seems to be using cycles in
> the semaphore code (in sem_wait_timeout()):

Semaphore?  How come it's using real semaphores ..  surely I implemented
my own locking?  I did do something _like_ semaphores in shared memory,
but I wouldn't have thought it was visible as system calls in a trace.

>     recv(9, 
> "%`\225\23\0\0\0\1\267\363\320@\0\0\0\0\0\233P\0\0\0\20\0\0\0\0\0\0\0\0\
>     16"..., 64, 0) = 64
>     clock_gettime(CLOCK_MONOTONIC, {1277506, 711333904}) = 0
>     rt_sigprocmask(SIG_BLOCK, [ALRM], NULL, 8) = 0
>     setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={29, 999356}}, NULL) 
> = 0
>     rt_sigaction(SIGALRM, {0x8058540, [ALRM], SA_RESTART}, {0x8058540, [ALRM], 
> SA_RE
>     START}, 8) = 0
>     rt_sigprocmask(SIG_UNBLOCK, [ALRM], NULL, 8) = 0
>     clock_gettime(CLOCK_MONOTONIC, {1277506, 711604626}) = 0
>     clock_gettime(CLOCK_MONOTONIC, {1277506, 711650698}) = 0
>     clock_gettime(CLOCK_MONOTONIC, {1277506, 711696028}) = 0
>     select(8, [7], NULL, NULL, {0, 10000})  = 1 (in [7], left {0, 10000})
>     clock_gettime(CLOCK_MONOTONIC, {1277506, 711805069}) = 0
>     select(8, [7], NULL, NULL, {0, 10000})  = 1 (in [7], left {0, 10000})
>     clock_gettime(CLOCK_MONOTONIC, {1277506, 711913044}) = 0
>     select(8, [7], NULL, NULL, {0, 10000})  = 1 (in [7], left {0, 10000})
>     clock_gettime(CLOCK_MONOTONIC, {1277506, 712021065}) = 0
>     select(8, [7], NULL, NULL, {0, 10000})  = 1 (in [7], left {0, 10000})
>     clock_gettime(CLOCK_MONOTONIC, {1277506, 712136156}) = 0
>     [this goes on for a while]

Ahhhhhhh. Select followed by time! Look, select is returning! It is
supposed to be watching fd 7 ('8' is ok .. it's supposed to be one more
than the max fd being watched). And that's the read set of fds. The
write and exception set are null. And the timeout is set for 10000us,
or 10ms (timeval is the struct, and it has tv_sec and tv_usec)... but
the select is returning immediately! That means the pipe is ready to be
read. Yet it's not! Because nobody does read it - there would be a read in
the strace if it happened.


> Had a look at the instructions generated for sem_try():
>     (gdb) x/i sem_try
>     0x8066a60 <sem_try>:    push   %ebp
>     0x8066a61 <sem_try+1>:  mov    %esp,%ebp
>     0x8066a63 <sem_try+3>:  push   %ebx
>     0x8066a64 <sem_try+4>:  sub    $0x14,%esp
>     0x8066a67 <sem_try+7>:  mov    0x8(%ebp),%ebx
>     0x8066a6a <sem_try+10>: mov    0xc(%ebx),%eax           ***

uhh, move (god, I wish I understood x86 assembler .. how can it do an
indirect or indexed MOVE!) from eax to contents of ebx + 12, or
whichever way move goes. Perhaps it's TO eax. That would make sense.

>     0x8066a6d <sem_try+13>: sub    $0x1,%eax                ***

Wel, that's a direct decrement of whatever's now in eax.

>     0x8066a70 <sem_try+16>: mov    %eax,0xc(%ebx)           ***

and that's a move back again, after decrement.

So that was something like a foo++, where foo was maybe some variable
on the stack, 12 bytes above the current frame bottom.


>     0x8066a73 <sem_try+19>: mov    0xc(%ebx),%eax           ***

Hmm .. going to do a test.

>     0x8066a76 <sem_try+22>: test   %eax,%eax                ***

What?

>     0x8066a78 <sem_try+24>: js     0x8066af4 <sem_try+148>
> 
> lines marked *** can cause problems if a context switch happens then.

Why? Aren't all the registers saved and restored?

It looks a bit of a long shot, but you get results. Would you have got
the same results by compiling sem.c with -O0? That's maybe what I would
do first.

> Made some changes to use the POSIX semaphore code and the CPU usage
> dropped down to the 1 to 2% range and the mkfs runs in 6 seconds
> (vs the 6 or 69 minutes that was previously happening).  Patches attached.
> Should add something to the configure script to check for the POSIX
> routines, but have just added a -D in the Makefile for now.

Well, I'm greatly indebted! Thanks a lot.

> The semaphore patch includes an added comment in nbdinfo.8 about
> merge requests being 2^n-1 (for those who need things spelled
> out :-)).

Ha.

> Also, have attached the complete code for pwprog that includes
> the code for udev online events (also includes a change
> to the repair function so it doesn't try to remove a device if
> the device is not currently part of the array).
> A patch for the rules file is also needed (unfortunately contains
> the path of the pwprog - udev seems to run with no PATH set!):

Great. I'll make sure you get full credit!

Peter