Re: init scripts problems ?

"Also sprach denis bonnenfant:"
[Charset ISO-8859-1 unsupported, filtering to ASCII...]
> Peter T. Breuer wrote:
> > "Also sprach denis bonnenfant:"
> 
> Good news, after many cycles, i was able to strace a blocked client :

OK! You did this by stracing (-p) clients while you ran your test? (I
think your test is something like "kill clients while under raid; start
new client", but I am not sure).

>   4092 select               S enbd-client se3-croise 12355 -i A-2 -n 2 -m -b 512 /dev/nda
>   4093 -                    D  \_ [enbd-client]
>   4102 sync_page            D  \_ enbd-client se3-croise 12355 -i A-2 -n 2 -m -b 512 /dev/nda
> 

OK - one parent, two slaves.

One point - when I test, I always run with -n 1 in order to
uncomplicate the situation and in order to make deadlocks more obvious.

> for 4102 :

The slave ...

> stat64("/dev/nda", {st_mode=S_IFBLK|0644, st_rdev=makedev(43, 0), ...}) = 0
> stat64("/dev/nda2", {st_mode=S_IFBLK|0644, st_rdev=makedev(43, 2), ...}) = 0
> open("/dev/nda2", O_RDWR|O_LARGEFILE

It is doing an open? OK - that MUST be that it is stuck doing a down()
on the bdev->bd_sem, which has already been taken by somebdy else.

> for 4093 :

The other slave ...

> write(2, "enbd-client  4093: # 156 unplug:"..., 107) = 107

Oh! Well, this is trying to die! Why? What is your test?

> ioctl(4, NBD_CLEAR_SOCK, 0xffd)         = -1 EINVAL (Invalid argument)

It did a clr sock and got INVAL?? Whaaaaaat?  I wonder what fd 4 was?
It must be the slave minor, nda2, no? The klog might say. It might say

   CLEAR_SOCK called on full device nda arg %lx

in which case all is OK. If it doesn't say that, I am puzzled. There is
nothing else it could be!  But why would the client run that ioctl on
the full device? Can you show the earlier parts of its trace?


> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7efd000
> read(5, "4092 \n", 4096)                = 6
> read(5, "", 4096)                       = 0
> close(5)                                = 0
> munmap(0xb7efd000, 4096)                = 0
> getpid()                                = 4093
> exit_group(0)                           = ?

You are saying it is not dead? It didn't die? It just stopped here?

> and 4094 ( the one that die just before 4102 born ) :

Is there a reason why it died?

> munmap(0xb7efd000, 4096)                = 0
> getpid()                                = 4094
> exit_group(0)                           = ?

It seemed to exit happily.

> 
> I hope this helps, for me it's not clear :)

Well, it's not conclusive. See comments above for what I want to know.

I think that both clients have been killed by signals, and the master
has started new clients to replace them.  But one slave is still trying
to die while one is trying to be born. The one that is trying to die
is the interesting one - WHY did its clr_scok call fail? It looks like
it is stuck somewhere in libc! It should call _exit() to die and it
hasn't!

Can you compare with a "normal" trace to see what it should have done
next?

Given that one client is stuck trying to die, it is natural that the
other is stuck trying to open, since the one dying has taken the 
device open/close semaphore. It is possible that some kind of flush is
being done .. it might be visible in nbdinfo, in the status line - the
counts of requests in that top line would be useful.

Peter