Re: race condition?

> Tab, et al,
> >   The sync stuff is definitely much too complex, not too mention the
> > performance would be terrible. I was trying to see how to do it with
> > just one port. If we used two per process that whole issue goes
away. 
> > Does kcomd buy anything other than fewer ports that can't be done
with a
> > restructuring of the code?  It's another process/daemon running and
if
> > it dies what happens to the migrated processes.   Granted, any
kernel
> > daemon that dies can screw things.   Scalability?
> 
> why would a kernel daemon dies ?

hardware problem; hence the comment on other daemons.  If the hardware
has problems, there are bigger issues than kcomd acting weird or dying.
:)

> 
> For the scability issue, that could be an issue to open only 2 ports
for
> all processes.
> 
> not sure though, since you lose lots of time, in the many port case,
> with lots of TCP handshake. And i would value stability and
correctness
> over performance.

Absolutely.

> 
> > Have two ports:  1 port for "node A -> node B" and 1 port for "node
B ->
> > node A". Replies and acks are sent back over same port.
> > 
> > A om_comm_send_with_ack(stuff) and a om_comm_send_with_reply(stuff)
> > which would talk to the other node directly. no kcomd, but the
process
> > would still sleep while it waits.
> 
> yeah but it would be waiting for *data* .. instead of *command*.
> data that could not be valid, or partial, ...

that's why I asked about types sent between the nodes.  If the home node
only ever sends either signals or migrate commands,  those can share a
struct with a bit designation, and there's no data vs command problem
since it's not on the same port as the remote node waiting on a syscall
return or page.

> 
> >  Each process having 2 ports open per process isn't all that many
ports
> > is it?  Even if there are 200 processes, that's only 400 ports. 
> > iptables handles WAY more than that with no problems.
> 
> even if that's unlikely that 200 processes from other nodes appear on
a
> single node, having 400 ports open for just housekeeping and stuff, is
> just gross. The kernel can certainly handle the case, that's not the
> question.

yea, For the per port vs kcomd, I'm just trying to get on the same page.
 I was thinking about extremes and corner/boundary cases.  I figure you
probably already have, but I'm new to this and didn't see much in the
archives to help me out.   

> 
> Having each process to handle all what could happen over a socket is
not
> very good. Instead you could have this kcomd that handles everything:
> communication dying, ports closed need to reopen, partial packets
> sent/received, queue of packet. and then hands a valid packet to the
> process.

> 
> (Now I'm not talking about duplicate code, since that's could all be
> factorize in functions to handle that, but each process doing the same
> logic.)
> 
> Plus having application to handle the network, mean you cannot do
> everything. like sending data through network in atomic code
> (recent bug: copy_from_user in atomic).
> now they just go to sleep, which give all sort of other problems ;)

> 
> > If all the calling functions and receive functions were merged into
one
> > each, the process would still be put to sleep while it waits.
> 
> it's *already* put to sleep by separate sending and receving function.
> now the thing is, it's waiting for data, not a command.
> if the remote send one byte, and then what's left to send, the deputy
> will read one byte, and die because.
> 
> What i'm proposing is having a separate daemon that wait for data, and
> construct all packets, and then wake up process when there's command
to
> do. The process would not have to mess with any file descriptor,
> network, ...
> 
> Plus it would be lots easier to do migration and fork/clone since you
> don't need to open ports at all during this phase (port is already
open
> by kcomd, and you can expect kcomd to open port if needed)

This makes sense.
Plus if it were ever desired, adding things to a central place would be
easier.. Things like:
1)  encrypting the traffic between nodes.
2)  d-bus messages on migration events


Does having kcomd help in the smp case?
Does the preempt case matter? om is ran on servers where preempty is
usually disabled. Probably a good idea to make it work anyway. ??

> 
> > How does 2.4 do things?
> 
> 2.4 does bad things. there's LOTS of assumption over the code, and any
> failure destroy processes in the best case, or just crash the node in
the
> worst case.
> And let's not forget that 2.6 is way different than 2.4, consider
> PREEMPT, softirq, ... so looking what 2.4 does is not a good idea.
> 
> 2.4 is skanky, and bad code. better not look what it's doing.

...skanky.  that's awesome! :)


I'm quite happy to go the kcomd way. It makes sense. I'm just trying to
play devil's advocate, (poorly :)). 

(from your email:)
  send_with_ack(stuff, node Y) ->
    kcomd(1) read the request, and put the process to sleep. when
    it's time to send the packet kcom received from the process(x),
    send it over the network to kcomd(2), and kcomd(2) send a ack if
    received. wake up process(x) with the ack response from kcomd(2)
    (ERR or SUCCESS).

  send_with_reply(stuff, &stuff2, node Y) ->
   kcomd(1) read the request, put the process to sleep. send the
    packet to kcomd(2). when kcomd(2) is going to reply what
    process(x) need, wake up process(x) and continue.


I need some help on kernel internals.  Specifically, what's the best way
for send_with_*() to interact with kcomd?  Do we set up a queue/list and
have send_with_ack()/reply() add to that queue and then kcomd processes
the queue?  Signals? (are there signals from kernelthread to
kernelthread?) Other?


Matt



> 
> > This might be a silly question but is it possible to have a daemon
> > 'kmigd' that takes care of all the housekeeping.  No om_pre_usermode
> > stuff at all.  The daemon handles signals and sets process flags
> > accordingly, rather than the process kernel stub doing it. Talks
with
> > other nodes and responds accordingly. checks the om_struct and puts
> > process to sleep and migrates it then wakes it up on remote node. 
That
> > way it really is just a small kernel module and patches are
unobtrusive?
> >  The syscall trap in entry.S would have to stay as I think the
syscall
> > table isn't trappable like it was in 2.4.  Is that correct?
> 
> The process deputy stub cannot go away. It keeps lots of needed stuff
> like file descriptors opened, signals mask, ...
> 
> Lots of code in the kernel expect that "current" task is doing stuff.
> so you really need to be in this process context to execute some
> function (and signal subsystem is one of them here), that why there's
a
> process stub staying on the deputy.
> 




-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642