Re: race condition?

Tab, et al,
  The sync stuff is definitely much too complex, not too mention the
performance would be terrible. I was trying to see how to do it with
just one port. If we used two per process that whole issue goes away. 
Does kcomd buy anything other than fewer ports that can't be done with a
restructuring of the code?  It's another process/daemon running and if
it dies what happens to the migrated processes.   Granted, any kernel
daemon that dies can screw things.   Scalability?

Have two ports:  1 port for "node A -> node B" and 1 port for "node B ->
node A". Replies and acks are sent back over same port.

A om_comm_send_with_ack(stuff) and a om_comm_send_with_reply(stuff)
which would talk to the other node directly. no kcomd, but the process
would still sleep while it waits.

 Each process having 2 ports open per process isn't all that many ports
is it?  Even if there are 200 processes, that's only 400 ports. 
iptables handles WAY more than that with no problems.

If all the calling functions and receive functions were merged into one
each, the process would still be put to sleep while it waits.

My understanding:  
2ports: 1 port for "node A -> node B" and 1 port for "node B -> node A" 

cases when remote node calls home:
1) syscalls, including fork and clone
2) migrate home command given on the remote node.
3) signals?

cases when home node calls remote:
1)  migrate home command given on the home node.
2)  signals

Replies and acks are sent back over same port.

Deadlock/race issues:
?


How does 2.4 do things?

This might be a silly question but is it possible to have a daemon
'kmigd' that takes care of all the housekeeping.  No om_pre_usermode
stuff at all.  The daemon handles signals and sets process flags
accordingly, rather than the process kernel stub doing it. Talks with
other nodes and responds accordingly. checks the om_struct and puts
process to sleep and migrates it then wakes it up on remote node.  That
way it really is just a small kernel module and patches are unobtrusive?
 The syscall trap in entry.S would have to stay as I think the syscall
table isn't trappable like it was in 2.4.  Is that correct?


Thoughts?
Matt




>>> Vincent Hanquez <tab-mTI/[email protected]> 05/18/06 7:35 AM >>>
On Wed, May 17, 2006 at 02:36:35PM -0600, Matt Dew wrote:
> Tab,
>   Expanding on your comments, here's the worst case conditions we need
> to account for (let me know if I've forgotten any.) Again, I'm not
much
> of a software guy so let me know if I'm heading in the wrong
direction.
> 
> 1)  remote node sends a syscall home for processing at the same time
the
> home node sends a migration home command (or signal) to the remote
node.
>  The network packets pass each other in flight.
> 
> 2)  The remote node sends a syscall home and the home node sends the
> response but the remote node errors out and sends home an error
> condition.  
> 
> (longer term)
> 3)  We have an SMP machine with a process that forked after it
migrated,
> so both processes share the same file descriptor.   Both processes
send
> syscalls home at the same time (one write and one read for example.). 

> Home node sends a migration command at the same time.
> 
> 
> A)  The home node has a queue for things it needs to send the remote
> node.
> B)  The remote node is the boss and if both nodes want to do
something,
> remote always wins.
> C)  Acks are always send before anything, so both know who is talking
> and who is listening."
> D)  There's a response/ack for each communication.
> E)  The structs have two more fields,  originating node
> ('home','remote'), and msg_num.
> 
> 
> "I'm sending syscall X to you, and give me back the result",
> "sending page X, then page Y, ..." ... 
> 
> becomes
> remote: "Ok, I'm not currently processing a signal or command from the
> home node. I'm gonna set a flag saying I'm doing the talking."
> remote:  "Hey, home node,  I'm gonna send you a syscall."
> home: "OK, go ahead, I have a signal in my queue I need to send you,
but
> you have priority, I'll try again later."
> remote: "I got your request but ignored it since my flag is set. Here
> comes my syscall. Let me know if you got it."
> home: "Ok, I got it. Response will be next"
> remote: "Awaiting response"
> home: "Here's the response, page X, let me know if you got it ok."
> remote:  "I have an error."
> home:  "Ok, I'll resend."
> remote: "I still have the error."
> home:  "Ok. Nothing I can do to help you."
> remote:  "I'm done with you, I'm returning an error code to the
calling
> function. and I'm clearing my flag."
> ...
> home: "Hey remote, I want to send you a signal."
> remote: "OK, I'll set a flag saying you're doing the talking."
> home: "here's the signal <msg_num=12>."
> remote: "Got it <and the msg_num is 12>. Any more?"
> home: "Nope. I'm done."
> remote: "OK, you're done, I'm clearing my flag."
> 
> Thoughts?
> 
> It's really verbose, but doing everything over one port is harder than
> having two (or more) ports.
> 
> How do you put a process to sleep for event number X when the events
are
> all just tcp packets?

That's seems overly complex to put that in the process kernel part. Plus
the logic have to be duplicate per process, instead of having simple
daemon that handle almost everything and hide everything to the process
kernel part.

actually having 1 port (or 2) open per node2node communication no matter
what are the number of process from this node seems even saner.

( - 1port: 1 port for "node A <-> node B"
  - 2ports: 1 port for "node A -> node B" and 1 port for "node B -> node
A" ?)

Some kernel thread (kcomd) is in charge to do the multiplexing/demux.

so you have something like

   kmigd  ----------\                                   /-> kmigd
   deputy process1 ---> kcomd(1) ---network--- kcomd(2) --> process1
   deputy process2 --/                                  \-> process2
   deputy process3 -/                                    \> process3

kcomd can do all the hard work, with a packet queue (even like an
iovec).
kcomd take the pid of the process to do the multiplex and don't stop
until a packet is fully received

all you have to do is put a request into the queue, with different
semantic. like:

   process(x) say:
   send(stuff, node Y) -> kcomd(1) read the request, and ack the request
                          to this process(x) (or not)

   send_with_ack(stuff, node Y) ->
	kcomd(1) read the request, and put the process to sleep. when
	it's time to send the packet kcom received from the process(x),
	send it over the network to kcomd(2), and kcomd(2) send a ack if
	received. wake up process(x) with the ack response from kcomd(2)
	(ERR or SUCCESS).

   send_with_reply(stuff, &stuff2, node Y) ->
   	kcomd(1) read the request, put the process to sleep. send the
	packet to kcomd(2). when kcomd(2) is going to reply what
	process(x) need, wake up process(x) and continue.

Now .. there's a great potential for deadlock there, but that should
be alright once we have the semantics right.
   
What do people think ?
-- 
Vincent Hanquez


-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services,
security?
Get stuff done quickly with pre-integrated technology to make your job
easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache
Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
openMosix-devel mailing list
openMosix-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/openmosix-devel



-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642