Re: Questions about SmartFrog

Zhang Qian wrote:
> Hi All,
> 
> Thanks for your replies about this topic.
> 
> I'd like to share more details about my cluster with you.
> As you know, it's a cluster includes hundreds of nodes. We divide
> these nodes into
> two categories: management nodes and computing nodes.

I see. We've tend to prefer the tactic of letting any node become a 
master (with agreement), because stops you having to decide which 
machines are in charge. Whatever boots up first can take over.

> For computing nodes, they just run the task arranged to them, do not
> have management roles, so we don't care it in this case.

OK -the workers are expected to fail and are told what to do; if they go 
away then something else gets the job.

> 
> For management nodes, we have a dozen of this kind of nodes in the
> cluster. Only
> one of them is the master node whose responsibility is to manage the
> entirely cluster, others are just the master candidates. The reason we
> do it in this way is to avoid single point failure, once the master
> node fails, a master candidate will take over its job, and become the
> new master node. So we have the heartbeat mechanism to detect the node
> status to realize fail-over.

OK. You're giving one machine charge of the resource management problem, 
but by sharing the data amongst candiates, if the master goes away you 
can have an election of some sort to decide who is the new master.



> Now there is a limitation: our cluster relies on shared-file system(such as NFS)
> which can be accessed by all the management nodes.That means all the config
> files placed on the shared-file system, all the management nodes need these
> config files. It's the master node's responsibilityto update these config file
> according to user's request, after a fail-over, the new master node
> will read these
> config file to know the latest configuration.

ah, so
  1. the NFS filestore is a failure point
  2. you need to save the configuration to a filesystem that doesnt go 
out of its way to enable locking


> 
> Now we want to remove the shared-file system dependency, each management
> node has config files in its local file system. So obviously, we need
> a mechanism
> to synchronize these config files on all the management nodes. That's
> why I asked
> that questions.

> I don't know whether there is a inbuilt component or service can
> provide this kind of mechanism in SmartFrog. Certainly I will
> investigate Anubis first, thanks for your sharing.

This is what anubis is designed for, to make a cluster out of a set of 
machines on a LAN. The papers and Paul can provide more details.

 > In addition, we have had a management console for user which will
 > communicate with our daemon in the master node, and deliver config
 > change to that daemon.
 > After receive the config change, this daemon will verify and activate
 > the change first,
 > then write it into the config file placed on the shared-file system.

I see. How does the management console deal with failure of the master? 
Does it discover it using some discovery protocol, or is the active 
master expected to update a dynamic DNS entry?




-------------------------------------------------------------------------
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell.  From the desktop to the data center, Linux is going
mainstream.  Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4