Re: Opinions wanted: Is smartfrog the right framework for my osgi-based system?

Guijarro, Julio wrote:
> Hi Stanley,
> =

>  =

> =

> See my answers below
> =

>  =

> =

> *From:* Stanley C. Burnitt [mailto:[email protected]]

> >3.  Use smartfrog to deploy and run bash scripts which install =

> >cloudera.repo files, then install cloudera's hadoop / hbase / zookeeper =

> >distributions (via yum),  edit configuration files, and issue startup & =

> >shutdown commands.
> =

> Steve has been working on some especial components to deploy and manage =

> Hadoop but he is using Apache=92s distro. He is on holiday for two weeks =

> and could give you an update on it once he is back.

Hell, I'm back.

I'm using my own patched version of trunk which -because it is trunk- is =

not something I'd keep files I value on. It's OK for short lived stuff, =

though as I haven't resynced with trunk for a while I will need to =

suffer to sync everything up and make sure that the reasons tests fail =

are not my fault.

It's patched because I've given the hadoop services a common base class, =

something I'd like to get back in there but which is tricky to do as =

trying to keep up with Hadoop changes is a full time job, and I'm only =

doing the hadoop work in the spare time. With the base classes we can =

then then bring up Hadoop in the SF JVM, providing live configuration, =

monitoring liveness, etc. This is nice, but introduces surprises.

we package up Hadoop in our own RPMs, just take the JARs I build myself =

and put them into the sf-hadoop RPM. This code only works with my =

modified hadoop version.

>> 4.  Use smartfrog to deploy and configure a solr cluster.  (There is no =

>> yum pkg for solr, to my knowledge.  Can I use smartfrog to deploy & run =

>> a bash script which scp's, extracts, and configures gzipped solr =

>> distributions on a cluster of solr servers?)


> =

> There are components for SSH and SCP, you can also write components that =

> get the file from a webserver for example and perform the operation =

> locally. For example, you deploy a component in a machine that then =

> downloads a package locally, unzips it and run the app. I would advise =

> you to try the Dynamic Webserver example included with SF. It does =

> exactly what you describe here and you could reuse most of it.
> =


You could also talk to the solr people about helping them create RPMs =

which suit your needs.

> =

> 5.  > Deploy and manage the dependencies and lifecycles of all of my =

> > system's OSGi components -- standalone product launchers on top of =

> > dozens of plugins.
> =

> Does this seem reasonable?  My entire system is implemented in java, and =

> for this reason I chose to look at smartfrog first, before bcfg2 or =

> puppet.  But this is an OSGi based system, not J2EE, and I will be =

> deploying (then extracting) gzipped eclipse products, not jars.
> Also, the OSGi bundles are started from native product launchers -- =

> again, not java jars.

Julio, do we have any components to unzip stuff? The code is all in =

Java, there's also the ant tasks to do better things -but nothing in =

Java sets up filesystem permissions very well, as Sun didn't want you to =

be able to do that. If you want to unzip/untar with permissions, exec =

the native programs.

> =

> With the stable release of SF you have to manage your OSGi components =

> using either Java and Scripts but there is a version of SF that has full =

> integration with OSGi (SF uses it but it also allows you to describe an =

> manage the entire environment but is not stable and has not yet been =

> used in production) The intention is to merge the branch into the trunk =

> soon but we do not know how long it is going to take to merge the code =

> and test it properly.
> =


I'm looking forward to this too, but am sure it will make =

hadoop-under-SF even more fun.

Know that following the merge we also plan to flip the license from LGPL =

to apache, just because people worry about the implications of LGPL and =

subclassing.


>> And I intend to run system / scalability tests on EC2.

We have some SF components to work with EC2 and S3, push out data and =

bring up/tear down VMs. There is a web front end, CloudFarmer, which =

does more: pushes out SmartFrog configurations to  machines when you =

create them, works to bring up Hadoop clusters, etc.

http://www.slideshare.net/steve_l/new-roles-for-the-cloud
http://www.slideshare.net/steve_l/deploying-on-ec2

CloudFarmer has four back ends: mock, physical, EC2 and HP Cells, the =

latter two being API-driven. But it isn't HA, can leak VMs, which is not =

what you want when paying for VMs; you really need a recovery mechanism =

(either store metadata in the VM config field of each EC2 VM or use =

SimpleDB). And it needs security, though that's a matter of bringing up =

Jetty with the right certificate settings, which is a matter of =

configuring the SF jetty components right. If you want the HA features, =

you get to start coding.

-Steve


---------------------------------------------------------------------------=
---
The Palm PDK Hot Apps Program offers developers who use the
Plug-In Development Kit to bring their C/C++ apps to Palm for a share =

of $1 Million in cash or HP Products. Visit us here for more details:
http://ad.doubleclick.net/clk;226879339;13503038;l?
http://clk.atdmt.com/CRS/go/247765532/direct/01/