Re: Moving away from cron jobs to some workflow manager

try GNUbatch

On 19/06/2018 09:06, Rabin Yasharzadehe
wrote:

Hi
all,

I
need some advice, currently I have a huge cron file which
schedules tasks one after anther, and each task is position
precisely (with some room for error) to start after it
predecessor.

So
if one job start at 00:00 and it will go and fetch some files
and it takes 3minutes

the
next job will be after start right after at ~00:05

and
so on

the
problem is that if one job fails, all other jobs which are
depend on him will fail as well, and then I get a shitload of
alerts, and the worst part is that if i have to manually start
a batch process I need to go to each machine and manually
start each job in the right order,

I
was looking to resolve this problem with a tool which can
manage this "pipe line"

and
I cam across several tools like Luigi and (apache-)AirFlow,
I started with Luigi but It didn't look

right
for the job, and then I tried airflow, but was not able to
make it to work, the jobs queue never executed. =(

Has
any one have experience with airflow, or other tool like it
which they can recommend ?

My
needs are to be able to execute my CURRENT shell/python/php
scripts and build the dependency between them, and I perfer
the option for remote exec so that I will have central

place
to manage and monitor all work flow whichs are executed on
several nodes.

Thanks
in advance,

Rabin

_______________________________________________
Linux-il mailing list
[email protected]
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

--
Moish
0546484411

_______________________________________________
Linux-il mailing list
[email protected]
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il