Re: Request for input - Batch processing using JAVA

This sounds to me like the right answer to the wrong question.

@Terry, you mentioned nothing of your requirements/constraints/intents. The language is really not that relevant. Your shop's choice of Java is based (I guess) either on the kind of talent (developers available) or the kind of libraries/code that could be reused for your processing. The IDE you use for development, Eclipse, not too relevant either. Johnny suggested to use Quartz, which uses the same syntax as cron. That would make your application a daemon. Also putting your quartz config in a properties file makes it easier to change, as easy as crontab.

Then you mention that the java code is packaged in .jar files, hard to search. Actually it's even worse, it's compiled into java bytecode and packaged into jars. If you want to look at the code, look at your code repository, why in the jar? I assume you have more experience with something like perl or shell scripting and you're used to looking/modifying the running code. Use a language you're more comfortable with then. Plenty of dynamic languages to chose from.

You also mention web services and SOA. That changes the topic from standalone, batch processing jobs to services, which implies interactions. Services (web or not) do processing on request, on a client call, and usually involves the processing of the input message passed by the caller (client). You could of course pass an empty message and use the call just trigger the batch processing job, or pass a simple selector to chose from a finite set of batch jobs, but that would be just a more complicated solution to a simple problem. A better use of services would be to accumulate the data passed by clients which could be batch processed later, but it doesn't look like your scenario.

There are quite a few people on this list experienced with SOA and if you provided more information about what exactly you want to achieve I am sure you'll get a lot of help.

Hadrian


On Mar 17, 2011, at 9:23 PM, Johnny Ren wrote:

> Have you considered using J2EE timer or Quartz timer instead of Unix cron to invoke your java application? This way you can package your jar files into your EJB or Tomcat container. 
> 
> --- On Tue, 3/15/11, Terry Waranch <[email protected]> wrote:
> 
> From: Terry Waranch <[email protected]>
> Subject: [Juglist] Request for input - Batch processing using JAVA
> To: [email protected]
> Date: Tuesday, March 15, 2011, 5:47 PM
> 
> Hi,
> 
>  
> I need some input.  I have sent this message to another email address but got what I think is a bounce back.  If this is a redundant message… sorry!  Here goes…
> 
>   
> Our shop runs a good many batch processes for daily reports, data extracts and data updates.  We are moving more toward java as our primary language.  Currently we use .jar files for these processes.  The programs are developed using Eclipse, exported to .jar files, and uploaded to our Linux server where they are executed.  The .jar will run under a Unix shell which is started by Cron.  A typical program will run under a minute to all day plus.  We use common .jar files for shared code.  Performance is good.  We have no problem distributing the programs. 
> 
>   
> The thing we don’t like about this strategy is that the code is packaged in .jar files which are hard to search and, when necessary, cumbersome to look through.
> 
>   
> We talked about organizing these processes into web services.  We could use Subversion to maintain the library.  This would give us a better tool for searching our code and organizing reusable functions. If we did that how would we kill an individual job?  I have also read articles that indicate that web servers will kill long running processes.  Are people using web services for batch processing.  Is that the right tool?
> 
>   
> As an alternative we talked about using large .jar files.  The code for batch processes could be stored in a common .jar file.  We could create individual .jar files with a simple MAIN to start a batch process.  The jobs would be easy to manage using Cron, easy to kill, and we would be able to use Subversion for library control. How large could a .jar file like this get?  Is this the right tool for this kind of processing?
> 
>   
> If you have any comments or something that I should look at please let me know. I attended a meeting several months ago where Service Oriented Architecture was the topic.  Does anyone remember the presenter’s name.  I would like to address this question to someone who has experience using SOA.
> 
>   
> Thanks for your help,
> 
> Terry
> 
>   
>   
> Terry N. Waranch
> Business Analyst/Developer
> REPLACEMENTS, LTD.
> PO Box 26029
> Greensboro, NC  27420-6029
> 336-697-3000, ext. 2298
> Fax:  336-697-3100
> [email protected]
> www.replacements.com
> 
>   
> NOTICE:  This message may contain information that is confidential, proprietary, and privileged.  If you are not the intended recipient, you may not use, print, copy, disseminate, or retain this message or any information contained in this message, including in any attachments to it.  If you have received this message in error, please notify the sender by reply email and delete this message and any attachments to it from your system(s).  Thank you for your cooperation. 
> -----Inline Attachment Follows-----
> 
> _______________________________________________
> Juglist mailing list
> [email protected]
> http://trijug.org/mailman/listinfo/juglist_trijug.org
> 
> _______________________________________________
> Juglist mailing list
> [email protected]
> http://trijug.org/mailman/listinfo/juglist_trijug.org

_______________________________________________
Juglist mailing list
[email protected]
http://trijug.org/mailman/listinfo/juglist_trijug.org