Re: Compacting multiple large files

Ken,

You kids.

When I took Analysis of Algorithms it was a graduate level class, but
that was 1973 or 1974.

I wonder of the subject matter was simplified or if the younger
generation is smarter.

Of course, we did our own homework.

Ivan 




 

-----Original Message-----
From: Discussion of advanced Java topics.
[mailto:[email protected]] On Behalf Of Ken Gentle
Sent: Wednesday, August 08, 2007 5:01 PM
To: [email protected]
Subject: Re: [ADVANCED-JAVA] Compacting multiple large files

Geez, this is one from "Analysis of Algorithms", CS-310 - the latter
half of a "merge" sort.  2nd or 3rd year undergraduate.  The merge sort
recursively breaks the input into smaller and smaller chunks until each
chunk is small enough to be sorted via a simple sorting algorithm then
the sorted chunks are merged.  Recursive algorithms can be converted to
iterative implementations and this one is left as an exercise for the
student.  ;-)

This is a problem that is relatively easy make parallel - the
constraints are going to be hardware (number of cores, amount of memory
and I/O devices (disks, ram-disks, whatever)) - the more of each that
you have, the better.
While it may be easy enough to make parallel or implement, you could end
up spending a lot of time trying to optimize performance.

There used to be commercial packages that specialized in this type of
problem (SyncSort comes to mind).  You might consider one as a step in a
batch process, not necessarily implemented in Java.

My two cents  (CS-310  happened around 1983, for me!)

Ken

On 8/8/07, [email protected] <[email protected]> wrote:
>
> Try the Unix sort command.  If you have at least 2 IO controllers, 
> that is a good start.  If this is all on one disk then :(
>
> > -----Original Message-----
> > From: Discussion of advanced Java topics.
> > [mailto:[email protected]] On Behalf Of Avinash 
> > Lakshman
> > Sent: Tuesday, August 07, 2007 6:56 PM
> > To: [email protected]
> > Subject: [ADVANCED-JAVA] Compacting multiple large files
> >
> > Hi
> >
> > I have a requirement where I need to merge multiple sorted large 
> > files into one single file. There could be at any given instant of 
> > time up to 50 files which could be a few GB in size. Is there any 
> > known efficient algorithm or technique in Java to achieve this ?
> >
> > Thanks in advance
> > Avinash
> >
> > _________________________________________________________________
> > Puzzles, trivia teasers, word scrambles and more. Play for your 
> > chance to win!
> > http://club.live.com/home.aspx?icid=CLUB_hotmailtextlink
> >
> > ===================================
> > This list is hosted by DevelopMentorR  http://www.develop.com
> >
> > View archives and manage your subscription(s) at 
> > http://discuss.develop.com
> >
> >
>
> ===================================
> This list is hosted by DevelopMentor(r)  http://www.develop.com
>
> View archives and manage your subscription(s) at 
> http://discuss.develop.com
>



--

J. Kenneth Gentle (Ken)
Gentle Software LLC
Phone:  484.371.8137
Mobile: 302.547.7151
Email:  mailto:[email protected]
Email:  mailto:[email protected]
http://www.gentlesoftware.com

===================================
This list is hosted by DevelopMentor(r)  http://www.develop.com

View archives and manage your subscription(s) at
http://discuss.develop.com

===================================
This list is hosted by DevelopMentor®  http://www.develop.com

View archives and manage your subscription(s) at http://discuss.develop.com