RE: Design pattern for processing a huge directory tree of files using GPars

If you are able to use a modern Java implementation, you can use pure-Java streams, eg:

https://stackoverflow.com/a/66044221

///
Files.walk(Paths.get("/path/to/root/directory")) // create a stream of paths
    .collect(Collectors.toList()) // collect paths into list to better parallize
    .parallelStream() // process this stream in multiple threads
    .filter(Files::isRegularFile) // filter out any non-files (such as directories)
    .map(Path::toFile) // convert Path to File object
    .sorted((a, b) -> Long.compare(a.lastModified(), b.lastModified())) // sort files date
    .limit(500) // limit processing to 500 files (optional)
    .forEachOrdered(f -> {
        // do processing here
        System.out.println(f);
    });
///

also read : https://www.airpair.com/java/posts/parallel-processing-of-io-based-data-with-java-streams

Hope this helps some.

BOB


From: Merlin Beedell <[email protected]>
Sent: Monday, 9 May 2022 8:12 PM
To: users-GSC0n/0aZo1d/SJB6HiN2Ni2O/[email protected]
Subject: Design pattern for processing a huge directory tree of files using GPars

I am trying to process millions of files, spread over a tree of directories.  At the moment I can collect the set of top level directories into a list and then process these in parallel using GPars with list processing (e.g. .eachParallel).
But what would be more efficient would be a 'parallel' for the File handling routines, for example:

               withPool() {
                              directory.eachFileMatchParallel (FILES, ~/($fileMatch)/) {aFile ->  ...

then I would be a very happy bunny!

I know I could copy the list of matching files into an Array list and then use the withPool { filesArray.eachParallel { ... - but this does not seem like an efficient solution - especially if there are several hundred thousand files in a directory.

What design pattern(s) might be better to consider using?

Merlin Beedell