Design pattern for processing a huge directory tree of files using GPars

Merlin Beedell <[email protected]>
Newsgroups gmane.comp.lang.groovy.user
Message-ID <LO0P123MB64465B35AD09E15E1907784FB9C69@LO0P123MB6446.GBRP123.PROD.OUTLOOK.COM>
I am trying to process millions of files, spread over a tree of directories.  At the moment I can collect the set of top level directories into a list and then process these in parallel using GPars with list processing (e.g. .eachParallel).
But what would be more efficient would be a 'parallel' for the File handling routines, for example:

               withPool() {
                              directory.eachFileMatchParallel (FILES, ~/($fileMatch)/) {aFile ->  ...

then I would be a very happy bunny!

I know I could copy the list of matching files into an Array list and then use the withPool { filesArray.eachParallel { ... - but this does not seem like an efficient solution - especially if there are several hundred thousand files in a directory.

What design pattern(s) might be better to consider using?

Merlin Beedell
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.