Design pattern for processing a huge directory tree of files using GPars
Merlin Beedell <[email protected]>
| Newsgroups | gmane.comp.lang.groovy.user |
|---|---|
| Message-ID | <LO0P123MB64465B35AD09E15E1907784FB9C69@LO0P123MB6446.GBRP123.PROD.OUTLOOK.COM> |
I am trying to process millions of files, spread over a tree of directories. At the moment I can collect the set of top level directories into a list and then process these in parallel using GPars with list processing (e.g. .eachParallel).
But what would be more efficient would be a 'parallel' for the File handling routines, for example:
withPool() {
directory.eachFileMatchParallel (FILES, ~/($fileMatch)/) {aFile -> ...
then I would be a very happy bunny!
I know I could copy the list of matching files into an Array list and then use the withPool { filesArray.eachParallel { ... - but this does not seem like an efficient solution - especially if there are several hundred thousand files in a directory.
What design pattern(s) might be better to consider using?
Merlin Beedell