RE: Splitting a large file of MARC records into smaller files

This isn't a perl solution, but it may work for you.

You can use the unix split command to split a file into several other
files with the same number of lines each. For that to work, you'll first
have to use tr to convert the ^] record separators into newlines. Then
use tr to convert them all back in each split file.

#> tr '^]' '\n' < filename > filename.nl
#> split -l $lines_per_file filename.nl SPLIT
#> for file in SPLIT*; do tr '\n' '^]' < $file > ${file%.nl}.rs

Or something like that.

/dev
-- 

Devon Smith
Consulting Software Engineer
OCLC Office of Research



-----Original Message-----
From: Nolte, Jennifer [mailto:[email protected]] 
Sent: Monday, January 25, 2010 9:48 AM
To: [email protected]
Subject: Splitting a large file of MARC records into smaller files

Hello-

I am working with files of MARC records that are over a million records
each. I'd like to split them down into smaller chunks, preferably using
a command line. MARCedit works, but is slow and made for the desktop.
I've looked around and haven't found anything truly useful- Endeavor's
MARCsplit comes close but doesn't separate files into even numbers, only
by matching criteria, so there could be lots of record duplication
between files.

Any idea where to begin? I am a (super) novice Perl person.

Thank you!

~Jenn Nolte


Jenn Nolte
Applications Manager / Database Analyst
Production Systems Team
Information Technology Office
Yale University Library
130 Wall St.
New Haven CT 06520
203 432 4878