Re: Splitting a large file of MARC records into smaller files

I also recommend using MARC::Batch. Attached is a simple script I wrote for
myself.

Saiful Amin
+91-9343826438


On Mon, Jan 25, 2010 at 8:33 PM, Robert Fox <[email protected]> wrote:

> Assuming that memory won't be an issue, you could use MARC::Batch to
> read in the record set and print out seperate files where you split on
> X amount of records. You would have an iterative loop loading each
> record from the large batch, and a counter variable that would get
> reset after X amount of records. You might want to name the sets using
> another counter that keeps track of how many sets you have and name
> each file something like batch_$count.mrc and write them out to a
> specific directory. Just concatenate each record to the previous one
> when you're making your smaller batches.
>
> Rob Fox
> Hesburgh Libraries
> University of Notre Dame
>
> On Jan 25, 2010, at 9:48 AM, "Nolte, Jennifer"
> <[email protected]> wrote:
>
> > Hello-
> >
> > I am working with files of MARC records that are over a million
> > records each. I'd like to split them down into smaller chunks,
> > preferably using a command line. MARCedit works, but is slow and
> > made for the desktop. I've looked around and haven't found anything
> > truly useful- Endeavor's MARCsplit comes close but doesn't separate
> > files into even numbers, only by matching criteria, so there could
> > be lots of record duplication between files.
> >
> > Any idea where to begin? I am a (super) novice Perl person.
> >
> > Thank you!
> >
> > ~Jenn Nolte
> >
> >
> > Jenn Nolte
> > Applications Manager / Database Analyst
> > Production Systems Team
> > Information Technology Office
> > Yale University Library
> > 130 Wall St.
> > New Haven CT 06520
> > 203 432 4878
> >
> >
>

#!c:/perl/bin/perl.exe
#
# Name: mbreaker.pl
# Version: 0.1
# Date: Jan 2009
# Author: Saiful Amin <[email protected]>
#
# Description: Extract MARC records based on command-line paramenters

use strict;
use warnings;
use Getopt::Long;
use MARC::Batch;

my $start	= 0;
my $end		= 1;

GetOptions ("start=i" => \$start,
			"end=i"   => \$end
);

my $batch = MARC::Batch->new('USMARC', $ARGV[0]);
$batch->strict_off();
$batch->warnings_off();

my $num = 0;
while (my $record = $batch->next() ) {
	$num++;
	next if $num < $start;
	last if $num > $end;
	print $record->as_usmarc();
	warn "$num records\n" if ( $num % 1000 == 0 );
}


__END__

=head1 NAME

mbreaker.pl

Breaks the MARC record file as per start and end position specified

=head1 SYNOPSIS

mbreaker.pl [options] file

	Options:
     -start			start position for reading records
	 -end			end position for reading records