MARC Records, XML, and encoding

Hi all,

I've attached a simple test script that converts a batch of records
from binary MARC to MARCXML and back again, converting to UTF-8
in the process and I've been running some tests on real records I'm
finding in the wild.  I've found that there are cases where the
script just dies with no warning, and there are cases where there is
serious data loss. I've attached a set of 192 records ... the file
size is 156778 bytes ... when I run the script the resulting file is
40250 bytes.

I'm running the following:
MARC::Record 2.0
MARC::File::XML 0.83 (from CPAN)
MARC::Charset 0.92
XML::SAX 0.14

Adding the line:

MARC::Charset->ignore_errors(1);

makes the resulting file 59192 bytes ...

Anyway, if anyone can shed some light on this I'd be grateful.

Cheers,

-- 
Joshua Ferraro               VENDOR SERVICES FOR OPEN-SOURCE SOFTWARE
President, Technology       migration, training, maintenance, support
LibLime                                Featuring Koha Open-Source ILS
[email protected] |Full Demos at http://liblime.com/koha |1(888)KohaILS

#!/usr/bin/perl
use strict; use warnings;
use MARC::Record;
use MARC::Batch;
use MARC::File::XML;
use MARC::Charset;
#MARC::Charset->ignore_errors(1);

my $infile = $ARGV[0];
my $outfile = $ARGV[1];

my $batch = MARC::Batch->new('USMARC',$infile);
$batch->strict_off();
open OUT,">:utf8",$outfile;

while( my $record = $batch->next() ) {
	my $xml = $record->as_xml;
	my $newrecord = MARC::Record::new_from_xml($xml, 'UTF-8');
	print OUT $newrecord->as_usmarc();
}
close OUT;