Re: Opening & writing to UTF-8 files; copyright symbol again
[email protected] (Jon Gorman) Fri, 13 Nov 2015 14:52:01 -0600
| Newsgroups | perl.perl4lib |
|---|---|
| Message-ID | <CABpHuQOCJynBWozREBqqXK_Ae1O9Lo=1C9RfWGCE7rTdMLa1mA@mail.gmail.com> |
--047d7b676b0a2afda50524723b2d
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
I'll ask the easiest solution first ;).
Are you sure the file 4788022.bib is in unicode and not marc-8? If it is in
unicode, is the leader 09 byte set to a?
I'm a bit rusty on the as_usmarc() call as well, you might want to check
the docs to make sure that doesn't do something like convert it to marc-8.
Finally, another thing I'd do is avoid the older conventions for opening
file handles (if you have time, effective perl, second edition is an
excellent book if you wan to do more perl. Nice and short with a good mix
of techniques).
So instead of
encoding()
A method for getting/setting the encoding for a record. The encoding for a
record is determined by position 09 in the leader, which is blank for
MARC-8 encoding, and 'a' for UCS/Unicode. encoding() will return a string,
either 'MARC-8' or 'UTF-8' appropriately.
If you want to set the encoding for a MARC::Record object you can use the
string values:
$record->encoding( 'UTF-8' );
NOTE: MARC::Record objects created from scratch have an a default encoding
of MARC-8, which has been the standard for years...but many online catlogs
and record vendors are migrating to UTF-8.
WARNING: you should be sure your record really does contain valid UTF-8
data when you manually set the encoding.
On Fri, Nov 13, 2015 at 2:01 PM, Highsmith, Anne L <[email protected]=
u
> wrote:
> This is related to my previous post (9/17/2015) about deleting 035 fields
> after RDA-ification. Jon Gorman solved that one for me by pointing out th=
at
> I probably had a problem with my perl libraries.
>
>
>
> But now, instead of creating the record from the database and writing it
> back to the database, I am reading from a file exported from my database,
> which is UTF-8. Specifically, the blasted copyright symbol again. As stor=
ed
> in the database, the copyright symbol is encoded as C2 A9, which if I rea=
d
> the tables correctly, is the correct UTF-8 encoding for copyright. But wh=
en
> I read the record from a file and write it back to the file after deletin=
g
> the problematic 035, the encoding for the copyright symbol has been turne=
d
> into A9.
>
>
>
> This =E2=80=9Ctransformation=E2=80=9D happens both when running the perl =
program on my pc
> and on the unix server. Interestingly, complicated Unicode seems to be
> okay. I took a record with Hebrew vernacular characters and edited it usi=
ng
> my program, then ran the source record and target record through xxd. I
> then diffed the files; it showed no difference. But the before and after =
of
> the record that has the copyright symbol munges the copyright by strippin=
g
> the C2.
>
>
>
> Here=E2=80=99s the program. If anybody can tell my what I=E2=80=99m doing=
wrong I=E2=80=99d really
> appreciate it.
>
>
> -------------------------------------------------------------------------=
---------------------------------
>
> use strict;
>
> use warnings;
>
> use MARC::Record;
>
> use MARC::Batch;
>
> my $infile=3D'4788022.bib';
>
> my $batch =3D MARC::Batch->new('USMARC',"$infile");
>
> my $outfile=3D'4788022.edited.bib';
>
> open(OUTPUT, ">$outfile");
>
>
>
> while (my $record =3D $batch->next) {
>
> my $f001 =3D $record->field('001');
>
> my $bib_id =3D $f001->as_string();
>
>
>
> my @a035 =3D $record->field('035');
>
> foreach my $f035 (@a035) {
>
> if (my $f035a =3D $f035->subfield('a')) {
>
> if ($f035a eq $bib_id) {
>
> $record->delete_field($f035);
>
> }
>
> }
>
> }
>
> print OUTPUT $record->as_usmarc();
>
> }
>
>
>
>
>
>
>
> Anne L. Highsmith
>
> Director, Consortia Systems
>
> TAMU Libraries
>
> 5000 TAMU
>
> College Station, TX 77843-5000
>
> 979 862 4234
>
> [email protected]
>
--047d7b676b0a2afda50524723b2d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div><div><div><div><br><br></div>I'll ask the easiest=
solution first ;).<br><br></div>Are you sure the file <span style=3D"font-=
family:"Courier New";color:rgb(31,73,125)">4788022.bib is in unic=
ode and not marc-8? If it is in unicode, is the leader 09 byte set to a?<br=
><br></span></div><span style=3D"font-family:"Courier New";color:=
rgb(31,73,125)">I'm a bit rusty on the as_usmarc() call as well, you mi=
ght want to check the docs to make sure that doesn't do something like =
convert it to marc-8. <br><br></span></div><span style=3D"font-family:"=
;Courier New";color:rgb(31,73,125)">Finally, another thing I'd do =
is avoid the older conventions for opening file handles (if you have time, =
effective perl, second edition is an excellent book if you wan to do more p=
erl. Nice and short with a good mix of techniques).<br></span><div><span st=
yle=3D"font-family:"Courier New";color:rgb(31,73,125)"><br></span=
></div><div><span style=3D"font-family:"Courier New";color:rgb(31=
,73,125)">So instead of <br><br></span></div><div><span style=3D"font-famil=
y:"Courier New";color:rgb(31,73,125)"><br><br></span><div><h2><a =
class=3D"" name=3D"encoding()">encoding()</a></h2>
<p>A method for getting/setting the encoding for a record. The encoding=20
for a record is determined by position 09 in the leader, which is blank=20
for MARC-8 encoding, and 'a' for UCS/Unicode. encoding() will retur=
n a=20
string, either 'MARC-8' or 'UTF-8' appropriately.</p>
<p>If you want to set the encoding for a MARC::Record object you can use th=
e string values:</p>
<pre class=3D""> $record->encoding( 'UTF-8' );</pre>
<p>NOTE: MARC::Record objects created from scratch have an a default=20
encoding of MARC-8, which has been the standard for years...but many=20
online catlogs and record vendors are migrating to UTF-8.</p>
<p>WARNING: you should be sure your record really does contain valid UTF-8 =
data when you manually set the encoding.</p><br><div><div><div><br><br><br>=
<br></div></div></div></div></div></div><div class=3D"gmail_extra"><br><div=
class=3D"gmail_quote">On Fri, Nov 13, 2015 at 2:01 PM, Highsmith, Anne L <=
span dir=3D"ltr"><<a href=3D"mailto:[email protected]" target=3D"=
_blank">[email protected]</a>></span> wrote:<br><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex">
<div link=3D"#0563C1" vlink=3D"#954F72" lang=3D"EN-US">
<div>
<p class=3D"MsoNormal"><span style=3D"color:#1f497d">This is related to my =
previous post (9/17/2015) about deleting 035 fields after RDA-ification. Jo=
n Gorman solved that one for me by pointing out that I probably had a probl=
em with my perl libraries.<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1f497d"><u></u>=C2=A0<u></u></=
span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1f497d">But now, instead of cr=
eating the record from the database and writing it back to the database, I =
am reading from a file exported from my database, which is UTF-8. Specifica=
lly, the blasted copyright symbol again.
As stored in the database, the copyright symbol is encoded as C2 A9, which=
if I read the tables correctly, is the correct UTF-8 encoding for copyrigh=
t. But when I read the record from a file and write it back to the file aft=
er deleting the problematic 035,
the encoding for the copyright symbol has been turned into A9. <u></u><u><=
/u></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1f497d"><u></u>=C2=A0<u></u></=
span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1f497d">This =E2=80=9Ctransfor=
mation=E2=80=9D happens both when running the perl program on my pc and on =
the unix server. Interestingly, complicated Unicode seems to be okay. I too=
k a record with Hebrew vernacular characters and edited
it using my program, then ran the source record and target record through =
xxd. I then diffed the files; it showed no difference. But the before and a=
fter of the record that has the copyright symbol munges the copyright by st=
ripping the C2.<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1f497d"><u></u>=C2=A0<u></u></=
span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1f497d">Here=E2=80=99s the pro=
gram. If anybody can tell my what I=E2=80=99m doing wrong I=E2=80=99d reall=
y appreciate it.<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1f497d">----------------------=
---------------------------------------------------------------------------=
---------<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">use strict;<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">use warnings;<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">use MARC::Record;<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">use MARC::Batch;<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">my $infile=3D'4788022.bib';<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">my $batch =3D MARC::Batch->new('USMARC',"$inf=
ile");<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">my $outfile=3D'4788022.edited.bib';<u></u><u></u></sp=
an></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">open(OUTPUT, ">$outfile");<u></u><u></u></span><=
/p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d"><u></u>=C2=A0<u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">while (my $record =3D $batch->next) {<u></u><u></u></span>=
</p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0 my $f001 =3D $record->field('=
001');<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0 my $bib_id =3D $f001->as_string()=
;<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0 <u></u>
<u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0 my @a035 =3D $record->field('=
035');<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0 foreach my $f035 (@a035) {<u></u><u>=
</u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =
if (my $f035a =3D $f035->subfield('a')) {<u></u><u></u></span></=
p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if ($f035a eq $bib_id) {<u></u><u></u></span=
></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 $record->de=
lete_field($f035);<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =
}=C2=A0=C2=A0=C2=A0
<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0 }<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0 print OUTPUT $record->as_usmarc()=
;=C2=A0=C2=A0
<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1f497d">}<u></u><u></u></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1f497d"><u></u>=C2=A0<u></u></=
span></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">Anne L. Highsmith<u></u><u></u></p>
<p class=3D"MsoNormal">Director, Consortia Systems<u></u><u></u></p>
<p class=3D"MsoNormal">TAMU Libraries<u></u><u></u></p>
<p class=3D"MsoNormal">5000 TAMU<u></u><u></u></p>
<p class=3D"MsoNormal">College Station, TX=C2=A0=C2=A0 77843-5000<u></u><u>=
</u></p>
<p class=3D"MsoNormal"><a href=3D"tel:979%20862%204234" value=3D"+197986242=
34" target=3D"_blank">979 862 4234</a><u></u><u></u></p>
<p class=3D"MsoNormal"><a href=3D"mailto:[email protected]" target=3D"_blank=
">[email protected]</a><u></u><u></u></p>
</div>
</div>
</blockquote></div><br></div>
--047d7b676b0a2afda50524723b2d--