Opening & writing to UTF-8 files; copyright symbol again
[email protected] ("Highsmith, Anne L") Fri, 13 Nov 2015 20:01:32 +0000
| Newsgroups | perl.perl4lib |
|---|---|
| Message-ID | <[email protected]> |
--_000_90d4c5e419db4c1bb2267b3ba3c9ddc5exch2pmbxt4adstamuedu_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
This is related to my previous post (9/17/2015) about deleting 035 fields a=
fter RDA-ification. Jon Gorman solved that one for me by pointing out that =
I probably had a problem with my perl libraries.
But now, instead of creating the record from the database and writing it ba=
ck to the database, I am reading from a file exported from my database, whi=
ch is UTF-8. Specifically, the blasted copyright symbol again. As stored in=
the database, the copyright symbol is encoded as C2 A9, which if I read th=
e tables correctly, is the correct UTF-8 encoding for copyright. But when I=
read the record from a file and write it back to the file after deleting t=
he problematic 035, the encoding for the copyright symbol has been turned i=
nto A9.
This "transformation" happens both when running the perl program on my pc a=
nd on the unix server. Interestingly, complicated Unicode seems to be okay.=
I took a record with Hebrew vernacular characters and edited it using my p=
rogram, then ran the source record and target record through xxd. I then di=
ffed the files; it showed no difference. But the before and after of the re=
cord that has the copyright symbol munges the copyright by stripping the C2=
.
Here's the program. If anybody can tell my what I'm doing wrong I'd really =
appreciate it.
---------------------------------------------------------------------------=
-------------------------------
use strict;
use warnings;
use MARC::Record;
use MARC::Batch;
my $infile=3D'4788022.bib';
my $batch =3D MARC::Batch->new('USMARC',"$infile");
my $outfile=3D'4788022.edited.bib';
open(OUTPUT, ">$outfile");
while (my $record =3D $batch->next) {
my $f001 =3D $record->field('001');
my $bib_id =3D $f001->as_string();
my @a035 =3D $record->field('035');
foreach my $f035 (@a035) {
if (my $f035a =3D $f035->subfield('a')) {
if ($f035a eq $bib_id) {
$record->delete_field($f035);
}
}
}
print OUTPUT $record->as_usmarc();
}
Anne L. Highsmith
Director, Consortia Systems
TAMU Libraries
5000 TAMU
College Station, TX 77843-5000
979 862 4234
[email protected]
--_000_90d4c5e419db4c1bb2267b3ba3c9ddc5exch2pmbxt4adstamuedu_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"EN-US" link=3D"#0563C1" vlink=3D"#954F72">
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><span style=3D"color:#1F497D">This is related to my =
previous post (9/17/2015) about deleting 035 fields after RDA-ification. Jo=
n Gorman solved that one for me by pointing out that I probably had a probl=
em with my perl libraries.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p> </o:p></spa=
n></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D">But now, instead of cr=
eating the record from the database and writing it back to the database, I =
am reading from a file exported from my database, which is UTF-8. Specifica=
lly, the blasted copyright symbol again.
As stored in the database, the copyright symbol is encoded as C2 A9, which=
if I read the tables correctly, is the correct UTF-8 encoding for copyrigh=
t. But when I read the record from a file and write it back to the file aft=
er deleting the problematic 035,
the encoding for the copyright symbol has been turned into A9. <o:p></o:p>=
</span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p> </o:p></spa=
n></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D">This “transforma=
tion” happens both when running the perl program on my pc and on the =
unix server. Interestingly, complicated Unicode seems to be okay. I took a =
record with Hebrew vernacular characters and edited
it using my program, then ran the source record and target record through =
xxd. I then diffed the files; it showed no difference. But the before and a=
fter of the record that has the copyright symbol munges the copyright by st=
ripping the C2.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p> </o:p></spa=
n></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D">Here’s the progr=
am. If anybody can tell my what I’m doing wrong I’d really appr=
eciate it.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D">----------------------=
---------------------------------------------------------------------------=
---------<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D">use strict;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D">use warnings;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D">use MARC::Record;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D">use MARC::Batch;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D">my $infile=3D'4788022.bib';<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D">my $batch =3D MARC::Batch->new('USMARC',"$infile"=
;);<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D">my $outfile=3D'4788022.edited.bib';<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D">open(OUTPUT, ">$outfile");<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D">while (my $record =3D $batch->next) {<o:p></o:p></span></p=
>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D"> my $f001 =3D $record->field('001'=
);<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D"> my $bib_id =3D $f001->as_string()=
;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D"> <o:p>
</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D"> my @a035 =3D $record->field('035'=
);<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D"> foreach my $f035 (@a035) {<o:p></o:p=
></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D"> =
if (my $f035a =3D $f035->subfield('a')) {<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D"> &=
nbsp; if ($f035a eq $bib_id) {<o:p></o:p></span></p=
>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D"> &=
nbsp; $record->del=
ete_field($f035);<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D"> &=
nbsp; }<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D"> =
}
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D"> }<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D"> print OUTPUT $record->as_usmarc()=
;
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c=
olor:#1F497D">}<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p> </o:p></spa=
n></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Anne L. Highsmith<o:p></o:p></p>
<p class=3D"MsoNormal">Director, Consortia Systems<o:p></o:p></p>
<p class=3D"MsoNormal">TAMU Libraries<o:p></o:p></p>
<p class=3D"MsoNormal">5000 TAMU<o:p></o:p></p>
<p class=3D"MsoNormal">College Station, TX 77843-5000<o:p></o:p=
></p>
<p class=3D"MsoNormal">979 862 4234<o:p></o:p></p>
<p class=3D"MsoNormal">[email protected]<o:p></o:p></p>
</div>
</body>
</html>
--_000_90d4c5e419db4c1bb2267b3ba3c9ddc5exch2pmbxt4adstamuedu_--