Re: Opening & writing to UTF-8 files; copyright symbol again
[email protected] (Jon Gorman) Fri, 13 Nov 2015 14:55:55 -0600
| Newsgroups | perl.perl4lib |
|---|---|
| Message-ID | <CABpHuQMkNs64Kcm2sf3emvy33o73iS-aySStgFZSupVnD8v+Lw@mail.gmail.com> |
--047d7b4720d22982320524724975 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Ack, sorry, various copying and pasting apparently caused Google Mail to have issues. As I as was saying,before I must have hit some keystroke that I'm sure makes sense in whatever editor I was just susing: Instead of having: open(OUTPUT, ">$outfile"); ... (whole bunch o code) print OUTPUT $record->as_usmarc(); I would do open $output_marc, '>:raw', $outfile or die "Couldn't open file $!" ; print $output_marc $record->as_usmarc() ; (or I might play with binmode). Sorry for any confusion with the two emails ;) Jon G On Fri, Nov 13, 2015 at 2:01 PM, Highsmith, Anne L <[email protected]= u > wrote: > This is related to my previous post (9/17/2015) about deleting 035 fields > after RDA-ification. Jon Gorman solved that one for me by pointing out th= at > I probably had a problem with my perl libraries. > > > > But now, instead of creating the record from the database and writing it > back to the database, I am reading from a file exported from my database, > which is UTF-8. Specifically, the blasted copyright symbol again. As stor= ed > in the database, the copyright symbol is encoded as C2 A9, which if I rea= d > the tables correctly, is the correct UTF-8 encoding for copyright. But wh= en > I read the record from a file and write it back to the file after deletin= g > the problematic 035, the encoding for the copyright symbol has been turne= d > into A9. > > > > This =E2=80=9Ctransformation=E2=80=9D happens both when running the perl = program on my pc > and on the unix server. Interestingly, complicated Unicode seems to be > okay. I took a record with Hebrew vernacular characters and edited it usi= ng > my program, then ran the source record and target record through xxd. I > then diffed the files; it showed no difference. But the before and after = of > the record that has the copyright symbol munges the copyright by strippin= g > the C2. > > > > Here=E2=80=99s the program. If anybody can tell my what I=E2=80=99m doing= wrong I=E2=80=99d really > appreciate it. > > > -------------------------------------------------------------------------= --------------------------------- > > use strict; > > use warnings; > > use MARC::Record; > > use MARC::Batch; > > my $infile=3D'4788022.bib'; > > my $batch =3D MARC::Batch->new('USMARC',"$infile"); > > my $outfile=3D'4788022.edited.bib'; > > open(OUTPUT, ">$outfile"); > > > > while (my $record =3D $batch->next) { > > my $f001 =3D $record->field('001'); > > my $bib_id =3D $f001->as_string(); > > > > my @a035 =3D $record->field('035'); > > foreach my $f035 (@a035) { > > if (my $f035a =3D $f035->subfield('a')) { > > if ($f035a eq $bib_id) { > > $record->delete_field($f035); > > } > > } > > } > > print OUTPUT $record->as_usmarc(); > > } > > > > > > > > Anne L. Highsmith > > Director, Consortia Systems > > TAMU Libraries > > 5000 TAMU > > College Station, TX 77843-5000 > > 979 862 4234 > > [email protected] > --047d7b4720d22982320524724975 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div><div>Ack, sorry, various copying and pasting apparent= ly caused Google Mail to have issues.<br><br></div>As I as was saying,befor= e I must have hit some keystroke that I'm sure makes sense in whatever = editor I was just susing:<br><br></div>Instead of having:<br><div><br><p cl= ass=3D"MsoNormal"><span style=3D"font-family:"Courier New";color:= rgb(31,73,125)">open(OUTPUT, ">$outfile");</span></p><p class= =3D"MsoNormal"><br></p><p class=3D"MsoNormal">... (whole bunch o code)<br><= /p><p class=3D"MsoNormal"><span style=3D"font-family:"Courier New"= ;;color:rgb(31,73,125)"></span></p><p class=3D"MsoNormal"><span style=3D"fo= nt-family:"Courier New";color:rgb(31,73,125)"><br></span></p>=C2= =A0 <span style=3D"font-family:"Courier New";color:rgb(31,73,125)">pr= int OUTPUT $record->as_usmarc();=C2=A0 <br><br></span></div><div><span s= tyle=3D"font-family:"Courier New";color:rgb(31,73,125)">I would d= o <br><br></span></div><div><span style=3D"font-family:"Courier New&qu= ot;;color:rgb(31,73,125)">open $output_marc, '>:raw', $outfile o= r die "Couldn't open file $!" ;<br><br></span></div><div><spa= n style=3D"font-family:"Courier New";color:rgb(31,73,125)">print = $output_marc $record->as_usmarc() ;<br><br></span></div><div><span style= =3D"font-family:"Courier New";color:rgb(31,73,125)">(or I might p= lay with binmode).<br><br></span></div><div><span style=3D"font-family:&quo= t;Courier New";color:rgb(31,73,125)">Sorry for any confusion with the = two emails ;)<br></span></div><div><span style=3D"font-family:"Courier= New";color:rgb(31,73,125)"><br></span></div><div><span style=3D"font-= family:"Courier New";color:rgb(31,73,125)">Jon G<br></span></div>= <div><span style=3D"font-family:"Courier New";color:rgb(31,73,125= )"><br><br><br> </span></div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote= ">On Fri, Nov 13, 2015 at 2:01 PM, Highsmith, Anne L <span dir=3D"ltr"><= <a href=3D"mailto:[email protected]" target=3D"_blank">hismith@libra= ry.tamu.edu</a>></span> wrote:<br><blockquote class=3D"gmail_quote" styl= e=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div link=3D"#0563C1" vlink=3D"#954F72" lang=3D"EN-US"> <div> <p class=3D"MsoNormal"><span style=3D"color:#1f497d">This is related to my = previous post (9/17/2015) about deleting 035 fields after RDA-ification. Jo= n Gorman solved that one for me by pointing out that I probably had a probl= em with my perl libraries.<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1f497d"><u></u>=C2=A0<u></u></= span></p> <p class=3D"MsoNormal"><span style=3D"color:#1f497d">But now, instead of cr= eating the record from the database and writing it back to the database, I = am reading from a file exported from my database, which is UTF-8. Specifica= lly, the blasted copyright symbol again. As stored in the database, the copyright symbol is encoded as C2 A9, which= if I read the tables correctly, is the correct UTF-8 encoding for copyrigh= t. But when I read the record from a file and write it back to the file aft= er deleting the problematic 035, the encoding for the copyright symbol has been turned into A9. <u></u><u><= /u></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1f497d"><u></u>=C2=A0<u></u></= span></p> <p class=3D"MsoNormal"><span style=3D"color:#1f497d">This =E2=80=9Ctransfor= mation=E2=80=9D happens both when running the perl program on my pc and on = the unix server. Interestingly, complicated Unicode seems to be okay. I too= k a record with Hebrew vernacular characters and edited it using my program, then ran the source record and target record through = xxd. I then diffed the files; it showed no difference. But the before and a= fter of the record that has the copyright symbol munges the copyright by st= ripping the C2.<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1f497d"><u></u>=C2=A0<u></u></= span></p> <p class=3D"MsoNormal"><span style=3D"color:#1f497d">Here=E2=80=99s the pro= gram. If anybody can tell my what I=E2=80=99m doing wrong I=E2=80=99d reall= y appreciate it.<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1f497d">----------------------= ---------------------------------------------------------------------------= ---------<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">use strict;<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">use warnings;<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">use MARC::Record;<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">use MARC::Batch;<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">my $infile=3D'4788022.bib';<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">my $batch =3D MARC::Batch->new('USMARC',"$inf= ile");<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">my $outfile=3D'4788022.edited.bib';<u></u><u></u></sp= an></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">open(OUTPUT, ">$outfile");<u></u><u></u></span><= /p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d"><u></u>=C2=A0<u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">while (my $record =3D $batch->next) {<u></u><u></u></span>= </p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0 my $f001 =3D $record->field('= 001');<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0 my $bib_id =3D $f001->as_string()= ;<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0 <u></u> <u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0 my @a035 =3D $record->field('= 035');<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0 foreach my $f035 (@a035) {<u></u><u>= </u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = if (my $f035a =3D $f035->subfield('a')) {<u></u><u></u></span></= p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if ($f035a eq $bib_id) {<u></u><u></u></span= ></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 $record->de= lete_field($f035);<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = }=C2=A0=C2=A0=C2=A0 <u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0 }<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0 print OUTPUT $record->as_usmarc()= ;=C2=A0=C2=A0 <u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Courier New";c= olor:#1f497d">}<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1f497d"><u></u>=C2=A0<u></u></= span></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">Anne L. Highsmith<u></u><u></u></p> <p class=3D"MsoNormal">Director, Consortia Systems<u></u><u></u></p> <p class=3D"MsoNormal">TAMU Libraries<u></u><u></u></p> <p class=3D"MsoNormal">5000 TAMU<u></u><u></u></p> <p class=3D"MsoNormal">College Station, TX=C2=A0=C2=A0 77843-5000<u></u><u>= </u></p> <p class=3D"MsoNormal"><a href=3D"tel:979%20862%204234" value=3D"+197986242= 34" target=3D"_blank">979 862 4234</a><u></u><u></u></p> <p class=3D"MsoNormal"><a href=3D"mailto:[email protected]" target=3D"_blank= ">[email protected]</a><u></u><u></u></p> </div> </div> </blockquote></div><br></div> --047d7b4720d22982320524724975--