Re: Editing marc records; program exits with Encode.pm error on copyright symbol
[email protected] (Jon Gorman) Fri, 18 Sep 2015 10:00:52 -0500
| Newsgroups | perl.perl4lib |
|---|---|
| Message-ID | <CABpHuQP4j+=h5UQyk33FmUYSVomHdxA64xr6M_0i4UDnaQtw5w@mail.gmail.com> |
--001a113d4bd24957bd052006cc24 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I haven't seen any responses, so I'll give this a shot. Warning though, my memory goes a little bit fuzzy about unicode ;). First, xBF looks suspiciously like part of a byte order mark (aka BOM aka EF BB BF in utf-8). I'd make sure that some change in the process hasn't started introducing some weirdness or the unicode format . Having some issues opening your screenshot (I'm not sure why you embedded it into Word? Mayhap just use a jpg or png? Or just copy/paste the hexdump?). I'd hope your vendor wouldn't be so clueless to introduce a BOM in a utf-8 file, but it's a common byproduct of using various Windows libraries. I'd be kinda surprised any such file could even be loaded in. Second, I'd probably try using binmode and perhaps even flag the incoming code as :raw. Very strange things can happen when parts of the chain try to convert an utf-8 encoding into utf-8 (aka double-encoding). However, since it's a database connection, you might need to look at it a bit. Have you changed the way your perl script connects to the database? Updated the perl libraries or the freetds/odbc libraries? I'm also a little confused. So the test record you made replicated the issue w/ the unicode in a record that previously worked? Or did it not? Sorry, I fear I have more questions than advice... Are you able to compare the raw record given by the vendor and what's stored in Voyager. (I wouldn't do any manipulation, I'd just have another script do something like... my $bib_marc =3D &get_bib_string($dbh, $dbase, $bib_id); open $dump_file,'>', "dump_file.mrc' ; binmode($dump_file,":raw") print $dump_file $bib_marc ; Jon G On Thu, Sep 17, 2015 at 1:53 PM, Highsmith, Anne L <[email protected]= u > wrote: > We just sent out our database to be RDA-ified and have reloaded it. We=E2= =80=99re > a Voyager site, so now I have to deal with the deadly problem of deleting > the 035 matching 001 that is added by the bib load process. (Voyager user= s > will know what I mean). > > > > Deleting that 035 is not the problem; I already had a program that did > that. But when I went to test that program this morning against the new > data, I found that it frequently exited with this error: > > =E2=80=9Cutf8 "\xBF" does not map to Unicode at C:/Perl/lib/Encode.pm lin= e 200.=E2=80=9D > > > > I discovered that the problematic field was a newly-added 264 with the > copyright symbol in 264$c. I printed out several sample records that > received this error and looked at the binary dumps. For the copyright > symbol, they all had C2 A9, which appears to me to be the proper code poi= nt > (am I using that correctly?) I added a copyright symbol to a test record > online using the voyager client, dumped it and looked at it, and saw that > the hex value for the copyright symbol added online was also C2 A9. So it > looks to me as though the RDA vendor used the character when adding the > copyright symbol. Or did they? > > > > So, what do I do to keep the program from blowing up? My program works by > creating a marc record string from marc record pieces stored in the > database, composing a marc record object from that string, changing the > object, changing it back to a string, then updating the database. I am no= t > working with an external file. The &get_bib_string is a locally written > subroutine that gets the pieces of the marc record as they=E2=80=99re sto= red in the > database and concatenates them into a string. I=E2=80=99ve been using it = for years > with no error, so that shouldn=E2=80=99t be the problem. > > > > The bulk of my code follows below the line. The error is generated by the > call to the new_from_usmarc function > > my $bib_marc =3D &get_bib_string($dbh, $dbase, $bib_id); > > my $record =3D MARC::Record->new_from_usmarc($bib_marc); > > my @a035 =3D $record->field('035'); > > foreach my $f035 (@a035) { > > if (my $f035a =3D $f035->subfield('a')) { > > if ($f035a eq $bib_id) { > > > $record->delete_field($f035); > > > $change_flag =3D 1; > > } > > } > > =E2=80=A6 [store updated marc record string to database] > > > _________________________________________________________________________= ___________________________________________________________________________= ________________________________________________________ > > The attachment shows the dump of part of the record; the copyright symbol > appears in the 6th line from the top. > --001a113d4bd24957bd052006cc24 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">I haven't seen any responses, so I'll give this a = shot.<div><br></div><div>Warning though, my memory goes a little bit fuzzy = about unicode ;).</div><div><br></div><div>First, xBF looks suspiciously like part of a byte order mark (aka BOM aka EF=20 BB BF in utf-8).=C2=A0 I'd make sure that some change in the process ha= sn't=20 started introducing some weirdness or the unicode format . Having some=20 issues opening your screenshot (I'm not sure why you embedded it into= =20 Word?=C2=A0 Mayhap just use a jpg or png? Or just copy/paste the hexdump?).= =20 I'd hope your vendor wouldn't be so clueless to introduce a BOM in = a=20 utf-8 file, but it's a common byproduct of using various Windows=20 libraries. I'd be kinda surprised any such file could even be loaded in= .</div><div><br></div><div>Second, I'd probably try using binmode and perhaps even flag the incoming code= =20 as :raw. Very strange things can happen when parts of the chain try to=20 convert an utf-8 encoding into utf-8 (aka double-encoding). However,=20 since it's a database connection, you might need to look at it a bit.</= div><div><br></div><div>Have you changed the way your perl script connects = to the database? Updated the perl libraries or the freetds/odbc libraries?<= /div><div><br></div><div>I'm also a little confused. So the test record you made replicated =C2=A0the= =20 issue w/ the unicode in a record that previously worked?=C2=A0 Or did it no= t?</div><div><br></div><div>Sorry, I fear I have more questions than advice= ...</div><div><br></div><div>Are you able to compare the raw record given by the vendor and what's=20 stored in Voyager. =C2=A0(I wouldn't do any manipulation, I'd just = have=20 another script do something like...</div><span class=3D"im"><div><br></div>= <div><span style=3D"color:rgb(31,73,125);font-family:Calibri,sans-serif;fon= t-size:14.6667px">my $bib_marc =3D &get_bib_string($dbh, $dbase, $bib_i= d);</span><br></div></span><div><span style=3D"color:rgb(31,73,125);font-fa= mily:Calibri,sans-serif;font-size:14.6667px">open $dump_file,'>'= , "dump_file.mrc' ;</span></div><div><font face=3D"Calibri, sans-s= erif" color=3D"#1f497d"><span style=3D"font-size:14.6667px">binmode($dump_f= ile,":raw")</span></font></div><div><font face=3D"Calibri, sans-s= erif" color=3D"#1f497d"><span style=3D"font-size:14.6667px">print $dump_fil= e $bib_marc ;</span></font></div><div><font face=3D"Calibri, sans-serif" co= lor=3D"#1f497d"><span style=3D"font-size:14.6667px"><br></span></font></div= ><div><font face=3D"Calibri, sans-serif" color=3D"#1f497d"><span style=3D"f= ont-size:14.6667px"><br></span></font></div><div><span style=3D"color:rgb(3= 1,73,125);font-family:Calibri,sans-serif;font-size:14.6667px"><br></span></= div><div><br></div><div>Jon G</div></div><div class=3D"gmail_extra"><br><di= v class=3D"gmail_quote">On Thu, Sep 17, 2015 at 1:53 PM, Highsmith, Anne L = <span dir=3D"ltr"><<a href=3D"mailto:[email protected]" target=3D= "_blank">[email protected]</a>></span> wrote:<br><blockquote clas= s=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pad= ding-left:1ex"> <div link=3D"#0563C1" vlink=3D"#954F72" lang=3D"EN-US"> <div> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d">We just sent out our database to be R= DA-ified and have reloaded it. We=E2=80=99re a Voyager site, so now I have = to deal with the deadly problem of deleting the 035 matching 001 that is added by the bib load process. (Voyager users will know what I= mean). <u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d"><u></u>=C2=A0<u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d">Deleting that 035 is not the problem;= I already had a program that did that. But when I went to test that progra= m this morning against the new data, I found that it frequently exited with this error:<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d">=E2=80=9Cutf8 "\xBF" does n= ot map to Unicode at C:/Perl/lib/Encode.pm line 200.=E2=80=9D<u></u><u></u>= </span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d"><u></u>=C2=A0<u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d">I discovered that the problematic fie= ld was a newly-added 264 with the copyright symbol in 264$c. I printed out = several sample records that received this error and looked at the binary dumps. For the copyright symbol, they all had C2 = A9, which appears to me to be the proper code point (am I using that correc= tly?) I added a copyright symbol to a test record online using the voyager = client, dumped it and looked at it, and saw that the hex value for the copyright symbol added online was a= lso C2 A9. So it looks to me as though the RDA vendor used the character wh= en adding the copyright symbol. Or did they?<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d"><u></u>=C2=A0<u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d">So, what do I do to keep the program = from blowing up? My program works by creating a marc record string from mar= c record pieces stored in the database, composing a marc record object from that string, changing the object, changing it ba= ck to a string, then updating the database. I am not working with an extern= al file. The &get_bib_string is a locally written subroutine that gets = the pieces of the marc record as they=E2=80=99re stored in the database and concatenates them into a string. I=E2=80=99ve b= een using it for years with no error, so that shouldn=E2=80=99t be the prob= lem.<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d"><u></u>=C2=A0<u></u></span></p> <div style=3D"border:none;border-bottom:solid windowtext 1.5pt;padding:0in = 0in 1.0pt 0in"> <p class=3D"MsoNormal" style=3D"border:none;padding:0in"><span style=3D"fon= t-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">The= bulk of my code follows below the line. The error is generated by the call= to the new_from_usmarc function<u></u><u></u></span></p> </div> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 my $bib_marc =3D &am= p;get_bib_string($dbh, $dbase, $bib_id);<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 my $record =3D MARC:= :Record->new_from_usmarc($bib_marc);<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 my @a035 =3D $record= ->field('035');<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 foreach my $f035 (@a= 035) {<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= if (my $f035a =3D $f035->subfield('a')) {<u></u><u></u></span><= /p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 if ($f035a eq $bib_id) {<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 $record->delete_field($f035);= <u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 $change_flag =3D 1;<u></u><u></u= ></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 }<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= }<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d">=E2=80=A6 [store updated marc record = string to database]<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d">_____________________________________= ___________________________________________________________________________= ___________________________________________________________________________= _________________<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;font-family:"Ca= libri",sans-serif;color:#1f497d">The attachment shows the dump of part= of the record; the copyright symbol appears in the 6<sup>th</sup> line fro= m the top.<u></u><u></u></span></p> </div> </div> </blockquote></div><br></div> --001a113d4bd24957bd052006cc24--