Re: Converting XML to MARC without reading a file?

--00000000000083ab91056a7a1868
Content-Type: text/plain; charset="UTF-8"

 The downside of that is that you would be parsing your XML twice though.
Once with XML::Twig and once with MARC::File::XML.

That said, if you have a huge result set, it would probably be better (or
even necessary) to use XML::Twig::parseurl() or roll your own using LWP
with callbacks and a pull parser like
http://search.cpan.org/dist/XML-LibXML/lib/XML/LibXML/Parser.pod#Pull_Parser_(Reader)
.

I built a OAI-PMH harvester for Koha, which can pull down a lot of huge
MARCXML records; I have a process handling the HTTP request with LWP and it
writes the XML to a pipe in chunks using a callback. I have a second
process read from the pipe using XML::LibXML::Reader and for each MARCXML
record that comes through it uses "new_from_xml()".

It works really really well.

On Thu, Apr 19, 2018 at 10:34 AM, Karl Billeter <[email protected]> wrote:

> On Thu, Apr 19, 2018 at 12:25:05AM +0000, Andy Kohler wrote:
>
> > Problem: Converting from MARCXML via MARC::File::XML seems to require
> > reading the records from a file.  I've already got the XML stored in a
> > variable, retrieved via LWP::Simple->get().
> >
> > Do I have to write the XML to a file, then read it in again to convert
> it?
> > Or am I just missing something obvious?
>
> Looking at the module documentation new_from_xml() should work but only
> for a
> single record chunk.  I'd probably use that and something like XML::Twig to
> split the records.
>
> K
>

--00000000000083ab91056a7a1868
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_extra">

<span style=3D"color:rgb(34,34,34);font-family:arial,sans-serif;font-size:s=
mall;font-style:normal;font-variant-ligatures:normal;font-variant-caps:norm=
al;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;t=
ext-transform:none;white-space:normal;word-spacing:0px;background-color:rgb=
(255,255,255);text-decoration-style:initial;text-decoration-color:initial;f=
loat:none;display:inline">The downside of that is that you would be parsing=
 your XML twice though. Once with XML::Twig and once with MARC::File::XML.<=
/span></div><div class=3D"gmail_extra"><span style=3D"color:rgb(34,34,34);f=
ont-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-=
ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:no=
rmal;text-align:start;text-indent:0px;text-transform:none;white-space:norma=
l;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:=
initial;text-decoration-color:initial;float:none;display:inline"><br></span=
></div><div class=3D"gmail_extra"><span style=3D"color:rgb(34,34,34);font-f=
amily:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligat=
ures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;=
text-align:start;text-indent:0px;text-transform:none;white-space:normal;wor=
d-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initi=
al;text-decoration-color:initial;float:none;display:inline">That said, if y=
ou have a huge result set, it would probably be better (or even necessary) =
to use XML::Twig::parseurl()=C2=A0or roll your own using LWP with callbacks=
 and a pull parser like <a href=3D"http://search.cpan.org/dist/XML-LibXML/l=
ib/XML/LibXML/Parser.pod#Pull_Parser_(Reader)">http://search.cpan.org/dist/=
XML-LibXML/lib/XML/LibXML/Parser.pod#Pull_Parser_(Reader)</a>.</span></div>=
<div class=3D"gmail_extra"><span style=3D"color:rgb(34,34,34);font-family:a=
rial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:no=
rmal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-al=
ign:start;text-indent:0px;text-transform:none;white-space:normal;word-spaci=
ng:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text=
-decoration-color:initial;float:none;display:inline"><br></span></div><div =
class=3D"gmail_extra">I built a OAI-PMH harvester for Koha, which can pull =
down a lot of huge MARCXML records; I have a process handling the HTTP requ=
est with LWP and it writes the XML to a pipe in chunks using a callback. I =
have a second process read from the pipe using XML::LibXML::Reader=C2=A0and=
 for each MARCXML record that comes through it uses &quot;new_from_xml()&qu=
ot;.=C2=A0</div><div class=3D"gmail_extra"><br></div><div class=3D"gmail_ex=
tra">It works really really well.</div><div class=3D"gmail_extra"><div clas=
s=3D"gmail_extra" style=3D"color:rgb(34,34,34);font-family:arial,sans-serif=
;font-size:small;font-style:normal;font-variant-ligatures:normal;font-varia=
nt-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-=
indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-dec=
oration-style:initial;text-decoration-color:initial"><br><div class=3D"gmai=
l_quote">On Thu, Apr 19, 2018 at 10:34 AM, Karl Billeter<span>=C2=A0</span>=
<span dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]" target=3D"_bla=
nk" style=3D"color:rgb(17,85,204)">[email protected]</a>&gt;</span><span>=
=C2=A0</span>wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0p=
x 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><s=
pan class=3D"gmail-">On Thu, Apr 19, 2018 at 12:25:05AM +0000, Andy Kohler =
wrote:<br><br>&gt; Problem: Converting from MARCXML via MARC::File::XML see=
ms to require<br>&gt; reading the records from a file.=C2=A0 I&#39;ve alrea=
dy got the XML stored in a<br>&gt; variable, retrieved via LWP::Simple-&gt;=
get().<br>&gt;<span>=C2=A0</span><br>&gt; Do I have to write the XML to a f=
ile, then read it in again to convert it?<br>&gt; Or am I just missing some=
thing obvious?<br><br></span>Looking at the module documentation new_from_x=
ml() should work but only for a<br>single record chunk.=C2=A0 I&#39;d proba=
bly use that and something like XML::Twig to<br>split the records.<br><span=
 class=3D"gmail-HOEnZb"><font color=3D"#888888"><br>K<br></font></span></bl=
ockquote></div></div><br class=3D"gmail-Apple-interchange-newline">

<br></div></div>

--00000000000083ab91056a7a1868--