Avoid startup overhead by saving compiled large XSD for Xerces-J validation

--000000000000b05e0005de359d6a
Content-Type: text/plain; charset="UTF-8"

(Apologies in advance if this is already answered. I did make an attempt to
search archives at lists.apache.org stack overflow etc., but did not find
this answered. )

We use Xerces-J to validate XML files. (XSD 1.0. We are not yet using XSD
1.1)

The schemas of these files are huge. Think 300+ fairly large XSD files all
included/imported together. Megabytes of XSD.

In contrast the XML documents we're validating are typically small. They
are a few hundred or thousand bytes of XML. But there are many of them, so
we're calling Xerces to parse+validate them in a loop.  We are already
using the Xerces APIs in such a way that the XSD is loaded once, and the
parser then called repeatedly for each input data document.

I imagine that to validate XML, Xerces does something akin to "compiling"
the XSD into lower-level data structures for faster use when actually
parsing (and validating) the incoming XML being parsed.

Question 1: Is that true? Is there much compiling/lowering of the XSD to
fast parser-runtime structures?

If the answer to that is yes, then my next question applies also.

Question 2: Is it possible to get this "compilation" of the large XSD
schema done, and then serialize the resulting java object to a file, and
reload this pre-compiled thing so as not to face this compiling overhead at
startup time?

I have seen some discussion of serializing an XSModel of the XSD schema but
that's more or less isomorphic to the XSD file objects, i.e., not really
saving any "compilation" overhead.

Any advice appreciated.

Mike Beckerle
Apache Daffodil PMC | daffodil.apache.org
OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
Owl Cyber Defense | www.owlcyberdefense.com

--000000000000b05e0005de359d6a
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_quote"><br><div dir=3D"ltr"><div style=
=3D"margin:0px;padding:0px;border:0px;font-variant-numeric:inherit;font-var=
iant-east-asian:inherit;font-stretch:inherit;font-size:12pt;line-height:inh=
erit;font-family:Calibri,Arial,Helvetica,sans-serif;vertical-align:baseline=
;color:rgb(0,0,0)"><span style=3D"font-size:12pt">(Apologies in advance if =
this is already answered. I did make an attempt to search archives at <a hr=
ef=3D"http://lists.apache.org" target=3D"_blank">lists.apache.org</a> stack=
 overflow etc., but did not find this answered. )</span><br></div><div styl=
e=3D"margin:0px;padding:0px;border:0px;font-variant-numeric:inherit;font-va=
riant-east-asian:inherit;font-stretch:inherit;font-size:12pt;line-height:in=
herit;font-family:Calibri,Arial,Helvetica,sans-serif;vertical-align:baselin=
e;color:rgb(0,0,0)"><br></div><div style=3D"margin:0px;padding:0px;border:0=
px;font-variant-numeric:inherit;font-variant-east-asian:inherit;font-stretc=
h:inherit;font-size:12pt;line-height:inherit;font-family:Calibri,Arial,Helv=
etica,sans-serif;vertical-align:baseline;color:rgb(0,0,0)">We use Xerces-J =
to validate XML files. (XSD 1.0. We are not yet using XSD 1.1)</div><div st=
yle=3D"margin:0px;padding:0px;border:0px;font-variant-numeric:inherit;font-=
variant-east-asian:inherit;font-stretch:inherit;font-size:12pt;line-height:=
inherit;font-family:Calibri,Arial,Helvetica,sans-serif;vertical-align:basel=
ine;color:rgb(0,0,0)"><br></div><div style=3D"margin:0px;padding:0px;border=
:0px;font-variant-numeric:inherit;font-variant-east-asian:inherit;font-stre=
tch:inherit;font-size:12pt;line-height:inherit;font-family:Calibri,Arial,He=
lvetica,sans-serif;vertical-align:baseline;color:rgb(0,0,0)">The schemas of=
 these files are huge. Think 300+ fairly large XSD files all included/impor=
ted together. Megabytes of XSD.=C2=A0</div><div style=3D"margin:0px;padding=
:0px;border:0px;font-variant-numeric:inherit;font-variant-east-asian:inheri=
t;font-stretch:inherit;font-size:12pt;line-height:inherit;font-family:Calib=
ri,Arial,Helvetica,sans-serif;vertical-align:baseline;color:rgb(0,0,0)"><br=
></div><div style=3D"margin:0px;padding:0px;border:0px;font-variant-numeric=
:inherit;font-variant-east-asian:inherit;font-stretch:inherit;font-size:12p=
t;line-height:inherit;font-family:Calibri,Arial,Helvetica,sans-serif;vertic=
al-align:baseline;color:rgb(0,0,0)">In contrast the XML documents we&#39;re=
 validating are typically small. They are a few hundred or thousand bytes o=
f XML. But there are many of them, so we&#39;re calling Xerces to parse+val=
idate them in a loop.=C2=A0 <span style=3D"font-size:12pt">We are already u=
sing the Xerces APIs in such a way that the XSD is loaded once, and the par=
ser then called repeatedly for each input data document.</span></div><div s=
tyle=3D"margin:0px;padding:0px;border:0px;font-variant-numeric:inherit;font=
-variant-east-asian:inherit;font-stretch:inherit;font-size:12pt;line-height=
:inherit;font-family:Calibri,Arial,Helvetica,sans-serif;vertical-align:base=
line;color:rgb(0,0,0)"><br></div><div style=3D"margin:0px;padding:0px;borde=
r:0px;font-variant-numeric:inherit;font-variant-east-asian:inherit;font-str=
etch:inherit;font-size:12pt;line-height:inherit;font-family:Calibri,Arial,H=
elvetica,sans-serif;vertical-align:baseline;color:rgb(0,0,0)">I imagine tha=
t to validate XML, Xerces does something akin to &quot;compiling&quot; the =
XSD into lower-level data structures for faster use when actually parsing (=
and validating) the incoming XML being parsed.</div><div style=3D"margin:0p=
x;padding:0px;border:0px;font-variant-numeric:inherit;font-variant-east-asi=
an:inherit;font-stretch:inherit;font-size:12pt;line-height:inherit;font-fam=
ily:Calibri,Arial,Helvetica,sans-serif;vertical-align:baseline;color:rgb(0,=
0,0)"><br></div><div style=3D"margin:0px;padding:0px;border:0px;font-varian=
t-numeric:inherit;font-variant-east-asian:inherit;font-stretch:inherit;font=
-size:12pt;line-height:inherit;font-family:Calibri,Arial,Helvetica,sans-ser=
if;vertical-align:baseline;color:rgb(0,0,0)">Question 1: Is that true? Is t=
here much compiling/lowering of the XSD to fast parser-runtime structures?<=
/div><div style=3D"margin:0px;padding:0px;border:0px;font-variant-numeric:i=
nherit;font-variant-east-asian:inherit;font-stretch:inherit;font-size:12pt;=
line-height:inherit;font-family:Calibri,Arial,Helvetica,sans-serif;vertical=
-align:baseline;color:rgb(0,0,0)"><br></div><div style=3D"margin:0px;paddin=
g:0px;border:0px;font-variant-numeric:inherit;font-variant-east-asian:inher=
it;font-stretch:inherit;font-size:12pt;line-height:inherit;font-family:Cali=
bri,Arial,Helvetica,sans-serif;vertical-align:baseline;color:rgb(0,0,0)">If=
 the answer to that is yes, then my next question=C2=A0applies also.</div><=
div style=3D"margin:0px;padding:0px;border:0px;font-variant-numeric:inherit=
;font-variant-east-asian:inherit;font-stretch:inherit;font-size:12pt;line-h=
eight:inherit;font-family:Calibri,Arial,Helvetica,sans-serif;vertical-align=
:baseline;color:rgb(0,0,0)"><br></div><div style=3D"margin:0px;padding:0px;=
border:0px;font-variant-numeric:inherit;font-variant-east-asian:inherit;fon=
t-stretch:inherit;font-size:12pt;line-height:inherit;font-family:Calibri,Ar=
ial,Helvetica,sans-serif;vertical-align:baseline;color:rgb(0,0,0)">Question=
 2: Is it possible to get this &quot;compilation&quot; of the large XSD sch=
ema done, and then serialize the resulting java object to a file, and reloa=
d this pre-compiled thing so as not to face this compiling overhead at star=
tup time?=C2=A0</div><div style=3D"margin:0px;padding:0px;border:0px;font-v=
ariant-numeric:inherit;font-variant-east-asian:inherit;font-stretch:inherit=
;font-size:12pt;line-height:inherit;font-family:Calibri,Arial,Helvetica,san=
s-serif;vertical-align:baseline;color:rgb(0,0,0)"><br></div><div style=3D"m=
argin:0px;padding:0px;border:0px;font-variant-numeric:inherit;font-variant-=
east-asian:inherit;font-stretch:inherit;font-size:12pt;line-height:inherit;=
font-family:Calibri,Arial,Helvetica,sans-serif;vertical-align:baseline;colo=
r:rgb(0,0,0)"><span style=3D"font-size:12pt">I have seen some discussion of=
 serializing an XSModel of the XSD schema but that&#39;s more or less isomo=
rphic to the XSD file objects, i.e., not really saving any &quot;compilatio=
n&quot; overhead.=C2=A0</span><br></div><div style=3D"margin:0px;padding:0p=
x;border:0px;font-variant-numeric:inherit;font-variant-east-asian:inherit;f=
ont-stretch:inherit;font-size:12pt;line-height:inherit;font-family:Calibri,=
Arial,Helvetica,sans-serif;vertical-align:baseline;color:rgb(0,0,0)"><br></=
div><div style=3D"margin:0px;padding:0px;border:0px;font-variant-numeric:in=
herit;font-variant-east-asian:inherit;font-stretch:inherit;font-size:12pt;l=
ine-height:inherit;font-family:Calibri,Arial,Helvetica,sans-serif;vertical-=
align:baseline;color:rgb(0,0,0)">Any advice appreciated.=C2=A0</div><div st=
yle=3D"margin:0px;padding:0px;border:0px;font-variant-numeric:inherit;font-=
variant-east-asian:inherit;font-stretch:inherit;font-size:12pt;line-height:=
inherit;font-family:Calibri,Arial,Helvetica,sans-serif;vertical-align:basel=
ine;color:rgb(0,0,0)"><br></div><div><div dir=3D"ltr" data-smartmail=3D"gma=
il_signature"><div dir=3D"ltr"><div>Mike Beckerle=C2=A0</div><div><span sty=
le=3D"font-size:x-small">Apache Daffodil PMC |</span><span style=3D"font-si=
ze:x-small">=C2=A0</span><a href=3D"http://daffodil.apache.org/" style=3D"f=
ont-size:x-small" target=3D"_blank">daffodil.apache.org</a><br></div><div><=
span style=3D"font-size:x-small">OGF</span><span style=3D"font-size:x-small=
">=C2=A0</span><span style=3D"font-size:x-small">DFDL</span><span style=3D"=
font-size:x-small">=C2=A0</span><span style=3D"font-size:x-small">Workgroup=
 Co-</span><span style=3D"font-size:x-small">Chair |=C2=A0</span><a href=3D=
"http://www.ogf.org/ogf/doku.php/standards/dfdl/dfdl" style=3D"font-size:x-=
small" target=3D"_blank">www.ogf.org/ogf/doku.php/standards/dfdl/dfdl</a><b=
r></div><div><font size=3D"1">Owl Cyber Defense |=C2=A0<a href=3D"http://ww=
w.owlcyberdefense.com/" target=3D"_blank">www.owlcyberdefense.com</a></font=
><br></div><font size=3D"1"><br></font><div><br></div></div></div></div></d=
iv>
</div></div>

--000000000000b05e0005de359d6a--