Re: [bdbxml] Insertion/Query on large Files

R=FCdiger,

> Hello,
> I think Iam going to be the wants-to-handle-large-files fellow of =20
> this forum ;-)... I have made some tests with different sizes of =20
> xml (up to 8.1g xml) files. More specific gxl-files (based on xml) =20
> which can store hierarchical graph structures. e.g.
>
> <gxl>
>   <graph>
>     <node>
>       <graph>
> >>      <node>
>           <attr name=3D"documentURL">
>             <string>The Title</string>
>           </attr>
>         </node>
>       </graph>
>     </node>
>   </graph>
> </gxl>
>
> The test query is supposed to retrieve a node containing a string-=20
> element with a specific text (see simplified example above).
>
> So my query looks like this:
>
> collection('gxl.dbxml')/gxl/graph/node/graph/node[attr/string=3D=20
> $myTitle]
>
> I have used the following indexes to try to speed things up:
>
> is.addIndex("","string", "edge-element-equality-string");
> is.addIndex("","documentURL", "edge-attribute-equality-string");

I think you meant to use:
is.addIndex("", "name", "edge-attribute-equality-string");

You need to use the attribute name ("name") not the value.

>
> My XQuery is a bit rusty- how would you add the restriction to attr-=20=

> Elements with die Attribute name=3D"documentURL"?

collection('gxl.dbxml')/gxl/graph/node/graph/node[attr/string=3D=20
$myTitle and attr/@name=3D'documentURL']

>
> Size of inserted Doc (as dbxml) and time for query
>  2,40M 22,00M 178,00M 1843,20M 27595,0234M
> 0,003s 0,010s  0,071s   0,695s   1127,866s
>
> The queries on the smaller files works pretty well. However the =20
> query on the 25G file takes roughly 19 minutes! This could possibly =20=

> also be achieved by SAX-parsing...
>
> Could you give be advise on how the speed things up?
>
> - Are the indexes ok? Should I rather use node- instead of edge?

That should not matter.  Edge may be more helpful.

> - Does "unique" speed things up or is it more a hindrance?

Uniqueness doesn't affect speed, just number of unique entries.

> - Is it possible to store the index-Data separate to the core data? =20=

> (Would keep dbxml file smaller)

Internally, it's kept separately.  Either way, this is not affecting =20
performance, just
size of the file.

> - Is there a possibility to improve the query?
You might try using collection('gxl.dbxml')//node[attr/string=3D$myTitle]

Usually people shy away from "//" and it can be expensive, but with
indexes, it may not be a problem.  I suspect your behavior will be about
the same, though.

You can also verify that the indexes are being used as expected, by =20
using
the queryPlan subcommand of the dbxml shell (or printing it using
XmlQueryExpression::getQueryPlan()).

BDB XML 2.2.13 can take a while to perform some queries on large =20
documents
when there are a lot (e.g. hundreds of thousands) of child nodes on =20
one or two
high-level elements, even when indexes are used.

This is a known problem and is being addressed, but may be affecting
your large documents.  It's one reason that we still recommend that
large, repetitive documents be split up into smaller documents if =20
possible.

In many cases, there is no pressing need to make a single large, tabular
document.  Your model looks more graph-like, and may not lend itself =20
to splitting, but
if you could, it'd help.

Regards,

George

>
> Thanks for any help you can offer!
>
> Regards,
>
> R=FCdiger


------------------------------------------
To remove yourself from this list, send an
email to [email protected]