Advantages of using a native XML database.

Nicky,

Over the last week or so I've been attempting to prove (one way or the
other) if an XML database is a credible solution for storing highly
structured data. The driving constraints for this have been to ensure
that the storage facilities maintain the full structure of the data,
allow for structural changes and provides a mechanism to recover
documents using a variety of query structures. 

For release 1.0 we used Lucene to index learning object metadata and
other structured data. Lucene is a great query engine with lightening
fast performance. The downside is that we lose the document structure as
it's flattened to allow lucene to create the indexes. We are also a
little limited when it comes to being able to handle identifiers as all
fields in the record are treated with the same filters (this is one of
the reasons it's difficult for us to turn off case sensitivity).

To provide a clearer picture of the problem lets work with an example,
searching for a learning object in the local repository.

At a high level this is what happens currently.

The query is sent to lucene.
The lucene query results are converted in a lazy fashion) lazy being
good in this case) to objects.
A generator converts the objects into a sax stream
The page is generated.

Using an XML database.

The query is sent to the database
The database returns a sax stream
The page is generated. 

Not alot of difference here. We have managed to eliminate the conversion
from result set to something that the generator can use.

Now say we want to enhance the result with list of versions for the
learning object.

The query is sent to lucene.
The lucene query results are converted in a lazy fashion) lazy being
good in this case) to objects.
A query is sent to the object model to recover the versions of the
objects.
A generator converts the objects into a sax stream
The page is generated.

Using an XML database.

The query is sent to the database
The database returns a sax stream
The page is generated. 

Now there are optimisations we can make to the lucene version. We could
decide to include the version information into the lucene index as a
comma separated list of values to eliminate the model query. That would
involve changing the index generation code, the object extraction code
and the sax event generator. What happens when we want the extract some
other piece of nested information? We change the index again, modify the
objects, change the generators etc. In short small changes can have a
large knock on effect because we have made arbitrary decisions regarding
the structure held in the index. With the xml database version we change
the query, a much smaller task.

In short the use of an xml database allows us maximum development
agility with acceptable performance characteristics and reduced
development effort.

Issues:

I have been able to corrupt the database by killing the system. Although
this initially looks like a show stopper it's not too bad, I had to try
really hard to make it happen and as the index is a redundant copy of
data stored elsewhere the indexes can be rebuilt with ease. I will
though be looking into this over the next week or so.

-k.

-- 
If you don't test then your code is only a collection of bugs which 
apparently behave like a working program. 

Website: http://www.rocketred.com.au/blogs/kevin/



-------------------------------------------------------
This SF.net email is sponsored by: ValueWeb: 
Dedicated Hosting for just $79/mo with 500 GB of bandwidth! 
No other company gives more support or power for your dedicated server
http://click.atdmt.com/AFF/go/sdnxxaff00300020aff/direct/01/