Re: [bdbxml] index performance problem

Qiao,

To be clear, we're talking about 2 different things here:
1.  insertion time for your data
2.  querying/results handling

> George:
>
> I am using default configuration without explicitly set cache size.  
> One clue for the slow reason is that I am using python interface, I  
> don't know whether the interface will bring so much overhead or not.

Are you creating a DBEnv object in Python, or just using a default  
XmlManager
object?  If your are doing the former (DBEnv), the default cache size
is tiny, and you really need to boost it.  Cache size affects  
insertion time
as well as querying.

Also, we've found in the past that a large majority of document  
insertion
time is often in the application, and how it gets the documents to  
hand to
BDB XML, rather than in the actual putDocument() call.

When doing performance analysis and optimization, it's important to
break your code down into smaller chunks to find the specific problem
areas.

>  Note that in per-doc performance, small document insertion is not  
> slow. (20 ms/small doc vs. 80 ms/large doc). Actually I just wonder  
> large document is too fast to index every children node.
>
> I found that John Snelson said this in one Apr 19 email: "This  
> pattern of XML document (a single element with a large number of  
> children) is currently quite inefficient in DB XML."
>

Some aspects are inefficient, that is true.  But performance is  
always application-dependent,
varying with the size/shape of the document(s), queries used, and  
handling of
results.



>  His suggestion is to use small documents. But small documents is  
> less efficient when I need to get all the children. I need a query  
> and get maybe hundreds of small documents (this is a frequent  
> action). While in large document schema, I need only get the  
> document. Do you have any suggestion?
>

One down side of a single, large document is that certain queries
can take more time than they would with separate documents.
But, if you are just "getting" the *entire* document, that's fast.

Alternatively with the right indexes, getting hundreds of small  
documents should
be pretty fast, but turning those documents into one single doc (if  
you need
to) in results handling can take some time.

My suggestion is more experimentation, as there is no one, single
rule that can apply to all applications.  If several aspects of your
performance are acceptable, and one is not, this list may be able
to provide some more specific help if you provide very specific
information and questions (e.g. specific queries, configuration,  
code, etc).

Regards,

George


------------------------------------------
To remove yourself from this list, send an
email to [email protected]