Re: [bdbxml] Regular expressions in bdbxml

On Aug 18, 2005, at 4:38 AM, Michael Beddow wrote:

> [Greg Burd]
>
>>>      Thanks for raising this issue.  How important is this feature?
>>> I happen to think that this will be a common use case.  If you do  
>>> too
>>> (this is a general question for the list not just Adam), drop me  
>>> some
>>> email and let me know.  The question is:
>>>
>>> "Should Berkeley DB XML support an index type that optimizes access
>>> to data using regular expressions?"
>>>
>>
>>
>
> I think this should be very much a part of the road-mapped  
> extension of
> indexing to sub-document level (i.e. using indexes to search within
> documents, rather than just to identify the documents containing  
> matches as
> at present).  Since regexes are part of the XPath2/XQuery specs, I  
> think
> users will expect them not just to be supported (as they indeed are at
> present) but supported in a way that gives optimum performance.

Right, I agree with what you're saying.  We simply didn't have the  
resources to cover all the bases before launch.  Regex and full text  
are two that we've put off until we have significant demand.  Seems  
like regex has that demand, thanks for speaking up.

>>> As a follow up, which is more important, regex or full-text?
>>>
>
> I doubt if it's possible to rank these two as far as importance to  
> users
> (and developers as bdbxml library users) is concerned. They are  
> different
> needs corresponding (mainly) to different use-cases.  As for what the
> relative importance should be to the sleepycat developers, I guess  
> that has
> to be a matter of internal project priorities. All I would say as an
> outsider is that developing true fulltext facilities is a big  
> undertaking in
> its own right, additional to the current bdxml agendas (there was a  
> thread
> exploring this a while back)  whereas I would regard addition of
> index-assistance for regexes as a necessary further step along a  
> path where
> bdbxml is already far advanced. So while I personally would regard the
> eventual introduction of fulltext features as awelcome  bonus, I'd  
> view
> long-term absence of index-supported regexes as a serious defect,  
> for my
> main use-cases at any rate.

I like your assessment.  I generally agree.  We need to do regex  
indexing, and we'd like to do full text (eventually).  For full text  
we're likely to integrate the leading open source solution at the  
time (if the license is complementary).  One candidate is the C++  
version of Lucene (http://sourceforge.net/projects/clucene/), are  
there others you know of?

> [Adam Rambousek]
>
>> For our application, regular expressions are very important. It's
>> application for managing large dictionaries in XML format (mostly  
>> WordNet
>> kind). We need various searching options and one of them is searching
>> literals using regex (so, regex is needed only for values of one  
>> element
>> that contains a word or two).
>
> Multi-lingual lexicography is my main field of activity, and I  
> don't think
> bdxml is suitable for it yet, precisely because it doesn't (yet)  
> provide the
> granularity of indexing that complex queries against highly-structured
> lexical data require. However, the road-mapped developments in  
> forthcoming
> versions will change that dramatically (and I had rather assumed  
> that regex
> support would be built into the extensions to current indexing  
> methods that
> those developments entail). For now, though, of the open-source  
> native XML
> databases, I find that only eXist has the sub-document level  
> indexing power
> and flexibility that my dictionary applications require.

I think that with 2.2's release you'll find that BDB XML does a  
better job of sub-document level indexing.

regards,

-greg


> Michael Beddow
>
> ------------------------------------------
> To remove yourself from this list, send an
> email to [email protected]



------------------------------------------
To remove yourself from this list, send an
email to [email protected]