Re: Re[2]: [bdbxml] Indexing suggestion

Hi Steve and John,

After spending an extensive amount of time with BDB XML for the past  
few days, and working with a fairly large dataset (14000 files, each  
representing an image), I have come to the conclusion that Steve is  
right, or at least, we need something that implements something of  
this sort.

My problem is that my XML files are generated by user created data  
(either from direct user input or from files the user created), and  
as such, is going to have both uppercase and lowercase letters.  
However, when users query for information, they do not wish to put  
query the string "eLiTe" if that is what they called their file, but  
rather they want to query the string "elite" and have it match the  
file which is called "eLiTe.jpg".

Currently, if I create an index on equality or substring, they will  
not be used, as I have to do the following query (assume we are using  
contains):
container(.....)/picture[contains(lower-case(./ 
filename),'searchstring')]

This will not invoke the index I created on filename. However, if we  
had a way to declare an index such as "add index on lower-case 
("filename")" - then it would work fine. For example, PostgreSQL lets  
you do this in SQL using:
CREATE INDEX test1_lower_col1_idx ON test1 (lower(col1));

I understand BDB XML is not an SQL database, but we really need to be  
able to specify more expressive indices. My problem really manifests  
itself when I have complicated queries, since they all use lower- 
case. This is really putting a strain on how I use BDB XML and  
whether or not it is the right thing to use for what I need.

Any response is welcome. I understand John's response that it is  
difficult to see which queries match, but I'm not sure why this is  
the case with Steve's suggestion or another suggestion.

Thanks for listening, regardless.

Itay




On Apr 16, 2006, at 2:11 AM, Steve Howe wrote:

> Hello John,
>
> Monday, April 10, 2006, 11:36:05 AM, you wrote:
>
>> Thanks for your suggestion, Steve. We have considered using paths to
>> specify indexes, and have rejected the idea before now.
>> The trouble comes when you try to match available indexes against a
>> given query to see which ones you can use - the more specific an  
>> index
>> the less queries you can use it in, and the harder it is to tell  
>> if you
>> can use it in a particular query.
> Thanks for your response, and I'm sorry for writing back only now.
>
> As I see, to write an appropriate index expression is is  
> responsibility
> of the programmer and not BDBXML's, right ? There are XPath  
> expressions
> that cane be very generic, such as "//*". To *allow* such indexes  
> to be
> written created is BDBXML's responsibility. Limiting what BDBXML  
> can do
> because the user could be not clever is weird and just too sad. The
> documentation could just instruct those users to add "//*" and such
> generic expressions. Let's not strip the power because users are  
> dumb...
> and if someone is dumb, the last thing he'll want to do is playing  
> with
> BDBXML.
>
> The problem with the current scheme is that it is too inflexible. For
> instance, if I have a container with two kinds of documents:
>
> <user>
>   ...
> </user>
>
> <group>
>   ...
> </group>
>
> ... and I want to index all nodes under "user" but not those under
> group, I can't do it, because default indexes will act on the whole
> document. I could do it, however, using two XPath indexes:
>
> /user/*
>
> /group
>
> And sometimes one wants to be *very* specific in its query, and  
> indexing
> using the current scheme simply isn't enough, for instance when are
> involved 3 edge nodes, a function, the "|" operator, etc. The
> possibilities would be as flexible as possible - because they would be
> the same BDBXML already knows and understands.
>
> Can you think of any kind of indexing scheme that is currently  
> addressed
> and couldn't be expressed as a XPath expression ? There can't be  
> because
> anything that can be searched is XPath anyway - right ?
>
> To match the current BDBXML indexes implementation, one could use:
>
> node presence:
> //xxx
>
> node equality:
> //xxx=$value
>
> edge presence:
> //xxx/*
>
> edge equality:
> //xxx/*=$value
>
> The metadata indexing should be covered the same way XPath expressions
> do: by using functions to access those nodes.
>
> Implementation doesn't seem to be too hard - just test the if node  
> in the
> current context matches the XPath index expression, and if it does,
> add an entry to the index database.
>
> I hate to be annoying, but I would like to ask you to reconsider
> implementing it unless there is some reason it could not be done. The
> current indexing API would not need to be changed; it would even be  
> just
> converted to generic XPath expressions meaning the same thing.
> Everything would be just compatible.
>
> Thanks for reading so far.
>
> -- 
> Best regards,
>  Steve                            mailto:[email protected]
>
>
>
> ------------------------------------------
> To remove yourself from this list, send an
> email to [email protected]
>

Itay A. Neeman
[email protected]


------------------------------------------
To remove yourself from this list, send an
email to [email protected]