Re: Index causes query to break

--Apple-Mail-83-280226739
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii


On Jun 17, 2010, at 8:05 PM, Liz Glasser wrote:

> We have xindice 1.1.
>=20
> I'm not really sure what you mean by value type, but I created the =
index with the following command:
>=20
> xindice add_indexer -c url -p "*@ID" -n idindex

Value type determines how the value is going to be interpreted, by =
default it is "string". Using other types, such as integer, can speed up =
queries that have comparisons with numbers (i.e. element[@attribute > =
5]).=20

> After sending my original email I made some progress. If I create just =
that index above the query for =
"/DatabaseRecord[//tns:LizsNode[@ID=3D'1234']]" works, but all queries =
for "/DatabaseRecord[//LizsNodeDesc[@ID=3D'1234']]" fail.
>=20
> However, if I add a second index for "LizsNodeDesc@ID" then both =
queries work. Further testing shows that the first index speeds up the =
first query drastically and the second index speeds up the second query =
drastically. So it appears this combination is what I need.
>=20

> But I don't understand why the first index a) doesn't speed up the =
second query and b) breaks the second query altogether. I also do not =
understand why the second query still works on my other server. I'm =
guessing that may be due to the fact that I only have a few records =
total in that database since I wiped it.

It appears that you are doing everything correctly, I don't see a reason =
for the index to break the query. I tried to reproduce the problem, =
however, I was not able to do that...=20

=46rom your description I  can tell that the second query uses the index =
and, for some reason, finds no matching documents. If there is another, =
more specific index, the query will use it instead of more generic one, =
which seems to work for you.=20

It is kind of difficult for me to troubleshoot the problem when I cannot =
reproduce it myself. If you can narrow it down somehow, it will help a =
lot (like having specific document structure that cause the problem). =
One thing you can try to do is to run index scanner program that I =
attached to see data in the index. It shows indexed value, corresponding =
document key and element/attribute where the value appears.=20

The scanner expects 3 parameters -
1. Path to the database directory.
2. Path to the target collection that does not include database name. =
For example, if in the context switch of the command line tool you use =
/db/my/collection then second parameter should be /my/collection.
3. Index name.

You can check if the value you are searching for actually appears in the =
index for the document that you know has that value and its element name =
is correct, including namespace. This can eliminate at least some =
possible problems.=09

Be sure to shutdown the database before running the scanner - it is a =
low level utility and scans index file directly. I did not have much =
time to test it, so I would not run it on production.

Regards,
Natalia


--Apple-Mail-83-280226739
Content-Disposition: attachment;
	filename=IndexScanner.java
Content-Type: application/octet-stream;
	name="IndexScanner.java"
Content-Transfer-Encoding: 7bit

package org.apache.xindice.tools;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.xindice.core.Collection;
import org.apache.xindice.core.Database;
import org.apache.xindice.core.data.Key;
import org.apache.xindice.core.data.Value;
import org.apache.xindice.core.filer.BTreeCallback;
import org.apache.xindice.core.filer.BTree;
import org.apache.xindice.core.indexer.Indexer;
import org.apache.xindice.core.indexer.ValueIndexer;
import org.apache.xindice.core.indexer.IndexMatch;
import org.apache.xindice.util.Configuration;
import org.apache.xindice.xml.dom.DOMParser;
import org.apache.xindice.xml.SymbolTable;

import java.io.File;

public class IndexScanner extends BTree {

    private static final Log log = LogFactory.getLog(IndexScanner.class);


    public static void main(String[] args) throws Exception {
        if (args.length != 3) {
            usage();
            System.exit(1);
        }

        File location = new File(args[0]);
        String path = location.getAbsolutePath();
        String name = location.getName();

        if ("".equals(name) || !location.exists() || !location.isDirectory()) {
            System.out.println("Database path must point to existing database directory");
            System.exit(1);
        }

        Database db = null;
        Indexer idx = null;
        try {
            // create minimal database configuration instead of trying to locate system.xml
            String config = "<root-collection dbroot='" + path + "/' name='" + name + "'/>";
            db = new Database();

            db.setConfig(new Configuration(DOMParser.toDocument(config)));
            Collection col = db.getCollection(args[1]);
            if (col == null) {
                System.out.println("Collection " + args[1] + " is not found.");
                return;
            }

            idx = col.getIndexer(args[2]);
            if (idx == null || !(idx instanceof ValueIndexer)) {
                System.out.println("Index " + args[2] + " is not found.");
                return;
            }

            idx.open();
            new IndexScanner().processIndex((ValueIndexer) idx, col.getSymbols());
        } finally {
            if (idx != null) {
                idx.close();
            }

            if (db != null) {
                db.close();
            }
        }

    }

    private static void usage() {
        System.out.println("Usage:");
        System.out.println("    IndexScanner <db location> <collection path> <index name>");
        System.out.println();
        System.out.println("DB Location - Directory containing Xindice database files.");
        System.out.println("Collection path - Path to the target collection (not including database).");
        System.out.println("Index name - Name of the string value index to scan. Must belong to the target collection.");
        System.out.println();
        System.out.println("Important: Shutdown database before proceeding!");
        System.out.println();
    }

    private IndexMatch getIndexMatch(Value v) {
        byte[] b = v.getData();
        int l = b.length - 13;
        Key key = new Key(b, 0, b.length - 13);

        int pos = ((b[l + 1] << 24) | (b[l + 2] << 16) | (b[l + 3] << 8) | b[l + 4]);
        int len = ((b[l + 5] << 24) | (b[l + 6] << 16) | (b[l + 7] << 8) | b[l + 8]);
        short elemID = (short) ((b[l + 9] << 8) | b[l + 10]);
        short attrID = (short) ((b[l + 11] << 8) | b[l + 12]);

        return new IndexMatch(key, pos, len, elemID, attrID);
    }

    private void processIndex(final ValueIndexer filer, final SymbolTable symbols) {

        try {
            filer.query(null, new BTreeCallback() {
                Value value;
                public boolean indexInfo(Value value, long pos) {
                    if (pos == -1000) {
                        IndexMatch match = getIndexMatch(value);
                        String elNS = symbols.getNamespaceURI(match.getElement());
                        String attrNS = symbols.getNamespaceURI(match.getAttribute());
                        String el = symbols.getName(match.getElement());
                        String attr = symbols.getName(match.getAttribute());
                        if (attr != null) {
                            System.out.println("value: " + this.value + "\tkey: " + match.getKey() + "\t" +
                                    (elNS != null ? "[" + elNS + "]" : "") + el + "@" +
                                    (attrNS != null ? "[" + attrNS + "]" : "") + attr);
                        } else {
                            System.out.println("value: " + this.value + "\tkey: " + match.getKey() + "\t" +
                                               (elNS != null ? "[" + elNS + "]" : "") + el);
                        }
                    } else {
                        this.value = value;
                        BTree.BTreeRootInfo root = new BTree.BTreeRootInfo(value, pos);
                        try {
                            filer.query(root, null, this);
                        } catch (Exception e) {
                            log.error("Got an exception while scanning the index", e);
                        }
                    }

                    return true;
                }
            });

        } catch (Exception e) {
            log.error("Got an exception while scanning the index", e);
        }
    }
}
--Apple-Mail-83-280226739
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii



> Any insight you can provide would help me to feel more confident in =
the ability for Xindice to work in a production environment.=20
> Thank you
>=20
> Liz
>=20

>=20
> Natalia Shilenkova wrote:
>> Liz,=20
>> What Xindice version do you use and what is the value type that you =
used for creating the index?=20
>> Natalia
>>=20
>> On Jun 17, 2010, at 4:12 PM, Liz Glasser wrote:
>>=20
>> =20
>>> Hopefully this won't be too hard to follow. Please let me know if I =
need to clarify something.
>>>=20
>>> I have two types of Nodes (let's call them LizsNode and =
LizsNodeDesc) that each have an attribute named ID. LizsNode uses a =
namespace (tns).
>>>=20
>>> I have had the LizsNode data around for months, but the LizNodeDesc =
are all new. I have added an index defined as "*@ID" recently.
>>>=20
>>> With the index defined, when use the xpath query =
"/DatabaseRecord[//tns:LizsNode[@ID=3D'1234']]" the node is returned =
correctly. But when I use the query =
"/DatabaseRecord[//LizsNodeDesc[@ID=3D'1234']]" 99.999% of the time =
nothing is returned. Occasionally the right result is returned. Also if =
I query for /DatabaseRecord[//LizsNodeDesc]" all those nodes are =
returned and I can see my ID is correct.
>>>=20
>>> If I remove the index all queries return correctly, but its way too =
slow.
>>>=20
>>> Also, I have one machine that this seemed to occur on but I deleted =
the entire collection and the newly created one is fine. I cannot do =
this to my production server however.
>>>=20
>>> Any ideas what is causing this? I have confirmed the issue by =
querying via commandline and via Java code.
>>>=20
>>> Liz
>>>   =20
>>=20
>>=20
>>=20
>> =20
>=20


--Apple-Mail-83-280226739--