Re: mRFC 0024: Full text indexing in Midgard

Hi,

On 1/3/06, Torben Nehmer <[email protected]> wrote:
> What is a "web frontend" in this context? A Host?

A host. As in a setup where the database is located on a separate backend host.

> You are saying that "Each Midgard database shall have its own full text index".
> This is a bit of a change compared to the current situation where each logical
> site (which could be driven by more then one host) has its own index. You can
> basically configure this using the indexname configuration directive (which
> defaults to hostname:port).

The idea here is to follow the current query builder model as closely
as possible to avoid introducing extra data structures. The tree model
support is introduced to make it easy to constraint queries to just
selected parts of the content tree.

> Right now, this'll change indexed field names. The current index is not tied to
> the MgdSchemas but to the Datamanger one's, at least in all cases where indexing
> is fully automated. The main reason for this is, that DM* will allow you to
> define your own custom additional fields and have simple names for it.
> Regardless *where* the data is actually stored. The uniqueness constraints on
> the DM field name listing makes indexing them safe in this context.

I'm relying on MgdSchema as the default mechanism for specifying the
fields so I don't see a need for an additional mapping for the
parameter names. If MidCOM were to use the proposed full text index,
then I suppose the best option would be to do the name mapping at the
PHP level where you have the Datamanager schemas available.

> I generally think that I like this example, but we just need to take this change
> of semantics into account when updating to the new Index to change all existing
> applications accordingly.

I think we'll need to have a rather long migration path for that, as
there is considerable difference between the MidCOM indexer model and
the proposed one. At the moment I'm targeting for a shared approach
where the indexer process would take care of *both* the MidCOM and the
core indexes. Hopefully by Midgard 2.0/MidCOM 3.0 we could be in a
situation where the indexes could be merged.

> Shouldn't multilang be queried implicitly?

My idea was to use the lang setting in the query builder to
automatically prefix unprefixed MultiLang property names. For example:

    $qb->set_lang("de");
    $qb->add_constraint("title", "CONTAINS", "foo"); // queries de:title

In fact I should probably use some other separator than colon as it is
already used in the Lucene query syntax.

> > The parent_down and parent_up fields contain special depth-first traversal
> > indexes of the object within the global content tree. This allows for very quick
> > determination of ancestor relations using predicates like:
> >
> > A is an ancestor of B :-
> >     A.parent_down < B.parent_down,
> >     A.parent_up > B.parent_up.
>
> I'm not sure if I fully understand this. I keep asking myself "why not having
> just a depth integer", but i have the distinct feeling that I miss something here.

It's the nested set model for storing hierarchical information in a
relational database (see
http://threebit.net/tutorials/nestedset/tutorial1.html). It gives you
a single query implementation of the "ancestor of" and "descendant of"
predicates. A depth integer doesn't cut it as it doesn't distinguish
between separate branches of the content tree.

> Sidenote: The first paragraph of "The query implementation" seems incomplete.

Ah, sorry. Should end with: "The CONTAINS operator specifies a Lucene
full text query for the specified property:"

> What about the main target area for the indexer: Object-independant site searches?
>
> A qb bound indexer doesn't help me a bit if I have to create a QB instance which
> is by definition bound to a single type.

Ah, a major omission in the spec! My plan was to allow queries on
"midgard_object" which would search all the MgdSchema types:

    $qb = new MidgardQueryBuilder("midgard_object");
    $qb->add_constraint("metadata:ancestor", "=", $root_guid);
    $qb->add_constraint("*", "CONTAINS", "foo");
    $qb->execute();

The returned objects would of course be correctly typed
(midgard_article, midgard_event, etc.).

I'll revise the spec to match these additions and clarifications.

BR,

Jukka Zitting

--
Yukatan - http://yukatan.fi/ - [email protected]
Software craftmanship, JCR consulting, and Java development