Re: mRFC 0024: Full text indexing in Midgard

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Jukka,

- --Jukka Zitting wrote on 2006-01-02 20:15:
> I've just posted a proposal for introducing full text indexing in Midgard 
> core. It's available as mRFC 0024: Full text indexing in Midgard 
> (http://www.midgard-project.org/development/mrfc/0024.html).

A few comments (especially in respect of current MidCOM indexing semantics):


> The proposed full text index shall be a Lucene index directory residing on 
> the host that runs the Midgard applications. If there are multiple web 
> frontend hosts accessing a shared backend database, then each frontend shall
> have its own full text index.

What is a "web frontend" in this context? A Host?

You are saying that "Each Midgard database shall have its own full text index".
This is a bit of a change compared to the current situation where each logical
site (which could be driven by more then one host) has its own index. You can
basically configure this using the indexname configuration directive (which
defaults to hostname:port).


> Parameters and attachment are treated as object properties in the full text
> index. Parameters are included as virtual properties named
> "parameter:domain:name", and attachments (whose content can be extracted) as
> properties named "attachment:name". All link fields are stored as GUID entries
> in the full text index.

Right now, this'll change indexed field names. The current index is not tied to
the MgdSchemas but to the Datamanger one's, at least in all cases where indexing
is fully automated. The main reason for this is, that DM* will allow you to
define your own custom additional fields and have simple names for it.
Regardless *where* the data is actually stored. The uniqueness constraints on
the DM field name listing makes indexing them safe in this context.

I generally think that I like this example, but we just need to take this change
of semantics into account when updating to the new Index to change all existing
applications accordingly.

The main problem I see here is that I currently take advantage of this
distinction: Several components I have index a "type" key which is not
neccessarily stored in the same object field everywhere (they could even be
derived values). They are used in more generic query interfaces which allow
optional limiting to a single object type. In the current solution I just add
the two restrictions (component and type), which takes the main logic of
distinguishing the various components/types into the indexer.


> MultiLang content shall be handled by prefixing all the MultiLang property names
> with the language code. A German version of a "title" property would be indexed
> as "de:title" while the default "title" property (MultiLang zero) would be
> indexed simply as "title". Also the general node content field will be versioned
> by language.

Shouldn't multilang be queried implicitly?

My current integration idea would contradict this API: I planned to have a
distinct document for each language available in the index. The framework would
add that language field constraint automatically then.

The only (performance) problem I currently see here is the known cavet of the
Lang 0 fallback semantics, but since I have to post-process the entire resultset
anyway (ACLs and the like), that should not be too hard in that respect.

In all cases the queries would always be independant of the actual object
language, you always query "title", not "lang:title". If you want to query only
german results, you'd query for "title" and "__LANG".


> The parent_down and parent_up fields contain special depth-first traversal
> indexes of the object within the global content tree. This allows for very quick
> determination of ancestor relations using predicates like:
> 
> A is an ancestor of B :-
>     A.parent_down < B.parent_down,
>     A.parent_up > B.parent_up.

I'm not sure if I fully understand this. I keep asking myself "why not having
just a depth integer", but i have the distinct feeling that I miss something here.


> "The query implementation"

Sidenote: The first paragraph of "The query implementation" seems incomplete.

What about the main target area for the indexer: Object-independant site searches?

A qb bound indexer doesn't help me a bit if I have to create a QB instance which
is by definition bound to a single type.


Live long and Prosper!
Torben Nehmer

- --
Torben Nehmer, Guenzburg, Bavaria, Germany
http://www.nathan-syntronics.de, mailto:[email protected]
PGP Public Key: https://www.link-m.de/pgp/t.nehmer.asc
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (MingW32)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFDujemJPh4Kn6d5FYRAlexAKCgPDS69iyfYK87FSfhKunHmdWlCQCgm5R5
zs67xnf5vAyGwQts+Zja8VU=
=V4gf
-----END PGP SIGNATURE-----