Re: TMQL - Proposal for a %base directive

Hi Lars,

On 29 Apr 2011, at 3:46 PM, Lars Heuer wrote:

> Hi Andrew,
> 
>> My (perhaps incorrect) understanding of one purpose of the tmql
>> local ontology was so that you could potentially assign additional
>> subject indicators to "local" identifiers to allow you to extend and
>> interpret the information in the effective map in particular ways to
>> ensure consistent query results.  The tmql:ontology type itself I
>> understand to represent entire namespaces.
> 
> I think that's almost correct.
> 
>  %prefix foo <http://psi.example.org/onto>
> 
> becomes (acc. to my understanding) in the effective map:
> 
>  foo isa tmql:ontology;
>     <http://psi.example.org/onto>
>  .
> 
> If the TMQL processor finds a QName, it asks the effective map for the
> prefix and concatenates the local part of the QName with the subject
> identifier.

From the usage in some of the unfinished supporting documentation, it would've seemed to me that you were also able to do some further definitions in-line, so it wasn't limited to just what could be retrieved and merged externally[1].

> It's the same mechanism which is used in CTM and other syntaxes, but
> with the nice side effect that you can query the effective map for
> 
>     //tmql:ontology
> 
> to retrieve all prefix definitions. At least the idea is nice, I am
> unsure if it has any advantages over "normal" prefixes like the ones
> used by CTM. IMO it simply adds unnecessary complexity. I am sure rho
> disagrees. :)

I think the flexibility in [1] is why you need to be able to more directly interact with artifacts from the ontology directly rather than being able to manipulate the prefixes.  Also, in CTM, etc. there doesn't seem to be any way to do full introspection from within the environment.  Usage of the tmql:ontology type as specified allows you to do this sort of thing directly.

All the above is inference and supposition on my part, however.

> I think it's possible to translate the %base directive as well to
> Topic Maps.
> 
>   %base <http://psi.example.org/base/>
> 
> could become:
> 
>   <http://psi.example.org/base/> isa tmql:ontology.
> 
> So, a tmql:ontology without a local identifier (item identifier) can
> be used as base IRI.

At first, I'm not sure I agreed with these semantics, but I think you're effectively right.  You're explicitly qualifying local identifiers.  However, unless you're doing so from the perspective of a fixed, externally-accessible address (for some relatively long-ish definition of fixed sans 30x redirects), I'm not sure that I see the advantage of doing this in practice.

Unless you're specifying an overlay ontology with your base directive that isn't reflected in the effective map, wouldn't the effective ontology of your query match the effective map by default anyway?  If they didn't have subject indicators, the interpretation of the semantics is context-specific; if they did have subject indicators, then you'd be able to leverage these to ensure fixed semantics because the local identifiers in the effective map would resolve to the same items as the local identifiers in the TMQL query.

Given that most of the rest of you seem to prefer fully-qualified names, that means you're likely to need to use multiple QNames in a given query anyway, so I'm not 100% sure I see the value in this use case.

For me, the same effective map can be potentially accessed through a number of different relative and absolute addresses, so in order to ensure relationships between maps for cross-map links remain consistent for a given starting point, the concept of a base context is pretty necessary.  "Relative" rules in my world.  Absolute references are deferred until the last possible moment as best I can to ensure the first sentence remains true.

However, for a fully-qualified, "flat world" view, I don't see how this would be of much value.  Obviously, I'm missing something.  The previous statements are made with the full realization that the "end user" view of a query and effective map is not necessarily the same view seen by the query processor or map persistence store, as all prefixed names would be expanded during query evaluation.

> [...]
>> This would then influence the resolution of any literal,
>> unqualified identifier appearing anywhere in a given query, right?
> 
> It would effect any IRI and unqualified identifiers, yes.
> 
>   %base <http://psi.example.org/onto/>
> 
>   select $album
>   where is-produced-by (production: $album, producer: tom-waits)
> 
> 
> would become:
> 
>   select $album
>   where ^<http://psi.example.org/onto/#is-produced-by>(
>         ^<http://psi.example.org/onto/#production>: $album,
>         ^<http://psi.example.org/onto/#producer>:
>         ^<http://psi.example.org/onto/#tom-waits>)
> 
> 
> (^ indicates an item identifier in CTM)

A related question:  what would be the IRIs of the returned items? Aren't they still defined in terms of the effective "base" IRI of the effective map defined by the method of access?

> Given that, I can take my query and go from TM A to TM B without
> re-parsing the query.

I'm not sure I really understand the implications of this statement.  The rules of the TMQL processing environment and the effective environment map are defined in the TMQL draft.  What you're talking about is resolution of surface syntax, not the semantics of the query.  Except for the difference of potentially 8 extra keystrokes by defining some prefix, I don't see why this is important, e.g.:

%prefix a ...

select $album where a:is-produced-by...

[bear with me as this is continued below]

> In tolog I cannot do that. A query is always
> bound to a topic map. If the topic map is stored at
> <http://www.example.org/map/a> the above query becomes:
> 
>   select $album
>   where ^<http://www.example.org/map/a#is-produced-by>(
>         ^<http://www.example.org/map/a#production>: $album,
>         ^<http://www.example.org/map/a#producer>:
>         ^<http://www.example.org/map/a#tom-waits>)
> 
> If the TM is stored at <http://www.example.org/map/b> it becomes:
> 
> 
>   select $album
>   where ^<http://www.example.org/map/b#is-produced-by>(
>         ^<http://www.example.org/map/b#production>: $album,
>         ^<http://www.example.org/map/b#producer>:
>         ^<http://www.example.org/map/b#tom-waits>)
> 
> If the TM is stored at <http://www.example.org/map/c> it becomes ...
> well, I think you got the point... ;)
> 
> If it's possible to query multiple maps with TMQL, each identifier
> like "tom-waits" has to be resolved relative to each map IRI.

This kind of resolution is mostly what I'd expect.  I think the difference is when the expansion actually occurs (at least to me).  In the first instance, you're using the TMQL query environment (as per spec) to expand any references for you to some kind of fixed subject indicator that's potentially used across map environments.  Where it isn't in the effective map, you get zero results.

In the second instance, I wouldn't expect the TMQL environment to do *any* expansion as you described above in the parsing phase of the query.  When it is actually *executing* the query, it would need to resolve those relative identifier references to the current map, but only using the normal access methods offered by the map implementation and completely outside the bounds of whether the query was portable or not.

Sure, you could be devious and define the meanings of each of the item identifiers differently using direct subject indicators in map A from map B.  This would then cause the interpretation of your results to potentially not be what you expect.

However, based on the byte-sequence-as-identity implied by using only local, unqualified identifiers, the semantics of the *query* would remain the same across maps and be consistent within the bounds of TMQL.  The only difference I see is in the expectations of the semantics reflected by query results.  In reality, while similar, they're actually asking two different questions, hence the variability of the results in the second case.

> [...]
>> Also, I'm assuming you're not specifying ordering of the directive
>> relative to prefixes, but rather the resolution precedence?  In
>> either case, why the MUST qualifier?  Wouldn't QNames (at least as
>> defined in CTM/TMQL as prefix-substitution constructs, automatically
>> be orthogonal to local identifier resolution anyway?
> 
> Well, the "MUST" avoids problems.
> 
> Example:
> 
>  %prefix a <#foo>
>  %base <http://www.example.org/>
>  %prefix b <#bar>
> 
> 
> Here we have to resolve the prefix IRI <#foo> against some (undefined)
> base IRI (application dependent) while the prefix IRI <#bar> is
> resolved against our base IRI <http://www.example.org/>.

First, it would've never occurred to me to do something like this... :)

Second, I think the example above actually illustrates the issues with prefixes vs. proper QNames we've discussed before.  The fact that you're really doing string-concatenation means that you're potentially more likely end up with non-sensical identifiers after the prefixes have been resolved because there aren't really many constraints on what constitutes a valid prefix value.  The spec says QIRI, which seems odd to me, because then you could have multiple levels of prefix nesting that may or may not be resolved depending on the way the environment was implemented.

	[49]	prefix-directive	::=	 %prefix   identifier QIRI

This is one of the definitions that's admittedly made me scratch my head a bit because I would've expected an IRI here instead of QIRI.  It wouldn't have solved the problem, because you could effectively still use prefixes as partial aliases, but at least the intention may be more clearly stated.

Admittedly, I haven't tried very hard, but I couldn't immediately think of an example where a prefix using a QName would be useful.

> While that would work, I thought that having the %base at the top is
> more clear and wouldn't lead to surprises.

In my own experience, the only reason to use any kind of base definition is to reduce the typing burden or to apply additional context to something more generally defined.  Either way, it's still a mechanism to help the humans, so, as such, I just didn't see why you wouldn't let people use it as they saw fit.  I wouldn't really see how it would make much difference to the parser.

I haven't implemented the willy-nilly base directive in any of my software yet, so I can't say for certain that it wouldn't create more surprises than it was worth. :)  I've thought about it quite a bit, though!

> 
>> Also, presumably, if you specified a base directive, it would only
>> remain in effect until the logical "end" of a query session or
>> another base directive was issued, e.g:
> 
> Yes, but I wouldn't allow more than one base directive, conservative
> as I am. The base IRI should be immutable. For all other use cases,
> the prefix mechanism should be used (at least this should work for
> TMDM, maybe you'll disagree since you seem to use local identifiers
> very much).

If it's really about saving keystrokes, I don't see the value in making it immutable.  The only scope of applicability is within a given TMQL query or, by extension, some application-defined query session.  It doesn't apply to the effective map, or are you thinking it would?

>>> An alternative to the base IRI directive might be a more radical
>>> "no-identifiers policy", so a query must only use absolute IRIs (and
>>> QNames), since relative identifiers have a doubtful value anyway.
> 
>> I think this is a bad idea (at least from my perspective and
>> implementation). In my world, relative identifiers have very high
>> value. If I wanted to require fully-qualified URIs for everything,
>> I'd just use RDF. ;)
> 
> :) I see.
> 
> 
>> More seriously though, my general modeling approach is more similar
>> to the use of xs:include with W3C XML Schema in that I use local
>> identifiers where convenient and then qualify them as necessary
>> using either subject indicators or subject locators to provide more
>> concrete identity assertions.  This means that local/relative
>> identifiers do have value within the context of the query.
> 
> I see. Well, that seems to match the tolog local identifier resolution
> mechanism (if you interpret the base IRI of a topic map as local
> namespace). That implies that your tolog queries may no longer work if
> you change the base IRI (that holds also true for the proposed %base
> directive). I think that the %base directive makes the local namespace
> simply more transparent/obvious and provides the advantage that a
> query processor may be self-contained: It can resolve IRIs
> independently of the concrete topic map (and its base IRI).

Yes, I absolutely assume a local namespace, but not necessarily a base IRI (if that makes sense).  These can vary completely independently for me.

This approach allows exposing maps or views of said map through different effective base IRIs but not disrupting the relationships between the proxies themselves.  There are consequences if you're resolving things across maps and the maps "move" in unexpected ways by not being resolvable, but I can live with this because there are additional ways to solve this problem once it's been discovered.

I think one of the differences (highlighted above) is in the way we carve up the responsibilities of the query processor.

I've never used tolog, so I can't say how closely my environment matches tolog.  However, as I understand the TMQL specification, the syntax of the queries themselves should be portable across maps.  The questions come down to:

	1) Does the map actually contain local relationships to satisfy the query?  If not, no results.
	2) Do the semantics of those local relationships match the semantics implied by my query?
	3) Do I care more about the structure or semantics of the query result?

If tolog actually pre-resolves the local identifiers relative to the local map as part of the parsing phase, I'd view that as an over-eager optimization.

One of the main downsides I see with the base directive is that it might be assumed that the IRI is applied to the effective map instead of the query.  That could potentially be more confusing to users since fully-qualified references to resulting items may have a totally different base IRI than specified as part of the query.  Either way, I think it's a documentation issue around specifying this behavior as being expected.

> My use case was something like this:
> 
>   stmt = conn.prepare("some query here")
> 
>   stmt.context.add_map(<IRI-1-HERE>)
>   stmt.context.add_map(<IRI-2-HERE>)
> 
>   res = stmt.execute()
> 
>   stmt.context.remove_map(<IRI-1-HERE>)
> 
>   res = stmt.execute()
> 
> I can modify the effective map without taking any of the topic map
> base IRIs into account since the query stays always the same. I don't
> have to recompile/re-parse the query just because my context map has
> changed; the query is immutable.


The goal certainly makes sense, but I wouldn't see any reason that what I describe would preclude this sequence.  Like always, if it really, really matters, pre-qualify exactly what you want to happen as much as possible, e.g. use prefixes or fully-qualified IRIs.

Cheers,

ast

[1] http://kill.devc.at/system/files/language.html#id2482018
--
Andrew S. Townley <[email protected]>
http://atownley.org