Re: Merging: was Re: Semantic Bleachers: was Re: Afghanistan War Diary as topic map in Maiana

Hi Patrick,

I'm not as good at snipping as you are... ;)

On 2 Nov 2010, at 8:34 PM, Patrick Durusau wrote:

> Andrew,
> 
> Two of three I think.
> 
> On Tue, 2010-11-02 at 19:59 +0000, Andrew S. Townley wrote:
>> On 1 Nov 2010, at 6:55 PM, Patrick Durusau wrote:
>> 
>>> Andrew,
>>> 
> 
> <snip>
> 
>>> I think the creative fashioning of merging rules, with boundaries for
>>> their application is largely unexplored territory. 
>>> 
>>> We would benefit from a very robust set of comparison operators. 
>> 
>> That's what I was thinking too, but I wasn't sure.  Again, I don't see how you can deal with Internet-scale datasets if you don't have these type of "overlay" facilities that can be linked to specific contexts.  Simple ones are "I'm user X" and "She's user Y", but others could be a lot more sophisticated.
>> 
>> However, from personal experience, this stuff gets pretty complicated (and CPU intensive) pretty quickly.  Still working on ways to deal with that issue to my satisfaction...
>> 
> 
> True but as I have been posting to my blog, there are techniques for
> clustering data and then performing additional operations on data that
> looks interesting.
> 
> In other words, merging isn't a one stage process or even one that
> requires all instances be compared. 

I wish I had time to read (m)any blogs, so don't feel offended when I say I haven't been keeping up with yours. :)

I guess I didn't explain myself very well (and this follows through to you the nature of your response below a bit too, I think).  Sure, you can do all kinds of intermediate clustering, staging, caching and the like (and we are, btw), to reduce the number of operations that are actually performed to resolve any particular query or proxy request.  That isn't what I'm talking about.

I'm actually talking to types of complexity.  One is simply the number of nodes and arcs required to capture the context that I'm talking about.  If this context applies to a large number of nodes in the map, then you've actually no choice but to traverse/resolve/filter on these relationships.  At the moment, I draw parallels between this problem and data normalization in an RDBMS.  Sometimes, you've just got to denormalize a bit to have a responsive system, and this is a conscious architectural trade-off that you make.  Which intermediate, cached, or other useful representations you need to generate to fulfill any particular request of the map is going to also be based on the architectural trade-offs that you make.

The problem I was highlighting is that as the complexity and interlinking of nodes in your graph get greater, your risk of things getting out of sync also increases (as does the time it takes to rematerialize any of these views).

One of the axes of context present for me is related to access control and authorization, so that is context that's pretty critical to ensure you get right.

I'm not saying I have the right balance yet, and I'm certainly not saying I'm not potentially doing some stupid things in either the modeling or the implementation.... It just happens to be something that I hit recently, and whether or not it impacts you depends a great deal on your model, your topic map legend and the implementations under the covers.  The current state of mine is moving from starting from the "correctness of model" perspective to the "it actually has to be, well, you know, usable, responsive, robust and secure" perspective... ;)

Of course, it also depends on what you mean by "merge" and when that operation is performed, as you say elsewhere.

> 
>>> 
>>>> I think you alluded to this the other day with some of your work, but I think that the idea behind what Manina is doing, what I understood was possible with what you have and the way that my implementation handles merging on a conditional rather than absolute basis illustrates that there's still work to be done in this area of the topic maps specifications.  Maybe some implementations take too literal a view on the whole XTM/TMDM merging requirements, but it makes a big deal if you define "equality" on the basis of property values, merge and then one or more of those property values change.
>>>> 
>>> 
>>> Well, those are separate questions:
>>> 
>>> a. How to merge on something in addition to the standard TMDM basis?
>> 
>> Or, in our case, "in spite of" or "happily ignoring" the TMDM basis... ;)
>> 
> 
> Well, maybe yes, maybe no.
> 
> The TMDM recognizes that there is likely to be merging beyond what it
> prescribes. 
> 
> And it would be possible to represent a complex set of merging
> conditions with the addition of a URL as subjectIdentifier so that
> fairly complex merging could be accomplished with fairly mundane topic
> map software. 
> 
> That is: Complex merging condition A, B and C, then add URI X to a
> topic. 
> 
> One of the topics (sorry!) that hasn't been discussed (mostly because it
> could not be standardized) is the pre-processing stage or even authoring
> stage for topic maps. 

Sorry, I guess tiredness caused me to be overly cheeky here.  Still, your example is interesting, and, if I understand what you wrote correctly, you're actually using a subjectIdentifier as additional context?  Or are you just saying that you do some sort of system-specific merging outside the TMDM and then for selected result proxies, you're adding subjectIdentifier references so that normal TMDM/XTM merging can be used to generate the desired result?

Probably the latter, right?

I certainly am not trying to prescribe overly-descriptive constraints on particular implementations or authoring techniques.  To me, these are really where the areas of innovation are in terms of distinguishing particular implementations that are based on the same fundamental models so that, at least at some base level, some core interoperability and portability of information remains both possible and plausible.

Anything else would get kinda boring, I think.... ;)

> 
>>> 
>>> b. What happens if after merging a value changes? (unexplored so far as
>>> I know, but good question)
>> 
>> The way I see it, "merging" by definition is just another context of viewing a particular dataset.  What's available and the legend will define the outcome.  This outcome is totally linked to the context, although the validity of the context can be "until further notice" or "from timestamp T where properties [...] also apply."
>> 
>> Complexity explosion?  Absolutely, but, again, I think it's essential to work through these scenarios.  "Truth" is relative, and the results of any view of any data set on any given day represent one particular view of "truth" in that domain.  Expand the scope and scale, and the differences between "truth" start to be interesting as artifacts of themselves, but without the traceability and the ability to do "what if" scenario modeling, this can be very difficult to determine.
>> 
>> Would really be interested in understanding what large TMDM users with real data do in these sorts of scenarios, or if they're just ignored.  Has anyone done any kind of analysis of the way the specs are applied and the corresponding trade-offs that result from going down these paths?
>> 
>> I'm not a fan of "best practices" because that tends to end up being a fluffy middle-ground of ambiguity that keeps people able to report upwards that "we're following best practices" when results are questioned.  What I think we need are more detailed analysis of the issues people are facing trying to employ topic maps (of any flavor) to solve interesting (and/or business) problems beyond single examples published at topic maps conferences.  Has this been done?  If so (and I hope), I'd love to see it, because I think it would be fascinating reading.
>> 
> 
> I will have to let others speak for detailed analysis of large user
> bases. I don't recall seeing such information in the literature. 
> 
> Complexity explosion? Perhaps but then one doesn't have to done all the
> merging that is possible. 
> 
> When you use a relational database you don't do joins on every table
> simply because you want to do a join on two tables.

Again, I didn't state this well.  Complexity in terms of the schema/model/ontology/graph and determining which ones are actually important to accessing a specific container/proxy/topic or attribute/property/occurrence.

> 
> Much the same should be true for topic maps. Yes, there may be way more
> merging possible than we can perform but if it isn't of any interest,
> say I am looking for cheap tickets to Bangkok, then merging of
> alt.politics posts probably isn't relevant for me. 

No arguments here for sure! :)

> Hope you are having a great evening!


With an interesting and lively topic maps discussion after two positive days on the road, how could I not! :)

Cheers,

ast
--
Andrew S. Townley <[email protected]>
http://atownley.org