Re: Merging: was Re: Semantic Bleachers: was Re: Afghanistan War Diary as topic map in Maiana

Andrew,

On Tue, 2010-11-02 at 22:26 +0000, Andrew S. Townley wrote:
> Hi Patrick,
> 
> I'm not as good at snipping as you are... ;)
> 

;-)

Last one for today though. Editing awaits.

> On 2 Nov 2010, at 8:34 PM, Patrick Durusau wrote:
> 
> > Andrew,
> > 
> > Two of three I think.
> > 
> > On Tue, 2010-11-02 at 19:59 +0000, Andrew S. Townley wrote:
> >> On 1 Nov 2010, at 6:55 PM, Patrick Durusau wrote:
> >> 
> >>> Andrew,
> >>> 
> > 
> > <snip>
> > 
> >>> I think the creative fashioning of merging rules, with boundaries for
> >>> their application is largely unexplored territory. 
> >>> 
> >>> We would benefit from a very robust set of comparison operators. 
> >> 
> >> That's what I was thinking too, but I wasn't sure.  Again, I don't see how you can deal with Internet-scale datasets if you don't have these type of "overlay" facilities that can be linked to specific contexts.  Simple ones are "I'm user X" and "She's user Y", but others could be a lot more sophisticated.
> >> 
> >> However, from personal experience, this stuff gets pretty complicated (and CPU intensive) pretty quickly.  Still working on ways to deal with that issue to my satisfaction...
> >> 
> > 
> > True but as I have been posting to my blog, there are techniques for
> > clustering data and then performing additional operations on data that
> > looks interesting.
> > 
> > In other words, merging isn't a one stage process or even one that
> > requires all instances be compared. 
> 
> I wish I had time to read (m)any blogs, so don't feel offended when I say I haven't been keeping up with yours. :)
> 
> I guess I didn't explain myself very well (and this follows through to you the nature of your response below a bit too, I think).  Sure, you can do all kinds of intermediate clustering, staging, caching and the like (and we are, btw), to reduce the number of operations that are actually performed to resolve any particular query or proxy request.  That isn't what I'm talking about.
> 
> I'm actually talking to types of complexity.  One is simply the number of nodes and arcs required to capture the context that I'm talking about.  If this context applies to a large number of nodes in the map, then you've actually no choice but to traverse/resolve/filter on these relationships.  At the moment, I draw parallels between this problem and data normalization in an RDBMS.  Sometimes, you've just got to denormalize a bit to have a responsive system, and this is a conscious architectural trade-off that you make.  Which intermediate, cached, or other useful representations you need to generate to fulfill any particular request of the map is going to also be based on the architectural trade-offs that you make.
> 
> The problem I was highlighting is that as the complexity and interlinking of nodes in your graph get greater, your risk of things getting out of sync also increases (as does the time it takes to rematerialize any of these views).
> 
> One of the axes of context present for me is related to access control and authorization, so that is context that's pretty critical to ensure you get right.
> 
> I'm not saying I have the right balance yet, and I'm certainly not saying I'm not potentially doing some stupid things in either the modeling or the implementation.... It just happens to be something that I hit recently, and whether or not it impacts you depends a great deal on your model, your topic map legend and the implementations under the covers.  The current state of mine is moving from starting from the "correctness of model" perspective to the "it actually has to be, well, you know, usable, responsive, robust and secure" perspective... ;)
> 
> Of course, it also depends on what you mean by "merge" and when that operation is performed, as you say elsewhere.
> 

Ah, ok, sure, different types of complexity.

Some due to size, modeling, other operational requirements.

Some due to subject identity tests, particularly when subject identities
are composed of subject identities, which are themselves subject to
merging. Can get complicated in a hurry. 

But as you say, it really is a matter making decisions on the basis of
operational requirements. Some merging may simply be "good enough."
Merging that results in inputs to X-ray equipment probably should have
stricter requirements. ;-)

I don't think of topic maps or subject identity tests as being complex
but rather that subject identity and handling it adequately as being
complicated. 

Hope you are having a great evening!

Patrick