Re: Saving derived views

[email protected] wrote:

> The basic question is "How can you store a derived view?".

This is technically tricky due to an age-old attached-vs-unattached 
view design choice.  It has to be copied to get stored.  In C++, there 
is an insertAt operator which can insert al rows/subviews of a view in 
one go.  In Python/Tcl, it is more work and slower, as the copy must be 
done explicitly.  The other tricky aspect is that you have to figure 
out the required structure and define the destination view to have that 
structure.

<Insert picture of me stammering that in hindsight it should all have 
been done differently - and will be when I get to redoing and 
completing MK from the ground up>

> What I'm currently faced with is a very large storage (about 2.5 
> million rows) and I would initially like to "subset" this into another 
> storage file.

That's precisely the scenario where I'd like to make things easier.  In 
a recent 2.4x release of MK, I made a start with a more generic form of 
serialization.  The hooks are there to load and save partial pieces of 
storages (subtrees to be precise), but not all details have been worked 
out, let alone implemented.

> Beyond that, it looks as though I'll ultimately be looking at a 
> storage with that many rows and, say 20,000 properties per row.  And 
> I'll want to create and save derived views from that as well.

Ouch.  It sounds as if you're taking the wrong direction - quite 
literally, I mean: why repeat in a horizontal (named columns) 
direction, when MK has a much more efficient vertical (numbered rows) 
direction and subviews?

What I mean is, why use:
	view[a1:I,a2:I,a3:I,...]
when you can do:
	view[all[a:I]]
and use double indexing?

It's bound to scale better than the first, especially for 20K property 
names.  If the types are not all integers, then you could use:
	view[ints[a:I],strings[a:S],floats[a:F]]

-jcw

_____________________________________________
Metakit mailing list  -  [email protected]
http://www.equi4.com/mailman/listinfo/metakit