Re: Saving derived views

mARK bLOORE wrote:

> i had a design where i was essentially duplicating a folder tree by
> storing the folder's path in a string, and the files in it in a
> subview:
>
> stuff[folderpath:S,files[id:I,name:S]]
> (and other data, of course.  note that it is not fully heirarchical,
> just two levels.)
>
> i found that when there were many rows (say 100,000) it was very slow
> to open.  it seems to go through them all to get some metadata on the
> subviews.

Yes.  With 100,000 subviews things can bog down.  The empty case is 
treated specially, but still.  A number of performance tweaks have been 
made over the past year to address this.  More are needed, there is 
some over-zealous memory buffer allocation going on as well.

> (a sidenote:  it looks like metakit couldn't do a real folder tree,
> since there  is no type which says "this is an arbitrary, dynamically
> allocated, metakit type".  i'm not sure such flexibility would be a
> good thing anyway, though the "store by columns" design would probably
> make it fairly easy.  hmm, maybe not such a bad idea, that could
> extend metakit from a relational database into the realm of
> object-oriented DBs.)

Yes, arbitrary subview structures is up there on my list.  I even have 
a notation for it ("view[*]") and have worked out most details w.r.t. 
restructuring: restructuring needs to be allowed on views, not just 
entire storages.  Doing so automatically redefines the parent view as 
having type "*", while restructuring the parent becomes a way to bring 
heterogeneous subview structures back into homogenous shape.

But there is a way out for simple nesting.  A relatively little known 
feature of MK is the ability to define self-recursive structures:
	blah[foo:S,yes[^],bar:S]
The ^ stands for "same structure as parent", so this can handle 
homogenous trees.  It's not necessarily always most convenient thing to 
do, a row (or tree node in this case) will need a variable amount of 
nested indexing to reach from the root.

Another efficient way to store a tree in a view, as used in CatFish and 
in Starkits, is to store either the parent or the child relationships 
explicitly:
	dirs[parent:I,name:S,files[name:S,date:I]]
Where "parent" is a row index into dirs, or -1 for the root.  This 
structure turns out to scale surprisingly well for filesystem-like data 
collections.  In this approach, <parent,name> is unique.

The general guideline is to look for a balance between # of rows in 
each view/subview and the total number of data columns present in a 
storage, since columns are the basic objects managed by MK.

-jcw

_____________________________________________
Metakit mailing list  -  [email protected]
http://www.equi4.com/mailman/listinfo/metakit