Re: Re[2]: newbie question - writing derived view back to db

On Tue, 18 Jan 2005 19:19:51 +0100, Marcin Krol <[email protected]> wrote:


> One could argue that I should exploit the column-wise data
> organization of MetaKit, but frankly, I don't have the faintest idea
> how to do it. And again it's probably non-trivial, given that e.g.
> file object iterators in Python obviously work in record/line oriented
> manner (though I didn't have them in this case, all data was in memory
> to prevent disk access trashing between the text file and the db and
> thus to isolate performance of db as much as possible), so there's a
> sort of mismatch between organization of the usual data sources and
> the data organization of storage.

i have the impression that although there is some documentation of  
metakit's column-oriented storage, there is very little knowledge about  
(1) why this should be a good idea, and (2) how to exploit this fact and  
in what use cases. some clarification of these two-and-a-half issues would  
be very welcome.

personally, i feel that i am working in row-oriented mode almost 100% of  
the time, the sole exception being when i imagine i have a collection of  
many objects and discover i want to add something to each of them i had  
forgotten -- then, obviously, adding a column in metakit would be less  
costly than in a row-oriented format.

however, the way things tend to be traditionally handled, this is either a  
design-time event that occurs not less often the more advanced the work in  
progress is. so any outstanding performance of a dbms to handle this  
particular situation is largely love's labour and lost for run-time.  
moreover, *if* there is a situation when aspects of the stored data  
objects should be made run-time expandible, *then*, surely, adding  
references from separate tables (pre-configured to hold such configurable  
data items) is more like the traditional solution. the alternative is to  
give run-time users the ability to modify the very structure of the data  
objects, something that will probably kill a majority of contemporary  
data-centric applications with the first bullet. (one might add that such  
additions will hardly typically not be of a nature that may be modelled by  
the introduction of a single atomar data field to each existing data  
object -- rather, new facts become apparent that have to be labelled and  
added at appropriate numbers to the select objects they pertain to.)

for these reasons i am much more interested in things working effectively  
with row-based access, no matter how they are implemented internally. let  
me remark that this particular point has made me look closer into sqlite  
as an alternative to metakit. while i am at it, i might further say that  
the other point is that i found that successive fetches of rows from a  
metakit table -- which, as i understand, is always entirely kept in  
memory, is outperformed by one or two orders of magnitude by an approach  
that first mirrors an entire table (with about 1k objects) into a python  
dictionary and then redirects fetches to this cache (meaning if you need  
more than 1% of the rows, you had better read *all* of them first). in  
effect, i then have two complete editions of one table in memory, which  
bothers me. i have not came about to trying these things with  
blocked/ordered/sorted views, one reason being my recent preoccupation  
with alternatives, the second being the somewhat awkward protocol i have  
to achieve in order to do so and the strange restriction of metakit that  
ordering is only possible on the 'first n' columns, and the third being my  
inability of figuring out how to effectively store derived views.

let me finish this longish posting by stating that i believe that metakit  
is a great product, but i also believe that its python interface needs at  
the very least a sound overhaul (i think i could contribute parts of an  
interface i have been building during recent months). also, it is hard to  
see why i should subscribe to a database working in memory that is  
outperfomed by a standard dictionary, something that i could have achieved  
by merely pickling my data. pointers anyone?

>>>> import mbs3
i am curious about this -- would you disclose your sources for us, marcin?

_wolfgang




-- 
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/

_____________________________________________
Metakit mailing list  -  [email protected]
http://www.equi4.com/mailman/listinfo/metakit