Re: [Evolution-hackers] Moving the struct instance heap space to mmap

Hmm, will it really reduce memory usage though?  I'm not so sure.  Remember
that although the summary file contains all strings on disk, in memory they
are actually unique-ized.  For a lot of load-types (e.g. mailing lists),
this saves a trememdous amount of memory - even after all the overhead of
the hashtable to manage it.  Address and subject strings in particular.  And
a mmap file isn't terribly different to performance to malloc'd memory -
once the latter is setup, in both cases the kernel is the one loading it off
disk (swap, or filesystem).

(an mmap file IS just memory, so if "mmap'd file size + in-memory support
tables" > "in-memory version", then you haven't generally won - even if 'ps'
shows you have).

Note also that all of the summary items, and all of the strings thus stored,
are using special allocators which reduce - significantly - the overhead of
malloc, on these small allocations.  In earlier versions I tried a different
string allocator for each messageinfo which mimicks much of what you're
doing here, but in memory.  Strings were stored in an array structure, but I
only stored a single pointer to the base of the array, and packed the
strings into a NUL separated right-sized buffer, and searched each time I
needed to access them, overhead was only about 5 bytes (pointer to the
'array' plus an array 'limit').   Performance was fine, but memory usage was
significantly larger than the current model, where strings are 'unique-ized'
- like 30% or more iirc.

Having extra indices on disk - you're really just writing a datbase table -
and all of the associated complexity, consistency and coherency issues that
implies.  And unless you do something 'proper' like a btree with a unique
key, you're going to have to end up storing plenty of extra support stuff in
memory anyway, e.g. a hashtable to find items by uid, the sorted list of
messages for the view, and so on.

We did a lot to reduce per-message overhead at the camel level, but its an
area of diminishing returns - the ui table view will still take as much
memory, etc.


On 08/09/06, Philip Van Hoof <[email protected]> wrote:
>
> On Thu, 2006-09-07 at 21:14 +0200, Philip Van Hoof wrote:
>
> > To read message n, you would simply do something like:
> >
> > from = sstart + *(istart + (sizeof (int) * 4 * n) + 1)
> > subject = sstart + *(istart + (sizeof (int) * 4 * n) + 2)
> > to = sstart + *(istart + (sizeof (int) * 4 * n) + 3)
> > flags = sstart + *(istart + (sizeof (int) * 4 * n) + 4)
> >
>
> No no no no no. I meant (something like) this of course:
>
>
> unsigned char *sstart = (uchar*)mmap(..), *istart = (uchar*)mmap(..);
> #define AMOUNT_OF_OFFSETS_PER_RECORD 4
> #define AOO AMOUNT_OF_OFFSETS_PER_RECORD
>
> from = sstart + *(istart + ((sizeof (uint32_t) * AOO * n) + sizeof
> (uint32_t)))
> subject = sstart + *(istart + ((sizeof (uint32_t) * AOO * n) + sizeof
> (uint32_t)))
> to = sstart + *(istart + ((sizeof (uint32_t) * AOO * n) + sizeof
> (uint32_t)))
> flags = sstart + *(istart + ((sizeof (uint32_t) * AOO * n) + sizeof
> (uint32_t)))
>
> To much pointers in my head ;)
>
> --
> Philip Van Hoof, software developer at x-tend
> home: me at pvanhoof dot be
> gnome: pvanhoof at gnome dot org
> work: vanhoof at x-tend dot be
> http://www.pvanhoof.be - http://www.x-tend.be
>
>

_______________________________________________
Performance-list mailing list
[email protected]
http://mail.gnome.org/mailman/listinfo/performance-list