Re: [NeoStats-Devel] [Commits] r2699 - in trunk: include src

M wrote:
> Justin Hammond wrote:
>>Now, regarding that "path" I was talking about. 
>>GDBM is currently consuming around 1.8Mb of memory for each 
>>open Database.
>>When you have a lot of modules opened up, that memory adds up 
>>(6Mb roughly if we say Set + modexclude + one table for each 
>>module) and go even higher with more tables. I believe its 
>>relating to the cache settings of GDBM which is fine (because 
>>the memory useage only jumps when we open a database and 
>>drops again when we close it. Found that out after hours of 
>>stepping through code with valgrind and gdb).
>>
>>But I was thinking to trim that memory useage down, can we 
>>close some of the databases (say the Set/modexclude Database 
>>after loading or changing a
>>value?) or Potentially put in a timer that will close 
>>"unused" databases after a period of time. 
> 
> 
> We can potentially keep most of the databases closed, most of the time.
> Currently my DBA layer assumes that a database is open for certain
> operations and errors if not but the general read is used to open them
> rather than an explicit open. There is no reason I cannot always open on
> demand but there would obviously be some performance loss in doing so if the
> database was likely to be closed. 
> 
> Since the majority of database access is config, for our current systems, an
> open read close at boot would suffice for most operations. Changing of
> settings would then also likely need an open write close sequence which is
> where we would suffer performance wise for config info. Other than first run
> installations, this is I guess unlikely to be a big issue since config
> settings are unlikely to change that often. 
> 
> A similar approach could be assumed for data lists such as excludes and
> access lists.
> 
> Data is generally adhoc access so might need to be more module specific E.g.
> StatServ would need to open writeblock close or open writeblock close within
> it's own code. A timer might work, but the chances of us timing a close at
> the right time from the core are likely to be slim since whatever time
> period we choose might be just before it gets reopened anyway giving us
> worse performance. StatServ is the most database intensive module so if we
> did not cache reads, but just read the whole database, we could manage with
> a block open/read/close at boot then just block open/write/close at runtime.
> Currently IIRC StatServ reads from the databases during runtime when
> channels are created.

It may be better to have a persistent setting like the bots do.
that way db's like StatServ and SeenServ which do access the data 
continuously don't get the performance hit of opening and closing all 
the time. if the flag is put on the db itself, then the config and 
excludes for those modules could still be closed, and the main data 
files remain open.

DNB