RE: [NeoStats-Devel] [Commits] r2699 - in trunk: include src

Justin Hammond wrote:
> You can see the change I made to the DBA layer in DBAFini was 
> to call DBMCloseTable rather than DBACloseTable. Reason was 
> that GET_CUR_MODULE() was empty, hence the lookup dbname in 
> the hashlookup in DBACloseTable was just exclusions.gdbm, 
> which, obviously doesn't exist. (which is why I'm so 
> confused, because it creates NeoStatsexclusions.gdbm, 
> indicating at start, its running fine. I think this has 
> *something* to do with the other problems I've been seeing 
> about the event in services.c not getting fired for whatever reasons) 

GET_CUR_MODULE() should never be empty since the zero level is the core
psuedo module. The codepath does not need run level for core level events
but I suppose it is possible. As mentioned in another email, the event
system now includes a tremendous amount of debug information in order to try
to track that issue so hopefully that will show up the problem. 

The DBA system also has additional debug information so if this issue still
occurs, I should be able to fix it from an appropriate debug.log file.

> Now, regarding that "path" I was talking about. 
> GDBM is currently consuming around 1.8Mb of memory for each 
> open Database.
> When you have a lot of modules opened up, that memory adds up 
> (6Mb roughly if we say Set + modexclude + one table for each 
> module) and go even higher with more tables. I believe its 
> relating to the cache settings of GDBM which is fine (because 
> the memory useage only jumps when we open a database and 
> drops again when we close it. Found that out after hours of 
> stepping through code with valgrind and gdb).
> 
> But I was thinking to trim that memory useage down, can we 
> close some of the databases (say the Set/modexclude Database 
> after loading or changing a
> value?) or Potentially put in a timer that will close 
> "unused" databases after a period of time. 

We can potentially keep most of the databases closed, most of the time.
Currently my DBA layer assumes that a database is open for certain
operations and errors if not but the general read is used to open them
rather than an explicit open. There is no reason I cannot always open on
demand but there would obviously be some performance loss in doing so if the
database was likely to be closed. 

Since the majority of database access is config, for our current systems, an
open read close at boot would suffice for most operations. Changing of
settings would then also likely need an open write close sequence which is
where we would suffer performance wise for config info. Other than first run
installations, this is I guess unlikely to be a big issue since config
settings are unlikely to change that often. 

A similar approach could be assumed for data lists such as excludes and
access lists.

Data is generally adhoc access so might need to be more module specific E.g.
StatServ would need to open writeblock close or open writeblock close within
it's own code. A timer might work, but the chances of us timing a close at
the right time from the core are likely to be slim since whatever time
period we choose might be just before it gets reopened anyway giving us
worse performance. StatServ is the most database intensive module so if we
did not cache reads, but just read the whole database, we could manage with
a block open/read/close at boot then just block open/write/close at runtime.
Currently IIRC StatServ reads from the databases during runtime when
channels are created.

Mark.