RE: [NeoStats-Devel] [Commits] r2699 - in trunk: include src

Hi,
Your going down a path that I wanted to look into later, touching on some
topics that I havnt brought up yet... But regardless, about excludes.

I wasn't clear in my previous emails, but I was refering to the global
exclusion database, not modexcludes. Valgrind was reporting that the
database (and memory resources etc) were still opened when it shutdown. The
Modexclude databases were cleaned up when module was unloaded successfully.

You can see the change I made to the DBA layer in DBAFini was to call
DBMCloseTable rather than DBACloseTable. Reason was that GET_CUR_MODULE()
was empty, hence the lookup dbname in the hashlookup in DBACloseTable was
just exclusions.gdbm, which, obviously doesn't exist. (which is why I'm so
confused, because it creates NeoStatsexclusions.gdbm, indicating at start,
its running fine. I think this has *something* to do with the other problems
I've been seeing about the event in services.c not getting fired for
whatever reasons) 

Now, regarding that "path" I was talking about. 
GDBM is currently consuming around 1.8Mb of memory for each open Database.
When you have a lot of modules opened up, that memory adds up (6Mb roughly
if we say Set + modexclude + one table for each module) and go even higher
with more tables. I believe its relating to the cache settings of GDBM which
is fine (because the memory useage only jumps when we open a database and
drops again when we close it. Found that out after hours of stepping through
code with valgrind and gdb).

But I was thinking to trim that memory useage down, can we close some of the
databases (say the Set/modexclude Database after loading or changing a
value?) or Potentially put in a timer that will close "unused" databases
after a period of time. 

This way, we can reap any benefits from a larger cache on those databases
that are often accessed (Data Databases, like SeenServ etc) and not waste so
much on the less used databases?

With a majority of the modules loaded, I'm running close to 30Mb of memory,
with about 29Mb swapped out to virtual ram, and 1Mb remaining resident
(after about 1/2 hour of running, which indicates that 29Mb of ram isn't
accessed!)

Considering that NeoStats ends up on shared shells, I think it would be a
good idea to try to keep the resource usage in check as much as possible.
Some shell providers ban NeoStats 2.5 right now because of the CPU
utilization, we fixed that, but now we are going to climb the memory
tables?!

Cheers

Justin

> -----Original Message-----
> From: M [mailto:[email protected]] 
> Sent: Monday, August 15, 2005 4:58 AM
> To: [email protected]
> Subject: RE: [NeoStats-Devel] [Commits] r2699 - in trunk: include src
> 
> 
> Justin Hammond wrote:
> > runlevel system, because I couldn't get the exculsion 
> > databases to close correctly using closetable. It was 
> > something I was going to look into further but havn't got to it yet.
> 
> The core should automatically close module exclusion 
> databases when a module
> unloads via the DBACloseDatabase call which closes all 
> "tables" for the
> affected module. Stepping through this process in the 
> debugger shows this is
> being called with the correct run level and is closing all 
> module databases
> including exclusions. A call to DBACloseTable should never be 
> needed to
> close an exclusions table for a module or the core. It is not 
> clear which
> one you are having issues with.
> 
> A test unload module results in: (exclusion highlighted with <----***)
> 
> 14/08/2005[21:40:00] DEBUGTX NeoStats - :stats2.ctcp.net ] :secureserv
> Module unloaded
> 14/08/2005[21:40:00] DEBUG1 SecureServ - DBACloseDatabase SecureServ
> 14/08/2005[21:40:00] DEBUG1 SecureServ - Closing Database SecureServ
> 14/08/2005[21:40:00] DEBUG1 SecureServ - Closing Table
> data/SecureServhelpers
> 14/08/2005[21:40:00] DEBUG1 SecureServ - Closing Table 
> data/SecureServconfig
> 14/08/2005[21:40:00] DEBUG1 SecureServ - Closing Table
> data/SecureServexclusions <----***
> 14/08/2005[21:40:00] DEBUG1 SecureServ - Closing Table
> data/SecureServmonchans
> 14/08/2005[21:40:00] DEBUG1 SecureServ - Closing Table
> data/SecureServrandomnicks
> 
> For the core, FiniDBA will close all core tables and any 
> remaining open
> module tables since it loops through all loaded databases and tables.
> 
> A test shutdown results in: (exclusion highlighted with <----***)
> 
> 14/08/2005[21:40:21] DEBUG1 SecureServ - DBACloseDatabase SecureServ
> 14/08/2005[21:40:21] DEBUG1 SecureServ - Closing Database SecureServ
> 14/08/2005[21:40:21] DEBUG1 SecureServ - Closing Table
> data/SecureServhelpers
> 14/08/2005[21:40:21] DEBUG1 SecureServ - Closing Table 
> data/SecureServconfig
> 14/08/2005[21:40:21] DEBUG1 SecureServ - Closing Table
> data/SecureServexclusions <----***
> 14/08/2005[21:40:21] DEBUG1 SecureServ - Closing Table
> data/SecureServmonchans
> 14/08/2005[21:40:21] DEBUG1 SecureServ - Closing Table
> data/SecureServrandomnicks
> 14/08/2005[21:40:21] DEBUG1 NeoStats - DBACloseDatabase NeoStats
> 14/08/2005[21:40:21] DEBUG1 NeoStats - Closing Database NeoStats
> 14/08/2005[21:40:21] DEBUG1 NeoStats - Closing Table 
> data/NeoStatsexclusions
> <----***
> 14/08/2005[21:40:21] DEBUG1 NeoStats - Closing Table 
> data/NeoStatsconfig
> 
> So it appears to be working as expected. 
> 
> Rev 2734 includes the debug text used to generate the above output.
> 
> How are you determining that a file is not closing?
> 
> You appear to have added a call to DBACloseTable("exclusions"); in
> FiniExcludes. Apart from the fact that this is not necessary, 
> if this is
> what is not working (and it should not), it is nothing to do 
> with run level,
> it is to do with the fact that a table name is path+root+name, i.e.
> "data/moduleexclusions", "data/neostatsexclusions " etc.
> 
> I can look at altering the way that DBACloseTable works but 
> this would also
> need a few changes to the DBA use of it. For now I am more 
> concerned with
> your suggestion that the table is not closing.
> 
> 
> Mark.
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: devel-unsubscribe-wool9L35kifE9wlyV4mCnKxOck334EZe@public.gmane.org
> For additional commands, e-mail: [email protected]
> 
>