RE: [NeoStats-Devel] Idea's for config/data storage

Note I have snipped a lot.

> Hey, as far as I'm concerned, and experience with previous 
> major releases, your not going to get a break with 3.0 goes 
> "GOLD"... Hehe.

I know there will be some things that come up post release but once it
settles down we will be back to "I want botserv" and "I didn't RTFM but help
me anyway" support rather than the main development.

> I don't think anybody is going to complain if you take a 
> break now anyway, as I see most of your changes are complete. :)

I didn't think anyone would, but I wouldn't be able to stay away without
thinking "I will just do this bit..." then be back where I started. :)
Besides, I have plenty to occupy myself in various areas and do not want to
create additional delays.

>  I was
> > under the impression that libevent was the only real major change 
> > planned from this point and had you not been able to 
> implement it, I 
> > would have been happy to release 3.0 without it and keep it on the 
> > todo list for 3.1.
> > 
> 
> To be honest, I'm not real keen to jump into a 3.1 series so 
> quickly once 3.0 is out. 
> The core needed attention so hence 3.0, but after 3.0 my 
> focus at least will shift back to the modules, as that what 
> provides the values to NeoStats....
> The core is useless without the modules

I agree completely, but at the time, being unfamiliar with libevent and what
it was supposed to do for us, it remained a core option that might have to
be ignored for 3.0 and remain a TODO for a future release and might be
serious enough a change to justify 3.1 rather than a 3.0.x update given the
changes required throughout the core and modules. I have no intention on
rewriting everything again once 3.0 comes out, but no objection to
postponing something we will not achieve in a decent timeframe until a newer
core version. I am also aware of the fact that no matter how wonderful we
make the 3.0 core, there will be some major thing that crops up that will
cause 3.1 to be the only real solution. However, I do hope that the 3.0 core
will be powerful enough that 3.1 is a much easier task than 3.0.

> Plus, considering the work we are putting into 3.0, I think 
> it's a good idea to make sure things are "right" now rather 
> than have to take on extra code to deal with those fringe 
> cases later... (not to say that your implementation isn't right)

I agree. But 3.0 development began one year ago today so I think we ought to
have a number of fundamental areas such as configuration and data storage on
which large parts of the core depend pretty much locked down by now which is
why I am keen for this to be resolved sooner rather than later. 

> I can get back into the development at least a couple of 
> times a week now.
> I'm involved in another project at the moment, that's for a 
> customer, but right now, its on hold as the customer now 
> starts work on their application.
> Its going to be months before they are ready to move again. 

OK cool. I just didn't want us reworking everything only for you to have to
be tied up for a few months then return with another idea for data storage
:)

> I think we can implement the config storage without major 
> changes to the core (other than maybe a rename of the 
> functions and some logic changes to say config data use this 
> storage "engine" and real data uses that storage "engine"

Yes we can. There will be odd things to work out here and there, but
changing the config format is not a huge task since the code for config was
written with this possibility in mind.

> As for the current socket code, right now I'm waiting for 
> some improvements on curl to be done around the socket code. 
> I want to get rid of the entire process flow where a Library 
> creates a POLLFDS/readfd/writefd/errorfd, passes it to the 
> core, does the select/poll and returns. Its just not 
> efficent. (my NeoStats on production spikes and spins on 100% 
> CPU when OPSB kicks off the scans as its creating/deleting 
> these (sometimes large) arrays every loop, and its probably 
> the cause of many people saying NeoStats uses too much CPU...)
> 
> The new code will work like the changes I've made to the adns 
> library, where we register the FD when the library opens it, 
> keep it statically loaded into libevent and only 
> change/remove it when the socket is closed, or we are now 
> interested in a different event... 

Remember we have two adns libraries. I do not remember exactly what is
different in the Win32 version but any changes will need to be made in that
one as well.

On a similar note, we ought to do a library review now before you change all
the libraries for libevent then we find we need to update the library to a
new revision. We may thus be able to combine ADNS into a single version
rather than two. The two RTA libs are of course more easily fixed by the
removing of RTA.

Also, does libevent support IPv6?

One other question, any ideas on timescale for libevent? I am just trying to
keep alpha updates reasonable at the moment so aiming for every 2 weeks but
hope to be able to separate core and module updates soon and libevent at
least is likely to be a full release. 

> Eg, a typical HTTP connect would go along the lines off:
> 
> Open Socket -> curl registers fd with libevent for POLLOUT -> 
> POLLOUT activity called -> CURL code writes data to socket -> 
> update libevent to say we are now interested in POLLIN -> 
> POLLIN activity called -> curl reads socket and closes socket 
> -> delete fd from libevent.
> 
> In the proposed case above, we only call into libevent 3 times... 
> 
> Where as the current implementation would call into libevent 
> once for every FD on every loop.... Not ideal obviously.... 
> 
> A local copy of my OPSB does the above already, as does ADNS... 

Do you have any ideas on how the module interface will look yet? I noticed
you change add_sock to cope with the new library so is this likely to be the
interface? My main thought here is to tie the socket API into everything
else so it changes to AddSock and DelSock. Minor thing really but it
standardises the API and gives us a complete break from 2.5.x so that no
confusion can arise with any published information on the older API.

> As for the lang support, as mentioned, that will be migrated 
> to use your DB abstraction....

That should solve the issue with bdb until it can be addressed properly and
means I can ignore it for Win32 completely. Although this does bring up the
question of how we distribute language files. Still until we have some, it
is not a huge concern :)

> I disagree that there is much difference to the current 
> implementation as there is to the old way in terms of 
> maintaining legacy code. Say StatServ gets a bunch of new 
> features over 10 releases.... This is then 10 different 
> getv<x> functions to handle the legacy code... 

Actually it isn't so much that. In the current system, a structure is
retrieved in a single call so does not have anywhere near the code overhead
of the original. On a related note however, if we require to change our data
structure 10 times, there is something wrong with the original data design. 

If you look at say network stats handling in statserv, the only code
required is:

	if (DBAFetch (NETWORK_TABLE, NETWORK_TABLE, &networkstats, sizeof
(networkstats)) == NS_SUCCESS) 

So, to support multiple versions, we simply have something along the lines
of:

	if (DBAFetch (NETWORK_TABLE, NETWORK_TABLE, &networkstats, sizeof
(networkstats)) == NS_SUCCESS) 
	else 	if (DBAFetch (NETWORK_TABLEv1, NETWORK_TABLEv1,
&networkstats, sizeof (networkstats)) == NS_SUCCESS) 

We do not necessarily need to handle removing old data since by ordering the
latest first, it will always read the latest if available and saving will be
handled by the next scheduled database update. The only caveat involved is
new structure fields must be placed at the end of the structure and we
cannot delete any field. 

Server stats would be a little more complex due to the use of callbacks per
record, but something similar could easily be developed by sending the size
read to the callback function for the memcpy.

> The original StatServ saved data for channels, servers and 
> network to different files, much like the same way we use 
> different tables. 

I hope you can see the new system of tables is not nearly as problematic for
compatibility or as involved from a coding perspective as the original
custom database format in statserv. Yes there is a small overhead, but it is
IMO trivial in comparison with the old statserv database systems and creates
only limited restrictions.

> > > but it
> > > doesn't offer the user any way to query the data in a native way.
> > 
> > Given the apparent lack of interest in SQLServ, is there sufficient 
> > interest in querying the data outside of NeoStats? Yes it would be 
> > great to do so, but is it considered essential at this stage. Also 
> > consider the obvious path the querying will take. The moment it is 
> > available there will be a request to write as well as query. This 
> > would represent a major change to the NeoStats systems 
> since NeoStats 
> > expects it's data to be written only by itself. Any implementation 
> > should be created with this in mind so that we either allow for 
> > external modification or prohibit it.
> > 
> 
> SQLServ had interest, the problem was that it was damn 
> difficult to get working. SQLServ never intended to support 
> write's either... And I never had a request for it... 

When Andy Church added the http module to display data in IRCServices, the
first thing people wanted was to be able to edit their nicks/channels/akills
etc online. It never happened since he refused, but people generally asssume
if it is easy to display it is easy to change. Maybe if sqlserv was too hard
to get running that is why the write requests were not made. Think about
customviri.dat in statserv. Almost immediately a request was made on the
forums for a write interface from IRC. More recently it appeared on the bugs
site as a feature request. Once we are displaying hostserv vhosts and
neostats excludes, I expect requests for a write interface to follow which
is why I thought it best to preempt the request at this stage.

> In 
> fact, if we can implement this, I believe we could drop 
> SQLServ and the headaches that go along with it (btw, its 
> currently very broken... Something is messed up in the SQL Parser)

Yes I noticed that. It works on Windows still but *nix doesn't. It did until
recently but I cannot find anything obvious to account for it. Everytime I
remember the svn site is down so I can't check the history and when the site
is up I forget to check :)

> For Config data, I think a external rehash option is 
> esential.... We have a basic infrastucture there to do it 
> already, and with all your SET work, I think it would be just 
> a matter of calling the Set processor in the core again... 
> With the flexibility that the SET option gives (in terms of 
> having callbacks etc) it should be trivial to update... (we 
> can put some cavets around this, for instance, if you change 
> StatServ's nickname and call rehash, its not going to work, 
> its going to require a restart.... 

Actually, if you change StatServ's nickname and rehash, it ought to work
since it will trigger the NICK callback and change nick live. On IRCds that
support it, realname, ident and host can also be changed in real time. I
know you were just aiming at an example but most settings should work from a
rehash. 

The current system was created to allow different writers to configuration
data to allow for transparent DCC support and we could support transparent
telnet etc in a similar manner. But this was designed to go via the standard
command path which just needed a Client to authorise in order to work. My
concern with writing was mainly due to the fact that we were avoiding the
command path but if it is restricted to configuration data as I define it,
it should work the same as running the SET command so the only caveats are
those noted for settings. 

> But if you update the 
> ExtAuth Service Roots entry, it can be added straight away..... 

The master root is defined in neostats.conf and is not currently handled by
the online systems. It would be something else we would have to implement to
get that working. ExtAuth entries are managed online and I consider them
data rather than configuration so they would not be handled by rehash.

> For actual Data.... Write support shouldn't be ever considered. 

OK, that is fine with me.

> There might be cases where we need to re-evaluate if our data 
> is really data, or config entries (for instance, HostServ 
> entries) and maybe re-implement that...

IMO, configuration is something that is used to influence operation of the
code, data is something the code generates and maintains (possibly in
multiples) as a result of it's operation. Thus access lists, excludes,
vhosts, stats etc are all data and NeoStats and modules will operate without
any defined. If you can influence the operation of NeoStats or a module with
something such as SET, it is configuration. 

If you start to use the config interface for data such as vhosts, then we
will soon run into similar problems to keeper.

> > > The below psudeo code *could* be optional depending on 
> the backend 
> > > storage we use. For instance, we might want to retain the current 
> > > gdb method of storing data, but if a user selects to store in a 
> > > config file or full SQL
> > > database, we can turn on the   code below and have separate 
> > > fields for each
> > > entry in a structure. 
> > 
> > There is nothing stopping a module choosing to store 
> separate fields 
> > for each structure field. The reason this is not done at 
> any location 
> > currently is the huge potential performance hit that keeper 
> > demonstrated by storing data in this manner. IIRC, you have 
> test data 
> > which prompted the move to BDB for StatServ in 2.5.x. I think this 
> > data must be tried with such a system before we can consider it for 
> > deployment and we should not rely on progressive write to 
> work around 
> > it since there is a chance that we lose large amounts of data 
> > unecessarily.
> > 
> 
> Well, in the other implemention, where its currently being 
> used in a semi beta enviroment for one of my customers 
> (MQServer was written for one of my Hosting Clients that 
> supplies news feeds to their clients), the entire "Client" 
> (it's the one that does the struct encodes/decodes) can 
> process around 900 messages a second (that's 
> encoding/decoding structures that have about 30 "fields", and 
> includes the latency in network communications and "lag" on 
> the server... )
> 
> And if its implemented in a way that's smart (ie, lets not 
> update each field in the SQL Database as each field is 
> processed, but rather save the updates for one SQL statement 
> (update table (one, two, three) values ("one", "two",
> "three");)
> 
> The overhead we had with StatServ was keeper's issue, because 
> it opened, read, and wrote the entire "data" file for each 
> update to a individual field... Ie, there was no way to do it 
> as a transaction.

OK, MySQL or whatever system is employed goes offline for an upgrade or
crashes or is busy serving a forum and a blog and a PHPNukeMe website. How
does this affect us? Exporting data so some other process can access it is
all very well, but do we really want to rely on an SQL process for run time
operation? Are there any services packages which use SQL for storage and
retrival rather than just providing information for some other process so it
is not essential to maintain operation? Serving a news feed is not a mission
critical task so such things are of little relevance, but a problem with the
external process here could stall NeoStats or make it inoperable. 

I am thinking worst case here for the simple reason that we might need to
have SQL + DBA so that NeoStats can run regardless of the status of the SQL
process. The DBA layer still handles having two outputs transparently to
modules but it affects how we code the backend. One alternative is to use
some sort of embedded SQL system such as SQLite which while less featured
would at least protect us against any problems with the SQL process over
which we have no control. 

Also, can we write lock our database so that someone does not develop their
own PHP interface which proceeds to change our data?

[bind style config]
> Porting libconfuse to replace what we have right now should 
> be simple...
> Plus I think anything it is a lot simpler that your standard (Unreal
> excluded) ircd config file... If they can master your I, O, 
> C/N lines...
> Then they should easily pick up anything else we can throw at them!

A lot of ircd.conf files now use a bind style config. Most users do not tend
to work out I lines etc, they harrass others to do it for them :)

> > > This is a good library called
> > > libconfuse that can handle that style nicely, as well as 
> providing 
> > > syntax checking etc. Reading data from confuse is easy, 
> and it also 
> > > has the ability to write updates back to the config file. 
> So if we 
> > > provide separate "config" files for each module. Obviously the 
> > > problems we had with keeper were related to us using it for data 
> > > storage as well, so we can discourage this option for data 
> > > variables.
> > 
> > I have no real objections to this and wish it had been the answer 2 
> > years or so ago when keeper was first deployed and config files 
> > removed.
> > 
> > My main concerns are:
> > 
> > 1) The addition of yet another library to NeoStats.
> 
> The confuse library is actually one two files and one 
> header... I'd just drop the files direct into the source 
> directory rather than set it up as a new lib in the /lib 
> directories... 

OK that sounds cool. I guess at the same time we can strip out the old
config stuff.

> > 2) After spending much time training users that they did 
> not need to 
> > worry about config files, we are looking to reintroduce them so we 
> > need to consider the potential additional support. I think 
> the bootup 
> > config must remain separate to any "run time" config 
> storage so that 
> > users can continue to ignore the system if they so desire.
> 
> Fully agree. Keep neostats.conf as the main file, and put 
> each modules config file into the data directory... 

OK. We might want to pick a new extension for the other config files then
since neostats will also generate one for it's internal options and this may
cause confusion.

> > 3) The current 3.0 config system has neostats create config entries 
> > where they do not exist at boot time and knows all defaults. It can 
> > thus automatically update the local config when new items are added 
> > while maintaining compatibility with existing config data. 
> If we are 
> > to distribute config files we have to maintain both them 
> and the run 
> > time defaults or come up with a system where we can update 
> the config 
> > file between updates thereby only maintaing one set of 
> defaults. Look 
> > at 2.5.x modules to see that there are various different default 
> > settings in that system which only provided run time support for 
> > defaults. If we do not distribute config files, we lose one of the 
> > biggest advantages of them, that of offline configuration prior to 
> > runtime.
> 
> This is why I like confuse... It can write config files out 
> as well... (but from memory, any comments in existing conf 
> files gets lost...)

That would be a great shame. Hopefully it will not do anything to config
files that are not changed online (i.e. the bootup neostats.conf).

> > 4) User readable config files suggest the need to support 
> some sort of 
> > rehash function which will require NeoStats to accept changes to 
> > configuration outside of the runtime. This relates to the 
> possibility 
> > of external writing to NeoStats data mentioned above.
> > 
> 
> Have the rehash signal call the ModuleConfig function, maybe 
> with a new flag, SET_REHASH... 

It should work without the need for another flag since it is equivalent to a
LOAD or CHANGE so keeps it simple for modules. Something we can investigate
when it is up and running. 

However, we do not necessarily know how to call ModuleConfig successfully
for a module since it is called from a module not automatically managed by
the core. Modules may call the routine several times with different data and
it triggers under other circumstance e.g. AddBot will call ModuleConfig on
the botinfo settings. So we would have to know what areas of a module to
rehash. We might be able to use the setting hash within a bot to do this but
I have not checked the code to ensure that this would work. It would of
course fail in the event a module does not actually have a bot but modules
without bots are not able to do much other than reporting and would not be
configurable online so we probably do not need to worry about this case.

> > If we go with this system, I say we provide this as the 
> sole means of 
> > configuration and give no options for storing configuration data in 
> > another format and it should be implemented immediately. I see no 
> > benefit in allowing multiple system for storing configuration data 
> > since it will be a lot more work for us to maintain and support.
> > 
> 
> Well, if we write a engine for your current DBA abstraction, 
> a initial implementation should be able do it without any 
> changes at all to the core.... Ie, still use DBAFetchConfig* 
> and DBAStoreConfig*, then the only update that we need to do 
> there is make it a "optional" option to specify a alternative 
> config engine... And if none is specified, then use the 
> Default one we already have (gdbm)

We would have to hack such an engine since the DBA assumes it is authoritive
and does not provide an ability to share part of the data handling between
different DBMs. Still, the config area is handled by macros so we can easily
redirect the config while leaving DBA authoritive for data.

The reason I am against having multiple DBA systems for the config is so
that we do not have the additional overhead in the event of a user posting
config problems, then we ask which DBA and he says gdbm but in fact that is
used for data and he is using libconfuse config files. We can simplify both
support and maintainance by having a single system for config files. Yes it
means no SQL for config but the thales proposal addresses that since it can
output such information itself. 

> The other change we should make is have the DBM option as a 
> "optional" item now. Set the default to GDBM, and only change 
> if its specified in the config, which further reduces the 
> number of options users need to choose to get up and running 
> (only hardcore junkies are really going to have a preference 
> over GDBM or BDB)

I have no objections to that... it is now implemented. SVN is unavailable
again at the moment though so not yet committed (server timeout not bdb
problems).

> > > Have a look at the code below before reading on.....
> > > 
> > > If we do some tricks in get_data (or save_data) when 
> using SQL, like 
> > > using the name field as the record index, and retrive the entire 
> > > record (all fields, not just the "mystruct[i].name" 
> field) and cache 
> > > it, we would could make this pretty fast. (eg, in the 
> first call to 
> > > get_data, we cache the entire record, so subsequent calls to 
> > > get_data just retrive out of the cache rather than make 
> another SQL 
> > > call) (in save_data, we could just have a commit_data call, that 
> > > actually sends the SQL update to the database).
> > 
> > As far as the current code is concerned, all such things 
> are the realm 
> > of the DBM module so if it can be done, it should not put too much 
> > overhead on module code or coders.
> > 
> 
> Agreed, as mentioned, if the user specifies, GDBM, then this 
> code isn't used.
> If they specify a DB such as SQL, then this code kicks into 
> gear... The changes to the core is that the Get/SetData 
> functions now take one additional argument, and that is the 
> structure "descriptor"

OK. Do we fallback to something else in the event a descriptor is not
provided so that modules do not have to do this from the outset or require
all modules to support it? See also thoughts below.

> > > The code below is actually inspired by the RTA library 
> code (though 
> > > very much simplified). I've got a working copy of it at 
> > > http://svn.neostats.net/cgi-bin/viewcvs.cgi/MQServer/Trunk
> > > (have a look in Client/testclient.c, Libs/encode_packet.c
> > > (pck_send_message_struct) or Libs/packet.c (pck_decode_message)
> > 
> > One of the biggest flaws in RTA is the fact is cannot handle some 
> > simple types through some sort of common callback. For example, the 
> > PSTR requiring a callback. There is no reason whatsoever 
> that a common 
> > routine could not handle a simple pointer. You have a length field, 
> > you have the pointer.
> > Moduleconfig gets it's versatility by only using pointers. 
> > This all needs to
> > be handled transparently in the core rather than modules having to 
> > write callbacks for handling such things.
> > 
> 
> Well, in fact, our Data should *NEVER* use a pointer with the 
> current implementation, otherwise when we save data, we are 
> saving the location the pointer "points" to, rather than the 
> data itself... When we load that "pointer" data again, it 
> might now be pointing to some other memory location that 
> doesn't contain anything valid. 

Funnily enough, our current system only uses pointers and a length. It knows
how to decode a string but it is of course the data pointed to of the
specified length that is stored not the pointer itself. Same way C handles
strings. 

> > Assuming we are keeping RTA, maybe we would be better aiming to 
> > integrate RTA with this idea so that we can use a single system for 
> > both. This would extend the reach of RTA as well since 
> there would be 
> > format information available to it for far more data.
> > 
> 
> Well, if we implement something like this, then lets drop 
> RTA... And for the user/server/channels data that RTA 
> currently provides, we would write a "thales" type module to 
> save the data every X minutes etc... And move all this into a 
> module as your tried to do once... :)

OK with me. Saves maintaining the two RTA libs (*nix and win32) as well.

RTA did work as a module, it was mainly the socket code that had to be
hacked and the fact statserv couldn't easily publish information or access
the structures so as a module it was reduced to core info. But if we are
maintaining a descriptor using a core based structure, that resolves that
issue.

In addition, we can maybe drop the html output from statserv using this
system.

Finally such a module resolves the potential backend issue wrt problems with
the SQL process since this module would handle SQL transactions and NeoStats
could continue to use a different backend. 

This might actually be the solution (assuming it could achieve all the aims
of any SQL provision), keep a dbm style backend for data with an optional
SQL export via an SQLServ module. We add the data descriptor to the DBA
calls which is defined in the core. If it is not provided, no SQL export is
possible. If it is and SQLServ is loaded, SQLServ will proceed to export SQL
updates to whichever SQL process is selected. Neostats does not have to rely
on a process it cannot control for reading data so cannot be affected by
issue affecting the SQL process including being offline and busy but the
data is made available to users than want it. This will provide automatic
write protection since NeoStats will never read the data and will avoid
issues where a module does not (yet) have descriptor support. 

However, we might end up with a situtation where SQL data is slightly out of
sync with NeoStats data. Stats type data would be updated at each database
update but say a vhost changed while SQL is down would be missed. We could
of course update SQL at startup but this would mean any disparity would
require a restart of NeoStats to fix. 

> > Anyway, I think we need to make a quick decision on this 
> area since it 
> > affects more than the data storage subsystems.  An 
> alternative config 
> > storage layer would more or less drop since I used macros to manage 
> > the config system to abstract them from the data storage. 
> By changing 
> > these to functions we should be able to change config data easily 
> > without a huge impact on the code base. The other ideas 
> however would 
> > likely require extensive reworking of current systems.
> > 
> 
> I could drop confuse in very quickly... As mentioned, I'm 
> included to implement confuse as a "engine" for the 
> DBAConfig* functions rather than repoint all these macros' at 
> confuse directly... 

Well you can't for reasons mentioned above but this is partly why they are
macros. One reason was to make a nice interface for modules to use for
config stuff and the other was to allow a separate system for config files. 

Get libconfuse in and let's go with it. Just by replacing the existing
config system for neostats.conf to start with since that is the primary
config file and we can drop the old config code. 

Then we need to decide whether to allow multiple config backends. My vote is
a libconfuse system only for config. I see no benefit in allowing config
data to also be stored in the DBA. If there is some desire to export config
information to SQL which is the only reason I can think an argument can be
made for a DBA config layer, then this can be implemented quite easily since
config data is readily available and processed by the core so can easily be
so output while NeoStats operates purely from the config files. The overhead
of "SELECT SCAN FROM OPSBCONFIG" is bound to be more than a read int from
text file and without external writing by SQL, largely pointless.

> Finally, in order to get the BDB system operational, do you 
> mind giving me access to your system so I can write some 
> configure checks etc... I've looked over the subversion code 
> that uses BDB heavily, and found they have checks to 
> determine what version of BDB your using, and thus code 
> apropriately... :)

No problem, I will set something up and let you have the details offlist.


I consider the DBA and config systems fundamental issues that the rest of
NeoStats rely on which is why I am keen to get this resolved once and for
all and lock it down in principle regardless of any tweaks we need as we
progress so to summarise: 

1) Let's add libconfuse now and remove the existing main config system. 
2) We need to decide whether configuration data has multiple backend
options. I am against multiple options for configuration data as discussed
above. I think we need to make this decision before moving config data to
libconfuse so that we can make any adjustments to the API.
3) We need to determine a definition of configuration items such that we
have a distinct seperation of configuration and data. I have given my
definition above but it can be boiled down to if you can SET it, it is
config otherwise it is data.
4) I do not object to SQL support as a backend but see problems in relying
on it. The thales proposal solves my concerns in that regard since SQL is an
additional feature rather than intergral to operation but would this solve
all the aims of an SQL backend? If the aims are solely external query, I
believe it does. The thales proposal also throws up a number of other
possibilities such as a non SQL user query system such as XML or HTML export
or even a local httpd that all utilise the suggested descriptor system.

I currently consider all other development secondary to this area so I think
if we can resolve these quickly, I think the next objective should be to
fully implement libconfuse and update the DBA API to reflect decisions on
the config system and account for the callback requirement outlined earlier
in the statserv compatibility comments. This will I think represent alpha 3
so we can get the new config system tested and introduce users to it. 

Obviously I do not expect us to have an SQL engine implemented for alpha 3,
but I intend to move RTA to the graveyard as soon as SVN returns which will
effectively retire it. Next I think we get the descriptor system setup and
implemented and APIs updated so that we can begin looking at SQL support. 

Justin, I don't want to keep you from libevent stuff so don't mind taking
time to work on the config stuff if you want to keep on libevent primarily. 


Mark.