RE: [NeoStats-Devel] Status

Justin Hammond wrote:
> I know that previously I have talked about other "changes" to 
> NeoStats (Database support etc) but now I figure that its 
> time to move with getting 3.0 out the door as a full release, 
> and work on these new changes as a 3.1 release instead.

I am happy for us to push towards a 3.0 release and postpone what we can't
get in to 3.1.

> For my thoughts about things outside 3.0 core, I think can 
> all be approached as modules, so there should be no 
> dependancies on us making any futher significant changes to 
> existing API. They are:
> 1) a version module, to basically just do a HTTP request to 
> see if you have the latest module

I assume this will depend on curl for it's requests? 

> 2) a channel linking module (dependant on update.c above)

Channel linking? As in linking networks?

> 3) A webserver module (prompted by a discussion with DNB 
> tonight.... Very early thoughts though)

Something I have also been thinking about for a long time but never had
chance to get a working implementation. 

> So Mark, Seeing as 3.0 core is being lead by you is there 
> anything else you want to see in 3.0 (apart from obvious 
> bugfixes or Win32 interface tidyups!!! (see following emails)).

I think other than the bugfixes and the Win32 interface the core is pretty
much there. One thing I will come back to later in this email.

There is one outstanding issue that I feel we must resolve before we go beta
and I have raised before that is the pcre/libeevent libraries. 

With PCRE, do we update to 5.x? Is it likely to cause us any issues? Whether
we do or not, I think we should absorb this into the core to resolve the
overhead in reassigning the malloc calls. This will simplify module coding
in that Win32 compatibility will not have this additional overhead.

With libevent, are we going to try and maintain compatibility with the
distro? We currently have custom changes for Win32 which do not match the
updates to the primary version last time I checked so if not we may as well
absorb this into the core as well and allow it to use our new abstraction
layer for socket calls etc.

> I guess I'm thinking for core 3.0, its overdue to move to 
> another alpha, and then feature freeze mode and only fix bugs 
> for 3.0 release. Lets get 3.0 final out the door, and put 3.0 
> into maintence mode (with bugfixes or features that do not 
> modify existing API's) and start thinking about 3.1

OK with me. 

> Finally, I know for 3.0 I have been a bottle neck, and made 
> some bad calls.
> I apologize for this, but honestly, what I have done with 
> regards to socket code or perl API are things that I felt 
> were critical (Socket code from point of view of performace, 
> and Perl API from perspective we needed better logic in 
> SecureServ dat files). (confuse/SQL support is something I 
> still want to look into, but now I realise it's a case of 
> "why fix something that's not broken", so it can wait till 3.1)

Since the confuse implementation went in I looked at getting modules using
it, but it is not versatile enough to fit in with our current SET interface.
If we could get confuse offering some sort of keeper like interface, it
would work well but at present it would require us to:

1) Generate the confuse tables in real time when a module loads to access
the information which is in itself trivial but removes any possibility of an
API to configuration options.
2) Require modules to provide the confuse tables which doubles up a lot of
the SET interface information and creates the potential for error during
config load should the tables become out of synch for things such as default
values.

Given that we do not currently use the config API other than within the core
and we have absorbed confuse, I guess the solution is to remove the config
API then create some merger of the SET tables and the confuse structures
optimised for our use. Unless we could potentially want config driven
options that are not available through a SET interface, I think this is the
best way forward but it could wait until 3.1 for the final solution which
will also give us a chance to see if we need a config API in addition to the
SET interface.

> As far as modules, well, these can be independent of any 3.0 
> release, so no problems there. The only thing I would like to 
> see (any volunteers?!) is a new webpage/website for the 
> "external" modules. And of course, I'd like to start 
> disucussions about proper documentation for not only the 
> core, but also the modules.

I will come back on documentation thoughts in a later email.

> Mark, One thing I was wondering about (I installed Visual 
> Studio finally) is that if you can put together a "programing 
> guideline" for NeoStats? Not just covering things like module 
> API, but also the abstraction interface you put in. (eg, 
> ns_malloc, versus os_malloc, versus malloc?!?! That still confuses
> me) 

Yes I can do this. 

As a quick note for our purposes, malloc/free should never be used in our
code. We should always use ns_malloc/free. The reasons for os_malloc/free
are mainly for libaries (pcre etc) that allow us to reassign the malloc/free
calls to avoid problems on Win32 linking the CRT in multiple locations and
to allow us to provide OS specific optimised versions without affecting the
code base.

os_xxxx calls are provided for most library functions mainly to solve the
Win32 CRT link issues so any library call should be os_ rather than the
standard function. The other reason for os_xxxx calls is largely down to the
fact that *nix treats files and sockets as streams using the same functions.
Win32 does not so close(socket) fails on Win32. os_sock_close(socket)
chooses the correct function on Win32. In addition, the os_xxxx calls
provide automatic logging and debug information.

This is the one remaining core area I would like to fully complete
abstraction of by replacing all ns_xxxx/os_xxxx type calls with a new full
abstraction layer. I think I have discussed the plans for the full
abstraction layer before on the list, but as a quick reminder, it would
remove all the os_xxxx, ns_xxxx calls plus the "irc" string functions and
provide a standard interface for handling all these functions while
standardising all our module code with this interface rather than the many
variants possible via C libraries. 

However, I do not mind postponing this to 3.1 since the current abstraction
layer will allow me to backport anything of note should 3.1 turn out to be a
long development time and it considered worthwhile to make the backport.

Mark.