Re: a slightly updated rpm spec file for Ger Hobbelt crm114 branch

Ger Hobbelt wrote:

> The vanilla crm114 contains no equivalent nor a replacement for those
> 'assertion' checks. They do not exist in vanilla crm114.
> A matter of differences in taste? Probably. (
> http://research.microsoft.com/apps/pubs/default.aspx?id=70290 )
> 

I've created and lived with assertation filled code.  it is a bear coming up
with the right assertions but, once they're there, once you make your code work
right, your code quality is easily an order of magnitude better than the code
most people generate today.

if I remember correctly Bertrand Meyer's programming by contract work predates
much of the stuff or at least formalized it.
http://en.wikipedia.org/wiki/Design_by_contract

http://archive.eiffel.com/doc/manuals/technology/contract/ariane/page.html
> rigorously now, including script 'fringe cases' and 'error
> conditions', but a _serious_ attempt to test the classifiers is still
> missing from vanilla megatest & GerH 'make check': there's a whole
> range of errors/'oddities' which is not caught by those (e.g. input
> series fed to train / classify). We're just darn lucky they don't
> happen... don't they?

Like the relatively weird scoring problems I've been having with the red side
filter?  Specifically how the red scores don't push below the training limit
like I would expect but instead pileup up against it.  I need to start doing
plots again.  In any case, here's some info from my files after only a week and
a half worth of training.  It will get worse from here.

 Sparse spectra file /var/tpblue/esj/crm114/spam.css statistics:

 Total available buckets          :      1048577
 Total buckets in use             :       263237
 Total in-use zero-count buckets  :            0
 Total buckets with value >= max  :            0
 Total hashed datums in file      :       338998
 Documents learned                :          316
 Features learned                 :       338999
 Average datums per bucket        :         1.29
 Maximum length of overflow chain :           16
 Average length of overflow chain :         1.51
 Average packing density          :         0.25

 bin value        0 found    785337 times
 bin value        1 found    248874 times
 bin value        2 found      8259 times
 bin value        3 found      2275 times
 bin value        4 found      1139 times
 bin value        5 found       609 times

Sparse spectra file /var/tpblue/esj/crm114/nonspam.css statistics:

 Total available buckets          :      1048577
 Total buckets in use             :       140641
 Total in-use zero-count buckets  :            0
 Total buckets with value >= max  :            0
 Total hashed datums in file      :       162055
 Documents learned                :           80
 Features learned                 :       162056
 Average datums per bucket        :         1.15
 Maximum length of overflow chain :           12
 Average length of overflow chain :         1.23
 Average packing density          :         0.13

 bin value        0 found    907933 times
 bin value        1 found    134282 times
 bin value        2 found      3842 times
 bin value        3 found       993 times
 bin value        4 found       414 times
 bin value        5 found       262 times

> for application invocation (did you run tests recently? Starting
> crm114, let alone script-forking it, is a prohibitive cost factor I
> can do without) nor unneeded file I/O (I'm not in the business of
> testing my /tmp partition I/O performance).

well, I can tell you that my time to invoke CRM 114 between 0.1 and 1.0 seconds
and this is on a lightly loaded system.  As the load increases, so does the time
it takes to invoke and run CRM 114.interestingly, I'm finding that using a
database to access messages is significantly faster than filesystem based
storage.  This leads me to believe that if you store CRM 114 css data as blogs,
it will run faster especially when you start switching filtering contexts.
Database engines manage storage far better than ad hoc tools.  Take advantage of
the performance gain.


---eric

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword