Re: New WGET and mainline version up: crm114-20080326-BlameSentansoken --> 64-bit SuSe 10.2 output

On Wed, Apr 2, 2008 at 9:34 PM, Paolo <[email protected]> wrote:
> On Wed, Apr 02, 2008 at 01:23:04PM -0400, Bill Y wrote:
>  >
>  > I.e. we have a machine that thinks it has int32_t.  BUT IT LIES!  It's
>  > really a 36-bit machine, and the int32_t is a lie.
>
>  then the OS/cc/build-chain would deadly broken and it would be masochistic
>  to waste time on that.

Indeed. Fortunately for us, 8-bit byte code assumptions have such a
_huge_ 'installed base' only some folks wearing tin foil hats still
consider to create new 9-bit (36-bit / 72-bit) machines.
The only other thing one might consider in actual and realistic future
is pushing crm114 into Unicode (wchar_t) Country as a stepping stone
towards handling non-Roman language tokenization. I see China as a big
market, and with proper care, it can be a big market for CRM114 too:
"you want mail from Hu Sing and others, but not from the bloody bloke
who's peddling the rhino dust every frakin' week at your door?
Download crm114, install and show it the way. The darn thing'll work
it out faster than you can get your capucino!"


>  > Don't count on that.  I bet it breaks the hash function itself.
>
>  I bet, if one ever manage to fill up a 2T cssfile, he wouldn't be much
>  happy on the usual 'oh, just retrain from scratch', eh? ;P

Might also be an attempt at performance improvement: I think for
certain purposes CSS files at 4 or even 8G cell count are realistic.
Especially if you want low fill at high data/token quantities: CRM114
uses linear probing hash tables for the production classifiers.
Research shows this type of hash table does not perform very well
anymore when the fill rate gets near or past the 50% mark.
One of the little experiments I need for my own data feeds - which is
definitely NOT mail but huge quantities of numbers - is testing the
thought what bucketed cuckoo hashing would do for me within CRM.
People reporting linear chain lengths of 600+ makes me think too.

So far, my own research for _my_ purposes indicates I might need to
handle Markovian token chains of up to 64 tokens (not the 5 or 7 or
what is it in crm right now?) to match the patterns that I'm looking
for: that means a single input train will get me 64 derived Markovian
hashes (using extended VT tables) per input token @ worst case (if I
fill the whole VT table with non-zeroes). That leads to a _serious_
number of tokens to store in the OSB/... hash tables, so, yes, >2G
hash table size does not sound too crazy to my ears. But then I might
wear a tin foil hat too, without knowing it, as one might question the
validity of using Markov chains so long for anything, because the
parameter count is going through the roof that way, but all I can say
now is that it's 'gut feel' and I need to see it before I can get
closer to a satisfactory answer.
That why I like some #define's in there that give me a CRM that can be
stretched and squeezed like Elastigirl, at least for VT + CSS file
sizes.

FYI: I'm looking for patterns in data that's quite close to pink
noise; ratios of 'good' to 'spam' can be up in the 1:10K's. Luck has
it that learning can be fully automated: it's like with earthquakes: a
little tremor isn't necessarily going to become one, but once it's
happened and it's all over the news, you sure know "yep, that was one
right there". Now imagine you're trying to teach crm114 to
'predict'(!) the earthquakes. In other words: teach crm114 to inform
you that with, say, >51% certainty, the *NEXT* email is going to be
'good'. And the DJ Tiesto party next door isn't going to do you and
your little seismograph any good. Timeseries are a real bitch.


>  he's in NL, they've got plenty of wind-power from offshore windfarms; such
>  hot iron would be fine as a ~resistive dumper/ballast load for the system.

hah :-))



-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace