Re: 'Floating point exception'

On Thu, Mar 19, 2009 at 4:20 PM, Paolo <[email protected]> wrote:
> On Thu, Mar 19, 2009 at 10:59:53AM -0400, Kurt Hackenberg wrote:
>> Thomas Michael Hagen wrote:
>> > i have also had several people using the program at the same time,
>> > writing to the same files.
>> >
>> That doesn't work.  There's no locking; files written concurrently are
>> likely to be trashed.
>
> correct (for GerH as well, afaikt)
>
>> Learning writes to the statistics files.  Classifying shouldn't, and I
>> think doesn't, though I don't know for sure.
>
> correct, concurrent CLASSIFY is fine.
[...]
On Mon, Mar 23, 2009 at 1:19 PM, Bill Yerazunis <[email protected]> wrote:
>   > Not trivial, yes, but mail systems have been reliably locking
>   > mailboxes placed on NFS storage for ages.
>
> Near as I can figure out, they're not.  There's a higher level server
> operating on the mailserver hosts acting as a lock manager.
>
> The killer is that NFS semantics allow clients to cache files _locally_,
> and not even inform the server that those files exist until it's
> "convenient" to do so.  Usually that's quick, but sometimes not.
>
> So, creating a lock file, twiddling for 10 milliseconds, and then
> deleting the lock file is only guaranteed visible on the local host.
> It may or may not ever be propagate in that 10 msec to the server,
> let alone to any other client machine.
>
> That's the problem that nobody seems to have fixed.

Note: below assumes clients can both init classify _and_ train (i.e.
read and write) operations.


Yup. NFS file locking is risky business.

There are networked locking mechanisms out there that provably work,
but I haven't seen those outside uni research labs.
Anything that's distributed and requires locking, generally uses some
sort of single point of contact aka locking manager. Compare with the
big boys in databases: DB2, Oracle, etc., which employ locking schemes
on a 'session' base (also similar to web servers handling stateful
HTTP transactions) at their 'front end' where a lot of effort is made
into ensuring each 'client' sticks to /one/ server for the duration of
one session, thus shoving the distributed locking issue into the
backend while placing this new peer-to-peer 'local-like' locking
system at the client/server interconnect.

As crm114 script is our 'client interface' (or rather: it's the crm114
the command line), you'd either need a crm locking manager to manage
access to the various CSS databases, or some other scheme which
ensures one crm114 is Boss over one (or more) CSS databases.
You can forget about operating systems providing guaranteed-to-work
network file locks. (They work for some. Mixed environments are more
trouble.)


the upscaling path I've been pondering for my own usage:

now we have a crm114 application, which uses file locking, which only
works on _local_ filesystems, i.e. on _local_ files. (Remember that
networked OS-based file locking support has caveats: it depends on
your file systems and your platforms, so is not the most portable
thing around.)

When you want to turn this into a 'distributed' concept, e.g. one
where training is done on box A and box B does the classification,
you'll have a bad case of network locking on your hands --> enter some
sort of 'database manager'.

The current idea I have is to do this in two stages:

first stage 2: get myself a crm114 _server_ as this will offer some
serious speedup over crm114 _application_ (no app start, no CSS mmap
cache loss, which now happens for all email users for every email as
crm114+mailreaver is started (and stopped) for every email. Poof goes
the mmap cache. Make crm114 app a crm114 server and your mmap cache
life increases tremendously. All you need is some RAM. And that is
bloody cheap these days.

stage 3: when such a 'does it all' server is not enough, there's the
'load distribution' thing database/web server style: multiple
networked crm114 servers. The simplest way to keep locking/collisions
workable while easy, is to have multiple 'individual' crm114 servers,
i.e. servers which perform 'classify' only (assuming 'classify' is a
read-only operation on the database, which for some classifiers, it is
not really; SVM et al indeed do special dances, but it's still a
write, so you need to 'merge' those writes somehow) while dedicated
machine(s) do the training, thus _locally_ creating fresh local copies
of the CSS databases, which are then transferred to each of the
classifying servers using a dedicated bit of protocol: safety is
accomplished by having such a 'classify server' put all classify
requests on hold / rejecting them, while the CSS databases are
transferred.
One could even 'hack' this using cronjobs and NTP-timesynced servers,
so the transfers (CSS database updates) coincide with temporary server
disable/enable.

stage 4: you need distributed servers, where each server does read AND
write to the CSS databases and clients can connect to any of the
(load-distributed) servers? Do the 'real thing': one 'master node'
manages the locks, while all the other servers request and adhere to
such locks while performing their classify/train/... operations. (I
don't expect I'll hit/need this stage. Would be a nice project for a
CS major in communications/operating systems.)
This, however, means you'll need to add your own
lock-requesting/releasing/probing network communication protocol
(preferrably on top of IP (TCP / UDP)) to the code.


The alternative is backstaging, i.e. provide a single point of contact
for clients, where classify/train/etc. requests enter the crm114
system; then this frontend/manager can distribute those requests among
distributed backends and manage the databases (and thus their locking)
at the same time. (Think 'instruction scheduling'.)


So far the drawing board ideas; not in GerH yet.



For today/right now there's solutions possible as well:

Yet another approach, which can be crm-scripted even, is to
'queue/offload', i.e. do all the read-only action you want (==
classify), but 'queue' the write operations (this introduces delays,
as did the 'sync' scheme mentioned above, between classify action and
train effect, so this means THTTTR and other 'looping' training
concepts must be 'queued' in their entirety to keep working as
expected). Then you can do a sync scheme as above when you have
multiple machines, which reduces to an either/or approach for one box:
the box which will process those queued training operations, will have
its own CSS databases (_the_ CSS databases in case of a single
machine), does _not_ accept any further client requests for the time
being, then processes each of the queued items in order (the
trainings), thus updating its local copy of the CSS databases
according to your wishes.
Once the queue has been processed, the CSS databases can be
distributed (sync scheme as mentioned earlier) to the other boxes once
they are available to accept such an 'update'. In case of a single
machine, no update/distrib work is needed: keep the CSS databases as
they are now and start accepting client requests again, like before,
where further write-operations (trainings) are, again, queued until
the next round.

In a non 24/7 usage pattern, the queue can be updated in the 'quiet
hours', for instance. (say 01:00-05:00 AM every day)

The most important bit here is that no client/user should be able to
access the CSS databases either through classify or train or any other
method, while such queued trainings are executed: during training, the
CSS database files are in a state of limbo which will produce
unpredictable output for simultaneous classify (or worst-worst case:
fatal errors).



A little 'twist' to this approach for instant gratification for
multiple users and instant training updates at the same time is to
take this approach and layer it in 'log/replay' scheme:

give each user his/her own personal copy of all the CSS databases, let
them use it any way they want, so train when training is desired,
etc., so no delays in training updates any time. Note the 'each his
own' here: ONE instance for EACH user.
How do users then benefit from each other's work? THAT is, alas,
delayed a little, as this concerns sharing, thus 'locking', thus the
'queued' processing.
Preparation per user: the code not only does the training on the local
copies, but also 'logs' each CSS state-altering operation AT LEAST
(NOT the individual trainings, but the 'need to train this input' bit:
message + what to do with it on a grander scale). Preferrably, ALL
user interactions are 'logged', as the 'merge' stage will/may behave a
little different. Thus, a user can classify/train to his heart's
desire every day, no fuss, and your code ALSO logs all the messages he
classifies and wishes to train like this:

scenario #1: message classifies to the users satisfaction, i.e. no
subsequent trainign rounds at all:
- item: content = XXX, action=classify, pR=10.23456,
pR_threshold=1.23456, determined_class=ABC, state=accepted
scenario #2: message is flagged as to be trained by the user:
- item, content = XXX, action = TOE/THTTTR/..., desired_class=ABC,
pR_threshold=1.23456, state=needs_training
(where the 'state' bit is redundant, as it's also in the 'action')

Next, once every user is off to lunch / home, another bit of the
machine kicks in (the log processor), which shuts down all the
client/user systems (nobody should touch those CSS files at this
time!), then picks up all the user logs, maybe even sorts them in time
order or other desirable order and then executes ('replays') those
actions, where:
scenario #1: you MAY find you need to train because this time around,
using the 'common'/master CSS databases, you find you don't get the
desired class at/beyond the desired pR-threshold for the given message
(hence the need to log ALL user actions, not just the 'state altering'
ones)
scenario #2: you MAY be lucky in that the training is not needed as
the classify already produces the correct result: after all, here you
are mixing the activity and instruction of multiple users into a
single set of CSS databases in a fully automated fashion, so it may
happen that the training due to another user's message(s) will
positively impact this message's classification

When all the 'user logs' have been processed (replayed into the
global/master CSS files), the code can copy/overwrite each users' CSS
database set with this 'common update'.

Again, it is essential that CSS updates/writes are done by a single
node at a time, while NO READS by any other nodes should occur during
that time.



This log/replay approach has the added benefit that you can take those
activity logs and use them to rerun/replay the entire scene at a later
time: since you not only logged the input, but also the
accepted/desired output with each user action, you have now created a
metadata format which can be used to reproduce your test results
and/or re-use this scene while testing different classifiers and/or
classifier settings in a fully automatable way. VERY handy when
evaluating classifiers and their performance.

Yes, it takes up additional harddisc space, and, yes, it takes time to
write that code, but the benefits are considerable IMO. Especially the
'can re-evaluate and/or compare this or other classifiers later on'
bit is a big bonus.


Given Thomas' current code, adding log/replay is just adding logging
script code to the existing code + writing the log processor, which is
the non-user-interactive copy of the user script: here, the 'what
category do you want this to be' interactive questioning gets it's
answers from the replay log file instead.
Plus additional shell scripts and maybe a bit of cronjobbing for
kickstarting this 'run this every night' automated log processor
script + copy/distribute CSS database files shell script.




-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web:    http://www.hobbelt.com/
        http://www.hebbut.net/
mail:   [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

------------------------------------------------------------------------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
easily build your RIAs with Flex Builder, the Eclipse(TM)based development
software that enables intelligent coding and step-through debugging.
Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com