Re: libcrm114 ready - but can't get into sourceforge --> libcrm114 usage and the p_db's
Ger Hobbelt <[email protected]> Tue, 5 Jul 2011 21:23:11 +0200
| Newsgroups | gmane.mail.spam.crm114 |
|---|---|
| Message-ID | <CAFP60frv7VghWsYRWTMpwyi8SBRP7qQZM3Ty06gsgRm3rW4ZWw@mail.gmail.com> |
--===============6209846587153754705==
Content-Type: multipart/alternative; boundary=0015176f02aa25374e04a7576a55
--0015176f02aa25374e04a7576a55
Content-Type: text/plain; charset=ISO-8859-1
>
> About mmap(): I was thinking that *if* I am not planning to "learn" or
> [...]
misclassified, it is not spam". On my small system, that will be done
> only a few times a day, so I plan to have it "one process invocation =
> *one* learn operation". Every time I fire this re-learning operation I
> will get a new state of the p_db's. And then I write them out. I suspect
> that I will need to write them to the new copies of the files, and tell
> the milter process that it needs to re-read (or re-mmap) the new files
> (or simply restart the milter). [Ideally, I would prefer to lock() the
> mmap()ed files, update them by the learning process, release the lock,
> and be sure that when the classifier process obtains the lock for the
> next classification round, it will "see" the new data and happily use
> it. But I somehow suspect that this will not be possible.]
>
Maybe this sounds stupid, but from what I read here, my suggestion is to
circumnavigate the whole lock+rewrite tangle all-together (filelocking is
notoriously flaky on networked storage and more) by using a 100% async
scheme like this:
1) do the mmap thing like you suggest for all readers. mmap may be
'marginally faster' as you load the data into RAM once for multiple runs
here, but for the scheme, it doesn't matter if you mmap of read().
2) (Not required) do training in bulk too: this is anticipating #3 below,
but for low volume you can go with 1-per-1 and do fine.
3) the crucial bit: you don't OVERWRITE but instead you create a new file
with the learned data.
4) when done with #3, the writer (the learn node) signals the readers that
the new version is available and they can discard and load at their leisure:
both versions are available, at least until all reader (classifier) nodes
have upgraded to the new version. 'Signalling' can be done using anything
from SIGHUP to 'polling' some signal file, heck, even a periodic dirscan
looking for new DB versions in the readers is fine. Another 'signal' way
would be to simply kill -9 all the readers==classifiers and construct the
system such that it will always start a new classifier when none is present
('live') on this node when the next mail is presented for classification.
This way you get a completely asynchronous system operation while not
requiring any locks what-so-ever, so you can distribute this layout to
multiple machines or whatever is your fancy. The tough part here is the
scalability of the writer node and the 'write db' I/O cost: when you use a
1-update-per-1-training, your first expected bottleneck is write:I/O; that
can be alleviated by going 'bulk', i.e. scheduled training: when at least 1
message has been accumulated in X time, process all msgs to be trained in a
single run and write a fresh classify db as a result. When 'bulking it'
doesn't suffice any more, you've entered a different game zone altogether,
and we're not talking 'small' anymore in any dimension. :-)
The benefit of the scheme is that you can be sure about not having any risk
regarding classify db corruption as the scheme is stronger than
single-writer, multiple-reader (which requires locks): this one is 'single
write, followed by infinite readers' (infinite ~ life until the next version
is signaled to the readers and they've all switched over).
If you can live with 'not 100% real time' classification updates, you can
use this approach; easier to code and less prone to
network/filesystem-specific sneaky locking troubles. The 'not 100%
real-time' sounds like a big loss, but consider the time it takes between
arrival of mail and the time when a mail M is read and thus identified as
'to be trained': there's a human in that loop, so it's going to be slow.
This way, though, system behaviour will visibly change for that human, so it
might be an issue anyhow (complaint: 'rest of my inbox doesn't
update instantaneously!')
If you want to be absolutely 'technically perfect' in your approach, you
could rerun (re-classify) yet-unprocessed email once a fresh learn cycle has
been detected as 'completed', though that sort of thing would only work on a
server when everyone is going in using IMAP instead of POP3.
My 2 cents.
PS: 'version history cleanup' of the DBs can be done as easy as through a
quota-based trigger: if disk is 50% filled with classify DBs, prune all the
ones dated before yesterday AND are not the latest two revisions. (so
classifier nodes never get pulled the rug from under them, no matter how
'sluggish' they are in switching to the latest&greatest.)
--
Met vriendelijke groeten / Best regards,
Ger Hobbelt
--------------------------------------------------
web: http://www.hobbelt.com/
http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------
--0015176f02aa25374e04a7576a55
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style=3D"margi=
n:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">About mmap(): I =
was thinking that *if* I am not planning to "learn" or<br>[...]</=
blockquote>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex;">misclassified, it is not spam". On my =
small system, that will be done<br>
only a few times a day, so I plan to have it "one process invocation =
=3D<br>
*one* learn operation". Every time I fire this re-learning operation I=
<br>
will get a new state of the p_db's. And then I write them out. I suspec=
t<br>
that I will need to write them to the new copies of the files, and tell<br>
the milter process that it needs to re-read (or re-mmap) the new files<br>
(or simply restart the milter). [Ideally, I would prefer to lock() the<br>
mmap()ed files, update them by the learning process, release the lock,<br>
and be sure that when the classifier process obtains the lock for the<br>
next classification round, it will "see" the new data and happily=
use<br>
it. But I somehow suspect that this will not be possible.]<br></blockquote>=
<div><br></div><div>Maybe this sounds stupid, but from what I read here, my=
suggestion is to circumnavigate the whole lock+rewrite tangle all-together=
(filelocking is notoriously flaky on networked storage and more) by using =
a 100% async scheme like this:</div>
<div><br></div><div>1) do the mmap thing like you suggest for all readers. =
mmap may be 'marginally faster' as you load the data into RAM once =
for multiple runs here, but for the scheme, it doesn't matter if you mm=
ap of read().</div>
<div><br></div><div>2) (Not required) do training in bulk too: this is anti=
cipating #3 below, but for low volume you can go with 1-per-1 and do fine.<=
/div><div><br></div><div>3) the crucial bit: you don't OVERWRITE but in=
stead you create a new file with the learned data.</div>
<div><br></div><div>4) when done with #3, the writer (the learn node) signa=
ls the readers that the new version is available and they can discard and l=
oad at their leisure: both versions are available, at least until all reade=
r (classifier) nodes have upgraded to the new version. =A0'Signalling&#=
39; can be done using anything from SIGHUP to 'polling' some signal=
file, heck, even a periodic dirscan looking for new DB versions in the rea=
ders is fine. Another 'signal' way would be to simply kill -9 all t=
he readers=3D=3Dclassifiers and construct the system such that it will alwa=
ys start a new classifier when none is present ('live') on this nod=
e when the next mail is presented for classification.</div>
<div><br></div><div><br></div><div><br></div><div>This way you get a comple=
tely asynchronous system operation while not requiring any locks what-so-ev=
er, so you can distribute this layout to multiple machines or whatever is y=
our fancy. The tough part here is the scalability of the writer node and th=
e 'write db' I/O cost: when you use a 1-update-per-1-training, your=
first expected bottleneck is write:I/O; that can be alleviated by going &#=
39;bulk', i.e. scheduled training: when at least 1 message has been acc=
umulated in X time, process all msgs to be trained in a single run and writ=
e a fresh classify db as a result. When 'bulking it' doesn't su=
ffice any more, you've entered a different game zone altogether, and we=
're not talking 'small' anymore in any dimension. :-)</div>
<div><br></div><div><br></div><div>The benefit of the scheme is that you ca=
n be sure about not having any risk regarding classify db corruption as the=
scheme is stronger than single-writer, multiple-reader (which requires loc=
ks): this one is 'single write, followed by infinite readers' (infi=
nite ~ life until the next version is signaled to the readers and they'=
ve all switched over).</div>
<div><br></div><div><br></div><div>If you can live with 'not 100% real =
time' classification updates, you can use this approach; easier to code=
and less prone to network/filesystem-specific sneaky locking troubles. The=
'not 100% real-time' sounds like a big loss, but consider the time=
it takes between arrival of mail and the time when a mail M is read and th=
us identified as 'to be trained': there's a human in that loop,=
so it's going to be slow. This way, though, system behaviour will visi=
bly change for that human, so it might be an issue anyhow (complaint: '=
rest of my inbox doesn't update=A0instantaneously!')</div>
<div>If you want to be absolutely 'technically perfect' in your app=
roach, you could rerun (re-classify) yet-unprocessed email once a fresh lea=
rn cycle has been detected as 'completed', though that sort of thin=
g would only work on a server when everyone is going in using IMAP instead =
of POP3.</div>
<div><br></div><div><br></div><div>My 2 cents.</div><div><br></div><div><br=
></div><div>PS: 'version history cleanup' of the DBs can be done as=
easy as through a quota-based trigger: if disk is 50% filled with classify=
DBs, prune all the ones dated before yesterday AND are not the latest two =
revisions. (so classifier nodes never get pulled the rug from under them, n=
o matter how 'sluggish' they are in switching to the latest&gre=
atest.)</div>
<div><br></div><div><br></div></div>-- <br>Met vriendelijke groeten / Best =
regards,<br><br>Ger Hobbelt<br><br>----------------------------------------=
----------<br>web:=A0 =A0 <a href=3D"http://www.hobbelt.com/">http://www.ho=
bbelt.com/</a><br>
=A0 =A0 =A0 =A0 <a href=3D"http://www.hebbut.net/">http://www.hebbut.net/</=
a><br>mail:=A0=A0 <a href=3D"mailto:[email protected]">[email protected]</a><br=
>mobile: +31-6-11 120 978<br>----------------------------------------------=
----<br><br>
--0015176f02aa25374e04a7576a55--
--===============6209846587153754705==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
--===============6209846587153754705==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general
--===============6209846587153754705==--