Re: Daemonizing CRM114 using FIFO

Thanks for your detailed response.  However, my question wasn't so much about how to solve this sort of problem in general but about how to do it specifically in CRM114.

First, let me tell you that this will be an asynchronous solution.  When it's all said and done, the client will send a unique ID along with every message and the :do_it: CRM function will parse the classify result and write the unique ID and its category number to result.txt.  There will be another process (a thread actually) that reads the result file at a later time.

I thought about using WINDOW statement to parse the content into messages but it can only read from stdin, correct?  In other words, can I use WINDOW statement to read the input from a FIFO?  Reading from a file is not an option for me...

Let's say I go with the lengh+content solution where 1+ message(s) are delivered at a time via a FIFO.  How would I write my :do_it: function so that when it's called (close() is received) it outputs the correct number of results before exiting?  Would I use MATCH and LIAF?  An example would help a lot since this would be my first attempt at writing a CRM114 script.


Thanks!

-James


----- Original Message ----
From: Ger Hobbelt <[email protected]>
To: James Lee <csejl-/[email protected]>
Cc: [email protected]
Sent: Monday, September 22, 2008 2:22:39 PM
Subject: Re: [Crm114-general] Daemonizing CRM114 using FIFO

I don't know how well versed you are will this sort of stuff
(communication protocol design), so I'll start at the beginning:


FIFO's are streams and suffer from the same issue as, for instance,
TCP streams. Using stream protocols to transfer messages is always a
hassle - as you found out.

Now there are a two levels of trouble, (a) one which is 'easily'
solved, (b) the other is much much harder to do 'properly'.


(a) is 'delineating' a message - the 'end of message' marker. Two
solutions there are used very often in generic communication
protocols:

1)- either define your messages as 'length + content': each message
starts with a 'header' which lists the number of 'content' bytes that
will follow. The easiest way when you go this route is to have the
header have a FIXED size. Since CRM114 is quite good at chopping text,
you might think up a header like, for instance, one line of text,
containing the length (in characters = bytes) of the content
following. To make 'looking at' the stuff while debugging a bit easier
you may consider adding a 'terminating' LF (end of line) character
after the content - it's not necessary this way, but it nicely
outlines the 'length' specifying header line when you look at the raw
traffic when debugging.

You can use crm114 script commands to 'cut out' the content again once
you've let CRM114 decode that length. It doesn't even HAVE to be fixed
as, using the example above, you can use the extra LF characters in a
regex to grab&decode the header for each message.

2)- another way communication protocols handle the 'delineation' issue
is through 'escaping' special codes - like you asked for. This method
does not require a 'header', nor the specification of a 'message
length' anywhere. Say we declare byte value 0 (ASCII NUL) to be our
EOM (End Of Message) marker. That means we can't have a NUL in our
content... unless we escape any NULs in there. For that, we need
/another/ byte value to signal the receiver an 'escape' is happening.
Assume we use ESC (byte value 27(decimal)) for that PLUS (as now ESC
can't show up in the transmitted content either) the agreement that
byte value 0 in content --> ESC '0' (ASCII ZERO = 48(decimal))
byte value ESC(27) in content --> ESC 'E'
(This is an example.)

Given those rules (very many protocols from the time when terminals
and RS232 lines were hot stuff work like this) we can transmit ANY
byte value in a message content, while a lone NUL(0) unambiguously
signals End Of Message.

Sample 7 byte message {ESC 'P' NUL "Hello"} would thus read as
(INCLUDING the End Of Message Marker!) when transmitted:

ESC 'E' 'P' ESC '0' 'H' 'e' 'l' 'l' 'o'
NUL

Again, CRM114 can handle this kind of thing using regexes to alter
those escapes back into the original byte values as soon as you've
isolated the individual messages.

----

These are the two basic solutions for 'delineating' your message -
they are not specific to CRM114, as both are generic concepts used in
a lot of communication protocols where messages have to be extracted
from data streams.

Of course, you can take the 'escape' idea and take it further like
Internet email once did: encode all content as base64 (the
historically sensitive would suggest UUencoding the data here ;-) ),
which leaves you a whole slew of byte values to use for End Of Message
markers and such (in fact: all byte values < 32, excluding CR and LF),
plus all the 'high ascii' values ie byte values >= 128(decimal).



(b) Which leaves the second issue, which you already ran into and
resolved by open+close on the one side plus wait on the other: this is
the hard nut to crack: asynchronous versus synchronous communications.

I can't provide you with any truely portable solution for it when
excahnging messages over stream protocols while 'keeping the
connection open', because there isn't. Several systems out there offer
'flushing' API calls which work on FIFO's; some systems can be
'tormented' into flushing outgoing data by cutting down the transmit
buffers to the very bare minimum (check your ioctl() and related
APIs); some systems cannot be tricked in any useful way but by
'padding' messages with extra data, that is: enough extra data to
ensure that particular system's buffers WILL be flushed by 'pushing
out' the message content through /overflowing/ the system buffers with
'padding bytes': the worst I've seen was 8Kbyte padding per message
and I am still not sure if that was really necessary given a bit of
proper research but the guy who wrote it was short on time and low on
OS knowledge. It worked for the sub-1K messages the system was
transmitting.

If you do 'padding', you can use anything, but it is useful to use a
'special byte code' for that as well, say FF (hex), so that the
receiver, who WILL have to handle all that extra data, at least can
quickly recognize the padding for what it is: fluff.


Why do I address this issue (b) separately and at all?

Because you should ask yourself if your system needs 'synchronous' or
'asynchronous' processing by CRM114: in other words: do you need to
receive the answer for each message before you can submit the next
(which means you're stuck with 'synchronous' communication)? Or can
you submit several messages without the need to receive the first
answer yet; just as long as the /order/ of transmitted messages is the
/same/ /order/ in which the answers return (==> asynch communication
is allowed)?

In the latter case, you can do without the waiting for each message;
you can transmit all messages, each properly delineated in one of the
two fashions listed above, and have CRM114 process them batch-wise,
returning the answers in order, though delayed -- as CRM114 will wait
for the 'input' to complete (= FIFO closed) before processing the
input. In the latter case, you basically construct batches for CRM114:
it'll be fast, but you have to ensure the total batch stays within the
buffer limits for CRM114 data processing -- see the CRM114 command
line options (I think it's -s, but not sure and can't check right now)
for when you wish to see or change that setting.


Does this help you along?







On Mon, Sep 22, 2008 at 7:33 PM, James Lee <csejl-/[email protected]> wrote:
> Hi,
>
> I'm in the process of daemonizing CRM114 and had a few questions.  First, here's what I have so far:
>
> - My C program (server) creates a FIFO (/tmp/dynamic_fifo) and runs this command: /usr/bin/crm /home/me/dynamic.crm < /home/me/start.txt
>
> - /home/me/dynamic.crm looks like this:
>
> {
>    isolate (:content:)
>
>  # classify dummy content to load the css files so that we don't have
> to reinitialize repeatedly in the :do_it: function below
>
> classify <osb unique microgroom> (\/home\/me\/css\/A.css
> \/home\/me\/css\/B.css \/home\/me\/css\/C.css \/home\/me\/css\/D.css)
> (:content:)
>    :loop:
>    syscall /:do_it: <\/tmp\/dynamic_fifo >>\/home\/me\/result\/result.txt /
>    goto /:loop:/
>    exit
> }
> {
>    :do_it:
>    input
>
>  classify <osb unique microgroom> (\/home\/me\/css\/A.css
> \/home\/me\/css\/B.css \/home\/me\/css\/C.css \/home\/me\/css\/D.css)
> (:_dw:)
>    output /lala :*:_dw:\n/
>    exit
> }
>
> - Another C program (client) opens the FIFO in write-only mode and writes some content (a "message" to be classified) in a loop.
>
> The
> first issue I'm having is that I have to open() and close() the FIFO
> every time I write inside the loop.  My finding is that the close()
> call simulates typing ctrl-D on the console.  The second issue is that
> I have to usleep() or run a dummy for-loop after the close() to make
> sure CRM114 receives and acts upon one chunk of data at a time.
> Without the wait, my :do_it: function only gets called once.  I think
> this has to do with how the kernel buffers and writes to FIFO.  It
> appears that it ignores the close() if the same FIFO is opened and
> written right away.
>
> My goal is to have the server program run
> forever and have the client send as many messages as possible as fast
> as possible.  Having to open() and close() every time I send a message
> slows down the process but it's not a biggie.  The wait part after each
> close() seems to be the major performance bottleneck since I can only
> sleep as little as 15-20 ms on the 2.4 kernel and 4-6 ms on the 2.6
> kernel.  The dummy for-loop takes too many CPU cycles and is just as
> slow.  And neither of them is a guaranteed solution.
>
> In this
> model, is there a way for me to use a special word or character to mark
> the end of a message so the CRM114 part doesn't depend on whether
> close() is received or not?  If not, is there a better way to
> accomplish this?
>
> Thanks in advance,
>
> -James
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Crm114-general mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/crm114-general
>
>



-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/