Re: Strings all the way down?

On Fri, 1 May 2009 01:52:22 +0200
paolo <[email protected]> wrote:

> > I have some large binary objects that I'm trying to handle - about
> > 2.6 MB. These are intermediates in an image processing routine that
> > would helpful to keep in memory rather than writing to a file. It
> > seems that they are getting truncated to 2.0 MB in memory. Run with
> > the -p option,  
> 
> hm, if I make a .crm like
> 
> # crm '-{window; isolate (:x: :v:) //; input (:x:) [/dev/urandom]
> #         eval (:v:) /:#:x:/; output /v=:*:v:\n/}'
> 
> 
> I get an error since :x: has grown up at 8M; and if it slurps in a
> 6MB file I get v=6M, so seems the only limit in effect is the buffer
> length controlled by -w opt. What do you get on your side?

Wow. If it was a memory issue, it would have failed much more
spactacularly. That's useful to know.

I was able to establish that it's an EOF issue by using syscall with
the <keep> flag to call the mapper program, but I don't have any way to
guarantee that I am getting *all* of the image, although I am capturing
*more* of the image output in general and all of it in *some* cases.
With the <keep> flag, what appears to be happening is that the control
is returned to the script when the first EOF marker is encountered.
Subsequent calls will retrieve more information from the pipe if
available, but it's still possible to come out short. My suspicion is
that there is more than one EOF in the pipe, but you can't read past
the first EOF in the pipe after the minion dies with any of the flags
available to syscall.

That suggests creating a file-like object so that I can either use the
window command with the <eofretry> option or use some form of input that
allows me to (discover and) specify the size of the file. A FIFO would
work, but doesn't have enough of an advantage over a physical file for
this application. 

How would I access file descriptors within a script? That's a topic
that I need for working with socat anyway.

[begin ramble]
In case anyone is curious as to how all this ties together, I run a play
by email Diplomacy game server based on a 15 y.o. server application. I
wrote scripts to convert MIME email into text/plain messages for the
minimal RFC 822 implementation of the server. Then I developed CGI
scripts that parse web input into RFC 822 messages for submission then a
script that intercepts outbound messages for alternate forms of
consumption while actually improving throughput. I need to be able to
munge internal data formats as well as I/O streams for processing by
both internal and external applications. In addition, the security
model for the base application is rather naive so the ability to filter
both input and output is necessary to scale the application responsibly.

Currently, players use external sites to access maps. These sites build
maps based on results received by email, creating no end of network
delay and reliability issues not to mention, well... The mapping script
that I am coding right now munges an archive file for consumption by a
Python program that parses the archive for results and modifies a
template to create a PostScript file. I'm also generating makefiles at
runtime in order to generate new maps only when there are new results.
Eventually I need to reverse engineer the PostScript process so that I
can use SVG instead, but in the interim the crm script already has
custody of the process, so I'm extending it to generate the bitmaps
with ImageMagick. The advantage of using crm over a shell script is
that I can store intermediate files in memory and feed them into
multiple processes. Also, certain file formats - especially PostScript
and SVG - can be further altered without modifying the original
templates... I can replace all occurrences of "Diplomacy" on a map with
"Diplomacy on USAK", for example.
[end ramble]

I'm not even using classifiers yet, although I have some applications
planned. It just happens that it's easy to think of this application as
a set of filters. In fact, the operations are so straightforward that
the biggest obstacle to moving to BlameBarack was the fact that :_pos:
variables are implemented differently.

Chris

------------------------------------------------------------------------------
Register Now & Save for Velocity, the Web Performance & Operations 
Conference from O'Reilly Media. Velocity features a full day of 
expert-led, hands-on workshops and two days of sessions from industry 
leaders in dedicated Performance & Operations tracks. Use code vel09scf 
and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf

_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iQEcBAEBAgAGBQJJ+1W5AAoJEASgqNsqZfCHnz8H/jtMV3KYNgwCkrSyy7oC9eAx
jE+2xHbYzVTTRiSWPI/qTcqlzQtvcoj3Ha2ZImmMagnLY+cRQnUN8eqy0axGVpAp
B+1+7Ds1Gvcqxl7Z8s25sObJLexfzbYJvCrXjJBlJXusnGkgVfsjt3OqdQtg7Vr1
euItlApNCHITJxBU4m0NE2l1MpwuWF9N4L+yAesFLLJIaJQPjXBiJPFsPq8FpECv
KUvAyEHT3NeW5w1nE0OpT1WbmVF8FC+jFoqViPC1lHATrwJDy45SCygKg86CM1dr
qGyxcC2tnq++I6bIrnz5JOum3WwmpkRZLhCeDZq49/70Zj3H6VthkbFfZ4t3YOw=
=/AgQ
-----END PGP SIGNATURE-----