Architecture advice requested

Hi, I'm hoping for some tips in architecting a fairly complex network 
app that I would like to port to POE.  It would be helpful to get some 
advice on how to best use POE to avoid as many dead ends as possible

The basic situation is that we have a VERY limited high latency network 
connection (2,400 baud, 1 second latency, $1.50/min) and we want to 
squeeze the maximum possible down that connection.

Right now I have a fairly advanced proxy application is written in 
vanilla perl with no special event architecture. However, for various 
reasons it needs a substantial update and I'm considering porting to POE 
to try and simplify the plumbing

Essentially we want to proxy SMTP, POP, IMAP (and HTTP) down a single 
TCP connection across the link, optimising packet size where possible, 
compressing all data using one of several (known in advance) compression 
algorithms which can change packet by packet (knowledge of the protocol 
allows minimal buffer flushing) and also some light rewriting of the 
protocol on the fly (for efficiency)

The  rough algorithm/components are as follows:

- Client side has an ordinary email program which talks to the proxy 
application
- Proxy application on the client side is relatively smart and parses 
the protocol (POP/SMTP) in some depth and rewrites the protocol a 
little, eg don't waste a packet sending jsut the username, batch 
username and password together in a single net packet.

- Server side can be less advanced because largely it's talking over a 
high speed network and replies are largely just streamed anyway.
- Server side does a small amount of protocol rewriting, but largely 
it's done at a higher level, eg dealing with connection drops/restarts
- Server side clearly deals with many, many more connections than the 
client side (order of a few hundred to a few thousand outstanding client 
connections at any one time).  Compression buffers consume a few MB 
each, therefore need to be careful of mem usage and ensure a forking 
solution which will return memory to the OS eventually

- Net protocol uses a single tcp connection and encapsulates using a 
simple protocol with a header stating original port, and compression 
protocol (this allows us to increase compression efficiency and manage 
tcp slow start more effectively)
- Compression buffer is flushed infrequently and the protocol layer 
knows when we need to flush (complicates where to place the compression 
code)
- We use a TCP CORK type emulation in the output buffer to decrease the 
number of packets where possible, eg received a "username ok", we happen 
to know that the "password ok/wrong" will be a millisecond behind, so 
try and batch them.  However, again the protocol layer knows when there 
is nothing else coming and can force a flush
- We can change the compression algorithm per packet.  Try to separate 
data and text for best compression.  Also if the connection is actually 
discovered to be over broadband then turn off the slow compression and 
use something more efficient
- I need to be able to get to the TCP connection handle to set a bunch 
of advanced options
- Perhaps most importantly on the client side we need to carefully apply 
backpressure on the application so that we don't end up with too much 
data in the OS net buffers.  With the network sending at 200 bytes per 
second, it's very easy to end up with relatively enormous buffered 
amounts of data and the client application quickly gets bored.  The two 
parts of the client proxy need to communicate and stop accepting data 
from the client as soon as we have more than a small amount buffered on 
the outbound side.


The question really is how best to break this up into POE components, 
wheels, filters, etc  Are there any gotchas to be aware of in terms of 
scalability, mem, ability to do intra communication, etc

Roughly each side looks like (say) POP server + POP Client back to back, 
but then there are some outer layers to wrap that inside another net 
protocol for transmission, and an outer layer on that to handle the 
compression.  Annoyingly the compression and net layer needs to 
communicate back to the inner layers to apply some backpressure and also 
to be told when there is an explicit flush (prioritisation of packets is 
also a future direction).  Also because the network is so slow there 
needs to be quite a lot of partial responses sent/received in order that 
each end doesn't get bored waiting for the whole response

How might I best break this up in POE?  What will my skeleton app look like?

This is a couple thousand lines of perl app today and not so easy to 
maintain. Hoping to improve on that

Thanks for any thoughts

Ed W