Re: [Roadrunner] Re: road runner on freebsd

Ryan Ripken <[email protected]> writes:

> I've been playing with RoadRunner with the tls and idxp profiles.  It
> looks great!

Hi Ryan,

Good to hear from you and that you're enjoing your journey through the valley
of RoadRunner. Even nicer that you like it so far. 

It's getting there, slowly but steadily. We've nailed some nasties and revised
the API now, and step by step closing in on a big 1.0 release. No date for
such a glorious event yet though, as that would probably be a fool's
challenge to the gods.

> You mentioned that you have to write a profile development tutorial
> sometime in the future.  Often these development tutorials follow the
> development of a trivial example from start to finish in a
> teach-by-example format.

Indeed. Also, it's usually an easier path to follow from both reader and
author perspective as well. This is, as you might've guessed, exactly what I
had in mind. 

> Here is my two cents on the subject, feel free to flame, ignore me, whatever...

Oh, I think i'll settle for whatever then and provide you with a normal answer
instead. :)

> I was wondering if you would consider writing a gzip profile as part of
> your tutorial?  Below I will outline the motivation for a gzip profile.

I'm aware of the discussion regarding various compression profiles and
mechanisms, including the variants proposed by Clipcode.com in the now-expired
draft-etal-common-beep-features (Still available on the web at
http://www.clipcode.org/peer/common-beep-features/).

First of all, the concept of utilizing the feature parameter is certainly an 
interesting one. Furthermore, the idea of doing intra-channel processing of
message content (rather than whole connection) is intriguing. A bit tricky to
implement efficiently, but we've been discussing it a couple of times over at
theRoadRunner mill. Maybe someday when a lightning bolt of inspiration
strikes...

For compression though, I agree with the point of view on the earlier beep-wg
discussion that as a tuning profile it's probably more useful than a
per-message one would be.

> One of the complaints about beep is that the use of xml adds unnecessary
> processing and bloat to the messages (
> http://ask.slashdot.org/comments.pl?sid=36070&cid=3892314 ).  I don't
> think this is necessarily true of just beep but more of a general
> complaint about xml.  The IDMEF standard, which I would like to see
> adopted, is also criticized for being bloated when compared with
> proprietary binary formats.  Although it would add to the amount of
> processing that needed to be done, compression can significantly reduce
> the size of xml messages.  In some tests that I have done I have found
> that gzip can compress idmef messages(which are xml) by a factor of 30.  I
> believe that beep's xml framing doesn't have quite the redundancy that IDMEF
> has but other custom beep profiles could also make heavy use of xml.

I don't agree that XML adds bloat to BEEP. It's usually an argument based on
ignorance and/or misunderstanding of the protocol itself and underlying
transport as well.

It's true that compression reduces traffic, although this happens at the
expense of (de)compression time, resulting in increased latency. In some
cases, you might well be spending more time handling compression than the
actual message, and you might end up with fewer messages/timeslice with
compression than without due to increased latency. 
For larger messages/streams where latency is not an issue, this will of
course not matter as much.

That said, I still certainly think there's room in this world for compression
profile[s] and is quite easily implemented. 

> *SNIP*
> If the tls implementation doesn't use compression then clearly some of the
> extra processing needed to compress a stream/message is saved because there is
> less data to encrypt.  I believe that it is also accepted that compression
> improves the security of a subsequent encryption.

Yes, and ASFAIK most encryption formats actually perform this step for that
reason, thus the compression being a side effect. I might be wrong though, as
I'm by no means well informed on the issue.

> I think a gzip profile would be very straightforward to do.  It should operate
> in a fashion very similar to the RoadRunner Rot13 profile with the exceptions
> that the gzip compression function is called instead of the rot13 function and
> care must be taken because the size of the messages will probably change
> (hopefully decrease ;) ) between the profiles that are reading and writing to
> the gzip profile in a session.

Quite right. Maybe some configuration exchange to select compression method,
level and possibly some other parameters. 

> For the idmef-over-idxp case compression would have obvious advantages
> (smaller messages and therefore less bandwidth used) and
> would only require that the client and server tune for compression
> initially if it is available and then they can tune to use tls and use the
> idxp profile in the normal fashion.  If the client or server don't support
> compression then it won't be used.  By having compression done by the
> transport the portability gained by using xml is maintained at the application
> layer.

Certainly so, as long as latency is not a key issue. In the case of a real
time analysis of IDMEF messages it might be better to avoid compression, as
RoadRunner can [almost now *evil grin*] do zero-copy message handling. That
means the memory block of an incoming message is never copied or moved once it
arrived on the network socket and stored in a buffer. In the IDXP/IDMEF case,
this would mean the pointer to the IDMEF XML block is passed directly to the
IDMEF handler (or XML parser if you will) from the network buffer, allowing
for some impressive performance. 

This might or might not be of importance to IDMEF and IDXP though, I'll leave
that to others to decide, but I'm sure the possibility will be appreciated by
somebody. Someday. Somewhere. Hopefully. 

> *snip*
> I think that people could clearly understand what a compression profile
> would do and why it would need to be used before tuning for security or
> similar.  As such it might be a good profile to implement in a tutorial, if
> that is how you were thinking about writing the tutorial.

Yes, I agree wholeheartedly. It's a very good and even probable idea for a
tutorial on how to write tuning profiles. It's simple, straightforward and
easy to understand.  

However, I think a tutorial on writing a simple exchange profile should come
firsthand. It is both a simpler and more common profile type encountered
when starting to write BEEP-based application protocols, and its purpose is
probably immediately easier to grasp to a newcomer. 

Please do correct me if I'm wrong though.

Finally, thanks for the ideas and shown interest, and of course also for the
feedback.

Cheers,

/D

PS. Please forgive me taking the liberty disclosing this response to the
roadrunner and idxp lists. It might well be of interest to the two
communities.

-- 
Daniel Lundin  | [email protected]
CodeFactory AB | http://www.codefactory.se/