Re: ANN: Etiquette: a protocol construction tool

Eugene Zaikonnikov <[email protected]> writes:

> Etiquette is an interaction protocol construction toolkit.  The
> project goal is to build a framework for rapid design of network
> communication code.

This is intriguing :-). I write a fair bit of code like this, and have
been pondering what a nice framework would be. Lately I've been
interested in using formalised state machines like your approach. It
seems good for making the code explicit and for getting good symbolic
debugging information. So I'm interested.

Could you post some more details of how Etiquette works, and which
particular programming aspects you are aiming to make easier? It would
be extra nice to see examples of non-Etiqutte protocol implementations
with pointers to the parts you want to make simpler.

I've been browsing the code a bit, let's see if I've understood it. It
looks like use of macros such as:

  (defprotocol smtp-mail ()
    (transfer request (:from :initiator)
              mail-request)
    (transfer-if response (mail-valid-p :from :respondent)
              reply-250
              reply-550))

Expands into some code that updates data structures to understand a
new state, and then these data structures become the input for
execution engine that's essentially an interpreter. Is that right?

It looks like the machines get a stack, i.e. TRANSFER* will return
after it is finished -- is that right?

Is the :from :initiator/:respondent there to distinguish between
things done by the client and the server, such that a single protocol
description defines both sides? If so, that sounds interesting -
what's the idea?

If that's totally wrong, please point me in the right directions
:-). Also, I'd appreciate a couple of sentences of comments for each
of the major data structures, saying what they're for and how they
relate to each other.

Anyway, before I make a fool of myself by talking too much about code
I don't understand properly :-) here's what I'm looking for in a
framework like this:

  Multitasking. One thread should be able to run many state machines,
  so that when one blocks on I/O another is allowed to execute.

  Complete. Either general enough that I can _easily_ extend it for
  "weird" unanticipated cases, or complete enough to anticipate
  everything. Some protocols have special requirements --
  e.g. considerable state that is abstracted out of the state machine
  description, or complex timeout logic. (TCP is a good example of
  both.) I may also e.g. need flexibility in how I parse my input
  depending on the current state.

  Protocol encoding. Some nice libraries for parsing/printing protocol
  messages can go a long way. Frode Vatvedt Fjeld's `binary-types'
  package is a superb example of such a library for bit-packed
  protocols like the lower layers of TCP/IP. Something for handling
  ASCII protocols like SMTP/POP3/HTTP with similar ease would be just
  great. (Gilbert Baumann is doing some nice stuff for lightweight
  lexing/parsing -- maybe something for this?) It is a crime for
  messages specified by context-free grammars to be parsed ad-hoc in a
  language like Lisp. Doubly so when you have to worry about avoiding
  blocking on I/O.

I'm guessing that to really get this stuff right requires implementing
at least a dozen protocols. Some friends did a framework along these
lines for Erlang, called the "bit syntax". They spent a year
implementing every protocol they could think of from the internet and
telecoms world, extending and refining it as they went along. The
result is very nice, and by all accounts it was great fun :-)

Anyway, I know a few links that might be interesting to you:

  Proposals for and Experiments with an Erlang Bit Syntax.
  http://www.erlang.se/euc/99/binaries.ps
  (Note: this describes a prototype that is more featureful than what
  the production Erlang currently has. Also note that Erlang itself is
  already designed for writing concurrent state machines easily, so
  and the Bit Syntax focuses on protocol encoding/decoding.)

  For writing an interpreter that can handle scheduling-out and still
  do subroutine calls, you want explicit continuations. Guy Steele
  posted such an interpreter to the ll1 mailing list, it's very nice
  and maybe inspirational -- I saved a copy:
  http://www.bluetail.com/~luke/misc/foo1.lisp

  If you read Emacs Lisp and have some basic familiarity with Erlang,
  you might get something out of my extensions to Elisp for
  erlang-style concurrent state machine programming. Described in the
  paper "Distel: Distributed Emacs Lisp (for Erlang)":
  http://www.bluetail.com/~luke/distel/distel-euc.pdf
  (Behind-the-scenes Distel also includes a framework for writing
  non-blocking network-attached state machines, used to implement the
  slightly complex Distributed Erlang protocol state machine.)

So - thanks for posting an interesting piece of software!

Cheers,
Luke