Re: ANN: Etiquette: a protocol construction tool

* "Luke" == Luke Gorrie <[email protected]> writes:

Luke>  Could you post some more details of how Etiquette works, and
Luke>  which particular programming aspects you are aiming to make
Luke>  easier? It would be extra nice to see examples of non-Etiqutte
Luke>  protocol implementations with pointers to the parts you want to
Luke>  make simpler.

I believe that gathering communication-related code into compact and
dedicated protocol descriptions gives benefits in debugging and
maintainability. It should be easier to understand and modify an
explicit protocol than one which is implicit and scattered throughout
application code. Protocols, or parts of them, can be traced. Error
handlers can also be specified on per-protocol basis, and even
automated to some degree (e.g. out of band or invalid messages are
reported by the framework). Underlying network transport can be
switched without altering protocol body.

Also, unified interface to protocols and communication transports
allows to encapsulate application protocols into transports (for
instance, SMTP, which runs over TCP, could be used as a transport for
some other protocol. Or, SMTP itself could be run over a protocol
other than TCP).

Of course Etiquette is not a panacea. But hopefully, for certain tasks
it could make life easier.

Luke>  I've been browsing the code a bit, let's see if I've understood
Luke>  it. It looks like use of macros such as:

Luke>  (defprotocol smtp-mail () (transfer request (:from :initiator)
Luke>  mail-request) (transfer-if response (mail-valid-p :from
Luke>  :respondent) reply-250 reply-550))

Luke>  Expands into some code that updates data structures to
Luke>  understand a new state, and then these data structures become
Luke>  the input for execution engine that's essentially an
Luke>  interpreter. Is that right?

Correct. The sequence of transfer forms results in corresponding PHASE
objects being attached to the protocol object during
initialisation. Each PHASE object encapsulates information about state
transitions possible at this point, and during protocol execution this
information is used to determine applicable actions and out of band
messages.

Luke>  It looks like the machines get a stack, i.e. TRANSFER* will
Luke>  return after it is finished -- is that right?

Etiquette protocols are mostly procedural descriptions ('mostly'
because at the moment it is possible to transfer control from protocol
error handlers). Phases (the transfer forms) are executed
sequentially; if a subprotocol was invoked, current protocol execution
will resume after its completion (unless communication was explicitly
terminated in the subprotocol).

Luke>  Is the :from :initiator/:respondent there to distinguish
Luke>  between things done by the client and the server, such that a
Luke>  single protocol description defines both sides?

Exactly. Protocol describes communication between the roles. During
instantiation, one have to specify the role in which this protocol
object is intended to be executed.

I moved away from 'client' and 'server' terms though, in favour of
'respondent' and 'initiator' as role designators, since I found them
confusing on several occasions. Incidentally, BEEP framework
developers seem to choose exactly the same terms, so this was probably
a right move.

Luke>  Also, I'd appreciate a couple of sentences of comments for each
Luke>  of the major data structures, saying what they're for and how
Luke>  they relate to each other.

You may want to look at the section 3 of design-notes.txt (it should
be in the tarball). The document outlines the data structures
employed. But in short, there are five major classes:

- INTERACTION-PROTOCOL, that represents, well, an interaction protocol
- PHASE, encapsulating state transition data for each protocol phase
- COMMUNICATION-AGENT, mainly used to tie protocol with application
  logic
- COMMUNICATOR, that handles actual data exchange for a protocol and
  its subprotocols
- TRANSPORT, which wraps specific communication transport

Now, every INTERACTION-PROTOCOL has a number of PHASEs and a
COMMUNICATOR that it shares with its subprotocols. COMMUNICATOR talks
to a specified TRANSPORT.

Each action and conditional test method is specialised on protocol,
its role (via EQL specialisers) and a communication agent. This allows
to flexibly mix agents (e.g. if one wants an HTTP proxy, they could
inherit from a server agent and a client agent).

If you'd like any clarifications, do not hesitate to ask.

Luke>  here's what I'm looking for in a framework like this:

Luke>  Multitasking. One thread should be able to run many state
Luke>  machines, so that when one blocks on I/O another is allowed to
Luke>  execute.

I don't quite understand how can execution proceed in the same context
when blocking input operation is performed. Could you please elaborate
on that? Maybe an example where that would be useful?

But in general, I see no obstacles to using Etiquette in multitasking
environment. It's just that managing multitasking itself is outside of
scope of the project.

Luke>  Complete. Either general enough that I can _easily_ extend it
Luke>  for "weird" unanticipated cases, or complete enough to
Luke>  anticipate everything.

Etiquette is by no way complete. It was mainly intended to make
writing new protocols easier; however, it became apparent that with
relatively little additional work I can extend it to support a number
of existing protocols (via detectors and aggregators). Still, there
are protocols around that are tedious to implement in
Etiquette. Hopefully its applicability will extend gradually, but I
refrain from grand claims on completeness or generality.

Luke>  Some protocols have special requirements -- e.g. considerable
Luke>  state that is abstracted out of the state machine description,
Luke>  or complex timeout logic.

Agents can be good places to store application state not directly
related to communication. Also, second argument to defprotocol is
intended to contain auxiliary info not directly related to state
machine (error handlers, transport specifiers and such). It shouldn't
be too hard to extend the list of allowed options.

As for timeouts, I think finer-grained control is needed. E.g. SMTP
RFC suggests various timeouts for various stages of protocol, hence
per-phase timeout control should be provided.

Luke>  I may also e.g. need flexibility in how I parse my input
Luke>  depending on the current state.

Aha.. I hope this can be reasonably done by specialising
detectors/aggregators to protocol and state designator. If this will
work out, it'll get into the next release.

Luke>  Protocol encoding. Some nice libraries for parsing/printing
Luke>  protocol messages can go a long way. Frode Vatvedt Fjeld's
Luke>  `binary-types' package is a superb example of such a library
Luke>  for bit-packed protocols like the lower layers of
Luke>  TCP/IP. Something for handling ASCII protocols like
Luke>  SMTP/POP3/HTTP with similar ease would be just great. (Gilbert
Luke>  Baumann is doing some nice stuff for lightweight lexing/parsing
Luke>  -- maybe something for this?) It is a crime for messages
Luke>  specified by context-free grammars to be parsed ad-hoc in a
Luke>  language like Lisp. Doubly so when you have to worry about
Luke>  avoiding blocking on I/O.

Thus far, Etiquette performs encoding/decoding only to extent
necessary for proper state identification. Message contents are
defined by particular applications. I agree that content encoding is
important, but not so sure whether it should be addressed by Etiquette
or left for choice of application implementor.

But if that sounds like a reasonable extension to people, I'll look
deeper into that.

Luke>  I'm guessing that to really get this stuff right requires
Luke>  implementing at least a dozen protocols. Some friends did a
Luke>  framework along these lines for Erlang, called the "bit
Luke>  syntax". They spent a year implementing every protocol they
Luke>  could think of from the internet and telecoms world, extending
Luke>  and refining it as they went along. The result is very nice,
Luke>  and by all accounts it was great fun :-)

My approach is similar: implementing various protocols to find and
polish the rough edges of the framework. Naturally, there still many
issues left; however, am coding it alone, in my free time and for only
three months, so I hope for leniency :)

Luke>  Anyway, I know a few links that might be interesting to you:

Thanks! I'll definitely look into that.

--
  Eugene