Re: Re: [Caml-list] Re: Common CGI interface

On Thu, 16 Jun 2005, Gerd Stolpmann <[email protected]> wrote:
> 
> we should discuss the various aspects in depth to see whether it is
> really worth changing the API. There is a very important downside
> [...] drive users away from using Ocamlnet.

That's why I proposed to develop the new modules in the sub-directory
netcgi/ at the same level as cgi/.  That way, nothing changes for
current users and one can develop the new API without hurry.  One
should also take this opportunity to remove the remaining limitations
in the library (e.g. FCGI string limit and develop AJP 1.3) and, IMHO,
to get rid of regexes (they are really simple so a convenience rather
than a real need; benefits are increased speed and, more importantly,
no dependency on a C lib which means easier porting -- I am thinking
especially about native win32 here).

> (a) [...] I call your approach "monolithic" and the current API "polylithic"

The fact that the interface is a single file is only a convenience to
present it.  I basically agree with your design (see below).

> (b) engineering. In the monolithic API, external people cannot add
> their own connectors to the module namespace. [...] difficult to
> have connectors optionally built.

I agree with that.  I am also thinking to add a mod_caml connector.

> (c) Code size. When the executable is linked, there is a step called
> "garbage collection" [...] only for top-level modules, [...]
> complaints why programs are so large.

Maybe it is a good time to reevaluate whether the current web of
dependencies is what we want.  See the attached Postscript files (in
the first one, the transitive dependencies are not show in the
interest of readability.

> As a compromise of the three mentioned aspects I can imagine the
> following: [...]

I agree with your division.  However I would not just add these
modules to the others (would probably just be more confusing) but to
develop them into a separate directory netcgi/, thus name them

Netcgi    (* base module, not connector specific; general aux fun if any *)
Netcgi_cgi         (* connectors sharing a common philosophy + specific *)
Netcgi_fcgi        (* functions per connector if needed *)
Netcgi_ajp
Netcgi_mod_caml
Netcgi_...


> 2) Now to the connectors.

The tough part. :)

> (a) One of your ideas is to have a "run" interface for sequential connectors.

The idea of [run] was more: no hassle, I'll use whatever default
concurrency is provided by the connector; I just want my script to
run.  I thought (maybe naively) that, in many cases, it is just what
one wants.

> (b) Regarding the socket interfaces: I don't understand how they work.
> Can you sketch some examples?

  open Netcgi_ajp

You create a socket with [socket()].  Then, several processes or
threads can accept connections on it.  Then you have to decide whether
you want to deal with connections in separate threads or processes
(e.g. with a pool of threads).  If one does not want these, one can
use [accept_threads] with [~thread:(fun f -> f())] (the default).  If
one wants a new thread or to access a pool of threads, define
[~thread] accordingly.  For [accept_fork], on passes a [fork]ing
function which must return the process id and a file descriptor to
communicate with the script (e.g. say "shutdown script").  A simple
example of fork is

  let fork f =
    let (infd, outfd) = Unix.pipe () in
    match Unix.fork() with
    | 0 -> f outfd; exit 0  (* or the double fork trick *)
    | n -> n, infd

but of course it is also possible to send the job to a pool of
processes.

After accept, one can run a ['a connection_handler].  It can be
created with [handle_connection ~cookie:StringCookie.marshal f]
running [f] on each connection and reacting appropriately if [f]
raises an exception -- i.e. log it and exit gracefully (do not let the
thread or process die abruptly).

There are a few details to get right in the signatures but you get the
idea.

> (c) Unfortunately, the URL-binding stuff of AJP has been removed.

I'll read more about AJP.  Do you have a good references besides the
spec?  How is this different from PATH_INFO ?

> type binding_list =
>   ( url_filter * action ) list
> 
> type binding =
>   [ `All of action
>         (* All URLs are handled by action *)
>   | `Select of binding_list
>         (* Select the first matching action *)
>   ... (* maybe more styles *)
>   ]

It is fine with me as long as one can discriminate the "end of URL"
with e.g. [#cgi_path_info] -- sometimes dynamic URLs are indeed
needed.  Now, if one cannot imagine an additional style, maybe

  type binding = (url_filter * action) list
  and url_filter = url -> bool

is enough -- (fun _ -> true) is easy enough to write -- or maybe a
[any_url] can be provided which also allows to have a "default" (catch
all) final case:

  [(fun s -> start_with s "admin/",  admin_script);
   (any_url, default_script)
  ]

If an URL that is not matched is requested, a default error message is
sent.

> For the coming HTTP daemon, one needs URL dispatching, too

I will have a look at it -- no time yet.

> In general, I would more like a unification of the connectors based
> on their semantics (or better, semantic implications).  [...]

This is a good idea.  My only worry is flexibility (see below).

> type runtime_model =
>   [ `Any
>   | `Sequential of ...
>        (* would be interpreted strictly, e.g. in the CGI case the
>         * activations would be serialized using mutexes
>         *)
>   | `Concurrent_processes of ...
>        (* arguments describe that in detail: number of processes,
>         * limit of the process lifetime etc. One can leave things
>         * unspecified, especially when the web server controls them
>         *)
>   | `Concurrent_threads of ...
>        (* Similar idea *)
>   ]
>   (* The arguments are objects, to allow subtyping *)
> 
> val cgi_entry : ?runtime_models:runtime_model list -> binding -> unit

(Here I do not understand why the [runtime_models] is a list.)  Does
[`Any] means "the default model of the connector"? -- if so, maybe
[`Default] or [`Dont_care] is better?

What is not clear to me is how to make this general enough not to need
the low level stuff anymore (at least 99.9% of the time).  For
example, how do you access an existing pool of threads?

I'll have to think over it more but in the interest of having your
point of view on other issues, I send the mail now... :)

> (* test_entry will be special because input does not come from
>  * environment *)

Yes, a convenient test connector has yet to be designed.  The problem
is that there are two kind of testing that are interesting: offline
testing (run in a shell) and "online" testing (run through the web
server).  In the later case, one may want to "frame" the output to add
debugging controls or information in other frames... or at least offer
the primitives that make that possible by a "template system".

> 3) Concerning the Random module, I think this is much better solved in
> Cryptokit. So better remove this module at all.

Ok.  A reference to it in the doc would be nice though.
(Done in http://ocaml-cgi.sourceforge.net/netcgi/Netcgi.html).

> There is a simplification for `Transactional. Well, transactions with
> file buffers are not very common, but sometimes one needs them.

Not sure what you mean here.

> Cookie module: This must be improved. Other languages have much
> better cookie support.  Especially one needs a cookie storage that
> is independent from the representation in the header.

You mean for setting and retrieving data?    There is unfortunately no
convenient way of storing any value in a cookie since we do not have
dynamic typing.  So we have to precise beforehand what types we want
to encode/decode (and how).  That implies that [environment] and
[cgi] will carry a parameter, the type of cookies.  The problem is how
to instantiate this type.

One possibility is to functorize all connectors.  This seems too heavy
to me.

So I adopted the idea of requiring a ~cookie argument to connectors to
handle encoding and decoding of these.  For a given connector, only 1
cookie type is possible (as a consequence of static typing) but of
course it may be a variant.  A default StringCookie is available.  A
functor creates Cookie modules with specialized types and encoding /
decoding functions.  See
http://ocaml-cgi.sourceforge.net/netcgi/Netcgi.html#2_Cookies for
further details (some names may need to be changed for better ones).

One may also imagine adding an [output_js : t cookie -> string] that
returns some javascript code for setting the cookie.

Is that the kind of thing you have in mind?  Otherwise, could you
detail and/or give pointers?

> Shorter names for argument methods: Well, this is hard to change
> now. It would break all programs at a lot of locations.

Since the old module will still exists, the new ones will only be used
for new programs.  [arg*] are likely to be used very often so short
meaningful names are an asset I believe.  (Now, one may provide a
script for all s/argument_value/arg_val/ ... if wanted.)

> What is the idea of arg_true? This sounds very Perlish, and imports bad
> habits ("0" is false and e.g. "0E0" is true).

Well, yes.  Maybe a #arg_exists is better (true only if the arg is
present, whatever its value).  It is just a convenience to toggle
things.

> "log" is primarily part of the environment.  [...] not a channel,
> because we cannot guarantee this. It is simply a function like val
> log : string -> unit

This is fine (and adopted in the current version).

> Maybe I overlooked other ideas in your proposal.

Well, I removed the possibility to mutate arguments as I argued in a
previous mail (it is better for libraries build on top not to
interfere).  Since you said nothing about that, I assume you agree. :)

ChriS

---
P.S.  I have brought up to date http://ocaml-cgi.sourceforge.net/netcgi/
with the above discussion.