Re: Miscellaneous questions on OCamlNet + news

Hi,

Sorry to reply only now but I am very busy ans this is not (even
related) to my main job.

Interface: http://ocaml-cgi.sourceforge.net/netcgi/doc/
Code: http://ocaml-cgi.sourceforge.net/netcgi/src/netcgi/
      (maintained under bzr -- http://bazaar-vcs.org/)

New connectors: AJP-1.3 and SCGI.

On Sun, 25 Sep 2005, "Gerd Stolpmann" <[email protected]> wrote:
> 
> Christophe TROESTLER said:
> > - Netmime.storage : it would be nice that if the user decided for
> >   `Memory but the data is too large to fit in a string, `File is
> >   automatically chosen instead of blowing up. [...]
> 
> Yes, this is possible, and only a matter of implementation. It is,
> however, the question what "too large to fit in a string"
> means. [...] I would say a limit of several KB would be reasonable.

Ok, thus the better is probably to leave it into the user hands.  That
made me think that [arg_store] may not be flexible enough.  For a
start I added a parameter of type [cgi_environment] so one can have
different configurations for, say, different [cgi_path_info]. 
I also added [`Memory_max of float],... return values so it is
possible to specify a maximum size for the argument.  If that maximum
is exceeded, an "oversized argument" is created meaning that the name
of the argument is accessible but its body is discarded and any call to
[#value] or [#open_value_rd] raises [Argument.Oversized].

> [...] [value] method should not be used for file arguments at all,
> or things may fail at run time. [...] argument without [value]
> accessor, but only with [open_value_rd].  However, this makes it
> very uncomfortable to access arguments at all.

There is no easy solution to this problem.  If we want to keep a
uniform access to arguments, the current solution is probably
preferable.  Any error occuring during [Netmime.decode_mime_body]
(e.g. string too small or disk full) will result in a oversized
argument.  Calling [#value] on a file that is too big hasn't changed
-- but one could decide to also raise [Argument.Oversized].

> > - Mimestring.read_multipart_body could also be useful in the form of
> >   "fold" (left and right)
> Yes, why not.

Good (this makes it easier to drop the `Discard arguments).  I believe
it is better you do it as you are the more familiar with Mimestring
internals.

> > - When the request method is not allowed or when it is unknown, the
> >   library raises Failure.  This is not a correct behavior however,
> >   instead a reply with the appropriate status (e.g. `Not_implemented)
> >   should be sent (and this should not be the task of the user: exn
> >   launched by the lib must be handled by it if it can).
> 
> This usually causes a 500 Server Error. This is acceptable as general
> fallback error.

Yes.  But one can do better!  Some errors naturally call for error
codes invented just for them such as "411 Length Required",...  I
agree that for the *user application* 500 is a good general fallback
(and that's indeed what the lib does), but not so much for *protocol
errors* reported by the supporting library.

> Currently, the library does not try to produce error pages at all
> (with a few exceptions).  Sending error pages is usually the task of
> the web server

That may not best because an uncaught exception kills the application
(I assume it doesn't forks or use threads) and, with a remote
connector like FCGI, SCGI, or AJP, the server cannot restart it --
thus _all_ current users of the web app. will suffer.  If a default
exception handler is in place, that at least still allows the correct
parts of the applications to be run (for a concrete example, think
about a forum which has a bug in the administrative part).

> or of the application built with the library.  The library should
> try not to do so, because error pages need normally to be customized.

The solution I propose does only provide a default fallback -- as much
as the standard error pages of the web server do but with more info
(and logging) helping to understand the cause of the error.  The user
can treat in a special way any exception he cares by setting an
optional argument of the type Netcgi.exn_handler defined as

  type exn_handler = cgi_environment -> (unit -> unit) -> unit

(see Netcgi.mli for an explanation).

> >   The way I solved this is to define an internal HTTP exception
> 
> Well, this may be a debugging aid, but I think it is not a good
> solution, as explained.

I am not sure I understand your point as this solution provides a nice
fallback, a debugging aid, increases robustness AND allows
customization.  If I did not get your point, please rephrase it.

> >   Also, since in the new interface all handlers are callbacks, what
> >   about automatically finalizing the arguments when the callback
> >   returns (normally or otherwise)?
> 
> I have no opinion on that. Often, arguments do not survive one call,
> but I cannot say whether users keep further references to them.

Maybe others feel otherwise but in my opinion, it is easy to forget to
call [#finalize] and have a "file leak" (especially because the
default storage for arguments is `Automatic).  Thus I think it is
better that, if the user wants to keep the file, it pays special
attention to it and move/hardlink it somewhere else.  All connectors
finalize all arguments when the request handler finishes (of course
also in the case an exception terminates it).

I have also added a method [cgi#at_exit] that allows to register more
functions to be executed when [#finalise] is.  It is important for
example to reuse database connections -- we need to free them back to
the pool at the end (which is better done automatically by the library
providing the pool for fear of the user forgetting).

> > - The Netmime.mime_header_ro defines
> >
> >   method content_type : unit -> string * (string * Mimestring.s_param) list
> >
> >   while Netcgi_env.cgi_environment says
> >
> >   method input_content_type : (string * (string * Mimestring.s_param) list)
> 
> Ok, but see below for my opinion about purely aesthetic changes.
> [...]
> I see little value in the many renamings.
> [...]
> I hope you can understand that I have a slightly different view. There
> are already lots of programs using Ocamlnet, and it is not a good idea to
> change the API in a radical way without having a very good reason.

I understand your point of view.  It is true I was more placing myself
in a position of someboby who had little code written in OCamlNet and
that I was worried that the lengthy methods names were going to be
burdensome.  Having moved my code, that worry is gone -- the long
names are even maybe easier to remember -- and I have reverted most of
my changes to OCamlnet conventions.  Let me go into that in more
details:

- Reverted to cgi_argument (for mutability see below).

- cgi_environment has been restored with the exception of
  [input_content_type] (see below) and the state management (which is
  not the user business).

- cgi: reverted to the old names (but left the class name cgi as this
  is where the code needs to be changed to use the callback style).

- Whenever possible, when the old name lacked coherence with the usual
  OCaml conventions, I deprecated it and provided an alternative.  As
  an example, cgi_environment has [#output_ch] while the cgi object
  has [#output].  The fact that the latter may be transactional is not
  enough in my opinion to justify two names.  Moreover, [#output] is
  an actual _outputting_function_, not a channel, in the standard lib
  -- and [#output#output_string] or [#output#output] reads poorly .  I
  have recommended the use of [#out_channel] for both.

The deprecation however does not always work and some names/methods
were changed in non backward compatible ways:

  * [cgi_argument#content_type] is now the same as for
    [Netmime.mime_header_ro].  Frankly having two different signatures
    for the very same thing does little besides confusing the user
    (and making it harder to remember).  The new API offers a good
    opportunity to restore the coherence.

  * [permitted_http_methods]: `GET,... are used in [cgi#request_method]
    so it is normal they are specified the same way and it is more typesafe.

  * I have a problem with cookies.  The record datastructure you
    choose is not extensible which is bad to support the newer cookie
    parameters (comments, ports) -- ok they haven't really hit the
    street but they may, eventually.  Thus I have maintained the new
    abstract cookie type [Cookie.t].  The method [#set_header] however
    keeps its optional argument [~set_cookie] accepting the old cookie
    type and has a new [~set_cookies] (more coherent) for the new type.


> > - It would be nice to be able to turn a mime_header read-only. [...]
> >   A simple solution is to make [mime_header] a class that takes a
> >   argument of type [mime_header_ro] -- it will still define a type so
> >   it does not influence the rest of the code (but appropriate doc
> >   should be provided).
> 
> One can already coerce mime_header to mime_header_ro.

Yes but there is no conversion function [mime_header_ro ->
mime_header].  That's why I proposed to turn the class declaration of
[mime_header] into one.

> Another point I don't like is the new class type for environments.
> You see it only from the users' perspective, and throw out the parts
> that are necessary for the role as glue container between the
> connectors and the "CGI level" interpretation of the channel
> data. Effectively, this makes your design more monolithic, and
> harder to extend from the outside. We talked already about that.

Then I am sorry but I do not understand your point.  I think it is
good desing to have a clear separation between what the user sould
care about (or what is the common interface implemented by all
connectors) and what the services offered to the developer of a new
connector are.  [Netcgi] is the interface a user must use.  It is also
the interface a new connector must respect (even if implemented in a
totally new way) and the interface that a developer can use to build a
higher level framework compatible with any connector.  Of course, most
connectors share quite a bit of ideas, hence code to help with them is
provided in Netcgi_common (maybe you would prefer to have it renamed
to Netcgi_dev ?).

To summarize my answers to the points you raised in a previous email :

- The parts that "that are necessary for the role as glue container
  between the connectors and the 'CGI level'" are set apart in
  Netcgi_common and not the privilege of a connector (CGI).

- The connector modules are separate.  Thus it is easy to add new
  connectors to OCamlNet.  In fact I have added AJP-1.3 and SCGI since
  our last discussion.

- The "code size" problem is also no longer an issue -- the code in
  Netcgi_common is basically used in all connectors so it does not
  seem necessary to divide is further (but one could; say one could
  create a subdir with many small development modules -- maybe you
  would prefer this).

> I think an API for self-developed connectors is missing.
> [...]
> Furthermore, in the past we had an API to develop independent
> connectors. This is part of the success of Ocamlnet, let's not drop it.

I am not sure what you precisely reproach to the Netcgi_common API.
My understanding is that it is more flexible and powerful than the
current one.  The ease with which the various connectors are developed
is a testimony to that...

> I also do not see what the advantage is from making all arguments
> read-only. It will make life very hard in some cases - arguments
> are some sort of untrusted data, and sometimes it is necessary to
> fix them up before processing them. This isn't pure style, but
> it is sometimes needed.

Could you give a concrete example?  I am thinking of a library
developed on top of Netcgi: the author of the library may want to
process arguments on its own way before providing (some of) them to
the user.  For that it is better if the user cannot modify the
arguments -- otherwise one needs to duplicate them just to be sure or
cannot transform them on demand (i.e. not used => not transformed).
On the other hand, arguments are strings or files, so nothing really
prevent them to be modified (one can just discourage it)...

In another mail, you also mention debugging as a reson to have mutable
arguments.  But, as I see it, debugging should be provided by a function

  debug : cgi -> (cgi -> unit) -> unit

that may modify the CGI parameters (and those of the environment)
before passing it to the main function [cgi -> unit].  It could also
modify the output channels to redirect their data to a frame (when the
content-type is text/html), other frames serving to display/set values.

> > - Cookie module that supports Netscape, RFC 2109, and RFC 2965 [...]
> 
> Another remark. In netstring, we have now the module Nethttp.Header
> where all kinds of parsers and printers for HTTP headers are
> collected. Maybe the low-level cookie stuff should go there.

Fine with me.

> > - The "run" functions has been kept for lack of a better name
> >   (suggestions welcome).  Netcgi_fcgi and Netcgi_ajp are ready to
> >   handle concurrent connections once a good model is found (for the
> >   other connectors, the script has no control about its concurrent
> >   behavior).
> 
> Well, I think it is too early to make a good design. We need the MPM
> modules first.

Agreed.

> Another, final point. You changed the license to LGPL. I don't see
> why. The LGPL is rather pointless for O'Caml code, and we have the
> much simpler zlib/libpng license because of this.

No problem with me, LGPL+exn is just the standard licence I use for
OCaml code and I insert it with a macro into new files...  I changed
the licence to the one you use (if I made mistakes, point them to me,
I do not care so much about the licence for this lib as long as it is
free).

I have added an FDF test -- updates a PDF form.

Regards,
ChriS

Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642