Re: Miscellaneous questions on OCamlNet + news

Hi,

Ok, I'll try to answer the questions and comment on the proposed changes.

Christophe TROESTLER said:
> - Netmime.storage : it would be nice that if the user decided for
>   `Memory but the data is too large to fit in a string, `File is
>   automatically chosen instead of blowing up.  From scanning Netmime,
>   a strategy could be that if the Netbuffer blows up, one copies its
>   content to a file and continues with it -- I haven't really thought
>   of the messy details though (not duplicating nor losing data).

Yes, this is possible, and only a matter of implementation. It is,
however, the question what "too large to fit in a string" means.
If taken verbatim, on a 32 bit platform arguments up to 16 MB are
kept in memory, and on a 64 bit platform arguments can be even
much larger. Normally, I would say a limit of several KB would be
reasonable.

The other thing is: Does it make sense? I mean, we have a real problem
here, because the [value] method should not be used for file arguments
at all, or things may fail at run time.

In general, I would say it is an option to have another kind of
argument without [value] accessor, but only with [open_value_rd].
However, this makes it very uncomfortable to access arguments at
all.

> - The FCGI Sys.max_string_length POST limitation is claimed to be
>   corrected.

No. There was another bug that lead to a 4K limitation. This is fixed.

> - Mimestring.read_multipart_body could also be useful in the form of
>   "fold" (left and right): e.g.
>
>   Mimestring.fold_left_multipart_body :
>     ('a -> Netstream.in_obj_stream -> 'a) -> 'a ->
>     string -> Netstream.in_obj_stream -> 'a

Yes, why not.

> - When the request method is not allowed or when it is unknown, the
>   library raises Failure.  This is not a correct behavior however,
>   instead a reply with the appropriate status (e.g. `Not_implemented)
>   should be sent (and this should not be the task of the user: exn
>   launched by the lib must be handled by it if it can).

This usually causes a 500 Server Error. This is acceptable as general
fallback error.

Currently, the library does not try to produce error pages at all
(with a few exceptions). Sending error pages is usually the task of
the web server or of the application built with the library. The
library should try not to do so, because error pages need normally
to be customized.

As far as I know there is no way to force the web server to send
one of its own error pages.

>   The way I solved this is to define an internal HTTP exception with a
>   status code and an error message (see Netcgi_common).  This
>   exception is caught by connectors and appropriate messages are sent
>   back to the client.  Note that your exn [Resources_exceeded] is now
>   a particular case of this (`Request_entity_too_large).

Well, this may be a debugging aid, but I think it is not a good
solution, as explained.

> - It does not seem that #finalize uses reference counting on the
>   files.  Concretely, it means that a MIME body can only be shared
>   between different MIME messages (as they will if one "clones" an
>   arg) with great care.

It was not designed for this case. This simplest solution would be
to clone on file level (hard link), but not all OS support that.

>   Also, since in the new interface all handlers are callbacks, what
>   about automatically finalizing the arguments when the callback
>   returns (normally or otherwise)?

I have no opinion on that. Often, arguments do not survive one call,
but I cannot say whether users keep further references to them.

> - The Netmime.mime_header_ro defines
>
>     method content_type : unit -> string * (string * Mimestring.s_param)
> list
>
>   while Netcgi_env.cgi_environment says
>
>     method input_content_type : (string * (string * Mimestring.s_param)
> list)
>
>   It would be nice for the two to be unified.  The same goes for
>   [content_length].

Ok, but see below for my opinion about purely aesthetic changes.

> - It would be nice to be able to turn a mime_header read-only.
>   Indeed, it is good to have it rw to fill it and then lock it ro.  Or
>   maybe a solution alike [Netmime.storage ~ro:true] can be envisioned.
>   A simple solution is to make [mime_header] a class that takes a
>   argument of type [mime_header_ro] -- it will still define a type so
>   it does not influence the rest of the code (but appropriate doc
>   should be provided).

One can already coerce mime_header to mime_header_ro.

> - One of the arguments for separating connectors in their own module
>   (which I agree with), was to reduce the code size.
>
>   [Mimestring], [Netmime], [Nethttp] and [Neturl] call [Netencoding]
>   for its [Base64], [QuotedPrintable], [Q] and [Url] modules.  However
>   [Netencoding] also has a [Html] module (not used for cgi/) which
>   links with [Netconversion], one of the bigger modules!  While I
>   agree it is the right place for [Html], this side effect is a bit
>   unfortunate.  Maybe a splitting [Netencoding_base64],... would be
>   nice.

On the other hand, Netconversion is one of the modules one needs
very quickly. For example, the only way to have a channel that
automatically performs HTML escaping is using Netconversion (in
a very tricky way).

> - It is not clear to me whether
>
>     let net_db_dir = "/usr/local/lib/ocaml/3.08.3/netstring" ;;
>
>   in Netdb is correctly set up at compile time.

It is. netdb.ml is generated from netdb.mlp. You can set the
directory with the -datadir option of configure.

> I'll appreciate comments on these.
>
> About the new interface, you can browse the interface at
> <http://ocaml-cgi.sourceforge.net/netcgi/>.  Notice that there is a
> Netcgi_compat module (implemented) which allows to convert between the
> new and old objects.  The documentation of this module also gather the
> motivation for the changes.

Well, I am not really convinced of the changes. For example, I see
little value in the many renamings. They will make any conversion
to your API very hard, and the gain is comparatively low.

Another point I don't like is the new class type for environments.
You see it only from the users' perspective, and throw out the
parts that are necessary for the role as glue container between
the connectors and the "CGI level" interpretation of the channel
data. Effectively, this makes your design more monolithic, and harder
to extend from the outside. We talked already about that.

I also do not see what the advantage is from making all arguments
read-only. It will make life very hard in some cases - arguments
are some sort of untrusted data, and sometimes it is necessary to
fix them up before processing them. This isn't pure style, but
it is sometimes needed.

> I am also happy to announce that I have started the implementation of
> the other modules (they depend on netstring)
> <http://cvs.sourceforge.net/viewcvs.py/ocaml-cgi/ocamlnet/netcgi/>.
> Highlights are:
>
> - Cookie module that supports Netscape, RFC 2109, and RFC 2965 cookies
>   both for sending and receiving (including quoted values).  Care has
>   been taken send cookies so that old browsers understand them (this
>   has not been tested however).
>
>   Remark: Cookies are just strings, contrarily to an earlier proposal.

Another remark. In netstring, we have now the module Nethttp.Header where
all kinds of parsers and printers for HTTP headers are collected. Maybe
the low-level cookie stuff should go there.

> - Correct responses to several errors by way of the [HTTP] exception
>   and handlers.
>
>   There is a default handler for user generated exceptions but it is
>   possible to override it.
>
> - All exceptions are acted upon correctly -- the user does not have to
>   write code to handle them (although he can if he wishes to).
>
> - One can decide to drop file arguments one knows we will not use --
>   save space.  See Netcgi.arg_store.

Seen that. This is useful.

> - Arguments are finalized on normal termination of the script or on
>   error.

Ok.

> - Netcgi_cgi and Netcgi_test: complete implementation with the new tools.
>
> - Netcgi_mod: "mod_caml" connector (complete).  The necessary subset
>   of mod_caml is included in the sources for convenience (but one may
>   as well use the standard mod_caml.so).
>
> - Netcgi_fcgi:
>
>   * An input object is created to handle incoming data without size
>     restrictions.

Very good.

>   * out_obj_channel now outputs correctly empty strings or strings
>     longer than 0xFFFF bytes.  Output is padded to reach a multiple of
>     8 bytes for performance.  Also, closing the output channel does
>     not close the error one -- they are independent.

I did not even know these bugs exist.
>
>   * All FCGI roles (Responder, Authorizer, Filter) are supported.
>
>   * Management records are handled appropriately.
>
>   The module has not been tested so it is likely to still contain
>   minor errors.  I am quite busy now so I'll do that in 3 weeks or so.
>
> - Netcgi_ajp: AJP 1.3 is planned but I have no time to do it in the
>   coming weeks (shouldn't be that difficult).
>
> - The "run" functions has been kept for lack of a better name
>   (suggestions welcome).  Netcgi_fcgi and Netcgi_ajp are ready to
>   handle concurrent connections once a good model is found (for the
>   other connectors, the script has no control about its concurrent
>   behavior).

Well, I think it is too early to make a good design. We need the MPM
modules first.

> - Netcgi_common shelter the main development functions while Netcgi is
>   the user interface.  I hope you like the design.

I think an API for self-developed connectors is missing.

> Misc.
>
> - Drop in replacement of Url: 10 * faster and never throws exceptions
>   (always decode).
>
> - Some examples have been ported to the new interface.
>
> As usual, all remarks are welcome.

I hope you can understand that I have a slightly different view. There are
already lots of programs using Ocamlnet, and it is not a good idea to
change the API in a radical way without having a very good reason.
We can create version 2 of the API, but it should be made simple to
migrate to it.

Furthermore, in the past we had an API to develop independent
connectors. This is part of the success of Ocamlnet, let's not
drop it.

Another, final point. You changed the license to LGPL. I don't see
why. The LGPL is rather pointless for O'Caml code, and we have the
much simpler zlib/libpng license because of this.

Gerd
> ---
> P.S.  It would be nice if the lines in *.mli files were not wider than
> the usual 80 chars, for compatibility with all screen layouts and
> because long lines are tiresome to read.  (Emacs M-q is a great help
> for that.)

Generally agreed, but that happens.

>
>
> -------------------------------------------------------
> SF.Net email is Sponsored by the Better Software Conference & EXPO
> September 19-22, 2005 * San Francisco, CA * Development Lifecycle
> Practices
> Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
> Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
> _______________________________________________
> Ocamlnet-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/ocamlnet-devel
>
>


------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany
[email protected]          http://www.gerd-stolpmann.de
------------------------------------------------------------




-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. 
Download it for free - -and be entered to win a 42" plasma tv or your very
own Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php