Re: Miscellaneous questions on OCamlNet + news

Christophe,

this is wonderful! Thank you for putting so much effort in the improved
API.

As we now have the MPM code, it makes a lot of sense to merge your
changes into the main repository. I would recommend we put it into a
directory parallel to the old Netcgi stuff so people can compare the API
side by side, and check what has been changed and maybe what is still
missing.

My proposal: Check your directory in to gps.dynxs.de (contact me
privately for password etc.) It is subversion, however, but svn is very
similar to cvs.

Am Freitag, den 23.06.2006, 19:57 +0200 schrieb Christophe TROESTLER:
> > Yes, this is possible, and only a matter of implementation. It is,
> > however, the question what "too large to fit in a string"
> > means. [...] I would say a limit of several KB would be reasonable.
> 
> Ok, thus the better is probably to leave it into the user hands.  That
> made me think that [arg_store] may not be flexible enough.  For a
> start I added a parameter of type [cgi_environment] so one can have
> different configurations for, say, different [cgi_path_info]. 
> I also added [`Memory_max of float],... return values so it is
> possible to specify a maximum size for the argument.  If that maximum
> is exceeded, an "oversized argument" is created meaning that the name
> of the argument is accessible but its body is discarded and any call to
> [#value] or [#open_value_rd] raises [Argument.Oversized].

Good.

> > [...] [value] method should not be used for file arguments at all,
> > or things may fail at run time. [...] argument without [value]
> > accessor, but only with [open_value_rd].  However, this makes it
> > very uncomfortable to access arguments at all.
> 
> There is no easy solution to this problem.  If we want to keep a
> uniform access to arguments, the current solution is probably
> preferable.  Any error occuring during [Netmime.decode_mime_body]
> (e.g. string too small or disk full) will result in a oversized
> argument.  Calling [#value] on a file that is too big hasn't changed
> -- but one could decide to also raise [Argument.Oversized].

Currently it depends on whether you run it on a 32 bit or 64 bit
platform. On the latter you can handle arbitrary size arguments.

Maybe we should leave this how it is (i.e. a "system issue" on string
lengths).

> > > - Mimestring.read_multipart_body could also be useful in the form of
> > >   "fold" (left and right)
> > Yes, why not.
> 
> Good (this makes it easier to drop the `Discard arguments).  I believe
> it is better you do it as you are the more familiar with Mimestring
> internals.

Ok.

> > > - When the request method is not allowed or when it is unknown, the
> > >   library raises Failure.  This is not a correct behavior however,
> > >   instead a reply with the appropriate status (e.g. `Not_implemented)
> > >   should be sent (and this should not be the task of the user: exn
> > >   launched by the lib must be handled by it if it can).
> > 
> > This usually causes a 500 Server Error. This is acceptable as general
> > fallback error.
> 
> Yes.  But one can do better!  Some errors naturally call for error
> codes invented just for them such as "411 Length Required",...  I
> agree that for the *user application* 500 is a good general fallback
> (and that's indeed what the lib does), but not so much for *protocol
> errors* reported by the supporting library.

Ok, if there are clearly better codes we should use these.

> > Currently, the library does not try to produce error pages at all
> > (with a few exceptions).  Sending error pages is usually the task of
> > the web server
> 
> That may not best because an uncaught exception kills the application
> (I assume it doesn't forks or use threads) and, with a remote
> connector like FCGI, SCGI, or AJP, the server cannot restart it --

I think this should be done in the server framework. Netplex (which I
propose to use for remote connectors) automatically restarts dead
processes/threads.

> thus _all_ current users of the web app. will suffer.  If a default
> exception handler is in place, that at least still allows the correct
> parts of the applications to be run (for a concrete example, think
> about a forum which has a bug in the administrative part).

But then you can never have backtraces. Maybe one should be able to turn
this handler on and off:

try ... with error when !error_pages_enabled -> ...

This is compatible with backtraces.

There is also the question whether the surrounding server should
generate error pages or the content generator. IMHO this depends very
much on the application. Sometimes it is better the server does it in
order to have uniform errors across all installed applications.

> > or of the application built with the library.  The library should
> > try not to do so, because error pages need normally to be customized.
> 
> The solution I propose does only provide a default fallback -- as much
> as the standard error pages of the web server do but with more info
> (and logging) helping to understand the cause of the error.  The user
> can treat in a special way any exception he cares by setting an
> optional argument of the type Netcgi.exn_handler defined as
> 
>   type exn_handler = cgi_environment -> (unit -> unit) -> unit
> 
> (see Netcgi.mli for an explanation).

Ok, we can go with that if it is possible to turn them off if absolutely
required.

> I am not sure I understand your point as this solution provides a nice
> fallback, a debugging aid, increases robustness AND allows
> customization.  If I did not get your point, please rephrase it.

Well, I formulated that 9 months ago, I don't remember. Currently I
think there are two arguments for being able to turn the default
behaviour off and let the exception fall through:

- Debugging: This makes backtraces possible (re-raising the   
  exception is no workaround because it is not recognized by 
  the compiler if too much code is run in the meantime and 
  the stack is destroyed) 

- Modularity: Maybe the surrounding code has already an error
  page (e.g. the server framework, or think of "nested CGIs"
  where an outer CGI calls an inner CGI with a faked 
  environment)

We can reach that by a simple switch variable.

> > >   Also, since in the new interface all handlers are callbacks, what
> > >   about automatically finalizing the arguments when the callback
> > >   returns (normally or otherwise)?
> > 
> > I have no opinion on that. Often, arguments do not survive one call,
> > but I cannot say whether users keep further references to them.
> 
> Maybe others feel otherwise but in my opinion, it is easy to forget to
> call [#finalize] and have a "file leak" (especially because the
> default storage for arguments is `Automatic).  Thus I think it is
> better that, if the user wants to keep the file, it pays special
> attention to it and move/hardlink it somewhere else.  All connectors
> finalize all arguments when the request handler finishes (of course
> also in the case an exception terminates it).
> 
> I have also added a method [cgi#at_exit] that allows to register more
> functions to be executed when [#finalise] is.  It is important for
> example to reuse database connections -- we need to free them back to
> the pool at the end (which is better done automatically by the library
> providing the pool for fear of the user forgetting).

Ok, we should go with that.

> > > - The Netmime.mime_header_ro defines
> > >
> > >   method content_type : unit -> string * (string * Mimestring.s_param) list
> > >
> > >   while Netcgi_env.cgi_environment says
> > >
> > >   method input_content_type : (string * (string * Mimestring.s_param) list)
> > 
> > Ok, but see below for my opinion about purely aesthetic changes.
> > [...]
> > I see little value in the many renamings.
> > [...]
> > I hope you can understand that I have a slightly different view. There
> > are already lots of programs using Ocamlnet, and it is not a good idea to
> > change the API in a radical way without having a very good reason.
> 
> I understand your point of view.  It is true I was more placing myself
> in a position of someboby who had little code written in OCamlNet and
> that I was worried that the lengthy methods names were going to be
> burdensome.  Having moved my code, that worry is gone -- the long
> names are even maybe easier to remember -- and I have reverted most of
> my changes to OCamlnet conventions.  Let me go into that in more
> details:
> 
> - Reverted to cgi_argument (for mutability see below).
> 
> - cgi_environment has been restored with the exception of
>   [input_content_type] (see below) and the state management (which is
>   not the user business).
> 
> - cgi: reverted to the old names (but left the class name cgi as this
>   is where the code needs to be changed to use the callback style).
> 
> - Whenever possible, when the old name lacked coherence with the usual
>   OCaml conventions, I deprecated it and provided an alternative.  As
>   an example, cgi_environment has [#output_ch] while the cgi object
>   has [#output].  The fact that the latter may be transactional is not
>   enough in my opinion to justify two names.  Moreover, [#output] is
>   an actual _outputting_function_, not a channel, in the standard lib
>   -- and [#output#output_string] or [#output#output] reads poorly .  I
>   have recommended the use of [#out_channel] for both.
> 
> The deprecation however does not always work and some names/methods
> were changed in non backward compatible ways:
> 
>   * [cgi_argument#content_type] is now the same as for
>     [Netmime.mime_header_ro].  Frankly having two different signatures
>     for the very same thing does little besides confusing the user
>     (and making it harder to remember).  The new API offers a good
>     opportunity to restore the coherence.
> 
>   * [permitted_http_methods]: `GET,... are used in [cgi#request_method]
>     so it is normal they are specified the same way and it is more typesafe.

This is all fine.

>   * I have a problem with cookies.  The record datastructure you
>     choose is not extensible which is bad to support the newer cookie
>     parameters (comments, ports) -- ok they haven't really hit the
>     street but they may, eventually.  Thus I have maintained the new
>     abstract cookie type [Cookie.t].  The method [#set_header] however
>     keeps its optional argument [~set_cookie] accepting the old cookie
>     type and has a new [~set_cookies] (more coherent) for the new type.

Maybe we should talk about cookies again. This is definitely an issue.

There is also the question of cookies from the server and from the
client perspective. Currently we model only the server perspective. For
web clients we would also need the possibility of a "cookie key ring".
IMHO we should reuse data structures as often as possible.

> > > - It would be nice to be able to turn a mime_header read-only. [...]
> > >   A simple solution is to make [mime_header] a class that takes a
> > >   argument of type [mime_header_ro] -- it will still define a type so
> > >   it does not influence the rest of the code (but appropriate doc
> > >   should be provided).
> > 
> > One can already coerce mime_header to mime_header_ro.
> 
> Yes but there is no conversion function [mime_header_ro ->
> mime_header].  That's why I proposed to turn the class declaration of
> [mime_header] into one.

Ok.

Actually, I already hit typing bugs where a mime_header was required by
a function where it would better have been only a mime_header_ro. This
makes me think of whether this distinction is too fine-grained.

> > Another point I don't like is the new class type for environments.
> > You see it only from the users' perspective, and throw out the parts
> > that are necessary for the role as glue container between the
> > connectors and the "CGI level" interpretation of the channel
> > data. Effectively, this makes your design more monolithic, and
> > harder to extend from the outside. We talked already about that.
> 
> Then I am sorry but I do not understand your point.  I think it is
> good desing to have a clear separation between what the user sould
> care about (or what is the common interface implemented by all
> connectors) and what the services offered to the developer of a new
> connector are.  [Netcgi] is the interface a user must use. 

Ok. Maybe I over-criticised you in this respect. Environments have
essentially two interfaces. One connects with the web server or whatever
is used instead, and one is the user side. So one has both aspects in a
single type. That's my point, and I think you got it.

>  It is also
> the interface a new connector must respect (even if implemented in a
> totally new way) and the interface that a developer can use to build a
> higher level framework compatible with any connector.  Of course, most
> connectors share quite a bit of ideas, hence code to help with them is
> provided in Netcgi_common (maybe you would prefer to have it renamed
> to Netcgi_dev ?).

Good question. I like good names.

Currently it contains base definitions. So common is acceptable (think
of "common base").

> I am not sure what you precisely reproach to the Netcgi_common API.
> My understanding is that it is more flexible and powerful than the
> current one.  The ease with which the various connectors are developed
> is a testimony to that...

Ok. I'll check whether I can integrate it into Nethttpd. If so, I'll
clap my hands :-) .

> > I also do not see what the advantage is from making all arguments
> > read-only. It will make life very hard in some cases - arguments
> > are some sort of untrusted data, and sometimes it is necessary to
> > fix them up before processing them. This isn't pure style, but
> > it is sometimes needed.
> 
> Could you give a concrete example?  I am thinking of a library
> developed on top of Netcgi: the author of the library may want to
> process arguments on its own way before providing (some of) them to
> the user.  For that it is better if the user cannot modify the
> arguments -- otherwise one needs to duplicate them just to be sure or
> cannot transform them on demand (i.e. not used => not transformed).
> On the other hand, arguments are strings or files, so nothing really
> prevent them to be modified (one can just discourage it)...

Yes, this is true.

I see that from the perspective of a working programmer. Imagine you
have an app to develop, you are overdue, and cannot do the sane
solution. Imagine the user types in a name, and it must not contain
slashes (because it will become part of a filename). So the quick and
dirty solution is to preprocess all arguments containing such names and
to throw out all slashes. (Otherwise you have a security problem.)

To some extent it is important for Ocamlnet to allow quick and dirty
solutions... It's a question of being accepted.

> In another mail, you also mention debugging as a reson to have mutable
> arguments.  But, as I see it, debugging should be provided by a function
> 
>   debug : cgi -> (cgi -> unit) -> unit
> 
> that may modify the CGI parameters (and those of the environment)
> before passing it to the main function [cgi -> unit].  It could also
> modify the output channels to redirect their data to a frame (when the
> content-type is text/html), other frames serving to display/set values.

Ok, debugging is no strict reason.

> > > - Cookie module that supports Netscape, RFC 2109, and RFC 2965 [...]
> > 
> > Another remark. In netstring, we have now the module Nethttp.Header
> > where all kinds of parsers and printers for HTTP headers are
> > collected. Maybe the low-level cookie stuff should go there.
> 
> Fine with me.
> 
> > > - The "run" functions has been kept for lack of a better name
> > >   (suggestions welcome).  Netcgi_fcgi and Netcgi_ajp are ready to
> > >   handle concurrent connections once a good model is found (for the
> > >   other connectors, the script has no control about its concurrent
> > >   behavior).
> > 
> > Well, I think it is too early to make a good design. We need the MPM
> > modules first.
> 
> Agreed.

Yes, we have them. Look at

https://gps.dynxs.de/svn/lib-ocamlnet2/trunk/code/src/nethttpd/nethttpd_plex.mli

how I integrated Nethttpd into Netplex. Essentially, there are two
functions:

- One is quite low-level: nethttpd_processor, for people who 
  need to control every bit

- One is high-level: nethttpd_factory.

For an example how to use the high-level function, see

https://gps.dynxs.de/svn/lib-ocamlnet2/trunk/code/examples/nethttpd/netplex.ml

One simply passes the created factory to the main entry function.

> > Another, final point. You changed the license to LGPL. I don't see
> > why. The LGPL is rather pointless for O'Caml code, and we have the
> > much simpler zlib/libpng license because of this.
> 
> No problem with me, LGPL+exn is just the standard licence I use for
> OCaml code and I insert it with a macro into new files...  I changed
> the licence to the one you use (if I made mistakes, point them to me,
> I do not care so much about the licence for this lib as long as it is
> free).

Thanks. My point is that LGPL+Ocaml exception is hard legal stuff.
Nobody really understands it. For maximum freedom, a simpler license is
better.

> I have added an FDF test -- updates a PDF form.

Which syntax is used by PDF to submit form data? Do you have a link?
(Sorry if you already sent it to me previously.)

-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
[email protected]          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
   Currently in Montain View, California, till July, 21.
------------------------------------------------------------


Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642