Re: A Python Web Application Package and Format

Hello,

I have few comments:

   - That file layout basically forces you to have your development
   environment as close to the production environment. This is especially
   visible if you're relying on python c extensions. Since you don't want to
   have the same environment constraints as appengine it should be more
   flexible in this regard and offer a way to generate the project dependencies
   somewhere else than the depeloper's machine.
   - There's no builtin support for logging configuration.
   - The update_fetch feels like a hack as it's not extensible to do
   lifecycle (hooks for shutdown, start, etc). Also, it's shouldn't be a
   application url because you'd want to run a hook before starting it or after
   stopping it. I guess you could accomplish that with a wsgi wrapper but there
   should be a clear separation between the app and hooks that manage the app.
   - I'm not entirely clear on why you avoid a build process (war-like)
   prior to deployment. It works fine for appengine - but you don't have it's
   constraints.

-- Ionel



On Fri, Apr 1, 2011 at 23:55, Ian Bicking <[email protected]> wrote:

> Hi all.  I wrote a blog post.  I would be interested in reactions from this
> crowd.
>
> http://blog.ianbicking.org/2011/03/31/python-webapp-package/
>
> Copied to allow responses:
>
> At PyCon there was an open space about deployment, and the idea of drop-in
> applications (Java-WAR-style<http://en.wikipedia.org/wiki/WAR_file_format_%28Sun%29>
> ).
>
> I generally get pessimistic about 80% solutions, and dropping in a WAR file
> feels like an 80% solution to me. I’ve used the Hudson/Jenkins installer
> (which I think is *specifically* a project that got WARs on people’s
> minds), and in a lot of ways that installer is nice, but it’s also kind of
> wonky, it makes configuration unclear, it’s not always clear when it
> installs or configures itself through the web, and when you have to do this
> at the system level, nor is it clear where it puts files and data, etc. So a
> great initial experience doesn’t feel like a great ongoing experience to me
> — and *it doesn’t have to be that way*. If those were *necessary*compromises, sure, but they aren’t. And because we
> *don’t have* WAR files, if we’re proposing to make something new, then we
> have every opportunity to make things better.
>
> So the question then is what we’re trying to make. To me: we want
> applications that are easy to install, that are self-describing,
> self-configuring (or at least guide you through configuration), reliable
> with respect to their environment (not dependent on system tweaking),
> upgradable, and respectful of persistence (the data that outlives the
> application install). A lot of this can be done by the "container" (to use
> Java parlance; or "environment") — if you just have the app packaged in a
> nice way, the container (server environment, hosting service, etc) can
> handle all the system-specific things to make the application actually work.
>
> At which point I am of course reminded of my Silver Lining<http://cloudsilverlining.org/>project, which defines something very much like this. Silver Lining isn’t
> *just* an application format, and things aren’t fully extracted along
> these lines, but it’s pretty close and it addresses a lot of important
> issues in the lifecycle of an application. To be clear: Silver Lining is an
> application packaging format, a server configuration library, a cloud server
> management tool, a persistence management tool, and a tool to manage the
> application with respect to all these services over time. It is a bunch of
> things, maybe too many things, so it is not unreasonable to pick out a
> smaller subset to focus on. Maybe an easy place to start (and good for
> Silver Lining itself) would be to separate at least the application format
> (and tools to manage applications in that state, e.g., installing new
> libraries) from the tools that make use of such applications (deploy, etc).
>
> Some opinions I have on this format, exemplified in Silver Lining:
>
>    - It’s not zipped or a single file, unlike WARs. Uploading zip files is
>    not a great API. Geez. I know there’s this desire to "just drop in a file";
>    but there’s no getting around the fact that "dropping a file" becomes a
>    *deployment protocol* *and* *it’s an incredibly impoverished protocol*.
>    The format is also not subtly git-based (ala Heroku) because git pushis not a good deployment protocol.
>    - But of course there isn’t really any deployment protocol inferred by
>    a format anyway, so maybe I’m getting ahead of myself ;) I’m saying a tool
>    that deploys should take as an argument a directory, not a single file. (If
>    the tool then zips it up and uploads it, fine!)
>    - Configuration "comes from the outside". That is, an application
>    requests services, and the *container* tells the application where
>    those services are. For Silver Lining I’ve used environmental variables. I
>    think this one point is really important — the container *tells* the
>    application. As a counter-example, an application that comes with a Puppet
>    deployment recipe is essentially *telling* the server how to arrange
>    itself to suit the application. This will never be reliable or simple!
>    - The application indicates what "services" it wants; for instance, it
>    may want to have access to a MySQL database. The container then provides
>    this to the application. In practice this means installing the actual
>    packages, but also creating a database and setting up permissions
>    appropriately. The alternative is never having *any* dependencies,
>    meaning you have to use SQLite databases or ad hoc structures, etc. But in
>    fact installing databases really isn’t that hard these days.
>    - *All* persistence has to use a service of some kind. If you want to
>    be able to write to files, you need to use a file service. This means the
>    container is fully aware of everything the application is leaving behind.
>    All the various paths an application should use are given in different
>    environmental variables (many of which don’t need to be invented anew, e.g.,
>    $TMPDIR).
>    - It uses vendor libraries exclusively for Python libraries. That means
>    the application bundles all the libraries it requires. Nothing ever gets
>    installed at deploy-time. This is in contrast to using a
>    requirements.txt list of packages at deployment time. If you want to
>    use those tools for development that’s fine, just not for deployment.
>    - There *is* also a way to indicate other libraries you might require;
>    e.g., you might lxml, or even something that isn’t quite a library,
>    like git (if you are making a github clone). You can’t do those as
>    vendor libraries (they include non-portable binaries). Currently in Silver
>    Lining the application description can contain a list of Ubuntu package
>    names to install. Of course that would have to be abstracted some.
>    - You can ask for scripts or a request to be invoked for an application
>    after an installation or deployment. It’s lame to try to test if
>    is-this-app-installed on *every* request, which is the frequent
>    alternative. Also, it gives the application the chance to signal that the
>    installation failed.
>    - It has a very simple (possibly/probably too simple) sense of
>    configuration. You don’t have to use this if you make your app
>    self-configuring (i.e., build in a web-accessible settings screen), but in
>    practice it felt like some simple sense of configuration would be helpful.
>
> Things that could be improved:
>
>    - There are some places where you might be encouraged to use routines
>    from the silversupport package. There are very few! But maybe an
>    alternative could be provided for these cases.
>    - A little convention-over-configuration is probably suitable for the
>    bundled libraries; silver includes tools to manage things, but it gets a
>    little twisty. When creating a new project I find myself creating several
>    .pth files, special customizing modules, etc. Managing vendor libraries
>    is also not obvious.
>    - Services <http://cloudsilverlining.org/services.html> are IMHO quite
>    important and useful, but also need to be carefully specified.
>    - There’s a bunch of runtime expectations that aren’t part of the
>    format, but in practice would be part of how the application is written. For
>    instance, I make sure each app has its own temporary directory, and that it
>    is cleared on update. If you keep session files in that location, and you
>    expect the environment to clean up old sessions — well, either all
>    environments should do that, or none should.
>    - The process model is not entirely clear. I tried to simply define one
>    process model (unthreaded, multiple processes), but I’m not sure that’s
>    suitable — most notably, multiple processes have a significant memory impact
>    compared to threads. An application should at least be able to indicate what
>    process models it accepts and prefers.
>    - Static files are *all* convention over configuration — you put static
>    files under static/ and then they are available. So static/style.csswould be at
>    /style.css. I think this is generally *good*, but putting all static
>    files under one URL path (e.g., /media/) can be good for other reasons
>    as well. Maybe there should be conventions for both.
>    - Cron jobs are important. Though maybe they could just be yet another
>    kind of service? Many extra features could be new services.
>    - Logging is also important; Silver Lining attempts to handle that
>    somewhat, but it could be specified much better.
>    - Silver Lining also supports PHP, which seemed to cause a bit of
>    stress. But just ignore that. It’s really easy to ignore.
>
> There is a description of the configuration file for apps<http://cloudsilverlining.org/appconfig.html>.
> The environmental variables<http://cloudsilverlining.org/envvariables.html>are also notably part of the application’s expectations. The file layout is
> explained (together with a bunch of Silver Lining-specific concepts) in Development
> Patterns <http://cloudsilverlining.org/devpatterns.html>. Besides all that
> there is admittedly some other stuff that is only really specified in code;
> but in Silver Lining’s defense, specified in code is better than unspecified
> ;) App Engine provides another example of an application format, and would
> be worth using as a point of discussion or contrast (I did that myself when
> writing Silver Lining).
>
> Discussing WSGI stuff with Ben Bangert at PyCon he noted that he didn’t
> really feel like the WSGI pieces needed that much more work, or at least
> that’s not where the interesting work was — the interesting work is in the
> tooling. An application format could provide a great basis for building this
> tooling. And I honestly think that the tooling has been held back more by
> divergent patterns of development than by the difficulty of writing the
> tools themselves; and a good, general application format could fix that.
>
>
> _______________________________________________
> Web-SIG mailing list
> [email protected]
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe:
> http://mail.python.org/mailman/options/web-sig/ionel.mc%40gmail.com
>
>

_______________________________________________
Web-SIG mailing list
[email protected]
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/gcpw-web-sig%40m.gmane.org