Re: Kevin's thoughts on modules

On Mon, Apr 11, 2011 at 3:08 AM, Thomas Leonard
<tal-v5nx5w6akNyLE8xUarVfuPLx9OUvmyODWmv/[email protected]> wrote:
> On 11/04/11 03:08, Kevin Reid wrote:
>> My rough ideas for a module system include these goals/characteristics
>> which are notably different from emakers:
>
> Here are a few comments on how this relates to what I've already
> implemented (and is in 0.9.3). The existing system is implemented in E,
> so it should also work on other systems, though I haven't tried it.
> Documentation is here:
>
>   http://wiki.erights.org/wiki/ELoader
>
> There are three ways of using it:
>
> - On its own (just the code implemented in E 0.9.3)
> - With 0install
> - With ebox
>
> ebox uses 0install to get all the dependences and wire them together,
> and then runs the program in a restricted environment (with a powerbox
> to request extra access).
>
> Using it with 0install (but without ebox), the program itself is run
> unconfined but its libraries get no special privileges. This is
> necessarily if you need more than the sandbox provides (e.g. access to
> an untamed Java library).
>
>> * No global namespace as in Java classpaths. (In Java, only convention
>> prevents a .jar from stepping on any name it wants to; this should be
>> *inexpressible* in a capability system.) Instead, once a 'module' has
>> been identified, it has its own namespace which is free to be entirely
>> flat (or not).
>
> ELoaders already work like this.
>
>> * It should be possible to identify a module in a highly future-proof
>> way, and have it automatically downloaded if needed. In particular, it
>> should be easy to say "def myModule :=<load the code which has the
>> SHA-whatever hash code 324587.... plus the following locator hints>".
>
> Already supported, except the SHA hashes don't go in the code, but in
> a separate package metadata file (XML). That also means you can depend
> on a range of possible versions, or different versions for different
> environments (e.g. an implementation localised for the user's preferred
> language).
>
> Hard-coding exact versions would make it hard to fix bugs in libraries,
> because all code using them would need to be edited. However, you can
> put version restrictions in the XML if needed.
>

It sounds to me like two issues are being conflated: how to specify
dependencies and where this specification lives.

Suppose dependencies are signed along with a version (with unambiguous
ordering) with a private key and referred to in source code by public
key with optional constraints on the version.  In this approach, the
author of dependent code relies on the authors of dependencies to
upgrade without breaking interfaces.  The dependent code doesn't need
to be redeployed when dependencies change (such as to fix a critical
security flaw).

Alternatively the author of dependent code may specify a set of
acceptable exact revisions in order to deny the authors of
dependencies the authority to alter the behavior of the dependent from
a known set of behaviors.

To me, it would be nice if a dependency management system allowed the
author of dependent code to choose either strategy (perhaps even
different strategies for different dependencies).


Separately, whether or not the dependency specification is stored in a
separate file of whatever format smells of bike-shedding to me.  (-but
I can't resist:)

As a matter of personal taste, I prefer Don't-Repeat-Yourself
minimization of unnecessary mappings.  I also prefer as few distinct
grammars as possible for a given technology stack, so I'm inclined to
prefer dependency specifications to be inline with E-code, as Kevin's
example suggested.


This brings up another issue, however: Must dependencies be specified
statically or are there benefits to doing so?  In a packaging system
that ensures dependencies are "ready" prior to running dependent code,
they must be statically specified.

Again, to me I would prefer both statically defined dependencies and a
runtime API so long as code authors understand the tradeoff.

> You can also, outside of the program, snapshot the exact library
> versions being used and run again later with the same versions.

This is a benefit of having a separate file.  My personal preference
is still for inline specification and tool to extract these if needed.


>
>> Goals:
>
>>     - that it be rare that one wishes to cut-and-paste code (without
>>       modification) rather than loading it as a module.
>>
>>     - that there is little barrier to using libraries which are not
>> part of
>>       standard E distributions.
>>
>> It would also be nice to, while being all nice and integrated about
>> module downloading, avoid inventing the equivalent of another package
>> installation system (CPAN, ruby gems, py-
>> whateverthey'redoingthesedays, quicklisp, etc.)


This is indeed a nice goal, but one that breaks down whenever
technology leaves its walled garden and interacts with the outside
world.

For example, does the given package rely on E-on-Java for a gui
framework?  If so, does this E packaging system manage specification
of Java dependencies?  If not, then it cannot be used for streamlined
installation, and all dependents of the handy GUI package also cannot
be installed in a streamlined fashion.

The rest of the packaging world (which doesn't seem to care about
confinement) embraces arbitrary system modification and external
dependencies to some degree, and it's always a hairy nightmare.

My preference is for the packaging system to ignore the outside world
in favor of simplicity and confinement.  Packages which rely on
dependents outside the packaging framework should include useful
information in exceptions which users can understand.

Does all code always have the authority to raise an exception?  If so
does it have the authority to define some part of the exception
message?


>
> Already working (using 0install). Works on Windows too now, not just
> POSIX systems.
>
>> * A module should be able to have private shared objects while its
>> implementation is broken across multiple files. (In the emaker system,
>> everything is either public or local to one .emaker file.)
>
> It doesn't do this.
>
>> * A module should be able to efficiently contain objects which are
>> unserialized (in the most general sense) data (e.g. images) without
>> exposing the distinction between these and code to clients.
>
> Already done.
>
>> There are probably other requirements I haven't remembered of just now.
>>
>>
>> Open questions is when modules should be instantiated (is there one of
>> a given module per vat, or not?); if they can be multiply
>> instantiated, whether that's 'deep' or 'shallow' wrt dependencies
>> (where do you stop?); and how to support (if at all) modules or graphs
>> of modules being granted references to privileged/application-specific
>> objects. (It would be nice to say that a module is always
>> unprivileged, but that results (I think) in certain types of
>> applications requiring massive application of the Author pattern,
>> which is undesirable in my opinion.)


A very simple packaging system would provide only statically defined
state.  I'm drawn to this because of its simplicity, but I haven't
seen enough code written in that style to judge it.

It sounds like the Author pattern requires the constructor author to
pass around a large number of references between components and ensure
the right connections are made.  Is this design pattern too cumbersome
in practice?

BTW- Is there a glossary of object capabilities patterns?  I'm relying
primarily on the first hit with this query, which isn't too specific:

https://encrypted.google.com/search?q=object+capabilities+author+pattern


>>
>>
>> The last time I was thinking about this, I actually wrote some code;
>> see the darcs repository at<http://switchb.org/darcs/e-modules/>.
>> This is not a complete system and probably has some design flaws in
>> its module format; but it works enough that I successfully loaded Den
>> in it.
>>
>> Whatever actually gets done should exist directly in E
>> implementations, not be an outside library, of course.
>
>
> --
> Dr Thomas Leonard
> Gamma House, Enterprise Road,
> Southampton SO16 7NS, UK
>
>
> tel: +44 23 8059 8866
>
> mailto:tal-v5nx5w6akNyLE8xUarVfuPLx9OUvmyODWmv/[email protected]
> http://www.it-innovation.soton.ac.uk/
> _______________________________________________
> e-lang mailing list
> [email protected]
> http://www.eros-os.org/mailman/listinfo/e-lang
>


Regards,
Nathan Wilcox