Re: E patches for review: ELoader, ESecureRandom

OK, I've committed those. Next up (on my "proposed" branch):

http://gitorious.org/~tal-itinnov/repo-roscidus/it-innovation/commits/proposed


ELoader is similar to <import>, but allows importing from any directory.
See: http://www.eros-os.org/pipermail/e-lang/2010-February/013416.html

The patches also add a loader called <this> to the privileged scope,
allowing the top-level .e file to import modules relative to itself.
Modules imported this way get <this> added to their own environment,
too. This is the same object (emakers loaded from a subdirectory can
still load anything under the loader's root, not merely from below their
own directory).

An ELoader can also be used to get a (read-only) directory object for
the directory it uses, which is useful for loading icons, documentation,
etc.

Updoc scripts also get a <this>, which always loads from the directory
containing the .updoc file (even if updoc is running recursively from a
higher-level directory).

An interactive session ("rune") gets a <this> for the current directory.

Running a .emaker directly doesn't give it <this>.

Loaders can also add other objects to the environment of the loaded
emakers. More complex programs will typically create a separate loader
for each module, putting loaders for the module's dependencies into the
module's environment.

One thing I'm slightly unhappy about is creating new vats. It's useful
to create a new vat with the same set of loaders, but you can't uncall
them at the moment. Possibly the uncall should be sealed. Thoughts?

The other change changes ESecureRandom to just SecureRandom in many
places, allowing you to pass in alternative implementations (though the
default is still ESecureRandom). This is useful for unit-tests, where
you don't want to stop and wait for high-quality entropy. This required
deprecating ESecureRandom.nextSwiss() in favour of
SwissTable.nextSwiss(). Is that OK? What does ESecureRandom do that the
default Java implementation doesn't?


On Mon, 2010-04-19 at 12:46 +0100, Thomas Leonard wrote:
> OK, I've committed the previous set of patches, except for the
> ImportLoader one (waiting for comments).
> 
> Next up for review on my "proposed" branch:
> 
> commit 04ab2776f62db166ad7fe08a8914a5d9efaf553c
> Author: Thomas Leonard <[email protected]>
> Date:   Sun Apr 4 16:26:55 2010 +0100
> 
>     Share a single SafeScope with each vat
> 
> commit cd440254247af1365c937440589748cfef8fca08
> Author: Thomas Leonard <tal-v5nx5w6akNyLE8xUarVfuPLx9OUvmyODWmv/[email protected]>
> Date:   Tue Apr 13 17:07:04 2010 +0100
> 
>     If any updoc tests fail, print the number of failures and exit with a non-zero exit status
> 
> commit b3afe6dcb699e57ca76bee1540688a73e38b63bf
> Author: Thomas Leonard <tal-v5nx5w6akNyLE8xUarVfuPLx9OUvmyODWmv/[email protected]>
> Date:   Thu Apr 15 09:37:02 2010 +0100
> 
>     Added EExpr.compile(scope)
>     
>     This converts the expression to transformed-E but then instead of executing it
>     (as EExpr.eval(scope) would), it wraps it in a CompiledE Thunk and returns it.
>     
>     This allows loaders to cache compiled code ready for execution. The static
>     memoized OurTransformer has been removed.
> 
> commit a28ed4257d49d35157e07e0e2ff453589ddc7ca0
> Author: Thomas Leonard <tal-v5nx5w6akNyLE8xUarVfuPLx9OUvmyODWmv/[email protected]>
> Date:   Thu Apr 15 14:18:03 2010 +0100
> 
>     Updated ImportLoader to use CompiledE
>     
>     Cache the CompiledE (which is always safe because merely compiling doesn't have
>     side-effects). Reevaluate the compiled E on each import.
> 
> 
> 
> On Thu, 2010-04-15 at 17:02 +0100, Thomas Leonard wrote:
> > On Tue, 2010-04-13 at 11:43 -0400, Kevin Reid wrote:
> > 
> > [ database quasi-parser ]
> > 
> > > This is a good point. In particular, since Java has an official SQL  
> > > API, E-on-Java should include appropriate wrappers for it. I retract  
> > > my objection on these grounds.
> > > 
> > > Have you fixed the synchronous access/determinism issues I previously  
> > > mentioned in
> > >    <http://www.eros-os.org/pipermail/e-lang/2010-March/013486.html>
> > > ?
> > 
> > I think that access to a database connection is sufficiently powerful
> > that trying to prevent non-determinism at this level is pointless. After
> > all, SQL is itself non-deterministic. e.g. the rows returned by a SELECT
> > can come in any order (unless you put an ORDER BY on the end).
> > 
> > In other words, you will always want to provide a Database Access Object
> > that provides an application-level interface with sensible E semantics.
> > For example, our wrappers only ever do queries with an ORDER BY, but
> > they are synchronous because our database is local and only accessed by
> > a single vat.
> > 
> > Making the calls asynchronous would cause problems in many cases.
> > Remember that a "database" is not necessarily a remote service. It can
> > quite easily be a local file accessed directly from the JVM (this is how
> > I use it) or even just an in-memory object.
> > 
> > So, I see SQL access as similar to <file> access in E: synchronous and
> > immediately subject to changes by other vats, to be wrapped where
> > needed.
> > 
> > > >>> * Share a single SafeScope within each vat
> > > >>
> > > >> This sounds like the right thing. Where is it stored?
> > > >
> > > > Each ImportLoader creates and stores one (since there's already one
> > > > ImportLoader per thread, and it's the thing that uses it).
> > > 
> > > Sounds reasonable. I'd like to take a closer look at it for  
> > > inappropriate dependencies, though. Unless MarkM can get to it sooner.
> > 
> > OK, will wait until next week before committing.
> > 
> > > >>> * In "not synchronously callable" error, give the verb
> > > >>
> > > >> E-on-CL generates "not synchronously callable: rec.verb(args...)" for
> > > >> the message. The implementations shouldn't be gratuitously different
> > > >> in this regard. Do you have a rationale for not printing the other
> > > >> components?
> > > >
> > > > Just the general issue of error messages leaking information. I'm a  
> > > > bit
> > > > concerned about ex.eStack() showing the arguments too, though it's
> > > > certainly useful.
> > > 
> > > eStack has enough information that you're utterly doomed.
> > > 
> > > Unfortunately there isn't a design issue page for it on the wiki, but  
> > > the solution is that *every* throw()n exception object, or  
> > > equivalently exception object you get from a catch block, is sealed.
> > > 
> > > Besides the debugging information, the throw-catch itself is  
> > > *dynamically scoped* and therefore does not follow capability  
> > > semantics; so we seal the information so that the only information a  
> > > catch give is 'unexpected failure within'.
> > > 
> > > Only that with debugging access can see either the message or the  
> > > eStack. Of course the REPL will automatically do that.
> > > 
> > > E-on-CL contains a partial implementation of these semantics.  
> > > Unfortunately, there's no formal writeup of the whole thing that I  
> > > know of, but there's some discussion in the mailing list archives.
> > > 
> > > http://www.google.com/search?q=site:eros-os.org+sealed+exceptions
> > > 
> > > If you'd like to help implement these semantics, please do -- it's one  
> > > of the big security problems currently in E-on-Java for untrusted-code  
> > > purposes, and one of the things that needs additional design work and  
> > > experience with Real Applications which need to do things like logging  
> > > errors.
> > 
> > OK, thanks for the clarification. What is the stack trace supposed to
> > look like once the E-to-JVM bytecode compiler is implemented? I guess
> > arguments won't be visible then anyway?
> > 
> > 
> 
> 


-- 
Dr Thomas Leonard
IT Innovation Centre
2 Venture Road
Southampton
Hampshire SO16 7NP

Tel: +44 0 23 8076 0834
Fax: +44 0 23 8076 0833
mailto:tal-v5nx5w6akNyLE8xUarVfuPLx9OUvmyODWmv/[email protected]
http://www.it-innovation.soton.ac.uk