Re: Idea: concept of a builder or transformer to compliment loaders

On Thu, May 28, 2015 at 9:11 AM, Brett Cannon <[email protected]> wrote:
> I should start off by saying I don't plan to pursue this idea, but I wanted
> to write it down for posterity and in case anyone else has thought about
> this.
>
> That being said, the idea of macros and other source-transforming things
> done to Python code has come up a few times on python-ideas as of late. Now
> experimenting with this sort of thing using a custom loader is not hard, and
> thanks to importlib.abc.ResourceLoader.source_to_code() it's fairly easy to
> do (by design; I tried to initially structure importlib's APIs to making
> alternative storage backends easy as well as alternative syntax stuff like
> Quixote from back in the day).
>
> But one thing I realized is that while finders and loaders are necessary for
> alternative code storage mechanisms, they are not the right abstraction for
> tweaking code semantics.

Agreed.

> Really all you need is a function that takes in
> source code and spits out a code object to use with exec() (hence
> ResourceLoader.source_to_code() even existing). It somewhat sucks that
> people who just want to tweak code semantics have to define a loader
> subclass and instantiate a new finder when all that is mostly stuff that
> doesn't concern them. It also sucks that they would have to do that for
> every storage type, e.g. local files and zip files.

Yep.

>
> Now I don't have a solid solution to propose for this niche use case. It
> makes me want to have some kind of way to register compiler functions, but

I had the same thought.

> that would be limiting if it went source -> code object. AST -> AST would
> allow for chaining much like Victor has proposed in the past, but it also
> means that people who want a transpiler to go source -> source are left out.

Yeah, it feels like there's an encapsulation there around the various
pieces of compilation.  Furthermore, I'd expect such an abstraction to
consider the needs of alternate Python implementations as well.

> And then there is the whole thing of how to get the loaders to know of these
> transpilers/transformers/compilers
> as adding more global state to sys feels
> dirty (maybe an attribute on finders that they can draw from if they so
> choose?), but maybe it isn't that big of a deal as long as they are just
> callables and people realize they must be re-entrant.

This is where something like ImportSystem (nee ImportEngine) would
help.  We'd just have sys.importsystem and add state there as
appropriate without further cluttering up the sys module.

FWIW, I've considered a number of minor additions similar to what
you're talking about for niche needs where it would still be nice to
have a convenient API because of the overhead of writing and managing
a finder/loader.

Perhaps it's just a matter of providing helper decorators along the
lines of contextlib.contextmanager, which convert your simple function
into the necessary format at register it in the correct place (e.g.
finder+loader -> sys.path/sys.metapath).

>
> As I said, I don't plan to work on this, but I wanted to get my ideas
> written down in case someone else cared.

:)

-eric