Re: move __init__.py's import of importlib into __getattr__?

On Wed, Aug 9, 2023 at 12:50 AM Nathan Jensen <[email protected]> wrote:

> Hi,
>
> On a project I work on we encountered a complicated issue when upgrading
> from scipy 1.8 to scipy 1.10, and we traced it to being introduced in scipy
> 1.9 with the changes to how scipy submodules are imported.  The issue is
> described here: https://github.com/ninia/jep/issues/487
>

Hi Nathan, thanks for bringing this up. That's a nice analysis in the
linked issue.


> Some background: The application uses Jep which enables Java processes to
> run CPython inside the process.  Jep supports two types of interpreters,
> SubInterpreters which use CPython's sub-interpreter API, and
> SharedInterpreters which do not.  The application is using SubInterpreters
> because it predates Jep's introduction of SharedInterpreters by many years
> and there are pros and cons of each type of interpreter and it's not
> trivial to switch.  Because most CPython extension modules are not coded or
> tested with sub-interpreters in mind, Jep introduced a concept called
> "shared modules" that shares a module across sub-interpreters and ensures
> it is only ever imported once per process and is never disposed of or
> garbage collected in that process.  scipy is imported as a shared module so
> it will work in sub-interpreters but then the usage of importlib is messing
> up the subpackage import.  It's complicated but explained in that issue I
> linked above.
>
> We found a fix for the issue is to move the import of importlib in scipy's
> __init__.py into the __getattr__ method.
> https://github.com/scipy/scipy/blob/main/scipy/__init__.py#L185
>
> So what I'm wondering is, if I submitted a pull request that moved the
> import of importlib into __getattr__, either before the if name in modules
> statement or inside the if name in modules statement, is that a change that
> the scipy team would consider accepting for a future release?
>

It looks like the overhead of this is ~70 ns on my machine per function
call if made like `scipy.submodule.func()`, because of repeating the
`import importlib` statement every time. That is also the typical overhead
of a Python function call; it's not negligible, but may be acceptable for
SciPy.

On the other hand, what we are using here is the exact pattern advertised
for lazy submodule importing from https://peps.python.org/pep-0562/. So
this is going to be used in many other packages (e.g., it's encouraged at
https://scientific-python.org/specs/spec-0001/ and used in scikit-learn,
NetworkX, and possibly soon in NumPy). This is exactly the kind of obscure
issue that tends to surface when doing things like lazy imports - this is
the first such issue I believe though, so far it's been pretty robust
compared to all the hacky ways of doing lazy imports from 10+ years ago.
It's not clear to me if this is fixable in either CPython or in Jep after
something is changed to enable this use case in `importlib`. So I'd suggest
to also open an issue on the CPython repo and see what can be done on the
importlib side.

Cheers,
Ralf

_______________________________________________
SciPy-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/scipy-dev.python.org/
Member address: [email protected]