Re: Fwd: Re: Use of "python" shebang an installation error?

On 2020-07-21 22:50, David Mathog wrote:
> (oops, had to resend, forgot to change the destination to
> <[email protected]>)
> 
> On Mon, Jul 20, 2020 at 12:38 PM John Thorvald Wodder II
> <[email protected]> wrote:
>>
>> On 2020 Jul 20, at 15:25, David Mathog <[email protected]> wrote:
>>> Lately I have been working on a CentOS 8 machine, and it has "python2"
>>> and "python3", but no "python".  Many packages install scripts with a
>>> shebang like:
>>>
>>>    #!/usr/bin/env python
>>>
>>> and those do not work on this OS.  Seems like rather a large missing
>>> dependency which goes by without triggering a fatal error.
>>
>> How exactly are these packages getting installed?  Last time I checked, both pip and setuptools automatically set the shebang in scripts (both console_script entry points and scripts declared with the "scripts" argument to `setup()`) to use the path of the running Python interpreter.  Are these packages installed using your system package manager?  If so, you should take the problem up with its maintainers.
> 
> Good point, I have been installing so many packages I get confused
> about which installer was used for which package.   It turned out that
> many (but not all) of the files which contained
> 
>     #!/usr/bin/env python
> 
> shebangs were installed using standard OS level tools (cmake,
> configure, make and the like).  Example package, hisat2. I guess there
> isn't much choice for those but to scan the directories for python
> scripts and fix the shebangs.

Hi,
For many of these tools, you can pass in something like 
PYTHON=/usr/bin/python3 at build or install time to select the interpreter.

> Installs that are initially into venvs and used pip3 are still an
> issue.  Example:
> 
> python3  -m venv johnnydep
> cd johnnydep
> grep -r '/usr/bin/env python$' .
> #finds:
> ./lib/python3.6/site-packages/pip/_vendor/appdirs.py:#!/usr/bin/env python
> ./lib/python3.6/site-packages/pip/_vendor/chardet/cli/chardetect.py:#!/usr/bin/env
> python
> ./lib/python3.6/site-packages/pip/_vendor/requests/certs.py:#!/usr/bin/env
> python
> ./lib/python3.6/site-packages/pkg_resources/_vendor/appdirs.py:#!/usr/bin/env
> python
> ./lib/python3.6/site-packages/johnnydep/pipper.py:#!/usr/bin/env python
> cd bin
> ls -1 | grep python
> lrwxrwxrwx. 1 modules modules    7 Jul 20 14:09 python -> python3
> lrwxrwxrwx. 1 modules modules   16 Jul 20 14:09 python3 -> /usr/bin/python3
> source activate
> pip3 install johnnydep
> head -1 johnnydep
> #!/home/common/lib/python3.6/Envs/johnnydep/bin/python
> #same for "tabulate" and all other shebangs in bin.
> cd ..
> grep -r '/usr/bin/env python$' .
> #same as before
> grep -r '/home/common/lib/python3.6/Envs/johnnydep/bin/python' .
> #just the files in the bin directory.
> 
> It looks like none of the "#!/usr/bin/env python" shebangs within the
> venv are going to be used after the install, so perhaps those are
> harmless.
> 
> The shebangs like
> 
>     #!/home/common/lib/python3.6/Envs/johnnydep/bin/python
> 
> are OK within the venv, but once they are "devirtualized" they become
> a problem.  That was a known problem though - my devirtualizer code
> already patches all of the ones in the bin directory.  I have not seen
> any elsewhere (yet) within the venv, but there is probably no rule
> that keeps them from appearing in "share" or elsewhere.
> 
> The "python" in use in the venv is just a symbolic link to "python3"
> which is itself a symbolic link to the actual program
> "/usr/bin/python3". It is constructed that way based on "python -m
> venv" which uses pieces which come from the CentOS 8
> python3-libs-3.6.8-23.el8.x86_64 RPM.  Is there some requirement that
> a venv have a "python"?  Odd that RedHat (and so CentOS) provide a
> "python" there, but not in the OS itself.

Yes, a venv will always have a "python".

For the OS itself, that was a deliberate (but tough) decision. It's not 
easy to find out what "python" should mean in a particular script (as 
you found out), and the errors you get when using the wrong one are not 
always enlightening. For an OS released before end of life of Python 2, 
but supported well after it, we* "refuse to guess" and don't provide 
"python" by default.
In a venv, though, all is clear: you get the Python you created the venv 
with.

There's some more info here:
- across Linux distros: https://www.python.org/dev/peps/pep-0394/
- for EL8: https://developers.redhat.com/blog/2018/11/14/python-in-rhel-8/


 From elsewhere in the thread:
> The best I can do now is run
> 
>    pdvctrl reshebang $TARGET_DIR
> 
> or
> 
>    pdvctrl reshebang $ROOT_DIR...
> 
> and fix them up after the fact.  (pdvctrl from python_devirtualizer here:
> https://sourceforge.net/projects/python-devirtualizer/).  Even then it
> usually has to guess that "python" means "python3" and not "python2",
> and sometimes it guesses wrong.
My recommendation is to try not to guess: either provide an option, or 
look at the Python in the venv you're devirtualizing.

Also note that a "reshebang" tool is included in Python source code[0], 
but it's not part of the standard library, so it doesn't always come 
with Python. (On Fedora/RHEL/Centos it's in the python3-devel package as 
/usr/bin/pathfix.py). We* use is quite often, and found some edge cases 
involving flags in shebangs. You might want to look into it if you run 
into those for devirtualizer's reshebang.


* here, "we" == Red Hat's Python maintenance team
[0]: https://github.com/python/cpython/blob/master/Tools/scripts/pathfix.py
--
Distutils-SIG mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at https://mail.python.org/archives/list/[email protected]/message/YHK37CXP6X4RR7CGYWS2F4TCCW2O3FG5/