Python3 UTF-8 file encoding error

Hi all.

Output of the command <file os-release>:
os-release: ISO-8859 text

Output of the command <cat os-release>:
NAME="GNU + Linux From Scratch"
VERSION="12.2"
ID=lfs
PRETTY_NAME="GNU + Linux Oriundo de C�digo Fonte"
VERSION_CODENAME="Jamenson Espindula"
HOME_URL="https://www.linuxfromscratch.org/lfs/"

Pay attention to the 'PRETTY_NAME' field: it holds a UTF-8 accented character.

Below is part of the log when building the 'libpwquality' package
(full log is at <https://pastebin.com/TfM0AX73>):

 = = = Transcription begins = = =

ERROR: Exception:
Traceback (most recent call last):

  File "/usr/lib/python3.12/site-packages/pip/_internal/cli/base_command.py",
line 105, in _run_wrapper
    status = _inner_run()
             ^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/pip/_internal/cli/base_command.py",
line 96, in _inner_run
    return self.run(options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/pip/_internal/cli/req_command.py",
line 67, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/pip/_internal/commands/wheel.py",
line 105, in run
    session = self.get_default_session(options)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/pip/_internal/cli/index_command.py",
line 76, in get_default_session
    self._session = self.enter_context(self._build_session(options))
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/pip/_internal/cli/index_command.py",
line 99, in _build_session
    session = PipSession(
              ^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/pip/_internal/network/session.py",
line 344, in __init__
    self.headers["User-Agent"] = user_agent()
                                 ^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/pip/_internal/network/session.py",
line 142, in user_agent
    linux_distribution = distro.name(), distro.version(), distro.codename()
                         ^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/pip/_vendor/distro/distro.py",
line 323, in name
    return _distro.name(pretty)
           ^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/pip/_vendor/distro/distro.py",
line 868, in name
    self.os_release_attr("name")

  File "/usr/lib/python3.12/site-packages/pip/_vendor/distro/distro.py",
line 1061, in os_release_attr
    return self._os_release_info.get(attribute, "")
           ^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/functools.py", line 993, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/pip/_vendor/distro/distro.py",
line 1100, in _os_release_info
    return self._parse_os_release_content(release_file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/pip/_vendor/distro/distro.py",
line 1121, in _parse_os_release_content
    tokens = list(lexer)
             ^^^^^^^^^^^

  File "/usr/lib/python3.12/shlex.py", line 300, in __next__
    token = self.get_token()
            ^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/shlex.py", line 109, in get_token
    raw = self.read_token()
          ^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/shlex.py", line 140, in read_token
    nextchar = self.instream.read(1)
               ^^^^^^^^^^^^^^^^^^^^^

  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf3 in position
91: invalid continuation byte

 = = = Transcription ends = = =

As you can see ('release_file'), the Python extractor had problems
with the file '/etc/os-release'.

That issue is easily solvable running the following commands (as 'root' user):

cd /etc/
mv -v os-release os-release-iso8859-1
iconv -f ISO-8591-1 -t UTF-8 < os-release-iso8859-1 > os-release

I believe it is helpful (especially for people outside North American)
to include such information in the BLFS book or another suitable
place.

Best regards

Jamenson Ferreira Espindula de Almeida Melo
Jaboatão dos Guararapes, Pernambuco, Brazil
GNU/Linux user #166197; LFS ID 24492
Key fingerprint: 234D 1914 4224 7C53 BD13 6855 2AE0 25C0 08A8 6180

-- 
http://lists.linuxfromscratch.org/sympa/info/blfs-dev
Unsubscribe: See the above information page