Re: Are the stdlib test.regrtest supposed to pass?

--===============0356177201777864504==
Content-Type: multipart/signed; micalg="pgp-sha512";
	protocol="application/pgp-signature"; boundary="=-1jQvoSoBZuvhe7WPOdhh"


--=-1jQvoSoBZuvhe7WPOdhh
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Thu, 2025-02-27 at 12:00 +0100, Miro Hron=C4=8Dok wrote:
> Hello PyPy folks.
>=20
> I have recently discovered that in Fedora, when building PyPy, we run=20
> test.regrtest but we don't make test failures terminate the build.
>=20
> For a couple times, I've reported test hangs because that was unfortunate=
ly the=20
> only problem that we were able to spot.
>=20
> Tests that run but don't fail the build are kinda useless. I want to reme=
dy=20
> that but I noticed that many tests fail for quite obvious reasons, for ex=
ample:
>=20
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> FAIL: test_getsitepackages (test.test_site.HelperFunctionsTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>    File=20
> "/builddir/build/BUILD/pypy3.10-v7.3.19-src/lib-python/3/test/test_site.p=
y",=20
> line 335, in test_getsitepackages
>      self.assertEqual(dirs[0], wanted)
> AssertionError: 'xoxo/lib64/pypy3.10/site-packages' !=3D=20
> 'xoxo/lib64/python3.10/site-packages'
> - xoxo/lib64/pypy3.10/site-packages
> ?              ^^
> + xoxo/lib64/python3.10/site-packages
> ?              ^^^^
> ----------------------------------------------------------------------
>=20
>=20
> I have examined the GitHub actions configuration in pypy/pypy but I don't=
 see=20
> the tests run.
>=20
> What are the expectations about the test results? Should we check that th=
e=20
> failure rate does not exceed some percentage? Or should we skip know fail=
ures=20
> one by one? Or report them to you?
>=20
> Or do we keep running the tests just that we could see the failure in the=
 logs=20
> but don't fail the build for that?
>=20
> How do you use those tests in PyPy?

Others can probably tell you more, but PyPy's running the test suite via
pytest.  As for failing, yes, some are expected to fail.  See the
buildbots [1] for the current status.

[1] https://buildbot.pypy.org/

--=20
Best regards,
Micha=C5=82 G=C3=B3rny


--=-1jQvoSoBZuvhe7WPOdhh
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: This is a digitally signed message part

-----BEGIN PGP SIGNATURE-----

iQFGBAABCgAwFiEEx2qEUJQJjSjMiybFY5ra4jKeJA4FAmfARxcSHG1nb3JueUBn
ZW50b28ub3JnAAoJEGOa2uIyniQOKh0H/iF4jeqC82N5BdBigfS3RCsdLNUHRfwE
mmuQRgoo0hrjKPofTymMkHOuFLJQGAoJVnAhnJ9h9THjwJBHmiLKFlzPVi1KSnR8
WtTvTzN+F5D92zmuIv+sLIZ88kePOT/V6qdbhmcxOe79115gv+9G7WsLlLvJJnqx
Zo3d+S7m2ZYLeRIFyiNwZpAiCdauxdKOjlaTB9/YzJ/6a5aLfVYM0COI4jSve3mA
FLlKWb9p1ncsckbB8zxVRBqd5ju5B/4RlcPTqlx20kBCKHYWWz/qhyJ4w3AfPpza
VZzUemmxbrlkOOeefgQ06mnO28FRsoiN90UlFwgYdn66fdmDyBHlMXo=
=vWX5
-----END PGP SIGNATURE-----

--=-1jQvoSoBZuvhe7WPOdhh--

--===============0356177201777864504==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
pypy-dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/pypy-dev.python.org/
Member address: [email protected]

--===============0356177201777864504==--