Re: setuptools reading simple index takes 30sec

What command are you using to result in the output? The root page lists all available packages on that index, and with 11MB of data there must be literally millions of entries to parse, and would definitely take a long time. But it definitely don’t need to be done under normal circumstances, and pip/setuptools definitely does not do it under normal circumstances.


Sent from Mail for Windows 10

From: [email protected]
Sent: 02 August 2019 05:06
To: [email protected]
Subject: [Distutils] setuptools reading simple index takes 30sec

Hello,
I have a python dev team that I support in our local jenkins/artifactory environment.  Recently this team switched over from an old local pypi legacy repository to a current pypi remote/virtual of pypi.org.  So we could get them updated libraries and such. Anyway, I am banging my head to explain the time taken for reading the simple index.  This index is about 11MB, takes 1 second to download itself, but then it takes ~30 seconds for setuptools to scan this and find the package it's looking for.  Multiple this by the ~170 libraries they have in requirements, and the build has gone from 30minutes to almost 3 hours.  And 90% comes from reading the simple index for each dependency.

Here's a sample output with timestamps
00:57:00.036 Searching for tenacity==4.8.0
00:57:00.037 Reading https://repo1.local/artifactory/api/pypi/py-pypi-virt/simple/
00:57:27.650 Reading https://repo1.local/artifactory/api/pypi/py-pypi-virt/simple/tenacity/
00:57:27.881 Downloading https://repo1.local/artifactory/api/pypi/py-pypi-virt/packages/packages/fc/e9/5499018e0d420f8d03a215c310ee7bc6e1a7e84adaa63f6ea208e864bdb6/tenacity-4.8.0-py2.py3-none-any.whl#sha256=efcf0672547f52fd49f96c2c1912e0f0e77d78a6630823aad54f99227a3c332d
00:57:28.037 Best match: tenacity 4.8.0
00:57:28.037 Processing tenacity-4.8.0-py2.py3-none-any.whl
00:57:28.038 Installing tenacity-4.8.0-py2.py3-none-any.whl to /home/build/workspace/workflows-pr/workflows-extension/.eggs
00:57:28.069 writing requirements to /home/build/workspace/workflows-pr/workflows-extension/.eggs/tenacity-4.8.0-py2.7.egg/EGG-INFO/requires.txt
00:57:28.114 Installed /home/build/workspace/workflows-pr/workflows-extension/.eggs/tenacity-4.8.0-py2.7.egg

Note the time on the first Reading.  I'm hoping there is some obvious simple solution to make this faster.
Any ideas or help would be greatly appreciated?

Thanks
--
Distutils-SIG mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at https://mail.python.org/archives/list/[email protected]/message/O7M274ZYQMMU3WT52PY2YIVJMCNOQJIR/

--
Distutils-SIG mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at https://mail.python.org/archives/list/[email protected]/message/M3R6JQ5X6MN4TRYAM5JYVL2OLO52BJW7/