Re: [Tiki-devel] composer.tiki.org slightly borken?
Victor Emanouilov via TikiWiki-devel <[email protected]>
| Newsgroups | gmane.comp.cms.tiki.devel |
|---|---|
| Message-ID | <[email protected]> |
Hello Jean-Marc and Ricardo, One more issue with composer.tiki.org - satis doesn't seem to update package required dependencies on time. Example cypht: https://composer.tiki.org/p/jason-munro/cypht$b38479f0a2b62a6adff8c6b00b6ff3b5d7d059a249ac031ba5371cb8861bcfed.json lists dev-master dependency of bacon/bacon-qr-code as ^1.0.3 while upstream was updated 2 days ago to ^1.0.3 || ^2.0.0: https://github.com/jasonmunro/cypht/commit/131daacbeb765671ad6d0c90fb4085cb2c9c2eae Any idea how to force satis to update these automatically. It is either a cache issue or something else happened recently while cleaning up packages. Is satis running normally these last 2 days? Regards, Victor On 7/11/22 12:25 PM, Victor Emanouilov via TikiWiki-devel wrote: > > Hi JM, > > I was offline a week, so seeing this just now. I think it might have > been a glitch or something - cannot really test from here without > access but I think you managed to fix the issue with Ricardo's help, > right? > > Let me know if you need anything else from me here. I still see a lot > of unused packages in composer.tiki.org/dist... > > Regards, > Victor > > On 6/30/22 3:55 AM, Jean-Marc Libs wrote: >> After a while, I imagine the satis update script was run and there >> are more packages >> #du -sh /composer/composer.tiki.org/ <http://composer.tiki.org/> >> 7.7G /composer/composer.tiki.org/ <http://composer.tiki.org/> >> >> The new tree command gives: >> https://composer.tiki.org/tree2.txt >> >> Cheers, >> J-M >> >> On Thu, Jun 30, 2022 at 2:27 AM Jean-Marc Libs >> <[email protected]> wrote: >> >> Thanks a lot, Victor! >> >> I needed more time than expected for doing the last remote backup >> (the remote backup server was running into quota issues) but it's >> done! >> >> I ran the script and it seems to have done *a lot* of cleanup. >> *Maybe too much?* >> >> Now the site takes only: >> root@hetzner01 /var/local/composer.tiki.org/www >> <http://composer.tiki.org/www> # du -sh >> /composer/composer.tiki.org/ <http://composer.tiki.org/> >> 3.1G /composer/composer.tiki.org/ <http://composer.tiki.org/> >> >> Here is the result of the tree command : >> https://composer.tiki.org/tree.txt >> >> I ran composer upgrade on a Tiki21 site and I got: >> web# sh setup.sh >> … >> Your choice [c]? >> Local PHP version 72 >= to required PHP version 72 - good >> Warning: You forced the install of 1.10.26 via --1, but 2.3.7 is >> the latest stable version. Updating to it via composer >> self-update --stable is recommended. >> You are already using composer version 1.10.26 (1.x channel). >> Do not run Composer as root/super user! See >> https://getcomposer.org/root for details >> You are using Composer 1 which is deprecated. You should upgrade >> to Composer 2, see >> https://blog.packagist.com/deprecating-composer-1-support/ >> Gathering patches from patch file. >> Removing package adodb/adodb-php so that it can be re-installed >> and re-patched. >> - Removing adodb/adodb-php (v5.21.3) >> > Tiki\Composer\RestoreVendors::restore >> > Tiki\Composer\CleanVendorAfterVendorBundledMigration::cleanLinks >> Loading composer repositories with package information >> Installing dependencies from lock file >> Warning: The lock file is not up to date with the latest changes >> in composer.json. You may be getting outdated dependencies. It is >> recommended that you run `composer update` or `composer update >> <package name>`. >> Package operations: 2 installs, 0 updates, 0 removals >> Gathering patches from patch file. >> Gathering patches for dependencies. This might take a minute. >> - Installing adodb/adodb-php (v5.21.3): Downloading (failed) >> Failed to download adodb/adodb-php from dist: The >> "https://composer.tiki.org/dist/adodb/adodb-php/adodb-adodb-php-c5415722049f36c446a4034d15f1d17943f11458-zip-e71b61.zip" >> file could not be downloaded (HTTP >> /1.1 404 Not Found) >> Now trying to download from source >> - Installing adodb/adodb-php (v5.21.3): Cloning c541572204 from >> cache >> - Applying patches for adodb/adodb-php >> ../installer/composer-patches/adodb-php__remove_references_to_adodb_extension.patch >> (Remove references to ADOdb Extension) >> Could not apply patch! Skipping. The error was: Cannot apply >> patch >> ../installer/composer-patches/adodb-php__remove_references_to_adodb_extension.patch >> ../installer/composer-patches/adodb-php__remove-the-check-if-the-adodb-php_ext-is-enabled.patch >> (Remove the check for ADOdb Extension) >> >> - Installing components/bootstrap (3.4.1): Downloading (failed) >> Failed to download components/bootstrap from dist: The >> "https://composer.tiki.org/dist/components/bootstrap/components-bootstrap-427f4d414fc6ae8385aee02b7efc8e391eb141ad-zip-393061.zip" >> file could not b >> e downloaded (HTTP/1.1 404 Not Found) >> Now trying to download from source >> - Installing components/bootstrap (3.4.1): Cloning 427f4d414f >> from cache >> Package container-interop/container-interop is abandoned, you >> should avoid using it. Use psr/container instead. >> Package evert/sitemap-php is abandoned, you should avoid using >> it. No replacement was suggested. >> Package league/commonmark-extras is abandoned, you should avoid >> using it. Use league/commonmark instead. >> Package robloach/component-installer is abandoned, you should >> avoid using it. Use oomphinc/composer-installers-extender instead. >> Package webuni/commonmark-attributes-extension is abandoned, you >> should avoid using it. Use league/commonmark instead. >> Package zendframework/zend-version is abandoned, you should avoid >> using it. Use ocramius/package-versions instead. >> Package zendframework/zendservice-akismet is abandoned, you >> should avoid using it. No replacement was suggested. >> Generating autoload files >> > ComponentInstaller\Installer::postAutoloadDump >> Compiling component files >> > Tiki\Composer\ThemeInstaller::setup >> > Tiki\Composer\BootstrapCompiler::build >> > Tiki\Composer\CleanVendors::clean >> > Tiki\Composer\CleanVendorAfterVendorBundledMigration::clean >> > Tiki\Composer\PatchCypht::setup >> >> Tiki setup.sh - your options >> ============================ >> … >> >> Should I restore the previous packages and wait for a new version >> of the script? >> >> Cheers, >> Jyhem >> >> >> >> On Thu, Jun 23, 2022 at 3:58 PM Victor Emanouilov via >> TikiWiki-devel <[email protected]> wrote: >> >> Hi Jean-Marc, >> >> I added a Tiki devtools script for this: >> >> https://gitlab.com/tikiwiki/tiki/-/commit/4ef8dc83e934ccee970a722e5f14237cbe8f6a73 >> >> Please configure the first variable to the actual local >> filesystem path of composer.tiki.org >> <http://composer.tiki.org> webroot and run it locally on that >> server. You can use any version of php 7. Running it without >> any arguments will list the files to be deleted. Running it >> with --cleanup argument will actually perform the delete >> operation. >> >> Regards, >> Victor >> >> On 6/22/22 5:57 PM, Victor Emanouilov via TikiWiki-devel wrote: >>> >>> JM, >>> >>> >>> As we talked on the roundtable meeting, this is the file >>> that lists all the package versions composer.tiki.org >>> <http://composer.tiki.org> supports: >>> >>> https://composer.tiki.org/packages.json >>> >>> Currently, it includes this one (it's huge!): >>> >>> https://composer.tiki.org/include/all$838361a2a0955d73dfb3e0370a109616b7e37fd5.json >>> >>> I will send you a script that pinpoints all the dist files >>> downloaded on local disk that are not references in the >>> packages.json file and provide an option to remove them. >>> This should clean up about half of the used space if not >>> even more. >>> >>> Regards, >>> Victor >>> >>> On 5/16/22 2:40 PM, Victor Emanouilov via TikiWiki-devel wrote: >>>> >>>> Alright, let's discuss it then! >>>> >>>> On 5/13/22 10:52 PM, Jean-Marc Libs wrote: >>>>> Hi Victor, >>>>> >>>>> I can't think of an issue with removing dev packages from >>>>> satis config. >>>>> >>>>> I have a hard time understanding the consequences of >>>>> removing old versions. Does it mean updating composer >>>>> packages using setup.sh will fail unless the Tiki source >>>>> is updated to the latest version first? >>>>> >>>>> I'm concerned about the consequences for people who update >>>>> old production Tikis. I am not familiar enough with >>>>> composer to make an opinion :-( >>>>> >>>>> Maybe we can discuss this during next Roundtable? I put >>>>> the topic here: >>>>> https://tiki.org/tiki-index.php?page=Roundtable-Meeting-2022-05 >>>>> >>>>> Cheers, >>>>> J-M >>>>> >>>>> >>>>> On Tue, May 3, 2022 at 11:08 AM Victor Emanouilov >>>>> <[email protected]> wrote: >>>>> >>>>> Hi J-M, >>>>> >>>>> Yes, I agree we can remove dev packages from satis >>>>> config and save space. The other thing I see is that >>>>> some packages have way too many versions as files on >>>>> the disk compared to the versions listed on >>>>> composer.tiki.org <http://composer.tiki.org> - I think >>>>> these are cached assets that probably disappeared >>>>> later and are no longer supported. Do you think that >>>>> we can do the following? >>>>> >>>>> 1. clean up satis config from dev packages >>>>> 2. delete the whole directory holding packages on >>>>> composer.tiki.org <http://composer.tiki.org> >>>>> 3. deploy and run satis again to download only the >>>>> ones that are currently in use >>>>> >>>>> Regards, >>>>> Victor >>>>> >>>>> On 4/11/22 1:32 AM, Jean-Marc Libs wrote: >>>>>> Hi Victor, here is the result of some more drilling down: >>>>>> http://composer.tiki.org/packages-phpstan.txt >>>>>> http://composer.tiki.org/packages-phpunit.txt >>>>>> http://composer.tiki.org/packages-plotly.txt >>>>>> http://composer.tiki.org/packages-sebastian.txt >>>>>> http://composer.tiki.org/packages-vimeo.txt >>>>>> http://composer.tiki.org/packages-aws.txt >>>>>> http://composer.tiki.org/packages-google.txt >>>>>> http://composer.tiki.org/packages-mpdf.txt >>>>>> >>>>>> >>>>>> I don't usually run unit-tests but I can't see why we >>>>>> keep a cache of old versions of dev tools. >>>>>> >>>>>> Cheers, >>>>>> J-M >>>>>> >>>>>> On Fri, Apr 8, 2022 at 11:28 AM Victor Emanouilov >>>>>> <[email protected]> wrote: >>>>>> >>>>>> JM, sorry, one more request. These are several >>>>>> extremely large, packages themselves are about >>>>>> 20-30mb but these take 10+GB on disk, can you >>>>>> check them as well (drill down)? >>>>>> >>>>>> 100G dist/phpstan >>>>>> 66G dist/phpunit >>>>>> 30G dist/plotly >>>>>> 29G dist/sebastian >>>>>> 26G dist/vimeo >>>>>> 25G dist/aws >>>>>> 15G dist/google >>>>>> 13G dist/mpdf >>>>>> >>>>>> Thanks, >>>>>> Victor >>>>>> >>>>>> On 4/7/22 7:10 PM, Jean-Marc Libs wrote: >>>>>>> Hi Victor, >>>>>>> >>>>>>> It's a huge file so I left it online: >>>>>>> That's a simple du -sh : >>>>>>> http://composer.tiki.org/packages_du-sh.txt >>>>>>> That one is sorted by size: >>>>>>> http://composer.tiki.org/packages_du-sh_sorted.txt >>>>>>> >>>>>>> No problem, >>>>>>> J-M >>>>>>> >>>>>>> On Wed, Apr 6, 2022 at 2:05 PM Victor Emanouilov >>>>>>> via TikiWiki-devel >>>>>>> <[email protected]> wrote: >>>>>>> >>>>>>> JM, can you `du -sh` on the satis storage >>>>>>> dirs and send us a list of the >>>>>>> directories that take the most amount of >>>>>>> space? We might limit semantic >>>>>>> versions of those packages to what Tiki >>>>>>> really needs... >>>>>>> >>>>>>> Thanks, >>>>>>> Victor >>>>>>> >>>>>>> On 4/5/22 6:42 PM, Jonny Bradley via >>>>>>> TikiWiki-devel wrote: >>>>>>> > Hi Victor >>>>>>> > >>>>>>> > Yes, that was going to be plan B, if just >>>>>>> listing what's needed currently in all the >>>>>>> branches didn't result in freeing up enough >>>>>>> space we could keep just the supported ones >>>>>>> (somehow, we don't actually have that info >>>>>>> in a format we can access easily afaik). >>>>>>> > >>>>>>> > The main thing needed is to change from >>>>>>> using the actual used version to working out >>>>>>> all the possible required ones, i expect we >>>>>>> can use whatever composer uses to calculate >>>>>>> that but i probably need to know more about >>>>>>> how satis working internally... >>>>>>> > >>>>>>> > And have more free time! ;) >>>>>>> > >>>>>>> > jb >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> >> On 5 Apr 2022, at 11:44, Victor >>>>>>> Emanouilov via TikiWiki-devel >>>>>>> <[email protected]> wrote: >>>>>>> >> >>>>>>> >> Hi Jonny, Jean-Marc, >>>>>>> >> >>>>>>> >> Thanks for fixing this! It has been a >>>>>>> lingering problem for quite some time. We >>>>>>> definitely don't need all the packages that >>>>>>> are downloaded on composer.tiki.org >>>>>>> <http://composer.tiki.org> but Jonny, that's >>>>>>> a static repository generator which means it >>>>>>> can't fall back to download a missing >>>>>>> package. I am thinking why don't we reduce >>>>>>> to the number of official Tiki releases we >>>>>>> support - we get all semantic versions of >>>>>>> packages and generate only those in satis? >>>>>>> Symfony packages, for example, will be >>>>>>> reduced to v3 or v4 only and not everything >>>>>>> from 2 to 6 >>>>>>> (https://composer.tiki.org/#symfony/config) >>>>>>> for example. Each official Tiki release can >>>>>>> have Jonny's improved script run and >>>>>>> generate the correct satis config. I believe >>>>>>> we should decrease storage requirements >>>>>>> several times this way. >>>>>>> >> >>>>>>> >> Regards, >>>>>>> >> Victor >>>>>>> >> >>>>>>> >> On 4/5/22 12:15 PM, Jonny Bradley via >>>>>>> TikiWiki-devel wrote: >>>>>>> >>> Hi Jean-Marc and all >>>>>>> >>> >>>>>>> >>> Had a spare hour last night so started >>>>>>> on this: >>>>>>> >>> >>>>>>> >>> >>>>>>> https://gitlab.com/tikiwiki/tiki/-/blob/master/doc/devtools/composer_packages_in_use.php >>>>>>> >>> >>>>>>> >>> which at the moment lists all the >>>>>>> versions currently in use by all the >>>>>>> branches in tiki on gitlab. >>>>>>> >>> >>>>>>> >>> However, it then occurred to me that >>>>>>> many of these branches will have outstanding >>>>>>> automatic updates pending, e.g. if we ran >>>>>>> `composer update` on tiki 22.x there almost >>>>>>> certainly would be updates to newer >>>>>>> packages, so maybe this isn't the right >>>>>>> strategy after all... >>>>>>> >>> >>>>>>> >>> I guess we need to take into account the >>>>>>> semantic versioning in all the composer.json >>>>>>> files rather than the actual ones in use in >>>>>>> the lock files, but at least it's a start? >>>>>>> >>> >>>>>>> >>> Would it be possible to remove some >>>>>>> packages form satis on composer.t.o, but if >>>>>>> something requests one of them it >>>>>>> re-downloads it? Then we could be more >>>>>>> confident in deleting stuff... >>>>>>> >>> >>>>>>> >>> Or just put a larger drive on it‽ :p >>>>>>> >>> >>>>>>> >>> Thanks for fixing it again! >>>>>>> >>> >>>>>>> >>> jonny >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> >>>> On 5 Apr 2022, at 01:09, Jean-Marc Libs >>>>>>> <[email protected]> wrote: >>>>>>> >>>> >>>>>>> >>>> Hi folks, >>>>>>> >>>> >>>>>>> >>>> root partition was full on >>>>>>> composer.tiki.org <http://composer.tiki.org> >>>>>>> 😞 I managed to find 26G and it works again >>>>>>> but the volume is staggering (409G, 103G >>>>>>> once compressed for backups) and goes up all >>>>>>> the time. >>>>>>> >>>> >>>>>>> >>>> Do we need so many third-party libs? >>>>>>> >>>> >>>>>>> >>>> Cheers, >>>>>>> >>>> J-M >>>>>>> >>>> >>>>>>> >>>> On Mon, Apr 4, 2022 at 5:17 PM Victor >>>>>>> Emanouilov via TikiWiki-devel >>>>>>> <[email protected]> wrote: >>>>>>> >>>> +1 same for me ... that URL returns 304 >>>>>>> not modified - maybe some >>>>>>> >>>> apache/nginx config is needed? Composer >>>>>>> expects a valid json file there... >>>>>>> >>>> >>>>>>> >>>> Regards, >>>>>>> >>>> Victor >>>>>>> >>>> >>>>>>> >>>> On 4/4/22 3:03 PM, Jonny Bradley via >>>>>>> TikiWiki-devel wrote: >>>>>>> >>>>> Hi all >>>>>>> >>>>> >>>>>>> >>>>> I'm trying to update composer for the >>>>>>> bootstrap5 branch following some rebasing >>>>>>> and it seem composer is failing now, and >>>>>>> it's the same in trunk/master/main now. >>>>>>> >>>>> >>>>>>> >>>>> When i do: >>>>>>> >>>>> >>>>>>> >>>>> $ php74 temp/composer.phar update >>>>>>> --working-dir vendor_bundled --prefer-dist >>>>>>> --optimize-autoloader >>>>>>> >>>>> >>>>>>> >>>>> I get: >>>>>>> >>>>> >>>>>>> >>>>>> In JsonFile.php line 345: >>>>>>> >>>>>> >>>>>>> "https://composer.tiki.org/include/all%2419ac295e729a70b8272116cec8bdb0716f3dd8d2.json" >>>>>>> does not contain valid JSON >>>>>>> >>>>>> Parse error on line 1: >>>>>>> >>>>>> >>>>>>> >>>>>> ^ >>>>>>> >>>>>> Expected one of: 'STRING', 'NUMBER', >>>>>>> 'NULL', 'TRUE', 'FALSE', '{', '[' >>>>>>> >>>>>> >>>>>>> >>>>> >>>>>>> https://composer.tiki.org/include/all%2419ac295e729a70b8272116cec8bdb0716f3dd8d2.json >>>>>>> is an empty file, which presumably is the >>>>>>> problem. >>>>>>> >>>>> >>>>>>> >>>>> Does anyone know why? (probably a >>>>>>> Jean-Marc thing i expect, maybe satis needs >>>>>>> updating/restarting/flushing?) >>>>>>> >>>>> >>>>>>> >>>>> jonny >>>>>>> >>>>> >>>>>>> >>>>> >>>>>>> >>>>> >>>>>>> >>>>> >>>>>>> >>>>> >>>>>>> _______________________________________________ >>>>>>> >>>>> TikiWiki-devel mailing list >>>>>>> >>>>> [email protected] >>>>>>> >>>>> >>>>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>>>> >>>> >>>>>>> _______________________________________________ >>>>>>> >>>> TikiWiki-devel mailing list >>>>>>> >>>> [email protected] >>>>>>> >>>> >>>>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>>>> >>>> >>>>>>> _______________________________________________ >>>>>>> >>>> TikiWiki-devel mailing list >>>>>>> >>>> [email protected] >>>>>>> >>>> >>>>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>>>> >>> >>>>>>> >>> >>>>>>> _______________________________________________ >>>>>>> >>> TikiWiki-devel mailing list >>>>>>> >>> [email protected] >>>>>>> >>> >>>>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>>>> >> >>>>>>> >> >>>>>>> _______________________________________________ >>>>>>> >> TikiWiki-devel mailing list >>>>>>> >> [email protected] >>>>>>> >> >>>>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> _______________________________________________ >>>>>>> > TikiWiki-devel mailing list >>>>>>> > [email protected] >>>>>>> > >>>>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> TikiWiki-devel mailing list >>>>>>> [email protected] >>>>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>>>> >>>> >>>> >>>> _______________________________________________ >>>> TikiWiki-devel mailing list >>>> [email protected] >>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>> >>> >>> _______________________________________________ >>> TikiWiki-devel mailing list >>> [email protected] >>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >> _______________________________________________ >> TikiWiki-devel mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >> > > > _______________________________________________ > TikiWiki-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel _______________________________________________ TikiWiki-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel