Re: [Tiki-devel] composer.tiki.org slightly borken?
Victor Emanouilov via TikiWiki-devel <[email protected]>
| Newsgroups | gmane.comp.cms.tiki.devel |
|---|---|
| Message-ID | <[email protected]> |
Hi JM, I was offline a week, so seeing this just now. I think it might have been a glitch or something - cannot really test from here without access but I think you managed to fix the issue with Ricardo's help, right? Let me know if you need anything else from me here. I still see a lot of unused packages in composer.tiki.org/dist... Regards, Victor On 6/30/22 3:55 AM, Jean-Marc Libs wrote: > After a while, I imagine the satis update script was run and there are > more packages > #du -sh /composer/composer.tiki.org/ <http://composer.tiki.org/> > 7.7G /composer/composer.tiki.org/ <http://composer.tiki.org/> > > The new tree command gives: > https://composer.tiki.org/tree2.txt > > Cheers, > J-M > > On Thu, Jun 30, 2022 at 2:27 AM Jean-Marc Libs > <[email protected]> wrote: > > Thanks a lot, Victor! > > I needed more time than expected for doing the last remote backup > (the remote backup server was running into quota issues) but it's > done! > > I ran the script and it seems to have done *a lot* of cleanup. > *Maybe too much?* > > Now the site takes only: > root@hetzner01 /var/local/composer.tiki.org/www > <http://composer.tiki.org/www> # du -sh > /composer/composer.tiki.org/ <http://composer.tiki.org/> > 3.1G /composer/composer.tiki.org/ <http://composer.tiki.org/> > > Here is the result of the tree command : > https://composer.tiki.org/tree.txt > > I ran composer upgrade on a Tiki21 site and I got: > web# sh setup.sh > … > Your choice [c]? > Local PHP version 72 >= to required PHP version 72 - good > Warning: You forced the install of 1.10.26 via --1, but 2.3.7 is > the latest stable version. Updating to it via composer self-update > --stable is recommended. > You are already using composer version 1.10.26 (1.x channel). > Do not run Composer as root/super user! See > https://getcomposer.org/root for details > You are using Composer 1 which is deprecated. You should upgrade > to Composer 2, see > https://blog.packagist.com/deprecating-composer-1-support/ > Gathering patches from patch file. > Removing package adodb/adodb-php so that it can be re-installed > and re-patched. > - Removing adodb/adodb-php (v5.21.3) > > Tiki\Composer\RestoreVendors::restore > > Tiki\Composer\CleanVendorAfterVendorBundledMigration::cleanLinks > Loading composer repositories with package information > Installing dependencies from lock file > Warning: The lock file is not up to date with the latest changes > in composer.json. You may be getting outdated dependencies. It is > recommended that you run `composer update` or `composer update > <package name>`. > Package operations: 2 installs, 0 updates, 0 removals > Gathering patches from patch file. > Gathering patches for dependencies. This might take a minute. > - Installing adodb/adodb-php (v5.21.3): Downloading (failed) > Failed to download adodb/adodb-php from dist: The > "https://composer.tiki.org/dist/adodb/adodb-php/adodb-adodb-php-c5415722049f36c446a4034d15f1d17943f11458-zip-e71b61.zip" > file could not be downloaded (HTTP > /1.1 404 Not Found) > Now trying to download from source > - Installing adodb/adodb-php (v5.21.3): Cloning c541572204 from > cache > - Applying patches for adodb/adodb-php > ../installer/composer-patches/adodb-php__remove_references_to_adodb_extension.patch > (Remove references to ADOdb Extension) > Could not apply patch! Skipping. The error was: Cannot apply > patch > ../installer/composer-patches/adodb-php__remove_references_to_adodb_extension.patch > ../installer/composer-patches/adodb-php__remove-the-check-if-the-adodb-php_ext-is-enabled.patch > (Remove the check for ADOdb Extension) > > - Installing components/bootstrap (3.4.1): Downloading (failed) > Failed to download components/bootstrap from dist: The > "https://composer.tiki.org/dist/components/bootstrap/components-bootstrap-427f4d414fc6ae8385aee02b7efc8e391eb141ad-zip-393061.zip" > file could not b > e downloaded (HTTP/1.1 404 Not Found) > Now trying to download from source > - Installing components/bootstrap (3.4.1): Cloning 427f4d414f > from cache > Package container-interop/container-interop is abandoned, you > should avoid using it. Use psr/container instead. > Package evert/sitemap-php is abandoned, you should avoid using it. > No replacement was suggested. > Package league/commonmark-extras is abandoned, you should avoid > using it. Use league/commonmark instead. > Package robloach/component-installer is abandoned, you should > avoid using it. Use oomphinc/composer-installers-extender instead. > Package webuni/commonmark-attributes-extension is abandoned, you > should avoid using it. Use league/commonmark instead. > Package zendframework/zend-version is abandoned, you should avoid > using it. Use ocramius/package-versions instead. > Package zendframework/zendservice-akismet is abandoned, you should > avoid using it. No replacement was suggested. > Generating autoload files > > ComponentInstaller\Installer::postAutoloadDump > Compiling component files > > Tiki\Composer\ThemeInstaller::setup > > Tiki\Composer\BootstrapCompiler::build > > Tiki\Composer\CleanVendors::clean > > Tiki\Composer\CleanVendorAfterVendorBundledMigration::clean > > Tiki\Composer\PatchCypht::setup > > Tiki setup.sh - your options > ============================ > … > > Should I restore the previous packages and wait for a new version > of the script? > > Cheers, > Jyhem > > > > On Thu, Jun 23, 2022 at 3:58 PM Victor Emanouilov via > TikiWiki-devel <[email protected]> wrote: > > Hi Jean-Marc, > > I added a Tiki devtools script for this: > > https://gitlab.com/tikiwiki/tiki/-/commit/4ef8dc83e934ccee970a722e5f14237cbe8f6a73 > > Please configure the first variable to the actual local > filesystem path of composer.tiki.org > <http://composer.tiki.org> webroot and run it locally on that > server. You can use any version of php 7. Running it without > any arguments will list the files to be deleted. Running it > with --cleanup argument will actually perform the delete > operation. > > Regards, > Victor > > On 6/22/22 5:57 PM, Victor Emanouilov via TikiWiki-devel wrote: >> >> JM, >> >> >> As we talked on the roundtable meeting, this is the file that >> lists all the package versions composer.tiki.org >> <http://composer.tiki.org> supports: >> >> https://composer.tiki.org/packages.json >> >> Currently, it includes this one (it's huge!): >> >> https://composer.tiki.org/include/all$838361a2a0955d73dfb3e0370a109616b7e37fd5.json >> >> I will send you a script that pinpoints all the dist files >> downloaded on local disk that are not references in the >> packages.json file and provide an option to remove them. This >> should clean up about half of the used space if not even more. >> >> Regards, >> Victor >> >> On 5/16/22 2:40 PM, Victor Emanouilov via TikiWiki-devel wrote: >>> >>> Alright, let's discuss it then! >>> >>> On 5/13/22 10:52 PM, Jean-Marc Libs wrote: >>>> Hi Victor, >>>> >>>> I can't think of an issue with removing dev packages from >>>> satis config. >>>> >>>> I have a hard time understanding the consequences of >>>> removing old versions. Does it mean updating composer >>>> packages using setup.sh will fail unless the Tiki source is >>>> updated to the latest version first? >>>> >>>> I'm concerned about the consequences for people who update >>>> old production Tikis. I am not familiar enough with >>>> composer to make an opinion :-( >>>> >>>> Maybe we can discuss this during next Roundtable? I put the >>>> topic here: >>>> https://tiki.org/tiki-index.php?page=Roundtable-Meeting-2022-05 >>>> >>>> Cheers, >>>> J-M >>>> >>>> >>>> On Tue, May 3, 2022 at 11:08 AM Victor Emanouilov >>>> <[email protected]> wrote: >>>> >>>> Hi J-M, >>>> >>>> Yes, I agree we can remove dev packages from satis >>>> config and save space. The other thing I see is that >>>> some packages have way too many versions as files on >>>> the disk compared to the versions listed on >>>> composer.tiki.org <http://composer.tiki.org> - I think >>>> these are cached assets that probably disappeared later >>>> and are no longer supported. Do you think that we can >>>> do the following? >>>> >>>> 1. clean up satis config from dev packages >>>> 2. delete the whole directory holding packages on >>>> composer.tiki.org <http://composer.tiki.org> >>>> 3. deploy and run satis again to download only the ones >>>> that are currently in use >>>> >>>> Regards, >>>> Victor >>>> >>>> On 4/11/22 1:32 AM, Jean-Marc Libs wrote: >>>>> Hi Victor, here is the result of some more drilling down: >>>>> http://composer.tiki.org/packages-phpstan.txt >>>>> http://composer.tiki.org/packages-phpunit.txt >>>>> http://composer.tiki.org/packages-plotly.txt >>>>> http://composer.tiki.org/packages-sebastian.txt >>>>> http://composer.tiki.org/packages-vimeo.txt >>>>> http://composer.tiki.org/packages-aws.txt >>>>> http://composer.tiki.org/packages-google.txt >>>>> http://composer.tiki.org/packages-mpdf.txt >>>>> >>>>> >>>>> I don't usually run unit-tests but I can't see why we >>>>> keep a cache of old versions of dev tools. >>>>> >>>>> Cheers, >>>>> J-M >>>>> >>>>> On Fri, Apr 8, 2022 at 11:28 AM Victor Emanouilov >>>>> <[email protected]> wrote: >>>>> >>>>> JM, sorry, one more request. These are several >>>>> extremely large, packages themselves are about >>>>> 20-30mb but these take 10+GB on disk, can you >>>>> check them as well (drill down)? >>>>> >>>>> 100G dist/phpstan >>>>> 66G dist/phpunit >>>>> 30G dist/plotly >>>>> 29G dist/sebastian >>>>> 26G dist/vimeo >>>>> 25G dist/aws >>>>> 15G dist/google >>>>> 13G dist/mpdf >>>>> >>>>> Thanks, >>>>> Victor >>>>> >>>>> On 4/7/22 7:10 PM, Jean-Marc Libs wrote: >>>>>> Hi Victor, >>>>>> >>>>>> It's a huge file so I left it online: >>>>>> That's a simple du -sh : >>>>>> http://composer.tiki.org/packages_du-sh.txt >>>>>> That one is sorted by size: >>>>>> http://composer.tiki.org/packages_du-sh_sorted.txt >>>>>> >>>>>> No problem, >>>>>> J-M >>>>>> >>>>>> On Wed, Apr 6, 2022 at 2:05 PM Victor Emanouilov >>>>>> via TikiWiki-devel >>>>>> <[email protected]> wrote: >>>>>> >>>>>> JM, can you `du -sh` on the satis storage >>>>>> dirs and send us a list of the >>>>>> directories that take the most amount of >>>>>> space? We might limit semantic >>>>>> versions of those packages to what Tiki >>>>>> really needs... >>>>>> >>>>>> Thanks, >>>>>> Victor >>>>>> >>>>>> On 4/5/22 6:42 PM, Jonny Bradley via >>>>>> TikiWiki-devel wrote: >>>>>> > Hi Victor >>>>>> > >>>>>> > Yes, that was going to be plan B, if just >>>>>> listing what's needed currently in all the >>>>>> branches didn't result in freeing up enough >>>>>> space we could keep just the supported ones >>>>>> (somehow, we don't actually have that info in >>>>>> a format we can access easily afaik). >>>>>> > >>>>>> > The main thing needed is to change from >>>>>> using the actual used version to working out >>>>>> all the possible required ones, i expect we >>>>>> can use whatever composer uses to calculate >>>>>> that but i probably need to know more about >>>>>> how satis working internally... >>>>>> > >>>>>> > And have more free time! ;) >>>>>> > >>>>>> > jb >>>>>> > >>>>>> > >>>>>> > >>>>>> >> On 5 Apr 2022, at 11:44, Victor Emanouilov >>>>>> via TikiWiki-devel >>>>>> <[email protected]> wrote: >>>>>> >> >>>>>> >> Hi Jonny, Jean-Marc, >>>>>> >> >>>>>> >> Thanks for fixing this! It has been a >>>>>> lingering problem for quite some time. We >>>>>> definitely don't need all the packages that >>>>>> are downloaded on composer.tiki.org >>>>>> <http://composer.tiki.org> but Jonny, that's >>>>>> a static repository generator which means it >>>>>> can't fall back to download a missing >>>>>> package. I am thinking why don't we reduce to >>>>>> the number of official Tiki releases we >>>>>> support - we get all semantic versions of >>>>>> packages and generate only those in satis? >>>>>> Symfony packages, for example, will be >>>>>> reduced to v3 or v4 only and not everything >>>>>> from 2 to 6 >>>>>> (https://composer.tiki.org/#symfony/config) >>>>>> for example. Each official Tiki release can >>>>>> have Jonny's improved script run and generate >>>>>> the correct satis config. I believe we should >>>>>> decrease storage requirements several times >>>>>> this way. >>>>>> >> >>>>>> >> Regards, >>>>>> >> Victor >>>>>> >> >>>>>> >> On 4/5/22 12:15 PM, Jonny Bradley via >>>>>> TikiWiki-devel wrote: >>>>>> >>> Hi Jean-Marc and all >>>>>> >>> >>>>>> >>> Had a spare hour last night so started on >>>>>> this: >>>>>> >>> >>>>>> >>> >>>>>> https://gitlab.com/tikiwiki/tiki/-/blob/master/doc/devtools/composer_packages_in_use.php >>>>>> >>> >>>>>> >>> which at the moment lists all the >>>>>> versions currently in use by all the branches >>>>>> in tiki on gitlab. >>>>>> >>> >>>>>> >>> However, it then occurred to me that many >>>>>> of these branches will have outstanding >>>>>> automatic updates pending, e.g. if we ran >>>>>> `composer update` on tiki 22.x there almost >>>>>> certainly would be updates to newer packages, >>>>>> so maybe this isn't the right strategy after >>>>>> all... >>>>>> >>> >>>>>> >>> I guess we need to take into account the >>>>>> semantic versioning in all the composer.json >>>>>> files rather than the actual ones in use in >>>>>> the lock files, but at least it's a start? >>>>>> >>> >>>>>> >>> Would it be possible to remove some >>>>>> packages form satis on composer.t.o, but if >>>>>> something requests one of them it >>>>>> re-downloads it? Then we could be more >>>>>> confident in deleting stuff... >>>>>> >>> >>>>>> >>> Or just put a larger drive on it‽ :p >>>>>> >>> >>>>>> >>> Thanks for fixing it again! >>>>>> >>> >>>>>> >>> jonny >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>>> On 5 Apr 2022, at 01:09, Jean-Marc Libs >>>>>> <[email protected]> wrote: >>>>>> >>>> >>>>>> >>>> Hi folks, >>>>>> >>>> >>>>>> >>>> root partition was full on >>>>>> composer.tiki.org <http://composer.tiki.org> >>>>>> 😞 I managed to find 26G and it works again >>>>>> but the volume is staggering (409G, 103G once >>>>>> compressed for backups) and goes up all the time. >>>>>> >>>> >>>>>> >>>> Do we need so many third-party libs? >>>>>> >>>> >>>>>> >>>> Cheers, >>>>>> >>>> J-M >>>>>> >>>> >>>>>> >>>> On Mon, Apr 4, 2022 at 5:17 PM Victor >>>>>> Emanouilov via TikiWiki-devel >>>>>> <[email protected]> wrote: >>>>>> >>>> +1 same for me ... that URL returns 304 >>>>>> not modified - maybe some >>>>>> >>>> apache/nginx config is needed? Composer >>>>>> expects a valid json file there... >>>>>> >>>> >>>>>> >>>> Regards, >>>>>> >>>> Victor >>>>>> >>>> >>>>>> >>>> On 4/4/22 3:03 PM, Jonny Bradley via >>>>>> TikiWiki-devel wrote: >>>>>> >>>>> Hi all >>>>>> >>>>> >>>>>> >>>>> I'm trying to update composer for the >>>>>> bootstrap5 branch following some rebasing and >>>>>> it seem composer is failing now, and it's the >>>>>> same in trunk/master/main now. >>>>>> >>>>> >>>>>> >>>>> When i do: >>>>>> >>>>> >>>>>> >>>>> $ php74 temp/composer.phar update >>>>>> --working-dir vendor_bundled --prefer-dist >>>>>> --optimize-autoloader >>>>>> >>>>> >>>>>> >>>>> I get: >>>>>> >>>>> >>>>>> >>>>>> In JsonFile.php line 345: >>>>>> >>>>>> >>>>>> "https://composer.tiki.org/include/all%2419ac295e729a70b8272116cec8bdb0716f3dd8d2.json" >>>>>> does not contain valid JSON >>>>>> >>>>>> Parse error on line 1: >>>>>> >>>>>> >>>>>> >>>>>> ^ >>>>>> >>>>>> Expected one of: 'STRING', 'NUMBER', >>>>>> 'NULL', 'TRUE', 'FALSE', '{', '[' >>>>>> >>>>>> >>>>>> >>>>> >>>>>> https://composer.tiki.org/include/all%2419ac295e729a70b8272116cec8bdb0716f3dd8d2.json >>>>>> is an empty file, which presumably is the >>>>>> problem. >>>>>> >>>>> >>>>>> >>>>> Does anyone know why? (probably a >>>>>> Jean-Marc thing i expect, maybe satis needs >>>>>> updating/restarting/flushing?) >>>>>> >>>>> >>>>>> >>>>> jonny >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> >>>>>> _______________________________________________ >>>>>> >>>>> TikiWiki-devel mailing list >>>>>> >>>>> [email protected] >>>>>> >>>>> >>>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>>> >>>> >>>>>> _______________________________________________ >>>>>> >>>> TikiWiki-devel mailing list >>>>>> >>>> [email protected] >>>>>> >>>> >>>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>>> >>>> >>>>>> _______________________________________________ >>>>>> >>>> TikiWiki-devel mailing list >>>>>> >>>> [email protected] >>>>>> >>>> >>>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>>> >>> >>>>>> >>> >>>>>> _______________________________________________ >>>>>> >>> TikiWiki-devel mailing list >>>>>> >>> [email protected] >>>>>> >>> >>>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>>> >> >>>>>> >> >>>>>> _______________________________________________ >>>>>> >> TikiWiki-devel mailing list >>>>>> >> [email protected] >>>>>> >> >>>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>>> > >>>>>> > >>>>>> > _______________________________________________ >>>>>> > TikiWiki-devel mailing list >>>>>> > [email protected] >>>>>> > >>>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> TikiWiki-devel mailing list >>>>>> [email protected] >>>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>>> >>> >>> >>> _______________________________________________ >>> TikiWiki-devel mailing list >>> [email protected] >>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >> >> >> _______________________________________________ >> TikiWiki-devel mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel > _______________________________________________ > TikiWiki-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel > _______________________________________________ TikiWiki-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel