Re: [Tiki-devel] composer.tiki.org slightly borken?
Victor Emanouilov via TikiWiki-devel <[email protected]>
| Newsgroups | gmane.comp.cms.tiki.devel |
|---|---|
| Message-ID | <[email protected]> |
Hi Jean-Marc, I added a Tiki devtools script for this: https://gitlab.com/tikiwiki/tiki/-/commit/4ef8dc83e934ccee970a722e5f14237cbe8f6a73 Please configure the first variable to the actual local filesystem path of composer.tiki.org webroot and run it locally on that server. You can use any version of php 7. Running it without any arguments will list the files to be deleted. Running it with --cleanup argument will actually perform the delete operation. Regards, Victor On 6/22/22 5:57 PM, Victor Emanouilov via TikiWiki-devel wrote: > > JM, > > > As we talked on the roundtable meeting, this is the file that lists > all the package versions composer.tiki.org supports: > > https://composer.tiki.org/packages.json > > Currently, it includes this one (it's huge!): > > https://composer.tiki.org/include/all$838361a2a0955d73dfb3e0370a109616b7e37fd5.json > > I will send you a script that pinpoints all the dist files downloaded > on local disk that are not references in the packages.json file and > provide an option to remove them. This should clean up about half of > the used space if not even more. > > Regards, > Victor > > On 5/16/22 2:40 PM, Victor Emanouilov via TikiWiki-devel wrote: >> >> Alright, let's discuss it then! >> >> On 5/13/22 10:52 PM, Jean-Marc Libs wrote: >>> Hi Victor, >>> >>> I can't think of an issue with removing dev packages from satis config. >>> >>> I have a hard time understanding the consequences of removing old >>> versions. Does it mean updating composer packages using setup.sh >>> will fail unless the Tiki source is updated to the latest version first? >>> >>> I'm concerned about the consequences for people who update old >>> production Tikis. I am not familiar enough with composer to make an >>> opinion :-( >>> >>> Maybe we can discuss this during next Roundtable? I put the topic >>> here: https://tiki.org/tiki-index.php?page=Roundtable-Meeting-2022-05 >>> >>> Cheers, >>> J-M >>> >>> >>> On Tue, May 3, 2022 at 11:08 AM Victor Emanouilov <[email protected]> >>> wrote: >>> >>> Hi J-M, >>> >>> Yes, I agree we can remove dev packages from satis config and >>> save space. The other thing I see is that some packages have way >>> too many versions as files on the disk compared to the versions >>> listed on composer.tiki.org <http://composer.tiki.org> - I think >>> these are cached assets that probably disappeared later and are >>> no longer supported. Do you think that we can do the following? >>> >>> 1. clean up satis config from dev packages >>> 2. delete the whole directory holding packages on >>> composer.tiki.org <http://composer.tiki.org> >>> 3. deploy and run satis again to download only the ones that are >>> currently in use >>> >>> Regards, >>> Victor >>> >>> On 4/11/22 1:32 AM, Jean-Marc Libs wrote: >>>> Hi Victor, here is the result of some more drilling down: >>>> http://composer.tiki.org/packages-phpstan.txt >>>> http://composer.tiki.org/packages-phpunit.txt >>>> http://composer.tiki.org/packages-plotly.txt >>>> http://composer.tiki.org/packages-sebastian.txt >>>> http://composer.tiki.org/packages-vimeo.txt >>>> http://composer.tiki.org/packages-aws.txt >>>> http://composer.tiki.org/packages-google.txt >>>> http://composer.tiki.org/packages-mpdf.txt >>>> >>>> >>>> I don't usually run unit-tests but I can't see why we keep a >>>> cache of old versions of dev tools. >>>> >>>> Cheers, >>>> J-M >>>> >>>> On Fri, Apr 8, 2022 at 11:28 AM Victor Emanouilov >>>> <[email protected]> wrote: >>>> >>>> JM, sorry, one more request. These are several extremely >>>> large, packages themselves are about 20-30mb but these take >>>> 10+GB on disk, can you check them as well (drill down)? >>>> >>>> 100G dist/phpstan >>>> 66G dist/phpunit >>>> 30G dist/plotly >>>> 29G dist/sebastian >>>> 26G dist/vimeo >>>> 25G dist/aws >>>> 15G dist/google >>>> 13G dist/mpdf >>>> >>>> Thanks, >>>> Victor >>>> >>>> On 4/7/22 7:10 PM, Jean-Marc Libs wrote: >>>>> Hi Victor, >>>>> >>>>> It's a huge file so I left it online: >>>>> That's a simple du -sh : >>>>> http://composer.tiki.org/packages_du-sh.txt >>>>> That one is sorted by size: >>>>> http://composer.tiki.org/packages_du-sh_sorted.txt >>>>> >>>>> No problem, >>>>> J-M >>>>> >>>>> On Wed, Apr 6, 2022 at 2:05 PM Victor Emanouilov via >>>>> TikiWiki-devel <[email protected]> wrote: >>>>> >>>>> JM, can you `du -sh` on the satis storage dirs and >>>>> send us a list of the >>>>> directories that take the most amount of space? We >>>>> might limit semantic >>>>> versions of those packages to what Tiki really needs... >>>>> >>>>> Thanks, >>>>> Victor >>>>> >>>>> On 4/5/22 6:42 PM, Jonny Bradley via TikiWiki-devel wrote: >>>>> > Hi Victor >>>>> > >>>>> > Yes, that was going to be plan B, if just listing >>>>> what's needed currently in all the branches didn't >>>>> result in freeing up enough space we could keep just >>>>> the supported ones (somehow, we don't actually have >>>>> that info in a format we can access easily afaik). >>>>> > >>>>> > The main thing needed is to change from using the >>>>> actual used version to working out all the possible >>>>> required ones, i expect we can use whatever composer >>>>> uses to calculate that but i probably need to know >>>>> more about how satis working internally... >>>>> > >>>>> > And have more free time! ;) >>>>> > >>>>> > jb >>>>> > >>>>> > >>>>> > >>>>> >> On 5 Apr 2022, at 11:44, Victor Emanouilov via >>>>> TikiWiki-devel <[email protected]> >>>>> wrote: >>>>> >> >>>>> >> Hi Jonny, Jean-Marc, >>>>> >> >>>>> >> Thanks for fixing this! It has been a lingering >>>>> problem for quite some time. We definitely don't need >>>>> all the packages that are downloaded on >>>>> composer.tiki.org <http://composer.tiki.org> but >>>>> Jonny, that's a static repository generator which >>>>> means it can't fall back to download a missing >>>>> package. I am thinking why don't we reduce to the >>>>> number of official Tiki releases we support - we get >>>>> all semantic versions of packages and generate only >>>>> those in satis? Symfony packages, for example, will be >>>>> reduced to v3 or v4 only and not everything from 2 to >>>>> 6 (https://composer.tiki.org/#symfony/config) for >>>>> example. Each official Tiki release can have Jonny's >>>>> improved script run and generate the correct satis >>>>> config. I believe we should decrease storage >>>>> requirements several times this way. >>>>> >> >>>>> >> Regards, >>>>> >> Victor >>>>> >> >>>>> >> On 4/5/22 12:15 PM, Jonny Bradley via >>>>> TikiWiki-devel wrote: >>>>> >>> Hi Jean-Marc and all >>>>> >>> >>>>> >>> Had a spare hour last night so started on this: >>>>> >>> >>>>> >>> >>>>> https://gitlab.com/tikiwiki/tiki/-/blob/master/doc/devtools/composer_packages_in_use.php >>>>> >>> >>>>> >>> which at the moment lists all the versions >>>>> currently in use by all the branches in tiki on gitlab. >>>>> >>> >>>>> >>> However, it then occurred to me that many of these >>>>> branches will have outstanding automatic updates >>>>> pending, e.g. if we ran `composer update` on tiki 22.x >>>>> there almost certainly would be updates to newer >>>>> packages, so maybe this isn't the right strategy after >>>>> all... >>>>> >>> >>>>> >>> I guess we need to take into account the semantic >>>>> versioning in all the composer.json files rather than >>>>> the actual ones in use in the lock files, but at least >>>>> it's a start? >>>>> >>> >>>>> >>> Would it be possible to remove some packages form >>>>> satis on composer.t.o, but if something requests one >>>>> of them it re-downloads it? Then we could be more >>>>> confident in deleting stuff... >>>>> >>> >>>>> >>> Or just put a larger drive on it‽ :p >>>>> >>> >>>>> >>> Thanks for fixing it again! >>>>> >>> >>>>> >>> jonny >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>>> On 5 Apr 2022, at 01:09, Jean-Marc Libs >>>>> <[email protected]> wrote: >>>>> >>>> >>>>> >>>> Hi folks, >>>>> >>>> >>>>> >>>> root partition was full on composer.tiki.org >>>>> <http://composer.tiki.org> 😞 I managed to find 26G >>>>> and it works again but the volume is staggering (409G, >>>>> 103G once compressed for backups) and goes up all the >>>>> time. >>>>> >>>> >>>>> >>>> Do we need so many third-party libs? >>>>> >>>> >>>>> >>>> Cheers, >>>>> >>>> J-M >>>>> >>>> >>>>> >>>> On Mon, Apr 4, 2022 at 5:17 PM Victor Emanouilov >>>>> via TikiWiki-devel >>>>> <[email protected]> wrote: >>>>> >>>> +1 same for me ... that URL returns 304 not >>>>> modified - maybe some >>>>> >>>> apache/nginx config is needed? Composer expects a >>>>> valid json file there... >>>>> >>>> >>>>> >>>> Regards, >>>>> >>>> Victor >>>>> >>>> >>>>> >>>> On 4/4/22 3:03 PM, Jonny Bradley via >>>>> TikiWiki-devel wrote: >>>>> >>>>> Hi all >>>>> >>>>> >>>>> >>>>> I'm trying to update composer for the bootstrap5 >>>>> branch following some rebasing and it seem composer is >>>>> failing now, and it's the same in trunk/master/main now. >>>>> >>>>> >>>>> >>>>> When i do: >>>>> >>>>> >>>>> >>>>> $ php74 temp/composer.phar update --working-dir >>>>> vendor_bundled --prefer-dist --optimize-autoloader >>>>> >>>>> >>>>> >>>>> I get: >>>>> >>>>> >>>>> >>>>>> In JsonFile.php line 345: >>>>> >>>>>> >>>>> "https://composer.tiki.org/include/all%2419ac295e729a70b8272116cec8bdb0716f3dd8d2.json" >>>>> does not contain valid JSON >>>>> >>>>>> Parse error on line 1: >>>>> >>>>>> >>>>> >>>>>> ^ >>>>> >>>>>> Expected one of: 'STRING', 'NUMBER', >>>>> 'NULL', 'TRUE', 'FALSE', '{', '[' >>>>> >>>>>> >>>>> >>>>> >>>>> https://composer.tiki.org/include/all%2419ac295e729a70b8272116cec8bdb0716f3dd8d2.json >>>>> is an empty file, which presumably is the problem. >>>>> >>>>> >>>>> >>>>> Does anyone know why? (probably a Jean-Marc >>>>> thing i expect, maybe satis needs >>>>> updating/restarting/flushing?) >>>>> >>>>> >>>>> >>>>> jonny >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> >>>>> TikiWiki-devel mailing list >>>>> >>>>> [email protected] >>>>> >>>>> >>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>> >>>> _______________________________________________ >>>>> >>>> TikiWiki-devel mailing list >>>>> >>>> [email protected] >>>>> >>>> >>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>> >>>> _______________________________________________ >>>>> >>>> TikiWiki-devel mailing list >>>>> >>>> [email protected] >>>>> >>>> >>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>> >>> >>>>> >>> _______________________________________________ >>>>> >>> TikiWiki-devel mailing list >>>>> >>> [email protected] >>>>> >>> >>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>> >> >>>>> >> _______________________________________________ >>>>> >> TikiWiki-devel mailing list >>>>> >> [email protected] >>>>> >> >>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>> > >>>>> > >>>>> > _______________________________________________ >>>>> > TikiWiki-devel mailing list >>>>> > [email protected] >>>>> > >>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>> >>>>> >>>>> _______________________________________________ >>>>> TikiWiki-devel mailing list >>>>> [email protected] >>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>> >> >> >> _______________________________________________ >> TikiWiki-devel mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel > > > _______________________________________________ > TikiWiki-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel _______________________________________________ TikiWiki-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel