Re: [Tiki-devel] composer.tiki.org slightly borken?
Victor Emanouilov via TikiWiki-devel <[email protected]>
| Newsgroups | gmane.comp.cms.tiki.devel |
|---|---|
| Message-ID | <[email protected]> |
Alright, let's discuss it then! On 5/13/22 10:52 PM, Jean-Marc Libs wrote: > Hi Victor, > > I can't think of an issue with removing dev packages from satis config. > > I have a hard time understanding the consequences of removing old > versions. Does it mean updating composer packages using setup.sh will > fail unless the Tiki source is updated to the latest version first? > > I'm concerned about the consequences for people who update old > production Tikis. I am not familiar enough with composer to make an > opinion :-( > > Maybe we can discuss this during next Roundtable? I put the topic > here: https://tiki.org/tiki-index.php?page=Roundtable-Meeting-2022-05 > > Cheers, > J-M > > > On Tue, May 3, 2022 at 11:08 AM Victor Emanouilov <[email protected]> wrote: > > Hi J-M, > > Yes, I agree we can remove dev packages from satis config and save > space. The other thing I see is that some packages have way too > many versions as files on the disk compared to the versions listed > on composer.tiki.org <http://composer.tiki.org> - I think these > are cached assets that probably disappeared later and are no > longer supported. Do you think that we can do the following? > > 1. clean up satis config from dev packages > 2. delete the whole directory holding packages on > composer.tiki.org <http://composer.tiki.org> > 3. deploy and run satis again to download only the ones that are > currently in use > > Regards, > Victor > > On 4/11/22 1:32 AM, Jean-Marc Libs wrote: >> Hi Victor, here is the result of some more drilling down: >> http://composer.tiki.org/packages-phpstan.txt >> http://composer.tiki.org/packages-phpunit.txt >> http://composer.tiki.org/packages-plotly.txt >> http://composer.tiki.org/packages-sebastian.txt >> http://composer.tiki.org/packages-vimeo.txt >> http://composer.tiki.org/packages-aws.txt >> http://composer.tiki.org/packages-google.txt >> http://composer.tiki.org/packages-mpdf.txt >> >> >> I don't usually run unit-tests but I can't see why we keep a >> cache of old versions of dev tools. >> >> Cheers, >> J-M >> >> On Fri, Apr 8, 2022 at 11:28 AM Victor Emanouilov >> <[email protected]> wrote: >> >> JM, sorry, one more request. These are several extremely >> large, packages themselves are about 20-30mb but these take >> 10+GB on disk, can you check them as well (drill down)? >> >> 100G dist/phpstan >> 66G dist/phpunit >> 30G dist/plotly >> 29G dist/sebastian >> 26G dist/vimeo >> 25G dist/aws >> 15G dist/google >> 13G dist/mpdf >> >> Thanks, >> Victor >> >> On 4/7/22 7:10 PM, Jean-Marc Libs wrote: >>> Hi Victor, >>> >>> It's a huge file so I left it online: >>> That's a simple du -sh : >>> http://composer.tiki.org/packages_du-sh.txt >>> That one is sorted by size: >>> http://composer.tiki.org/packages_du-sh_sorted.txt >>> >>> No problem, >>> J-M >>> >>> On Wed, Apr 6, 2022 at 2:05 PM Victor Emanouilov via >>> TikiWiki-devel <[email protected]> wrote: >>> >>> JM, can you `du -sh` on the satis storage dirs and send >>> us a list of the >>> directories that take the most amount of space? We might >>> limit semantic >>> versions of those packages to what Tiki really needs... >>> >>> Thanks, >>> Victor >>> >>> On 4/5/22 6:42 PM, Jonny Bradley via TikiWiki-devel wrote: >>> > Hi Victor >>> > >>> > Yes, that was going to be plan B, if just listing >>> what's needed currently in all the branches didn't >>> result in freeing up enough space we could keep just the >>> supported ones (somehow, we don't actually have that >>> info in a format we can access easily afaik). >>> > >>> > The main thing needed is to change from using the >>> actual used version to working out all the possible >>> required ones, i expect we can use whatever composer >>> uses to calculate that but i probably need to know more >>> about how satis working internally... >>> > >>> > And have more free time! ;) >>> > >>> > jb >>> > >>> > >>> > >>> >> On 5 Apr 2022, at 11:44, Victor Emanouilov via >>> TikiWiki-devel <[email protected]> wrote: >>> >> >>> >> Hi Jonny, Jean-Marc, >>> >> >>> >> Thanks for fixing this! It has been a lingering >>> problem for quite some time. We definitely don't need >>> all the packages that are downloaded on >>> composer.tiki.org <http://composer.tiki.org> but Jonny, >>> that's a static repository generator which means it >>> can't fall back to download a missing package. I am >>> thinking why don't we reduce to the number of official >>> Tiki releases we support - we get all semantic versions >>> of packages and generate only those in satis? Symfony >>> packages, for example, will be reduced to v3 or v4 only >>> and not everything from 2 to 6 >>> (https://composer.tiki.org/#symfony/config) for example. >>> Each official Tiki release can have Jonny's improved >>> script run and generate the correct satis config. I >>> believe we should decrease storage requirements several >>> times this way. >>> >> >>> >> Regards, >>> >> Victor >>> >> >>> >> On 4/5/22 12:15 PM, Jonny Bradley via TikiWiki-devel >>> wrote: >>> >>> Hi Jean-Marc and all >>> >>> >>> >>> Had a spare hour last night so started on this: >>> >>> >>> >>> >>> https://gitlab.com/tikiwiki/tiki/-/blob/master/doc/devtools/composer_packages_in_use.php >>> >>> >>> >>> which at the moment lists all the versions currently >>> in use by all the branches in tiki on gitlab. >>> >>> >>> >>> However, it then occurred to me that many of these >>> branches will have outstanding automatic updates >>> pending, e.g. if we ran `composer update` on tiki 22.x >>> there almost certainly would be updates to newer >>> packages, so maybe this isn't the right strategy after >>> all... >>> >>> >>> >>> I guess we need to take into account the semantic >>> versioning in all the composer.json files rather than >>> the actual ones in use in the lock files, but at least >>> it's a start? >>> >>> >>> >>> Would it be possible to remove some packages form >>> satis on composer.t.o, but if something requests one of >>> them it re-downloads it? Then we could be more confident >>> in deleting stuff... >>> >>> >>> >>> Or just put a larger drive on it‽ :p >>> >>> >>> >>> Thanks for fixing it again! >>> >>> >>> >>> jonny >>> >>> >>> >>> >>> >>> >>> >>>> On 5 Apr 2022, at 01:09, Jean-Marc Libs >>> <[email protected]> wrote: >>> >>>> >>> >>>> Hi folks, >>> >>>> >>> >>>> root partition was full on composer.tiki.org >>> <http://composer.tiki.org> 😞 I managed to find 26G and >>> it works again but the volume is staggering (409G, 103G >>> once compressed for backups) and goes up all the time. >>> >>>> >>> >>>> Do we need so many third-party libs? >>> >>>> >>> >>>> Cheers, >>> >>>> J-M >>> >>>> >>> >>>> On Mon, Apr 4, 2022 at 5:17 PM Victor Emanouilov >>> via TikiWiki-devel >>> <[email protected]> wrote: >>> >>>> +1 same for me ... that URL returns 304 not >>> modified - maybe some >>> >>>> apache/nginx config is needed? Composer expects a >>> valid json file there... >>> >>>> >>> >>>> Regards, >>> >>>> Victor >>> >>>> >>> >>>> On 4/4/22 3:03 PM, Jonny Bradley via TikiWiki-devel >>> wrote: >>> >>>>> Hi all >>> >>>>> >>> >>>>> I'm trying to update composer for the bootstrap5 >>> branch following some rebasing and it seem composer is >>> failing now, and it's the same in trunk/master/main now. >>> >>>>> >>> >>>>> When i do: >>> >>>>> >>> >>>>> $ php74 temp/composer.phar update --working-dir >>> vendor_bundled --prefer-dist --optimize-autoloader >>> >>>>> >>> >>>>> I get: >>> >>>>> >>> >>>>>> In JsonFile.php line 345: >>> >>>>>> >>> "https://composer.tiki.org/include/all%2419ac295e729a70b8272116cec8bdb0716f3dd8d2.json" >>> does not contain valid JSON >>> >>>>>> Parse error on line 1: >>> >>>>>> >>> >>>>>> ^ >>> >>>>>> Expected one of: 'STRING', 'NUMBER', 'NULL', >>> 'TRUE', 'FALSE', '{', '[' >>> >>>>>> >>> >>>>> >>> https://composer.tiki.org/include/all%2419ac295e729a70b8272116cec8bdb0716f3dd8d2.json >>> is an empty file, which presumably is the problem. >>> >>>>> >>> >>>>> Does anyone know why? (probably a Jean-Marc thing >>> i expect, maybe satis needs updating/restarting/flushing?) >>> >>>>> >>> >>>>> jonny >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> _______________________________________________ >>> >>>>> TikiWiki-devel mailing list >>> >>>>> [email protected] >>> >>>>> >>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>> >>>> _______________________________________________ >>> >>>> TikiWiki-devel mailing list >>> >>>> [email protected] >>> >>>> >>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>> >>>> _______________________________________________ >>> >>>> TikiWiki-devel mailing list >>> >>>> [email protected] >>> >>>> >>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>> >>> >>> >>> _______________________________________________ >>> >>> TikiWiki-devel mailing list >>> >>> [email protected] >>> >>> >>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>> >> >>> >> _______________________________________________ >>> >> TikiWiki-devel mailing list >>> >> [email protected] >>> >> >>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>> > >>> > >>> > _______________________________________________ >>> > TikiWiki-devel mailing list >>> > [email protected] >>> > >>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>> >>> >>> _______________________________________________ >>> TikiWiki-devel mailing list >>> [email protected] >>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>> _______________________________________________ TikiWiki-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel