Re: [Tiki-devel] composer.tiki.org slightly borken?

I was prodding at this again a few weeks ago, it's a "fun" little puzzle!

I found this satis command (which i thought i shared before, but seems not)

    https://github.com/composer/satis#purge

which (if i read it correctly, takes a composer.json file and removes all the packages not in it from satis... hmm, not quite sadly, need to read the code more in \Composer\Satis\Console\Command\PurgeCommand

So towards that goal i've updated that script to get a combined version string for each package used in tiki, so we can (probably) remove all the ones not needed...

    https://gitlab.com/tikiwiki/tiki/-/commit/f082712c

Checked some of them on here

    https://semver.madewithlove.com/?package=smarty%2Fsmarty&constraint=^3.1.33|^4.0&stability=stable

and it seems to work...

The command `php doc/devtools/composer_packages_in_use.php > possible-satis-composer.json` produced this file:



Does that help at at all?

As you say, something for the RTM...

jonny



> On 15 May 2022, at 04:43, Marc Laporte <[email protected]> wrote:
> 
> If we "delete the whole directory holding packages on composer.tiki.org" and then "deploy and run satis again to download only the ones that are currently in use", we may be unable to get them all back. Some of the older stuff is no longer available from where got them.
> 
> I suppose running a test with supported versions (18.x +) would tell us.
> 
> Thanks!
> 
> Marc
> 
> On Tue, 3 May 2022 12:07:13 +0300 Tiki developers [email protected] said
> 
>> Hi J-M, Yes, I agree we can remove dev packages from satis config and
>>     save space. The other thing I see is that some packages have way
>>     too many versions as files on the disk compared to the versions
>>     listed on composer.tiki.org - I think these are cached assets that
>>     probably disappeared later and are no longer supported. Do you
>>     think that we can do the following? 1. clean up satis config from dev packages 2. delete the whole directory holding packages on
>>     composer.tiki.org 3. deploy and run satis again to download only the ones that are
>>     currently in use Regards, Victor On 4/11/22 1:32 AM, Jean-Marc Libs
>>     wrote: Hi Victor, here is the result of some more drilling down:
>> http://composer.tiki.org/packages-phpstan.txt http://composer.tiki.org/packages-phpunit.txt http://composer.tiki.org/packages-plotly.txt http://composer.tiki.org/packages-sebastian.txt http://composer.tiki.org/packages-vimeo.txt http://composer.tiki.org/packages-aws.txt http://composer.tiki.org/packages-google.txt http://composer.tiki.org/packages-mpdf.txt I don't usually run unit-tests but I can't see why we keep
>>         a cache of old versions of dev tools. Cheers, J-M On Fri, Apr 8, 2022 at 11:28
>>         AM Victor Emanouilov < [email protected] >
>>         wrote: JM, sorry, one more request. These are several extremely
>>             large, packages themselves are about 20-30mb but these
>>             take 10+GB on disk, can you check them as well (drill
>>             down)? 100G	dist/phpstan
>> 66G	dist/phpunit
>> 30G	dist/plotly
>> 29G	dist/sebastian
>> 26G	dist/vimeo
>> 25G	dist/aws
>> 15G	dist/google
>> 13G	dist/mpdf Thanks, Victor On 4/7/22 7:10 PM, Jean-Marc Libs wrote: Hi Victor, It's a huge file so I left it online: That's a simple du -sh : http://composer.tiki.org/packages_du-sh.txt That one is sorted by size: http://composer.tiki.org/packages_du-sh_sorted.txt No problem, J-M On Wed, Apr 6, 2022 at
>>                 2:05 PM Victor Emanouilov via TikiWiki-devel <
>> [email protected] >
>>                 wrote: JM, can you `du
>>                 -sh` on the satis storage dirs and send us a list of
>>                 the directories that take the most amount of space? We
>>                 might limit semantic versions of those packages to what Tiki really
>>                 needs... Thanks, Victor On 4/5/22 6:42 PM, Jonny Bradley via TikiWiki-devel
>>                 wrote: > Hi Victor > > Yes, that was going to be plan B, if just listing
>>                 what's needed currently in all the branches didn't
>>                 result in freeing up enough space we could keep just
>>                 the supported ones (somehow, we don't actually have
>>                 that info in a format we can access easily afaik). > > The main thing needed is to change from using the
>>                 actual used version to working out all the possible
>>                 required ones, i expect we can use whatever composer
>>                 uses to calculate that but i probably need to know
>>                 more about how satis working internally... > > And have more free time! ;) > > jb > > > >> On 5 Apr 2022, at 11:44, Victor Emanouilov
>>                 via TikiWiki-devel < [email protected]
>> >
>>                 wrote: >> >> Hi Jonny, Jean-Marc, >> >> Thanks for fixing this! It has been a
>>                 lingering problem for quite some time. We definitely
>>                 don't need all the packages that are downloaded on
>> composer.tiki.org but Jonny, that's a static repository generator which
>>                 means it can't fall back to download a missing
>>                 package. I am thinking why don't we reduce to the
>>                 number of official Tiki releases we support - we get
>>                 all semantic versions of packages and generate only
>>                 those in satis? Symfony packages, for example, will be
>>                 reduced to v3 or v4 only and not everything from 2 to
>>                 6 ( https://composer.tiki.org/#symfony/config )
>>                 for example. Each official Tiki release can have
>>                 Jonny's improved script run and generate the correct
>>                 satis config. I believe we should decrease storage
>>                 requirements several times this way. >> >> Regards, >> Victor >> >> On 4/5/22 12:15 PM, Jonny Bradley via
>>                 TikiWiki-devel wrote: >>> Hi Jean-Marc and all >>> >>> Had a spare hour last night so started on
>>                 this: >>> >>>     
>> >https://gitlab.com/tikiwiki/tiki/-/blob/master/doc/devtools/composer_packages_in_use.php
>> >
>> >>> >>> which at the moment lists all the
>>                 versions currently in use by all the branches in tiki
>>                 on gitlab. >>> >>> However, it then occurred to me that many
>>                 of these branches will have outstanding automatic
>>                 updates pending, e.g. if we ran `composer update` on
>>                 tiki 22.x there almost certainly would be updates to
>>                 newer packages, so maybe this isn't the right strategy
>>                 after all... >>> >>> I guess we need to take into account the
>>                 semantic versioning in all the composer.json files
>>                 rather than the actual ones in use in the lock files,
>>                 but at least it's a start? >>> >>> Would it be possible to remove some
>>                 packages form satis on composer.t.o, but if something
>>                 requests one of them it re-downloads it? Then we could
>>                 be more confident in deleting stuff... >>> >>> Or just put a larger drive on it‽ :p >>> >>> Thanks for fixing it again! >>> >>> jonny >>> >>> >>> >>>> On 5 Apr 2022, at 01:09, Jean-Marc
>>                 Libs < [email protected] >
>>                 wrote: >>>> >>>> Hi folks, >>>> >>>> root partition was full on composer.tiki.org 😞 I managed to find 26G
>> > and it works again but the
>>                 volume is staggering (409G, 103G once compressed for
>>                 backups) and goes up all the time. >>>> >>>> Do we need so many third-party libs? >>>> >>>> Cheers, >>>> J-M >>>> >>>> On Mon, Apr 4, 2022 at 5:17 PM Victor
>>                 Emanouilov via TikiWiki-devel <
>> [email protected] >
>>                 wrote: >>>> +1 same for me ... that URL returns
>>                 304 not modified - maybe some >>>> apache/nginx config is needed?
>>                 Composer expects a valid json file there... >>>> >>>> Regards, >>>> Victor >>>> >>>> On 4/4/22 3:03 PM, Jonny Bradley via
>>                 TikiWiki-devel wrote: >>>>> Hi all >>>>> >>>>> I'm trying to update composer for
>>                 the bootstrap5 branch following some rebasing and it
>>                 seem composer is failing now, and it's the same in
>>                 trunk/master/main now. >>>>> >>>>> When i do: >>>>> >>>>> $ php74 temp/composer.phar
>>                 update  --working-dir vendor_bundled --prefer-dist
>>                 --optimize-autoloader >>>>> >>>>> I get: >>>>> >>>>>> In JsonFile.php line 345: >>>>>>                             
>>                                                        
>>              
>>                                                        
>>   "
>> https://composer.tiki.org/include/all%2419ac295e729a70b8272116cec8bdb0716f3dd8d2.json
>> "
>>                 does not contain valid JSON >>>>>>     Parse error on line 1: >>>>>> >>>>>>     ^ >>>>>>     Expected one of:
>>                 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '[' >>>>>> >>>>>
>> > https://composer.tiki.org/include/all%2419ac295e729a70b8272116cec8bdb0716f3dd8d2.json
>> > is an empty file, which presumably is the problem. >>>>> >>>>> Does anyone know why? (probably a
>>                 Jean-Marc thing i expect, maybe satis needs
>>                 updating/restarting/flushing?) >>>>> >>>>> jonny >>>>> >>>>> >>>>> >>>>> >>>>>
>>                 _______________________________________________ >>>>> TikiWiki-devel mailing list >>>>> [email protected] >>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>
>>                 _______________________________________________ >>>> TikiWiki-devel mailing list >>>> [email protected] >>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>>>
>>                 _______________________________________________ >>>> TikiWiki-devel mailing list >>>> [email protected] >>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >>> >>>
>>                 _______________________________________________ >>> TikiWiki-devel mailing list >>> [email protected] >>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel >> >>
>>                 _______________________________________________ >> TikiWiki-devel mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel > > > _______________________________________________ > TikiWiki-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel _______________________________________________ TikiWiki-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
> 
> 
> 
> 
> _______________________________________________
> TikiWiki-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel

_______________________________________________
TikiWiki-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel