Re: [Tiki-devel] composer.tiki.org slightly borken?

After a while, I imagine the satis update script was run and there are more
packages
# du -sh /composer/composer.tiki.org/
7.7G    /composer/composer.tiki.org/

The new tree command gives:
https://composer.tiki.org/tree2.txt

Cheers,
J-M

On Thu, Jun 30, 2022 at 2:27 AM Jean-Marc Libs <[email protected]>
wrote:

> Thanks a lot, Victor!
>
> I needed more time than expected for doing the last remote backup (the
> remote backup server was running into quota issues) but it's done!
>
> I ran the script and it seems to have done *a lot* of cleanup. *Maybe too
> much?*
>
> Now the site takes only:
> root@hetzner01 /var/local/composer.tiki.org/www # du -sh /composer/
> composer.tiki.org/
> 3.1G    /composer/composer.tiki.org/
>
> Here is the result of the tree command :
> https://composer.tiki.org/tree.txt
>
> I ran composer upgrade on a Tiki21 site and I got:
> web# sh setup.sh
> …
> Your choice [c]?
>
> Local PHP version 72 >= to required PHP version 72 - good
> Warning: You forced the install of 1.10.26 via --1, but 2.3.7 is the
> latest stable version. Updating to it via composer self-update --stable is
> recommended.
> You are already using composer version 1.10.26 (1.x channel).
> Do not run Composer as root/super user! See https://getcomposer.org/root
> for details
> You are using Composer 1 which is deprecated. You should upgrade to
> Composer 2, see https://blog.packagist.com/deprecating-composer-1-support/
> Gathering patches from patch file.
> Removing package adodb/adodb-php so that it can be re-installed and
> re-patched.
>  - Removing adodb/adodb-php (v5.21.3)
> > Tiki\Composer\RestoreVendors::restore
> > Tiki\Composer\CleanVendorAfterVendorBundledMigration::cleanLinks
> Loading composer repositories with package information
> Installing dependencies from lock file
> Warning: The lock file is not up to date with the latest changes in
> composer.json. You may be getting outdated dependencies. It is recommended
> that you run `composer update` or `composer update <package name>`.
> Package operations: 2 installs, 0 updates, 0 removals
> Gathering patches from patch file.
> Gathering patches for dependencies. This might take a minute.
>  - Installing adodb/adodb-php (v5.21.3): Downloading (failed)    Failed to
> download adodb/adodb-php from dist: The "
> https://composer.tiki.org/dist/adodb/adodb-php/adodb-adodb-php-c5415722049f36c446a4034d15f1d17943f11458-zip-e71b61.zip"
> file could not be downloaded (HTTP
> /1.1 404 Not Found)
>    Now trying to download from source
>  - Installing adodb/adodb-php (v5.21.3): Cloning c541572204 from cache
>  - Applying patches for adodb/adodb-php
>    ../installer/composer-patches/adodb-php__remove_references_to_adodb_extension.patch
> (Remove references to ADOdb Extension)
>   Could not apply patch! Skipping. The error was: Cannot apply patch
> ../installer/composer-patches/adodb-php__remove_references_to_adodb_extension.patch
>
>    ../installer/composer-patches/adodb-php__remove-the-check-if-the-adodb-php_ext-is-enabled.patch
> (Remove the check for ADOdb Extension)
>
>  - Installing components/bootstrap (3.4.1): Downloading (failed)    Failed
> to download components/bootstrap from dist: The "
> https://composer.tiki.org/dist/components/bootstrap/components-bootstrap-427f4d414fc6ae8385aee02b7efc8e391eb141ad-zip-393061.zip"
> file could not b
> e downloaded (HTTP/1.1 404 Not Found)
>    Now trying to download from source
>  - Installing components/bootstrap (3.4.1): Cloning 427f4d414f from cache
> Package container-interop/container-interop is abandoned, you should avoid
> using it. Use psr/container instead.
> Package evert/sitemap-php is abandoned, you should avoid using it. No
> replacement was suggested.
> Package league/commonmark-extras is abandoned, you should avoid using it.
> Use league/commonmark instead.
> Package robloach/component-installer is abandoned, you should avoid using
> it. Use oomphinc/composer-installers-extender instead.
> Package webuni/commonmark-attributes-extension is abandoned, you should
> avoid using it. Use league/commonmark instead.
> Package zendframework/zend-version is abandoned, you should avoid using
> it. Use ocramius/package-versions instead.
> Package zendframework/zendservice-akismet is abandoned, you should avoid
> using it. No replacement was suggested.
> Generating autoload files
> > ComponentInstaller\Installer::postAutoloadDump
> Compiling component files
> > Tiki\Composer\ThemeInstaller::setup
> > Tiki\Composer\BootstrapCompiler::build
> > Tiki\Composer\CleanVendors::clean
>
> > Tiki\Composer\CleanVendorAfterVendorBundledMigration::clean
> > Tiki\Composer\PatchCypht::setup
>
>
> Tiki setup.sh - your options
>
> ============================
> …
>
> Should I restore the previous packages and wait for a new version of the
> script?
>
> Cheers,
> Jyhem
>
>
>
> On Thu, Jun 23, 2022 at 3:58 PM Victor Emanouilov via TikiWiki-devel <
> [email protected]> wrote:
>
>> Hi Jean-Marc,
>>
>> I added a Tiki devtools script for this:
>>
>>
>> https://gitlab.com/tikiwiki/tiki/-/commit/4ef8dc83e934ccee970a722e5f14237cbe8f6a73
>>
>> Please configure the first variable to the actual local filesystem path
>> of composer.tiki.org webroot and run it locally on that server. You can
>> use any version of php 7. Running it without any arguments will list the
>> files to be deleted. Running it with --cleanup argument will actually
>> perform the delete operation.
>>
>> Regards,
>> Victor
>> On 6/22/22 5:57 PM, Victor Emanouilov via TikiWiki-devel wrote:
>>
>> JM,
>>
>>
>> As we talked on the roundtable meeting, this is the file that lists all
>> the package versions composer.tiki.org supports:
>>
>> https://composer.tiki.org/packages.json
>>
>> Currently, it includes this one (it's huge!):
>>
>>
>> https://composer.tiki.org/include/all$838361a2a0955d73dfb3e0370a109616b7e37fd5.json
>>
>> I will send you a script that pinpoints all the dist files downloaded on
>> local disk that are not references in the packages.json file and provide an
>> option to remove them. This should clean up about half of the used space if
>> not even more.
>>
>> Regards,
>> Victor
>> On 5/16/22 2:40 PM, Victor Emanouilov via TikiWiki-devel wrote:
>>
>> Alright, let's discuss it then!
>> On 5/13/22 10:52 PM, Jean-Marc Libs wrote:
>>
>> Hi Victor,
>>
>> I can't think of an issue with removing dev packages from satis config.
>>
>> I have a hard time understanding the consequences of removing old
>> versions. Does it mean updating composer packages using setup.sh will fail
>> unless the Tiki source is updated to the latest version first?
>>
>> I'm concerned about the consequences for people who update old production
>> Tikis. I am not familiar enough with composer to make an opinion :-(
>>
>> Maybe we can discuss this during next Roundtable? I put the topic here:
>> https://tiki.org/tiki-index.php?page=Roundtable-Meeting-2022-05
>>
>> Cheers,
>> J-M
>>
>>
>> On Tue, May 3, 2022 at 11:08 AM Victor Emanouilov <[email protected]>
>> wrote:
>>
>>> Hi J-M,
>>>
>>> Yes, I agree we can remove dev packages from satis config and save
>>> space. The other thing I see is that some packages have way too many
>>> versions as files on the disk compared to the versions listed on
>>> composer.tiki.org - I think these are cached assets that probably
>>> disappeared later and are no longer supported. Do you think that we can do
>>> the following?
>>>
>>> 1. clean up satis config from dev packages
>>> 2. delete the whole directory holding packages on composer.tiki.org
>>> 3. deploy and run satis again to download only the ones that are
>>> currently in use
>>>
>>> Regards,
>>> Victor
>>> On 4/11/22 1:32 AM, Jean-Marc Libs wrote:
>>>
>>> Hi Victor, here is the result of some more drilling down:
>>> http://composer.tiki.org/packages-phpstan.txt
>>> http://composer.tiki.org/packages-phpunit.txt
>>> http://composer.tiki.org/packages-plotly.txt
>>> http://composer.tiki.org/packages-sebastian.txt
>>> http://composer.tiki.org/packages-vimeo.txt
>>> http://composer.tiki.org/packages-aws.txt
>>> http://composer.tiki.org/packages-google.txt
>>> http://composer.tiki.org/packages-mpdf.txt
>>>
>>>
>>> I don't usually run unit-tests but I can't see why we keep a cache of
>>> old versions of dev tools.
>>>
>>> Cheers,
>>> J-M
>>>
>>> On Fri, Apr 8, 2022 at 11:28 AM Victor Emanouilov <[email protected]>
>>> wrote:
>>>
>>>> JM, sorry, one more request. These are several extremely large,
>>>> packages themselves are about 20-30mb but these take 10+GB on disk, can you
>>>> check them as well (drill down)?
>>>>
>>>> 100G	dist/phpstan
>>>> 66G	dist/phpunit
>>>> 30G	dist/plotly
>>>> 29G	dist/sebastian
>>>> 26G	dist/vimeo
>>>> 25G	dist/aws
>>>> 15G	dist/google
>>>> 13G	dist/mpdf
>>>>
>>>> Thanks,
>>>> Victor
>>>> On 4/7/22 7:10 PM, Jean-Marc Libs wrote:
>>>>
>>>> Hi Victor,
>>>>
>>>> It's a huge file so I left it online:
>>>> That's a simple du -sh :
>>>> http://composer.tiki.org/packages_du-sh.txt
>>>> That one is sorted by size:
>>>> http://composer.tiki.org/packages_du-sh_sorted.txt
>>>>
>>>> No problem,
>>>> J-M
>>>>
>>>> On Wed, Apr 6, 2022 at 2:05 PM Victor Emanouilov via TikiWiki-devel <
>>>> [email protected]> wrote:
>>>>
>>>>> JM, can you `du -sh` on the satis storage dirs and send us a list of
>>>>> the
>>>>> directories that take the most amount of space? We might limit
>>>>> semantic
>>>>> versions of those packages to what Tiki really needs...
>>>>>
>>>>> Thanks,
>>>>> Victor
>>>>>
>>>>> On 4/5/22 6:42 PM, Jonny Bradley via TikiWiki-devel wrote:
>>>>> > Hi Victor
>>>>> >
>>>>> > Yes, that was going to be plan B, if just listing what's needed
>>>>> currently in all the branches didn't result in freeing up enough space we
>>>>> could keep just the supported ones (somehow, we don't actually have that
>>>>> info in a format we can access easily afaik).
>>>>> >
>>>>> > The main thing needed is to change from using the actual used
>>>>> version to working out all the possible required ones, i expect we can use
>>>>> whatever composer uses to calculate that but i probably need to know more
>>>>> about how satis working internally...
>>>>> >
>>>>> > And have more free time! ;)
>>>>> >
>>>>> > jb
>>>>> >
>>>>> >
>>>>> >
>>>>> >> On 5 Apr 2022, at 11:44, Victor Emanouilov via TikiWiki-devel <
>>>>> [email protected]> wrote:
>>>>> >>
>>>>> >> Hi Jonny, Jean-Marc,
>>>>> >>
>>>>> >> Thanks for fixing this! It has been a lingering problem for quite
>>>>> some time. We definitely don't need all the packages that are downloaded on
>>>>> composer.tiki.org but Jonny, that's a static repository generator
>>>>> which means it can't fall back to download a missing package. I am thinking
>>>>> why don't we reduce to the number of official Tiki releases we support - we
>>>>> get all semantic versions of packages and generate only those in satis?
>>>>> Symfony packages, for example, will be reduced to v3 or v4 only and not
>>>>> everything from 2 to 6 (https://composer.tiki.org/#symfony/config)
>>>>> for example. Each official Tiki release can have Jonny's improved script
>>>>> run and generate the correct satis config. I believe we should decrease
>>>>> storage requirements several times this way.
>>>>> >>
>>>>> >> Regards,
>>>>> >> Victor
>>>>> >>
>>>>> >> On 4/5/22 12:15 PM, Jonny Bradley via TikiWiki-devel wrote:
>>>>> >>> Hi Jean-Marc and all
>>>>> >>>
>>>>> >>> Had a spare hour last night so started on this:
>>>>> >>>
>>>>> >>>
>>>>> https://gitlab.com/tikiwiki/tiki/-/blob/master/doc/devtools/composer_packages_in_use.php
>>>>> >>>
>>>>> >>> which at the moment lists all the versions currently in use by all
>>>>> the branches in tiki on gitlab.
>>>>> >>>
>>>>> >>> However, it then occurred to me that many of these branches will
>>>>> have outstanding automatic updates pending, e.g. if we ran `composer
>>>>> update` on tiki 22.x there almost certainly would be updates to newer
>>>>> packages, so maybe this isn't the right strategy after all...
>>>>> >>>
>>>>> >>> I guess we need to take into account the semantic versioning in
>>>>> all the composer.json files rather than the actual ones in use in the lock
>>>>> files, but at least it's a start?
>>>>> >>>
>>>>> >>> Would it be possible to remove some packages form satis on
>>>>> composer.t.o, but if something requests one of them it re-downloads it?
>>>>> Then we could be more confident in deleting stuff...
>>>>> >>>
>>>>> >>> Or just put a larger drive on it‽ :p
>>>>> >>>
>>>>> >>> Thanks for fixing it again!
>>>>> >>>
>>>>> >>> jonny
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>>> On 5 Apr 2022, at 01:09, Jean-Marc Libs <[email protected]>
>>>>> wrote:
>>>>> >>>>
>>>>> >>>> Hi folks,
>>>>> >>>>
>>>>> >>>> root partition was full on composer.tiki.org 😞 I managed to
>>>>> find 26G and it works again but the volume is staggering (409G, 103G once
>>>>> compressed for backups) and goes up all the time.
>>>>> >>>>
>>>>> >>>> Do we need so many third-party libs?
>>>>> >>>>
>>>>> >>>> Cheers,
>>>>> >>>> J-M
>>>>> >>>>
>>>>> >>>> On Mon, Apr 4, 2022 at 5:17 PM Victor Emanouilov via
>>>>> TikiWiki-devel <[email protected]> wrote:
>>>>> >>>> +1 same for me ... that URL returns 304 not modified - maybe some
>>>>> >>>> apache/nginx config is needed? Composer expects a valid json file
>>>>> there...
>>>>> >>>>
>>>>> >>>> Regards,
>>>>> >>>> Victor
>>>>> >>>>
>>>>> >>>> On 4/4/22 3:03 PM, Jonny Bradley via TikiWiki-devel wrote:
>>>>> >>>>> Hi all
>>>>> >>>>>
>>>>> >>>>> I'm trying to update composer for the bootstrap5 branch
>>>>> following some rebasing and it seem composer is failing now, and it's the
>>>>> same in trunk/master/main now.
>>>>> >>>>>
>>>>> >>>>> When i do:
>>>>> >>>>>
>>>>> >>>>> $ php74 temp/composer.phar update  --working-dir vendor_bundled
>>>>> --prefer-dist --optimize-autoloader
>>>>> >>>>>
>>>>> >>>>> I get:
>>>>> >>>>>
>>>>> >>>>>> In JsonFile.php line 345:
>>>>> >>>>>>
>>>>>                                                               "
>>>>> https://composer.tiki.org/include/all%2419ac295e729a70b8272116cec8bdb0716f3dd8d2.json"
>>>>> does not contain valid JSON
>>>>> >>>>>>     Parse error on line 1:
>>>>> >>>>>>
>>>>> >>>>>>     ^
>>>>> >>>>>>     Expected one of: 'STRING', 'NUMBER', 'NULL', 'TRUE',
>>>>> 'FALSE', '{', '['
>>>>> >>>>>>
>>>>> >>>>>
>>>>> https://composer.tiki.org/include/all%2419ac295e729a70b8272116cec8bdb0716f3dd8d2.json
>>>>> is an empty file, which presumably is the problem.
>>>>> >>>>>
>>>>> >>>>> Does anyone know why? (probably a Jean-Marc thing i expect,
>>>>> maybe satis needs updating/restarting/flushing?)
>>>>> >>>>>
>>>>> >>>>> jonny
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> _______________________________________________
>>>>> >>>>> TikiWiki-devel mailing list
>>>>> >>>>> [email protected]
>>>>> >>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>>> >>>> _______________________________________________
>>>>> >>>> TikiWiki-devel mailing list
>>>>> >>>> [email protected]
>>>>> >>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>>> >>>> _______________________________________________
>>>>> >>>> TikiWiki-devel mailing list
>>>>> >>>> [email protected]
>>>>> >>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>>> >>>
>>>>> >>> _______________________________________________
>>>>> >>> TikiWiki-devel mailing list
>>>>> >>> [email protected]
>>>>> >>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>>> >>
>>>>> >> _______________________________________________
>>>>> >> TikiWiki-devel mailing list
>>>>> >> [email protected]
>>>>> >> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>>> >
>>>>> >
>>>>> > _______________________________________________
>>>>> > TikiWiki-devel mailing list
>>>>> > [email protected]
>>>>> > https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> TikiWiki-devel mailing list
>>>>> [email protected]
>>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>>>
>>>>
>>
>> _______________________________________________
>> TikiWiki-devel mailing listTikiWiki-devel-5NWGOfrQmneRv+LV9MX5ugtCRVl27V+i0wdF1cv0I5s@public.gmane.org://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>
>>
>>
>> _______________________________________________
>> TikiWiki-devel mailing listTikiWiki-devel-5NWGOfrQmneRv+LV9MX5ugtCRVl27V+i0wdF1cv0I5s@public.gmane.org://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>
>> _______________________________________________
>> TikiWiki-devel mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>
>

_______________________________________________
TikiWiki-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel