Re: [Tiki-devel] composer.tiki.org slightly borken?

Hi all,

Checked with Jean-Marc and restored the existing backup, composer.tiki.org
should be working again now to install packages.

There are a few things pending:
* is a backup from last monday
* satis updates are disabled
* I'll run a few extra tests and will attempt a cleanup using the same
script we used in the past for composer.t.o to shrink the list of files
* After all those, satis updates will be turned on again.

On Thu, Jun 30, 2022 at 1:56 AM Jean-Marc Libs <[email protected]>
wrote:

> After a while, I imagine the satis update script was run and there are
> more packages
> # du -sh /composer/composer.tiki.org/
> 7.7G    /composer/composer.tiki.org/
>
> The new tree command gives:
> https://composer.tiki.org/tree2.txt
>
> Cheers,
> J-M
>
> On Thu, Jun 30, 2022 at 2:27 AM Jean-Marc Libs <[email protected]>
> wrote:
>
>> Thanks a lot, Victor!
>>
>> I needed more time than expected for doing the last remote backup (the
>> remote backup server was running into quota issues) but it's done!
>>
>> I ran the script and it seems to have done *a lot* of cleanup. *Maybe
>> too much?*
>>
>> Now the site takes only:
>> root@hetzner01 /var/local/composer.tiki.org/www # du -sh /composer/
>> composer.tiki.org/
>> 3.1G    /composer/composer.tiki.org/
>>
>> Here is the result of the tree command :
>> https://composer.tiki.org/tree.txt
>>
>> I ran composer upgrade on a Tiki21 site and I got:
>> web# sh setup.sh
>> …
>> Your choice [c]?
>>
>> Local PHP version 72 >= to required PHP version 72 - good
>> Warning: You forced the install of 1.10.26 via --1, but 2.3.7 is the
>> latest stable version. Updating to it via composer self-update --stable is
>> recommended.
>> You are already using composer version 1.10.26 (1.x channel).
>> Do not run Composer as root/super user! See https://getcomposer.org/root
>> for details
>> You are using Composer 1 which is deprecated. You should upgrade to
>> Composer 2, see
>> https://blog.packagist.com/deprecating-composer-1-support/
>> Gathering patches from patch file.
>> Removing package adodb/adodb-php so that it can be re-installed and
>> re-patched.
>>  - Removing adodb/adodb-php (v5.21.3)
>> > Tiki\Composer\RestoreVendors::restore
>> > Tiki\Composer\CleanVendorAfterVendorBundledMigration::cleanLinks
>> Loading composer repositories with package information
>> Installing dependencies from lock file
>> Warning: The lock file is not up to date with the latest changes in
>> composer.json. You may be getting outdated dependencies. It is recommended
>> that you run `composer update` or `composer update <package name>`.
>> Package operations: 2 installs, 0 updates, 0 removals
>> Gathering patches from patch file.
>> Gathering patches for dependencies. This might take a minute.
>>  - Installing adodb/adodb-php (v5.21.3): Downloading (failed)    Failed
>> to download adodb/adodb-php from dist: The "
>> https://composer.tiki.org/dist/adodb/adodb-php/adodb-adodb-php-c5415722049f36c446a4034d15f1d17943f11458-zip-e71b61.zip"
>> file could not be downloaded (HTTP
>> /1.1 404 Not Found)
>>    Now trying to download from source
>>  - Installing adodb/adodb-php (v5.21.3): Cloning c541572204 from cache
>>  - Applying patches for adodb/adodb-php
>>    ../installer/composer-patches/adodb-php__remove_references_to_adodb_extension.patch
>> (Remove references to ADOdb Extension)
>>   Could not apply patch! Skipping. The error was: Cannot apply patch
>> ../installer/composer-patches/adodb-php__remove_references_to_adodb_extension.patch
>>
>>    ../installer/composer-patches/adodb-php__remove-the-check-if-the-adodb-php_ext-is-enabled.patch
>> (Remove the check for ADOdb Extension)
>>
>>  - Installing components/bootstrap (3.4.1): Downloading (failed)
>>    Failed to download components/bootstrap from dist: The "
>> https://composer.tiki.org/dist/components/bootstrap/components-bootstrap-427f4d414fc6ae8385aee02b7efc8e391eb141ad-zip-393061.zip"
>> file could not b
>> e downloaded (HTTP/1.1 404 Not Found)
>>    Now trying to download from source
>>  - Installing components/bootstrap (3.4.1): Cloning 427f4d414f from cache
>> Package container-interop/container-interop is abandoned, you should
>> avoid using it. Use psr/container instead.
>> Package evert/sitemap-php is abandoned, you should avoid using it. No
>> replacement was suggested.
>> Package league/commonmark-extras is abandoned, you should avoid using it.
>> Use league/commonmark instead.
>> Package robloach/component-installer is abandoned, you should avoid using
>> it. Use oomphinc/composer-installers-extender instead.
>> Package webuni/commonmark-attributes-extension is abandoned, you should
>> avoid using it. Use league/commonmark instead.
>> Package zendframework/zend-version is abandoned, you should avoid using
>> it. Use ocramius/package-versions instead.
>> Package zendframework/zendservice-akismet is abandoned, you should avoid
>> using it. No replacement was suggested.
>> Generating autoload files
>> > ComponentInstaller\Installer::postAutoloadDump
>> Compiling component files
>> > Tiki\Composer\ThemeInstaller::setup
>> > Tiki\Composer\BootstrapCompiler::build
>> > Tiki\Composer\CleanVendors::clean
>>
>> > Tiki\Composer\CleanVendorAfterVendorBundledMigration::clean
>> > Tiki\Composer\PatchCypht::setup
>>
>>
>> Tiki setup.sh - your options
>>
>> ============================
>> …
>>
>> Should I restore the previous packages and wait for a new version of the
>> script?
>>
>> Cheers,
>> Jyhem
>>
>>
>>
>> On Thu, Jun 23, 2022 at 3:58 PM Victor Emanouilov via TikiWiki-devel <
>> [email protected]> wrote:
>>
>>> Hi Jean-Marc,
>>>
>>> I added a Tiki devtools script for this:
>>>
>>>
>>> https://gitlab.com/tikiwiki/tiki/-/commit/4ef8dc83e934ccee970a722e5f14237cbe8f6a73
>>>
>>> Please configure the first variable to the actual local filesystem path
>>> of composer.tiki.org webroot and run it locally on that server. You can
>>> use any version of php 7. Running it without any arguments will list the
>>> files to be deleted. Running it with --cleanup argument will actually
>>> perform the delete operation.
>>>
>>> Regards,
>>> Victor
>>> On 6/22/22 5:57 PM, Victor Emanouilov via TikiWiki-devel wrote:
>>>
>>> JM,
>>>
>>>
>>> As we talked on the roundtable meeting, this is the file that lists all
>>> the package versions composer.tiki.org supports:
>>>
>>> https://composer.tiki.org/packages.json
>>>
>>> Currently, it includes this one (it's huge!):
>>>
>>>
>>> https://composer.tiki.org/include/all$838361a2a0955d73dfb3e0370a109616b7e37fd5.json
>>>
>>> I will send you a script that pinpoints all the dist files downloaded on
>>> local disk that are not references in the packages.json file and provide an
>>> option to remove them. This should clean up about half of the used space if
>>> not even more.
>>>
>>> Regards,
>>> Victor
>>> On 5/16/22 2:40 PM, Victor Emanouilov via TikiWiki-devel wrote:
>>>
>>> Alright, let's discuss it then!
>>> On 5/13/22 10:52 PM, Jean-Marc Libs wrote:
>>>
>>> Hi Victor,
>>>
>>> I can't think of an issue with removing dev packages from satis config.
>>>
>>> I have a hard time understanding the consequences of removing old
>>> versions. Does it mean updating composer packages using setup.sh will fail
>>> unless the Tiki source is updated to the latest version first?
>>>
>>> I'm concerned about the consequences for people who update old
>>> production Tikis. I am not familiar enough with composer to make an opinion
>>> :-(
>>>
>>> Maybe we can discuss this during next Roundtable? I put the topic here:
>>> https://tiki.org/tiki-index.php?page=Roundtable-Meeting-2022-05
>>>
>>> Cheers,
>>> J-M
>>>
>>>
>>> On Tue, May 3, 2022 at 11:08 AM Victor Emanouilov <[email protected]>
>>> wrote:
>>>
>>>> Hi J-M,
>>>>
>>>> Yes, I agree we can remove dev packages from satis config and save
>>>> space. The other thing I see is that some packages have way too many
>>>> versions as files on the disk compared to the versions listed on
>>>> composer.tiki.org - I think these are cached assets that probably
>>>> disappeared later and are no longer supported. Do you think that we can do
>>>> the following?
>>>>
>>>> 1. clean up satis config from dev packages
>>>> 2. delete the whole directory holding packages on composer.tiki.org
>>>> 3. deploy and run satis again to download only the ones that are
>>>> currently in use
>>>>
>>>> Regards,
>>>> Victor
>>>> On 4/11/22 1:32 AM, Jean-Marc Libs wrote:
>>>>
>>>> Hi Victor, here is the result of some more drilling down:
>>>> http://composer.tiki.org/packages-phpstan.txt
>>>> http://composer.tiki.org/packages-phpunit.txt
>>>> http://composer.tiki.org/packages-plotly.txt
>>>> http://composer.tiki.org/packages-sebastian.txt
>>>> http://composer.tiki.org/packages-vimeo.txt
>>>> http://composer.tiki.org/packages-aws.txt
>>>> http://composer.tiki.org/packages-google.txt
>>>> http://composer.tiki.org/packages-mpdf.txt
>>>>
>>>>
>>>> I don't usually run unit-tests but I can't see why we keep a cache of
>>>> old versions of dev tools.
>>>>
>>>> Cheers,
>>>> J-M
>>>>
>>>> On Fri, Apr 8, 2022 at 11:28 AM Victor Emanouilov <[email protected]>
>>>> wrote:
>>>>
>>>>> JM, sorry, one more request. These are several extremely large,
>>>>> packages themselves are about 20-30mb but these take 10+GB on disk, can you
>>>>> check them as well (drill down)?
>>>>>
>>>>> 100G	dist/phpstan
>>>>> 66G	dist/phpunit
>>>>> 30G	dist/plotly
>>>>> 29G	dist/sebastian
>>>>> 26G	dist/vimeo
>>>>> 25G	dist/aws
>>>>> 15G	dist/google
>>>>> 13G	dist/mpdf
>>>>>
>>>>> Thanks,
>>>>> Victor
>>>>> On 4/7/22 7:10 PM, Jean-Marc Libs wrote:
>>>>>
>>>>> Hi Victor,
>>>>>
>>>>> It's a huge file so I left it online:
>>>>> That's a simple du -sh :
>>>>> http://composer.tiki.org/packages_du-sh.txt
>>>>> That one is sorted by size:
>>>>> http://composer.tiki.org/packages_du-sh_sorted.txt
>>>>>
>>>>> No problem,
>>>>> J-M
>>>>>
>>>>> On Wed, Apr 6, 2022 at 2:05 PM Victor Emanouilov via TikiWiki-devel <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> JM, can you `du -sh` on the satis storage dirs and send us a list of
>>>>>> the
>>>>>> directories that take the most amount of space? We might limit
>>>>>> semantic
>>>>>> versions of those packages to what Tiki really needs...
>>>>>>
>>>>>> Thanks,
>>>>>> Victor
>>>>>>
>>>>>> On 4/5/22 6:42 PM, Jonny Bradley via TikiWiki-devel wrote:
>>>>>> > Hi Victor
>>>>>> >
>>>>>> > Yes, that was going to be plan B, if just listing what's needed
>>>>>> currently in all the branches didn't result in freeing up enough space we
>>>>>> could keep just the supported ones (somehow, we don't actually have that
>>>>>> info in a format we can access easily afaik).
>>>>>> >
>>>>>> > The main thing needed is to change from using the actual used
>>>>>> version to working out all the possible required ones, i expect we can use
>>>>>> whatever composer uses to calculate that but i probably need to know more
>>>>>> about how satis working internally...
>>>>>> >
>>>>>> > And have more free time! ;)
>>>>>> >
>>>>>> > jb
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >> On 5 Apr 2022, at 11:44, Victor Emanouilov via TikiWiki-devel <
>>>>>> [email protected]> wrote:
>>>>>> >>
>>>>>> >> Hi Jonny, Jean-Marc,
>>>>>> >>
>>>>>> >> Thanks for fixing this! It has been a lingering problem for quite
>>>>>> some time. We definitely don't need all the packages that are downloaded on
>>>>>> composer.tiki.org but Jonny, that's a static repository generator
>>>>>> which means it can't fall back to download a missing package. I am thinking
>>>>>> why don't we reduce to the number of official Tiki releases we support - we
>>>>>> get all semantic versions of packages and generate only those in satis?
>>>>>> Symfony packages, for example, will be reduced to v3 or v4 only and not
>>>>>> everything from 2 to 6 (https://composer.tiki.org/#symfony/config)
>>>>>> for example. Each official Tiki release can have Jonny's improved script
>>>>>> run and generate the correct satis config. I believe we should decrease
>>>>>> storage requirements several times this way.
>>>>>> >>
>>>>>> >> Regards,
>>>>>> >> Victor
>>>>>> >>
>>>>>> >> On 4/5/22 12:15 PM, Jonny Bradley via TikiWiki-devel wrote:
>>>>>> >>> Hi Jean-Marc and all
>>>>>> >>>
>>>>>> >>> Had a spare hour last night so started on this:
>>>>>> >>>
>>>>>> >>>
>>>>>> https://gitlab.com/tikiwiki/tiki/-/blob/master/doc/devtools/composer_packages_in_use.php
>>>>>> >>>
>>>>>> >>> which at the moment lists all the versions currently in use by
>>>>>> all the branches in tiki on gitlab.
>>>>>> >>>
>>>>>> >>> However, it then occurred to me that many of these branches will
>>>>>> have outstanding automatic updates pending, e.g. if we ran `composer
>>>>>> update` on tiki 22.x there almost certainly would be updates to newer
>>>>>> packages, so maybe this isn't the right strategy after all...
>>>>>> >>>
>>>>>> >>> I guess we need to take into account the semantic versioning in
>>>>>> all the composer.json files rather than the actual ones in use in the lock
>>>>>> files, but at least it's a start?
>>>>>> >>>
>>>>>> >>> Would it be possible to remove some packages form satis on
>>>>>> composer.t.o, but if something requests one of them it re-downloads it?
>>>>>> Then we could be more confident in deleting stuff...
>>>>>> >>>
>>>>>> >>> Or just put a larger drive on it‽ :p
>>>>>> >>>
>>>>>> >>> Thanks for fixing it again!
>>>>>> >>>
>>>>>> >>> jonny
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>> On 5 Apr 2022, at 01:09, Jean-Marc Libs <[email protected]>
>>>>>> wrote:
>>>>>> >>>>
>>>>>> >>>> Hi folks,
>>>>>> >>>>
>>>>>> >>>> root partition was full on composer.tiki.org 😞 I managed to
>>>>>> find 26G and it works again but the volume is staggering (409G, 103G once
>>>>>> compressed for backups) and goes up all the time.
>>>>>> >>>>
>>>>>> >>>> Do we need so many third-party libs?
>>>>>> >>>>
>>>>>> >>>> Cheers,
>>>>>> >>>> J-M
>>>>>> >>>>
>>>>>> >>>> On Mon, Apr 4, 2022 at 5:17 PM Victor Emanouilov via
>>>>>> TikiWiki-devel <[email protected]> wrote:
>>>>>> >>>> +1 same for me ... that URL returns 304 not modified - maybe some
>>>>>> >>>> apache/nginx config is needed? Composer expects a valid json
>>>>>> file there...
>>>>>> >>>>
>>>>>> >>>> Regards,
>>>>>> >>>> Victor
>>>>>> >>>>
>>>>>> >>>> On 4/4/22 3:03 PM, Jonny Bradley via TikiWiki-devel wrote:
>>>>>> >>>>> Hi all
>>>>>> >>>>>
>>>>>> >>>>> I'm trying to update composer for the bootstrap5 branch
>>>>>> following some rebasing and it seem composer is failing now, and it's the
>>>>>> same in trunk/master/main now.
>>>>>> >>>>>
>>>>>> >>>>> When i do:
>>>>>> >>>>>
>>>>>> >>>>> $ php74 temp/composer.phar update  --working-dir vendor_bundled
>>>>>> --prefer-dist --optimize-autoloader
>>>>>> >>>>>
>>>>>> >>>>> I get:
>>>>>> >>>>>
>>>>>> >>>>>> In JsonFile.php line 345:
>>>>>> >>>>>>
>>>>>>                                                               "
>>>>>> https://composer.tiki.org/include/all%2419ac295e729a70b8272116cec8bdb0716f3dd8d2.json"
>>>>>> does not contain valid JSON
>>>>>> >>>>>>     Parse error on line 1:
>>>>>> >>>>>>
>>>>>> >>>>>>     ^
>>>>>> >>>>>>     Expected one of: 'STRING', 'NUMBER', 'NULL', 'TRUE',
>>>>>> 'FALSE', '{', '['
>>>>>> >>>>>>
>>>>>> >>>>>
>>>>>> https://composer.tiki.org/include/all%2419ac295e729a70b8272116cec8bdb0716f3dd8d2.json
>>>>>> is an empty file, which presumably is the problem.
>>>>>> >>>>>
>>>>>> >>>>> Does anyone know why? (probably a Jean-Marc thing i expect,
>>>>>> maybe satis needs updating/restarting/flushing?)
>>>>>> >>>>>
>>>>>> >>>>> jonny
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>> _______________________________________________
>>>>>> >>>>> TikiWiki-devel mailing list
>>>>>> >>>>> [email protected]
>>>>>> >>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>>>> >>>> _______________________________________________
>>>>>> >>>> TikiWiki-devel mailing list
>>>>>> >>>> [email protected]
>>>>>> >>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>>>> >>>> _______________________________________________
>>>>>> >>>> TikiWiki-devel mailing list
>>>>>> >>>> [email protected]
>>>>>> >>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>>>> >>>
>>>>>> >>> _______________________________________________
>>>>>> >>> TikiWiki-devel mailing list
>>>>>> >>> [email protected]
>>>>>> >>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>>>> >>
>>>>>> >> _______________________________________________
>>>>>> >> TikiWiki-devel mailing list
>>>>>> >> [email protected]
>>>>>> >> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>>>> >
>>>>>> >
>>>>>> > _______________________________________________
>>>>>> > TikiWiki-devel mailing list
>>>>>> > [email protected]
>>>>>> > https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> TikiWiki-devel mailing list
>>>>>> [email protected]
>>>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>>>>
>>>>>
>>>
>>> _______________________________________________
>>> TikiWiki-devel mailing listTikiWiki-devel-5NWGOfrQmneRv+LV9MX5ugtCRVl27V+i0wdF1cv0I5s@public.gmane.org://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>
>>>
>>>
>>> _______________________________________________
>>> TikiWiki-devel mailing listTikiWiki-devel-5NWGOfrQmneRv+LV9MX5ugtCRVl27V+i0wdF1cv0I5s@public.gmane.org://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>
>>> _______________________________________________
>>> TikiWiki-devel mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>
>> _______________________________________________
> TikiWiki-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>

_______________________________________________
TikiWiki-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel