Re: [Tiki-devel] composer.tiki.org slightly borken?

Thanks a lot, Victor!

I needed more time than expected for doing the last remote backup (the
remote backup server was running into quota issues) but it's done!

I ran the script and it seems to have done *a lot* of cleanup. *Maybe too
much?*

Now the site takes only:
root@hetzner01 /var/local/composer.tiki.org/www # du -sh /composer/
composer.tiki.org/
3.1G    /composer/composer.tiki.org/

Here is the result of the tree command :
https://composer.tiki.org/tree.txt

I ran composer upgrade on a Tiki21 site and I got:
web# sh setup.sh
…
Your choice [c]?

Local PHP version 72 >= to required PHP version 72 - good
Warning: You forced the install of 1.10.26 via --1, but 2.3.7 is the latest
stable version. Updating to it via composer self-update --stable is
recommended.
You are already using composer version 1.10.26 (1.x channel).
Do not run Composer as root/super user! See https://getcomposer.org/root
for details
You are using Composer 1 which is deprecated. You should upgrade to
Composer 2, see https://blog.packagist.com/deprecating-composer-1-support/
Gathering patches from patch file.
Removing package adodb/adodb-php so that it can be re-installed and
re-patched.
 - Removing adodb/adodb-php (v5.21.3)
> Tiki\Composer\RestoreVendors::restore
> Tiki\Composer\CleanVendorAfterVendorBundledMigration::cleanLinks
Loading composer repositories with package information
Installing dependencies from lock file
Warning: The lock file is not up to date with the latest changes in
composer.json. You may be getting outdated dependencies. It is recommended
that you run `composer update` or `composer update <package name>`.
Package operations: 2 installs, 0 updates, 0 removals
Gathering patches from patch file.
Gathering patches for dependencies. This might take a minute.
 - Installing adodb/adodb-php (v5.21.3): Downloading (failed)    Failed to
download adodb/adodb-php from dist: The "
https://composer.tiki.org/dist/adodb/adodb-php/adodb-adodb-php-c5415722049f36c446a4034d15f1d17943f11458-zip-e71b61.zip"
file could not be downloaded (HTTP
/1.1 404 Not Found)
   Now trying to download from source
 - Installing adodb/adodb-php (v5.21.3): Cloning c541572204 from cache
 - Applying patches for adodb/adodb-php
   ../installer/composer-patches/adodb-php__remove_references_to_adodb_extension.patch
(Remove references to ADOdb Extension)
  Could not apply patch! Skipping. The error was: Cannot apply patch
../installer/composer-patches/adodb-php__remove_references_to_adodb_extension.patch

   ../installer/composer-patches/adodb-php__remove-the-check-if-the-adodb-php_ext-is-enabled.patch
(Remove the check for ADOdb Extension)

 - Installing components/bootstrap (3.4.1): Downloading (failed)    Failed
to download components/bootstrap from dist: The "
https://composer.tiki.org/dist/components/bootstrap/components-bootstrap-427f4d414fc6ae8385aee02b7efc8e391eb141ad-zip-393061.zip"
file could not b
e downloaded (HTTP/1.1 404 Not Found)
   Now trying to download from source
 - Installing components/bootstrap (3.4.1): Cloning 427f4d414f from cache
Package container-interop/container-interop is abandoned, you should avoid
using it. Use psr/container instead.
Package evert/sitemap-php is abandoned, you should avoid using it. No
replacement was suggested.
Package league/commonmark-extras is abandoned, you should avoid using it.
Use league/commonmark instead.
Package robloach/component-installer is abandoned, you should avoid using
it. Use oomphinc/composer-installers-extender instead.
Package webuni/commonmark-attributes-extension is abandoned, you should
avoid using it. Use league/commonmark instead.
Package zendframework/zend-version is abandoned, you should avoid using it.
Use ocramius/package-versions instead.
Package zendframework/zendservice-akismet is abandoned, you should avoid
using it. No replacement was suggested.
Generating autoload files
> ComponentInstaller\Installer::postAutoloadDump
Compiling component files
> Tiki\Composer\ThemeInstaller::setup
> Tiki\Composer\BootstrapCompiler::build
> Tiki\Composer\CleanVendors::clean

> Tiki\Composer\CleanVendorAfterVendorBundledMigration::clean
> Tiki\Composer\PatchCypht::setup

Tiki setup.sh - your options

============================
…

Should I restore the previous packages and wait for a new version of the
script?

Cheers,
Jyhem



On Thu, Jun 23, 2022 at 3:58 PM Victor Emanouilov via TikiWiki-devel <
[email protected]> wrote:

> Hi Jean-Marc,
>
> I added a Tiki devtools script for this:
>
>
> https://gitlab.com/tikiwiki/tiki/-/commit/4ef8dc83e934ccee970a722e5f14237cbe8f6a73
>
> Please configure the first variable to the actual local filesystem path of
> composer.tiki.org webroot and run it locally on that server. You can use
> any version of php 7. Running it without any arguments will list the files
> to be deleted. Running it with --cleanup argument will actually perform the
> delete operation.
>
> Regards,
> Victor
> On 6/22/22 5:57 PM, Victor Emanouilov via TikiWiki-devel wrote:
>
> JM,
>
>
> As we talked on the roundtable meeting, this is the file that lists all
> the package versions composer.tiki.org supports:
>
> https://composer.tiki.org/packages.json
>
> Currently, it includes this one (it's huge!):
>
>
> https://composer.tiki.org/include/all$838361a2a0955d73dfb3e0370a109616b7e37fd5.json
>
> I will send you a script that pinpoints all the dist files downloaded on
> local disk that are not references in the packages.json file and provide an
> option to remove them. This should clean up about half of the used space if
> not even more.
>
> Regards,
> Victor
> On 5/16/22 2:40 PM, Victor Emanouilov via TikiWiki-devel wrote:
>
> Alright, let's discuss it then!
> On 5/13/22 10:52 PM, Jean-Marc Libs wrote:
>
> Hi Victor,
>
> I can't think of an issue with removing dev packages from satis config.
>
> I have a hard time understanding the consequences of removing old
> versions. Does it mean updating composer packages using setup.sh will fail
> unless the Tiki source is updated to the latest version first?
>
> I'm concerned about the consequences for people who update old production
> Tikis. I am not familiar enough with composer to make an opinion :-(
>
> Maybe we can discuss this during next Roundtable? I put the topic here:
> https://tiki.org/tiki-index.php?page=Roundtable-Meeting-2022-05
>
> Cheers,
> J-M
>
>
> On Tue, May 3, 2022 at 11:08 AM Victor Emanouilov <[email protected]> wrote:
>
>> Hi J-M,
>>
>> Yes, I agree we can remove dev packages from satis config and save space.
>> The other thing I see is that some packages have way too many versions as
>> files on the disk compared to the versions listed on composer.tiki.org -
>> I think these are cached assets that probably disappeared later and are no
>> longer supported. Do you think that we can do the following?
>>
>> 1. clean up satis config from dev packages
>> 2. delete the whole directory holding packages on composer.tiki.org
>> 3. deploy and run satis again to download only the ones that are
>> currently in use
>>
>> Regards,
>> Victor
>> On 4/11/22 1:32 AM, Jean-Marc Libs wrote:
>>
>> Hi Victor, here is the result of some more drilling down:
>> http://composer.tiki.org/packages-phpstan.txt
>> http://composer.tiki.org/packages-phpunit.txt
>> http://composer.tiki.org/packages-plotly.txt
>> http://composer.tiki.org/packages-sebastian.txt
>> http://composer.tiki.org/packages-vimeo.txt
>> http://composer.tiki.org/packages-aws.txt
>> http://composer.tiki.org/packages-google.txt
>> http://composer.tiki.org/packages-mpdf.txt
>>
>>
>> I don't usually run unit-tests but I can't see why we keep a cache of old
>> versions of dev tools.
>>
>> Cheers,
>> J-M
>>
>> On Fri, Apr 8, 2022 at 11:28 AM Victor Emanouilov <[email protected]>
>> wrote:
>>
>>> JM, sorry, one more request. These are several extremely large, packages
>>> themselves are about 20-30mb but these take 10+GB on disk, can you check
>>> them as well (drill down)?
>>>
>>> 100G	dist/phpstan
>>> 66G	dist/phpunit
>>> 30G	dist/plotly
>>> 29G	dist/sebastian
>>> 26G	dist/vimeo
>>> 25G	dist/aws
>>> 15G	dist/google
>>> 13G	dist/mpdf
>>>
>>> Thanks,
>>> Victor
>>> On 4/7/22 7:10 PM, Jean-Marc Libs wrote:
>>>
>>> Hi Victor,
>>>
>>> It's a huge file so I left it online:
>>> That's a simple du -sh :
>>> http://composer.tiki.org/packages_du-sh.txt
>>> That one is sorted by size:
>>> http://composer.tiki.org/packages_du-sh_sorted.txt
>>>
>>> No problem,
>>> J-M
>>>
>>> On Wed, Apr 6, 2022 at 2:05 PM Victor Emanouilov via TikiWiki-devel <
>>> [email protected]> wrote:
>>>
>>>> JM, can you `du -sh` on the satis storage dirs and send us a list of
>>>> the
>>>> directories that take the most amount of space? We might limit semantic
>>>> versions of those packages to what Tiki really needs...
>>>>
>>>> Thanks,
>>>> Victor
>>>>
>>>> On 4/5/22 6:42 PM, Jonny Bradley via TikiWiki-devel wrote:
>>>> > Hi Victor
>>>> >
>>>> > Yes, that was going to be plan B, if just listing what's needed
>>>> currently in all the branches didn't result in freeing up enough space we
>>>> could keep just the supported ones (somehow, we don't actually have that
>>>> info in a format we can access easily afaik).
>>>> >
>>>> > The main thing needed is to change from using the actual used version
>>>> to working out all the possible required ones, i expect we can use whatever
>>>> composer uses to calculate that but i probably need to know more about how
>>>> satis working internally...
>>>> >
>>>> > And have more free time! ;)
>>>> >
>>>> > jb
>>>> >
>>>> >
>>>> >
>>>> >> On 5 Apr 2022, at 11:44, Victor Emanouilov via TikiWiki-devel <
>>>> [email protected]> wrote:
>>>> >>
>>>> >> Hi Jonny, Jean-Marc,
>>>> >>
>>>> >> Thanks for fixing this! It has been a lingering problem for quite
>>>> some time. We definitely don't need all the packages that are downloaded on
>>>> composer.tiki.org but Jonny, that's a static repository generator
>>>> which means it can't fall back to download a missing package. I am thinking
>>>> why don't we reduce to the number of official Tiki releases we support - we
>>>> get all semantic versions of packages and generate only those in satis?
>>>> Symfony packages, for example, will be reduced to v3 or v4 only and not
>>>> everything from 2 to 6 (https://composer.tiki.org/#symfony/config) for
>>>> example. Each official Tiki release can have Jonny's improved script run
>>>> and generate the correct satis config. I believe we should decrease storage
>>>> requirements several times this way.
>>>> >>
>>>> >> Regards,
>>>> >> Victor
>>>> >>
>>>> >> On 4/5/22 12:15 PM, Jonny Bradley via TikiWiki-devel wrote:
>>>> >>> Hi Jean-Marc and all
>>>> >>>
>>>> >>> Had a spare hour last night so started on this:
>>>> >>>
>>>> >>>
>>>> https://gitlab.com/tikiwiki/tiki/-/blob/master/doc/devtools/composer_packages_in_use.php
>>>> >>>
>>>> >>> which at the moment lists all the versions currently in use by all
>>>> the branches in tiki on gitlab.
>>>> >>>
>>>> >>> However, it then occurred to me that many of these branches will
>>>> have outstanding automatic updates pending, e.g. if we ran `composer
>>>> update` on tiki 22.x there almost certainly would be updates to newer
>>>> packages, so maybe this isn't the right strategy after all...
>>>> >>>
>>>> >>> I guess we need to take into account the semantic versioning in all
>>>> the composer.json files rather than the actual ones in use in the lock
>>>> files, but at least it's a start?
>>>> >>>
>>>> >>> Would it be possible to remove some packages form satis on
>>>> composer.t.o, but if something requests one of them it re-downloads it?
>>>> Then we could be more confident in deleting stuff...
>>>> >>>
>>>> >>> Or just put a larger drive on it‽ :p
>>>> >>>
>>>> >>> Thanks for fixing it again!
>>>> >>>
>>>> >>> jonny
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>> On 5 Apr 2022, at 01:09, Jean-Marc Libs <[email protected]>
>>>> wrote:
>>>> >>>>
>>>> >>>> Hi folks,
>>>> >>>>
>>>> >>>> root partition was full on composer.tiki.org 😞 I managed to find
>>>> 26G and it works again but the volume is staggering (409G, 103G once
>>>> compressed for backups) and goes up all the time.
>>>> >>>>
>>>> >>>> Do we need so many third-party libs?
>>>> >>>>
>>>> >>>> Cheers,
>>>> >>>> J-M
>>>> >>>>
>>>> >>>> On Mon, Apr 4, 2022 at 5:17 PM Victor Emanouilov via
>>>> TikiWiki-devel <[email protected]> wrote:
>>>> >>>> +1 same for me ... that URL returns 304 not modified - maybe some
>>>> >>>> apache/nginx config is needed? Composer expects a valid json file
>>>> there...
>>>> >>>>
>>>> >>>> Regards,
>>>> >>>> Victor
>>>> >>>>
>>>> >>>> On 4/4/22 3:03 PM, Jonny Bradley via TikiWiki-devel wrote:
>>>> >>>>> Hi all
>>>> >>>>>
>>>> >>>>> I'm trying to update composer for the bootstrap5 branch following
>>>> some rebasing and it seem composer is failing now, and it's the same in
>>>> trunk/master/main now.
>>>> >>>>>
>>>> >>>>> When i do:
>>>> >>>>>
>>>> >>>>> $ php74 temp/composer.phar update  --working-dir vendor_bundled
>>>> --prefer-dist --optimize-autoloader
>>>> >>>>>
>>>> >>>>> I get:
>>>> >>>>>
>>>> >>>>>> In JsonFile.php line 345:
>>>> >>>>>>
>>>>                                                             "
>>>> https://composer.tiki.org/include/all%2419ac295e729a70b8272116cec8bdb0716f3dd8d2.json"
>>>> does not contain valid JSON
>>>> >>>>>>     Parse error on line 1:
>>>> >>>>>>
>>>> >>>>>>     ^
>>>> >>>>>>     Expected one of: 'STRING', 'NUMBER', 'NULL', 'TRUE',
>>>> 'FALSE', '{', '['
>>>> >>>>>>
>>>> >>>>>
>>>> https://composer.tiki.org/include/all%2419ac295e729a70b8272116cec8bdb0716f3dd8d2.json
>>>> is an empty file, which presumably is the problem.
>>>> >>>>>
>>>> >>>>> Does anyone know why? (probably a Jean-Marc thing i expect, maybe
>>>> satis needs updating/restarting/flushing?)
>>>> >>>>>
>>>> >>>>> jonny
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> _______________________________________________
>>>> >>>>> TikiWiki-devel mailing list
>>>> >>>>> [email protected]
>>>> >>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>> >>>> _______________________________________________
>>>> >>>> TikiWiki-devel mailing list
>>>> >>>> [email protected]
>>>> >>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>> >>>> _______________________________________________
>>>> >>>> TikiWiki-devel mailing list
>>>> >>>> [email protected]
>>>> >>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>> >>>
>>>> >>> _______________________________________________
>>>> >>> TikiWiki-devel mailing list
>>>> >>> [email protected]
>>>> >>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>> >>
>>>> >> _______________________________________________
>>>> >> TikiWiki-devel mailing list
>>>> >> [email protected]
>>>> >> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > TikiWiki-devel mailing list
>>>> > [email protected]
>>>> > https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>>
>>>>
>>>> _______________________________________________
>>>> TikiWiki-devel mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>>>
>>>
>
> _______________________________________________
> TikiWiki-devel mailing listTikiWiki-devel-5NWGOfrQmneRv+LV9MX5ugtCRVl27V+i0wdF1cv0I5s@public.gmane.org://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>
>
>
> _______________________________________________
> TikiWiki-devel mailing listTikiWiki-devel-5NWGOfrQmneRv+LV9MX5ugtCRVl27V+i0wdF1cv0I5s@public.gmane.org://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>
> _______________________________________________
> TikiWiki-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>

_______________________________________________
TikiWiki-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel