CI: disabling recursion on MediaWiki extension dependencies

Hello,

On CI, if your MediaWiki extension has any dependencies, those are 
processed recursively. As of today, I am disabling recursion on some 
extensions when the CI jobs are known to pass without it.

Most extensions have a small set of hard requirements (via 
extension.json 
<https://www.mediawiki.org/wiki/Manual:Extension_registration#Requirements_(dependencies)>). 
As developers added integration tests (which is great) more soft 
dependencies have been added to the tree. Due to recursion, CI can clone 
up to 65 extensions and, since it runs every single tests, this cause 
build runtime to be fairly long.

Timo Tijhof wrote a very nice problem statement on 
https://phabricator.wikimedia.org/T389998. For the last few weeks I have 
worked on the problem to make it possible to disable recursion and went 
to patch several extensions to allow the removal of some dependencies.  
  The recursion is disabled via a recurse boolean parameter which can be 
checked at 
https://gerrit.wikimedia.org/g/integration/config/+/refs/heads/master/zuul/dependencies.yaml CI 
adds to the builds a MW_ZUUL_RECURSE parameter ("true" or "false") to 
expose whether recursion is enabled. You can find it in the builds 
parameters, for example: 
https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php81/lastSuccessfulBuild/parameters/

Although I have verified the jobs still pass, there might be issues with 
tests from dependencies due to their own dependencies no more being 
cloned. In this case, those tests should be skipped. With PHPUnit this 
can be done using:

$this->markTestSkippedIfExtensionNotLoaded( <dependency> );

For example WikimediaMessages 
<https://www.mediawiki.org/wiki/Extension:WikimediaMessages> (which 
holds Wikimedia specific localization messages) depends on the cldr 
<https://www.mediawiki.org/wiki/Extension:CLDR> extension. 
WikimediaMessages is also used as a dependency by multiple extensions 
which do not require cldr nor do any integration testing with cldr. When 
recursion is disabled, those extensions no more have cldr added and that 
would cause WikimediaMessages test to fail. I have thus marked the test 
to be skipped: 
https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaMessages/+/1188422. 
When CI runs for WikimediaMessages, cldr is cloned (since it is a direct 
dependency) and the test does run.

As part of disabling recursion, there might be issues due to a missing 
class/extension. In this case, please file a task in Phabricator 
<https://phabricator.wikimedia.org/maniphest/task/edit/form/1/> against 
project #ci-test-error and as a subtask of T389998 
<https://phabricator.wikimedia.org/T389998> mentioning the Gerrit 
change, the values of MW_ZUUL_RECURSE and EXT_DEPENDENCIES and a copy of 
the error found in the build console.

cheers,

Antoine "hashar" Musso
Wikimedia Release Engineering

_______________________________________________
Wikitech-l mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/