gitlab CI configuration for the Debian website (Re: [Delayed] Summary of the web team BoF at DC18)

Hello Wouter, all

Sorry for taking so long to answer about this.

El 02/10/18 a las 17:31, Wouter Verhelst escribió:
> On Wed, Sep 26, 2018 at 07:36:03PM +0200, Wouter Verhelst wrote:
>> I have been working on improving the setup to split out a full build
>> into multiple jobs -- see
>> https://salsa.debian.org/wouter/webwml/pipelines/19296 for an example.
>> Since the timeout is per job and not per pipeline, that would work
>> around this problem. However, it doesn't seem to be working properly
>> yet; the "make install" step seems to be calling "wml" occasionally,
>> which means things take time, and we don't want that (because it would
>> make things time out). I'm not exactly sure whether that's my fault or
>> the build system's.
> 
> I've now fixed this by making every individual language step do "make
> install" rather than just "make all", and by having the "pages" phase
> rsync each individual language directory into the main "public"
> directory, which should then do the final publication.
> 

Thanks for all the work on this, I just merged it[1] into the main
webwml repo.

[1]: https://salsa.debian.org/webmaster-team/webwml/merge_requests/30

> It works, except that deploying the created output fails with an error
> message that the produced data is too large. I'm sure this is something
> which the salsa administrators could fix if we asked them nicely.
> 

As we have talked in IRC during the tests, the current status is that
the CI jobs build the website but it is not published as "pages",
because of a size limitation (100MB) and (if I understood correctly) we
should review the documentation and try to use deployment environments:
https://docs.gitlab.com/ee/ci/environments.html

Any help is appreciated on this, and I will do my best to reply quicker
with tests and feedback.

Meanwhile:

* we can learn if commits break the builds (I'm not sure who gets
notifications (I get them, but not sure if because of my particular mix
of notification-levels and project-membership), you can probably tweak
your profile settings in Salsa if you don't get them and want to get
them), and we can also have a look at the CI results prior to merge some
merge request.

* People can browse/download the results of the CI builds in the Salsa
web interface, going to the webwml project, CI/CD, Jobs (or Pipelines).
Note that the artifacts corresponding to languages (other than English)
will exist (containing the /english folder) even if that language was
not built (if the language was built, the zip will contain the /english
and the /$language folders).

> At any rate, I plan to replace the code in my merge request with the
> code from this branch[1], it seems like a better option.
> 
> There are still a few things that could be done to improve this,
> however:
> 
> - The webwml Makefile is currently set up to ignore failures and
>   continue. For a continuous integration system, this is suboptimal,
>   although I understand that it makes sense for the main website. I
>   would like to suggest that ignoring failures is not done if some
>   environment variable is set; the CI jobs can then just set that
>   variable and allow things to fail if something is broken
> - The jobs start off with running "apt-get install" for various
>   packages. This works, but it is more efficient to create an image on
>   the docker hub which already has those packages preinstalled. I'll
>   look at doing so.

I will create bugs to track these two proposals.


> - I'm not sure what happens if changes are made to multiple languages in
>   a single git push. There are two possibilities: either gitlab will
>   merge pushed caches from a single stage so that the cached result will
>   have all files from all translations (which is good, that's what we
>   want), or pushing a cache will overwrite all other caches from the
>   same phase (in which case the publish stage will just push the English
>   version of the site together with and whichever translation happened
>   to be finished last). I'll need to experiment and see if that needs
>   fixing. Even if the latter of those two possibilities is the case,
>   however, I still think that seeing an auto build break is a good idea,
>   so regardless of that I still think this is progress.

+1
I'll keep an eye on things in the following days, and report back to the
mailing list.

Cheers