Bug#924172: www.debian.org: differences under english/ between builds in stretch and buster

Hi all,

On Mon, 8 Jun 2020 00:35:26 +0200
Cyril Brulebois <[email protected]> wrote:

> (Looping in wml package maintainer + original author)
> 

Some notes:

1. I am not the original author of wml, just the current upstream maintainer
(though I believe that I have invested quite a lot of work in it).

2. The newest release of wml is 2.28.0 (see
https://github.com/thewml/website-meta-language/releases ) while Debian sid is
stuck at 2.12.x (see https://packages.debian.org/sid/wml ). I'd rather not
support such an old release, so if the new version still exhibits some
regressions, please send a failing testcase to
https://github.com/thewml/website-meta-language/tree/master/src/wml_test and
I'll try to fix it.

For a joke about it, see:
https://www.shlomifish.org/humour/fortunes/tinic.html#hamakor-discs-mozilla-1.1-1

3. I recommend the Debian site to gradually transition away from using wml,
because while wml is fairly powerful and flexible, it is slow, quirky and hard
to learn. It is a hindrance that the debian site takes 3 hours to build on a
modern computer:

https://xkcd.com/303/

Converting some of my sites away from wml has shortened their build times
considerably.

Regards,

	Shlomi

> Hi!
> 
> Laura Arjona Reina <[email protected]> (2020-06-07):
> > Thanks Cyril for your work here.
> > 
> > We're trying to fix all the issues so the migration of www-master to
> > buster can be done as soon as possible.  
> 
> You're welcome, helping towards that goal was my original intent.
> 
> > I'll comment on the issues you found:
> > 
> > 1. Changes related to canonicalization(?) of the URL
> > 
> > The files affected are:
> > 
> > 1.1.- */News/news.*.rdf  (all languages)
> > 
> > All those files are built using /english/News/news.rdf.in which includes
> > this line:
> > 
> > <?xml-stylesheet href="$(HOME)/security/dsa-rdf.css" type="text/css"?>
> > 
> > it seems that the variable $(HOME) is interpreted differently by the
> > make or wml commands in buster.
> > 
> > I have played with the "-D HOME~." line in /english/.wmlrc but it does
> > not solve the issue.
> > 
> > The file is built (in english and all the other languages except
> > Chinese) by lines 36-37 of the /english/News/Makefile:
> > 
> > 	$(WML) $(shell egrep '^-D (CUR_|CHAR)' ../.wmlrc) \
> > 	    $(ENGLISHDIR)/News/news.rdf.in
> > 
> > If I change that to
> > 
> > 	$(WML) $(shell egrep '^-D (CUR_|CHAR)' ../.wmlrc) news.rdf.in
> > 
> > then the English file is built with correct URL to the CSS, but the
> > other languages fail.
> > 
> > I don't know how to solve this, except using an absolute reference to
> > the CSS file in the /english/News/news.rdf.in instead of the $(HOME)
> > variable.  
> 
> I'll skip this for now, to concentrate on the topic I've debugged this
> evening; this might be related though!
> 
> > 1.2. */sitemap.*.html  (all languages)
> > 
> > I guess it's the same problem (that the $(HOME) variable is used and
> > interpreted wrongly with the new make or wml), but this file and how it
> > is built is more cryptic for me so I wouldn't know how to start.
> > 
> > The sitemap would be completely broken until this issue is fixed, if
> > we migrate www-master to buster :/
> > 
> > Help needed!  
> 
> For the record, we're talking about:
>   https://salsa.debian.org/webmaster-team/webwml
> 
> The problem can be triggered with:
>   make -C english sitemap.en.html
> 
> Another file in the same directory can be obtained with:
>   make -C english index.en.html
> 
> 
> OK, so I've spent some time on this. Based on the HOME mention, I tried
> to switch it from `-D HOME~.` to `-D HOME=.` in english/.wmlrc but that
> wasn't sufficient. I had to turn *all* such defines in that file from
> `~` to `=` to get a reasonable sitemap.
> 
> The wml_intro.7 manpage quickly explains the difference between both,
> but that seems to be happening in passes 2 and 3; english/sitemap.wml is
> one of the only files were pass protection is used, but that turned out
> to be a red herring…
> 
> 
> Adding the `-v2` flag to the `$(WML)` calls in english/Makefile made the
> issue apparent: as early as pass 1, we have different parameters being
> sent to wml_p1_ipp between stretch (2.0.12) and buster (2.12.2), among
> which:
> 
>  * stretch:
> 
>         "-DBUGS=Bugs"
>         "-DINTRO=intro"
> 
>  * buster:
>         "-DBUGS=../english/Bugs"
>         "-DINTRO=../english/intro" 
> 
> for our english/sitemap.wml
> 
> BUT
> 
> that's not the case for index.en.html, built from english/index.wml!
> 
> At this point, I suspected the origin of the change in behavior was the
> path to the input file:
> 
>   $(WML) … index.wml is OK
>   $(WML) … ../english/sitemap.wml is not OK
> 
> because the latter triggers the insertion of `../english/` in lots of
> places (defined with `~` instead of `=`) with 2.12.2, which was not the
> case with 2.0.12.
> 
> 
> Dropping the ../english/ prefix for sitemap.wml leads to a reasonable
> diff for the English sitemap (sitemap.en.html):
> 
>     --- stretch.en.html	2020-06-08 00:07:43.916768223 +0200
>     +++ buster.en.html	2020-06-08 00:07:47.248781809 +0200
>     @@ -4,8 +4,8 @@
>        <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
>        <title>    Site map for Debian web pages </title>
>        <link rel="author" href="mailto:[email protected]">
>     -  <meta name="Generator" content="WML 2.0.12 (16-Apr-2008)">
>     -  <meta name="Modified" content="2020-06-07 22:07:43">
>     +  <meta name="Generator" content="WML 2.12.2">
>     +  <meta name="Modified" content="2020-06-08 00:07:46">
>        <meta name="viewport" content="width=device-width">
>        <meta name="mobileoptimized" content="300">
>        <meta name="HandheldFriendly" content="true">
>     @@ -377,7 +377,7 @@
>      <p>
>      Last Modified: Sun, Jun 7 21:49:47 UTC 2020
>      &emsp;
>     -Last Built: Sun, Jun 7 22:07:43 UTC 2020
>     +Last Built: Sun, Jun 7 22:07:46 UTC 2020
>        <br>
>        Copyright &copy; 1997-2020
>       <a href="https://www.spi-inc.org/">SPI</a> and others; See <a
> href="./license" rel="copyright">license terms</a><br>
> 
> Therefore, I suspect we *might* be able to get away with this if we were
> to add a symlink from all language directories, from $LANG/sitemap.wml
> to ../english/sitemap.wml, as a companion change to dropping the
> ../english prefix mentioned above (only tested successfully on English).
> I'll be checking this hypothesis and possibly pushing a branch / opening
> an MR if that works fine.
> 
> I'm sending this mail to reach out to upstream so they can comment on
> whether the change was intentional or an unwanted side effect.
> 
> I might look at the News/RDF thing in the meanwhile.
> 
> 
> > 3. Changes in a log file
> > 
> > Not important (and probably we shouldn't provide the card there, maybe
> > in the debian-flyers repo?)
> > 
> > 4. Changes in ordering of coordinators
> > 5. Changes in ordering under wnpp
> > 6. Changes in order under l10n
> > 8. More ordering changes (architectures, DSAs)
> > 
> > Thanks for reporting, and for the work towards reproducibility.
> > I think these are not blockers for the migration to buster of
> > www-master.debian.org
> > 
> > Maybe we could open a specific bug about these "reproducibility issues"
> > and see if somebody is willing/able to work on it?  
> 
> I'm very fine with such a thing. I just wanted to classify differences
> between stuff that were possibly unimportant and stuff that might need
> fixing.
> 
> 
> Cheers,



-- 

Shlomi Fish       https://www.shlomifish.org/
Understand what Open Source is - https://shlom.in/oss-fs

“God is dead.” — Nietzsche
“Nietzsche is dead.” — God                 (Unknown Source)

Please reply to list if it's a mailing list post - https://shlom.in/reply .