Re: build of installation-guide on web server broken [Was: Re: debian-doc files on the Debian web server]

Hi,

Osamu Aoki <[email protected]> wrote (Wed, 02 Feb 2022 00:29:40 +0900):
> Hi,
> 
> I am going to sleep :-)
> 
> I will look into this on my week end.
> 
> I suppose path needs some adjustment.  I don't know where should it be.
> 
> Since I have no direct access to the build environment, I need to run cron job to
> debug...

Ok. 
I set up a test environment here locally and debugged the situation.
Found the error:


The log had:
starting the lessoften cron job at Mon Jan 31 14:49:01 UTC 2022
Already up to date.
run-parts: executing /srv/www.debian.org/cron/lessoften-parts/1ftpfiles
I: Download http://ftp.de.debian.org/debian/dists/sid/main/source/Sources.xz
I: Download: pool/main/i/installation-guide/installation-guide_20220129.dsc
I: Download: pool/main/i/installation-guide/installation-guide_20220129.tar.xz
I: Download http://ftp.de.debian.org/debian/dists/stable/main/source/Sources.xz
I: Download: pool/main/i/installation-guide/installation-guide_20210730.dsc
I: Download: pool/main/i/installation-guide/installation-guide_20210730.tar.xz
I: Download http://ftp.de.debian.org/debian/dists/oldstable/main/source/Sources.xz
I: Download: pool/main/i/installation-guide/installation-guide_20190622.dsc
I: Download: pool/main/i/installation-guide/installation-guide_20190622.tar.xz


So, it downloads the *.dsc and *.tar.xz files everytime.
That's because the check for the existence of the files used the wrong file path
(it searched for 
"pool/main/i/installation-guide/installation-guide_20190622.dsc"
and not for
"installation-guide_20190622.dsc"

That caused the line
	rm -f ${srcmask}_*
to be executed everytime, and wiped out the just downloaded *.dsc file, when 
going through the for loop the second time (during processing the *.tar.xz file).


I fixed that, and now it says (here locally):

ned@t520:~/deb_holgerw/website/on-salsa/cron/cron$ ./lessoften-parts/1ftpfiles
I: Download http://ftp.de.debian.org/debian/dists/sid/main/source/Sources.xz
I: Skip: pool/main/i/installation-guide/installation-guide_20220129.dsc (already downloaded)
I: Skip: pool/main/i/installation-guide/installation-guide_20220129.tar.xz (already downloaded)
I: Download http://ftp.de.debian.org/debian/dists/stable/main/source/Sources.xz
I: Skip: pool/main/i/installation-guide/installation-guide_20210730.dsc (already downloaded)
I: Skip: pool/main/i/installation-guide/installation-guide_20210730.tar.xz (already downloaded)
I: Download http://ftp.de.debian.org/debian/dists/oldstable/main/source/Sources.xz
I: Skip: pool/main/i/installation-guide/installation-guide_20190622.dsc (already downloaded)
I: Skip: pool/main/i/installation-guide/installation-guide_20190622.tar.xz (already downloaded)

and now *.tar.xz and *.dsc files are both there in the target dir.

So I guess it will work tomorrow as it should.


Best regards

Holger





> Osamu
> -----Original Message-----
> From: Holger Wansing <[email protected]>
> To: Osamu Aoki <[email protected]>
> Cc: [email protected]
> Subject: build of installation-guide on web server broken [Was: Re: debian-doc files
> on the Debian web server]
> Date: Mon, 31 Jan 2022 19:06:29 +0100
> 
> Hi Osamu,
> 
> 
> Osamu Aoki <[email protected]> wrote (Thu, 04 Nov 2021 11:34:06 +0900):
> > Hi,  (Sorry)
> > 
> > On Wed, 2021-11-03 at 18:14 +0100, Holger Wansing wrote:
> > > Hi Osamu,
> > > 
> > > Am 3. November 2021 15:45:44 MEZ schrieb Osamu Aoki <[email protected]>:
> > > > Hi,
> > > > 
> > > > I finally made update to the cron script so we don't rely on FTP protocol.
> > > > url = [email protected]:webmaster-team/cron.git  
> > > > 
> > > >  526ebd0 ("Download deb via HTTP", 2021-11-01)
> > > > 
> > > >    No more use of FTP protocol
> > > >    
> > > >    Ensure to avoid unnecessary downloads of non-sid packages and already
> > > >    downloaded packages.
> > > >    
> > > >    unpack once and do not reprocess
> > > >    
> > > >    Signed-off-by: Osamu Aoki <[email protected]>
> > > >    
> > > > 
> > > > I stop making a copy of doc.  I don't see the following data are used any more.
> > > > > $crondir/ftpfiles/doc
> > > > 
> > > > Although it was tested on my machine, since this is rather big change, I am
> > > > afraid
> > > > something may break.
> > > 
> > > As far as I can see, the installation-guide source
> > > packages are
> > > no longer downloaded by 1ftpfiles.
> > > This was handled by this line:
> > > 
> > > wgetfiles installation-guide installation-guide_* 
> > 
> > I thought this was some kind of left-over.  Sorry, you are getting *.dsc file and
> > tarball here.  When I wrote wgetfiles, I didn't anticipate such uses.
> > 
> > > Now, building of installation-guide can no longer work.
> > > (the lessoften-parts/1installation-guide script relies on the
> > > source files to be existing in $ftpdir/pool dir.)
> > > 
> > > Or did I miss something?
> > > 
> > 
> > Give me sometime (2 days), I have some idea.  Instead of Packages, we need to
> > download Sources there.  Then it is trivial to download correct version.  No
> > rmadison
> > complication needed.
> 
> Two days ago, we had the first upload of installation-guide package since 
> the above changings.
> 
> Now I see, that the build is broken.
> At https://www-master.debian.org/build-logs/webwml/lessoften.log I see:
> 
> ---------------------------------------------------------------------------------
> starting the lessoften cron job at Sun Jan 30 14:49:01 UTC 2022
> Already up to date.
> run-parts: executing /srv/www.debian.org/cron/lessoften-parts/1ftpfiles
> I: Download http://ftp.de.debian.org/debian/dists/sid/main/source/Sources.xz
> I: Download: pool/main/i/installation-guide/installation-guide_20220129.dsc
> I: Download: pool/main/i/installation-guide/installation-guide_20220129.tar.xz
> I: Download http://ftp.de.debian.org/debian/dists/stable/main/source/Sources.xz
> I: Download: pool/main/i/installation-guide/installation-guide_20210730.dsc
> I: Download: pool/main/i/installation-guide/installation-guide_20210730.tar.xz
> I: Download http://ftp.de.debian.org/debian/dists/oldstable/main/source/Sources.xz
> I: Download: pool/main/i/installation-guide/installation-guide_20190622.dsc
> I: Download: pool/main/i/installation-guide/installation-guide_20190622.tar.xz
> 
> 1ftpfiles finished (at Sun Jan 30 14:49:08 UTC 2022)
> 
> run-parts: executing /srv/www.debian.org/cron/lessoften-parts/1installation-guide
> no new installation-guide for stable (bullseye), skipping build
> extracting installation-guide source for testing (bookworm) ...
> dpkg-source: error: cannot read
> /srv/www.debian.org/cron/ftpfiles/sid/installation-guide_20220129.dsc: No such file
> or directory
> building installation-guide for testing (bookworm) ...
> /srv/www.debian.org/cron/lessoften-parts/1installation-guide: line 116: cd:
> installation-guide-20220129/build: No such file or directory
> FAILED to build the installmanual for testing (bookworm) ...
> no new installation-guide for oldstable (buster), skipping build
> ---------------------------------------------------------------------------------
> 
> So, even if 1ftpfiles says, it downloads the *.dsc file, it is not there.
> The 1installation-guide script however needs it.
> 
> 
> Could you take a look?
> 
> 
> Thanks
> Holger
> 
> 
> 
> 
> 


-- 
Holger Wansing <[email protected]>
PGP-Fingerprint: 496A C6E8 1442 4B34 8508  3529 59F1 87CA 156E B076