Bug#884229: Bug #884229: [packages.debian.org] 500 error/Internal Server Error (in packages-pkgmirror-csail.debian.org)

Cyril Brulebois <[email protected]> Fri, 26 Jan 2018 08:37:49 +0100

Newsgroups	gmane.linux.debian.devel.www
Organization	Debian
Message-ID	<[email protected]>

Hi Olly,

Olly Betts <[email protected]> (2018-01-26):
> Yes.  The directory on pkgmirror-csail actually had two different
> versions of the database using different database backends, so the
> files which aren't in the directory on picconi are essentially an
> old version of the same database.
> 
> That alone shouldn't cause the search to fail though - it should just
> open one or the other (it looks like both Xapian 1.2.x and 1.4.x will
> open the old one in this rather odd situation).
> 
> Xapian's replication should cleanly handle the database being replicated
> switching backends so I'm guessing the replication here is using rsync
> (without --delete-after) or similar?  That might also explain why the
> old database is broken as the databases aren't safe to rsync if an
> update is in progress - if the last rsync of the old database happened
> while an update was in progress, it could have been left broken, which
> would typically result in particular searches failing.
> 
> Updating with rsync can also break searches on the mirror while the
> rsync is in progress even if the master isn't updating, so perhaps we
> should switch to using Xapian's replication?  That's already being
> used for the main website search (I set it up with weasel at Debconf
> 15).
> 
> Happy to assist with that, though I'll probably need ssh access to
> pkgmirror-csail (seems I currently only have it for picconi).  My
> Debian username is "olly".

Looking around without knowing much about pkg stuff:
 - picconi has: conf/pushpdo.mirror → csail pkgmirror-csail.debian.org pkg_user
 - those bits are read by bin/pushpdo, which calls rsync this way:
     rsync -e "ssh -i ${MKEYFILE} -${MPROTO} ${SSH_OPTS}" -av --stats  "${MIRRORPATH}"  ${MUSER}@${MHOSTNAME}:/does/not/matter >"${LOGDIR}/pushpdo.${MLNAME}.log"

I suppose help is welcome to possibly fix/improve that, but I'm no pkg
expert (I just happened to have helped one or twice in the past, and to
still be in the pkg_maint group…)


Cheers,
-- 
Cyril Brulebois ([email protected])            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

signature.asc (application/pgp-signature, 833 B)