Bug#884229: Bug #884229: [packages.debian.org] 500 error/Internal Server Error (in packages-pkgmirror-csail.debian.org)

Control: tag -1 patch

Hi Laura,

Thanks for mentioning this on #-www today!

Laura Arjona Reina <[email protected]> (2018-01-24):
> I can reproduce the problem with some other packages (I tried several from the
> list https://packages.debian.org/unstable/main/newpkg and got the error in the
> links to buster and sid).
> 
> I also tried
> https://packages.debian.org/stretch-backports/kernel-image-4.14.0-0.bpo.2-arm64-di
> with the same results (error 500).
> 
> Thanks to some help in the IRC channel we've found that only one of our wo
> machines serving packages.debian.org fails:
> 
> https://packages-pkgmirror-csail.debian.org/sid/elpa-find-file-in-project
> gives the 500 Internal Server Error
> but
> https://packages-picconi.debian.org/sid/elpa-find-file-in-project
> works.
> 
> If I can be of any help to solve this, just tell. I'll try to look at the logs
> later, but I'm not sure I have the corresponding permissions (will try).

From the apache error log on the mirror:
> mod_fcgid: stderr: Can't call method "get_document" on an undefined value at ../lib/Packages/Search.pm line 264.
> End of script output before headers: dispatcher.fcgi

Looking at the code, that's a method call on a xapian query result.

Looking at the xapian directory on picconi:
> pkg_user@picconi:/srv/packages.debian.org$ ls -l files/db/xapian/
> total 302384
> -rw-r--r-- 1 pkg_user pkg_maint   1974272 Jan 24 15:56 docdata.glass
> -rw-r--r-- 1 pkg_user pkg_maint         0 Jan 24 15:53 flintlock
> -rw-r--r-- 1 pkg_user pkg_maint       138 Jan 24 15:56 iamglass
> -rw-r--r-- 1 pkg_user pkg_maint 181952512 Jan 24 15:56 position.glass
> -rw-r--r-- 1 pkg_user pkg_maint  80699392 Jan 24 15:56 postlist.glass
> -rw-r--r-- 1 pkg_user pkg_maint  44998656 Jan 24 15:56 termlist.glass

Looking at the xapian directory on the mirror:
> pkg_user@pkgmirror-csail:/srv/packages.debian.org$ ls -l files/db/xapian/
> total 537584
> -rw-r--r-- 1 pkg_user pkg_maint   1974272 Jan 24 09:48 docdata.glass
> -rw-r--r-- 1 pkg_user pkg_maint         0 Jan 24 09:45 flintlock
> -rw-r--r-- 1 pkg_user pkg_maint        28 Nov  8 09:26 iamchert
> -rw-r--r-- 1 pkg_user pkg_maint       138 Jan 24 09:48 iamglass
> -rw-r--r-- 1 pkg_user pkg_maint      1699 Nov  8 09:30 position.baseA
> -rw-r--r-- 1 pkg_user pkg_maint      1725 Nov  8 09:30 position.baseB
> -rw-r--r-- 1 pkg_user pkg_maint 111788032 Nov  8 09:30 position.DB
> -rw-r--r-- 1 pkg_user pkg_maint 181944320 Jan 24 09:48 position.glass
> -rw-r--r-- 1 pkg_user pkg_maint      1254 Nov  8 09:30 postlist.baseA
> -rw-r--r-- 1 pkg_user pkg_maint      1263 Nov  8 09:30 postlist.baseB
> -rw-r--r-- 1 pkg_user pkg_maint  81616896 Nov  8 09:30 postlist.DB
> -rw-r--r-- 1 pkg_user pkg_maint  80707584 Jan 24 09:48 postlist.glass
> -rw-r--r-- 1 pkg_user pkg_maint        54 Nov  8 09:30 record.baseA
> -rw-r--r-- 1 pkg_user pkg_maint        56 Nov  8 09:30 record.baseB
> -rw-r--r-- 1 pkg_user pkg_maint   2523136 Nov  8 09:30 record.DB
> -rw-r--r-- 1 pkg_user pkg_maint       691 Nov  8 09:30 termlist.baseA
> -rw-r--r-- 1 pkg_user pkg_maint       703 Nov  8 09:30 termlist.baseB
> -rw-r--r-- 1 pkg_user pkg_maint  44883968 Nov  8 09:30 termlist.DB
> -rw-r--r-- 1 pkg_user pkg_maint  45006848 Jan 24 09:48 termlist.glass

For files dated Jan 24, the timestamps don't match, but that's probably
just a sync waiting to happen, and that doesn't explain the issues which
have been happening for so long. I suspected the stray files instead,
dating back to November. I've created an “old-files” directory and moved
them there, and the mirror seems to be behaving properly now.

I'm tagging this bug report with “pending” for the time being, to give
people a chance to comment. But I suppose it should be safe to remove
those old files entirely?

FTR, that would mean cleaning this:

    230M /srv/packages.debian.org/files/db/xapian/old-files/


Cheers,
-- 
Cyril Brulebois ([email protected])            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEtg6/KYRFPHDXTPR4/5FK8MKzVSAFAlpotSkACgkQ/5FK8MKz
VSDTsxAAgBeHXTmAmGBqmje5PEbaKDvC3qgwILAH5a5kMV9cMCfMdSrIftOTFlyY
4+yowxEUtrRwEHUYpCpHpmqd29e+v7cTie8yXoWk3C6/1LoHmJZ/Z/5HQCQLqHXp
7y0Ap25U6eT6xzL8ibcFg2Hs7/cchVVRz75fvL7LIhd1S4ca5LNzqnGCgkSBNZ3f
MNAi4aeqxXGx3XMgbcuYC9zoitBKZp9I6jfgi8QZ10f0/MavZF75BzCcDJnRka83
pwVoXtQqxvG6j1c1PCPozivfu4hm3fj0Ba9BO0C0MVZiM5wqyLHuGYXJRz1e+pK8
cI5vwdgp9r2HLF8CZSR9EeAp/X7trsMPGYgTeg6j0YjiDg4X9TAEHT7QXuuo2o/R
g8nh3VmLnlfjKn1LZYGi86JGXgyTlWZfcdNSa/9C33ZO66OdFth6mGyxrh0axnS6
uyz1rqNNq9QMJ8W/7mGhDVlLz3SDFmOqCWd3uCPerGSlBuQ4po8UJCYMGitj7Igw
c3lEBrG/mw/e8jXde1ObBOtGP06govgq3omCfKDfzjwXrnmiKHuWzGkx2ZqefvAL
fEPW3u4s5e0pP9MiY8lrMZxwzrHGl2iBkIbpHBVktgRiP9PqMaCf/uPFklVmQtfY
XuKC4i4E+PJ2//KDzAhjfFGE3kujunZ7bsCI1vdGkxZD+OA7xGA=
=EBOD
-----END PGP SIGNATURE-----