Re: scoring related slowdown in group loading

On 2009-03-31, Grant Edwards <[email protected]> wrote:
> On 2009-03-31, Grant Edwards <[email protected]> wrote:
>
>> Recently (within the past few weeks) the loading rate of
>> groups suddently dropped from 100-200 articles/sec down to
>> maybe 5-10 articles per second.
>>
>> Removing the following two lines from my .score file eliminates
>> the slowdown.
>>
>>  Score: =0
>>   Newsgroups: gmane.os.ecos.general
>
> One more piece of info: the above two lines cause a slowdown
> using the supernews/giganews server (for "real" Usenet), but do
> not cause a slowdown when using the gmane.org server (for
> gatewayed mailing lists).

It appears that having a .score rule for a newsgroup makes slrn
think that it needs the "path" header, which means it has to do
individual HEAD commands.  That doesn't make sense to me, but
here's the evidence.  Note that at line 55 in the slow case,
slrn sends "XHDR path", gets "unsupported header field" as the
response, and then closes the TCP connection.  Slrn then
reopens the TCP connection and does individual HEAD commands
for every article.

Why does slrn think it needs the path header?


NNTP session transcript for the "fast" case:

     1	200 News.GigaNews.Com
     2	MODE READER
     3	480 authentication required
     4	AUTHINFO USER <elided>
     5	381 more authentication required
     6	AUTHINFO PASS <elided>
     7	281 News.GigaNews.Com
     8	MODE READER
     9	200 News.GigaNews.Com
    10	XOVER
    11	412 no group selected
    12	LIST OVERVIEW.FMT
    13	215 overview format
    14	Subject:
    15	From:
    16	Date:
    17	Message-ID:
    18	References:
    19	Bytes:
    20	Lines:
    21	Xref:full
    22	.
    23	GROUP comp.graphics.appps.gnuplot
    24	GROUP comp.graphics.gnuplot
    25	GROUP comp.unix.bsd.freebsd.announce
    26	GROUP alt.os.linux.gentoo
    27	GROUP comp.dcom.telecom.tech
    28	GROUP comp.lang.python
    29	GROUP comp.lang.python.announce
    30	GROUP comp.mail.mutt
    31	GROUP comp.arch.embedded
    32	GROUP comp.os.linux.announce
    33	GROUP comp.os.linux.development.apps
    34	GROUP comp.os.linux.development.system
    35	GROUP comp.os.linux.embedded
    36	GROUP comp.os.linux.misc
    37	GROUP rec.autos.makers.mazda.miata
    38	211 14 2 15 comp.graphics.appps.gnuplot
    39	211 36 250 285 comp.graphics.gnuplot
    40	211 7 853 859 comp.unix.bsd.freebsd.announce
    41	211 25501 6422 31922 alt.os.linux.gentoo
    42	211 9081 55315 64395 comp.dcom.telecom.tech
    43	211 343627 292624 636250 comp.lang.python
    44	211 4992 2865 7856 comp.lang.python.announce
    45	211 9273 20464 29736 comp.mail.mutt
    46	211 123634 167711 291344 comp.arch.embedded
    47	211 2878 15603 18480 comp.os.linux.announce
    48	211 31077 124977 156053 comp.os.linux.development.apps
    49	211 26427 132530 158956 comp.os.linux.development.system
    50	211 12275 9099 21373 comp.os.linux.embedded
    51	211 193131 521923 715053 comp.os.linux.misc
    52	211 30008 151673 181680 rec.autos.makers.mazda.miata
    53	GROUP comp.os.linux.embedded
    54	211 12275 9099 21373 comp.os.linux.embedded
    55	XOVER 21274-21373
    56	224 xover information follows
    57	21274	Re: Backporting  V4L2 (Video 4 Linux 2)	anon <[email protected]>	Thu, 12 Mar 2009 09:22:14 +0100	<[email protected]>	<[email protected]>	1270	7	Xref: number1.nntp.dca.giganews.com comp.os.linux.embedded:21274
    58	21275	How to write data to config file in kernel mode	Kid <[email protected]>	Thu, 12 Mar 2009 01:45:58 -0700 (PDT)	<[email protected]>		1504	12	Xref: number1.nntp.dca.giganews.com comp.os.linux.embedded:21275
    59	21276	Re: How to write data to config file in kernel mode	Kristof Provost <[email protected]>	Thu, 12 Mar 2009 10:31:25 GMT	<[email protected]>	<[email protected]>	1459	17	Xref: number1.nntp.dca.giganews.com comp.os.linux.embedded:21276
    60	21277	Re: Backporting V4L2 (Video 4 Linux 2)	=?ISO-8859-1?B?R/xy/FBy4OfkZA==?= <[email protected]>	Thu, 12 Mar 2009 18:39:02 -0700 (PDT)	<833251ed-fa00-4024-b34e-6ef347a65a17@w24g2000prd.googlegroups.com>	<[email protected]>  <[email protected]>	1815	18	Xref: number1.nntp.dca.giganews.com comp.os.linux.embedded:21277
    61	21278	Re: Video for Linux (V4L2)	"StoneThrower" <digi_64-public[removethis]@yahoo.com>	Sat, 14 Mar 2009 18:36:26 +0100	<[email protected]>	<c223ad2e-8e47-41bd-b9f6-183362544b74@s19g2000vbp.googlegroups.com>	1224	6	Xref: number1.nntp.dca.giganews.com comp.os.linux.embedded:21278
[...]
   118	21359	Re: embedded Linux for product development	Chris <[email protected]>	Thu, 26 Mar 2009 13:23:27 -0700 (PDT)	<c3932efb-8576-46b2-917a-fe09a3f22d47@k19g2000prh.googlegroups.com>	<[email protected]> <[email protected]>	3365	53	Xref: number1.nntp.dca.giganews.com comp.os.linux.embedded:21359
   119	21368	buildroot + ubifs now working ?	bugkiller <[email protected]>	Mon, 30 Mar 2009 14:50:47 -0700 (PDT)	<c65ae7bd-422f-45d6-9628-ebc60ee722bb@v12g2000vbb.googlegroups.com>		1337	10	Xref: number1.nntp.dca.giganews.com comp.os.linux.embedded:21368
   120	21372	Re: Free Solaris 10 root account and training	Ed Prochak <[email protected]>	Tue, 31 Mar 2009 07:22:06 -0700 (PDT)	<07b60865-2fb1-428f-a629-a6216bdaa393@l38g2000vba.googlegroups.com>	<43be7c59-5f9f-4df6-8232-78aa7b62ec92@q30g2000prq.googlegroups.com>	2506	33	Xref: number1.nntp.dca.giganews.com comp.os.vxworks:71611 comp.os.qnx:56717 comp.os.linux.embedded:21372 comp.realtime:40694
   121	21373	Re: embedded Linux for product development	"Johnson L" <[email protected]>	Tue, 31 Mar 2009 09:32:58 -0600	<[email protected]>	<[email protected]> <[email protected]> <c3932efb-8576-46b2-917a-fe09a3f22d47@k19g2000prh.googlegroups.com>	3319	57	Xref: number1.nntp.dca.giganews.com comp.os.linux.embedded:21373
   122	.


NNTP transcript for the slow case:

     1	200 News.GigaNews.Com
     2	MODE READER
     3	480 authentication required
     4	AUTHINFO USER <elided>
     5	381 more authentication required
     6	AUTHINFO PASS <elided>
     7	281 News.GigaNews.Com
     8	MODE READER
     9	200 News.GigaNews.Com
    10	XOVER
    11	412 no group selected
    12	LIST OVERVIEW.FMT
    13	215 overview format
    14	Subject:
    15	From:
    16	Date:
    17	Message-ID:
    18	References:
    19	Bytes:
    20	Lines:
    21	Xref:full
    22	.
    23	GROUP comp.graphics.appps.gnuplot
    24	211 14 2 15 comp.graphics.appps.gnuplot
    25	GROUP comp.graphics.gnuplot
    26	GROUP comp.unix.bsd.freebsd.announce
    27	GROUP alt.os.linux.gentoo
    28	GROUP comp.dcom.telecom.tech
    29	GROUP comp.lang.python
    30	GROUP comp.lang.python.announce
    31	GROUP comp.mail.mutt
    32	GROUP comp.arch.embedded
    33	GROUP comp.os.linux.announce
    34	GROUP comp.os.linux.development.apps
    35	GROUP comp.os.linux.development.system
    36	GROUP comp.os.linux.embedded
    37	GROUP comp.os.linux.misc
    38	GROUP rec.autos.makers.mazda.miata
    39	211 36 250 285 comp.graphics.gnuplot
    40	211 7 853 859 comp.unix.bsd.freebsd.announce
    41	211 25501 6422 31922 alt.os.linux.gentoo
    42	211 9081 55315 64395 comp.dcom.telecom.tech
    43	211 343627 292624 636250 comp.lang.python
    44	211 4992 2865 7856 comp.lang.python.announce
    45	211 9273 20464 29736 comp.mail.mutt
    46	211 123634 167711 291344 comp.arch.embedded
    47	211 2878 15603 18480 comp.os.linux.announce
    48	211 31077 124977 156053 comp.os.linux.development.apps
    49	211 26427 132530 158956 comp.os.linux.development.system
    50	211 12275 9099 21373 comp.os.linux.embedded
    51	211 193131 521923 715053 comp.os.linux.misc
    52	211 30008 151673 181680 rec.autos.makers.mazda.miata
    53	GROUP comp.os.linux.embedded
    54	211 12275 9099 21373 comp.os.linux.embedded
    55	XHDR Path
    56	503 unsupported header field
[slrn closes the connection]    
    57	503 time out
[server closes the connection]
[slrn opens a new connection]
    58	200 News.GigaNews.Com
    59	MODE READER
    60	480 authentication required
    61	AUTHINFO USER <elided>
    62	381 more authentication required
    63	AUTHINFO PASS <elided>
    64	281 News.GigaNews.Com
    65	MODE READER
    66	200 News.GigaNews.Com
    67	GROUP comp.os.linux.embedded
    68	211 12275 9099 21373 comp.os.linux.embedded
    69	XHDR Path
    70	503 unsupported header field
    71	HEAD 21275
    72	221 21275 <[email protected]>
    73	Path: border1.nntp.dca.giganews.com!nntp.giganews.com!postnews.google.com!v5g2000prm.googlegroups.com!not-for-mail
    74	From: Kid <[email protected]>
    75	Newsgroups: comp.os.linux.embedded
    76	Subject: How to write data to config file in kernel mode
    77	Date: Thu, 12 Mar 2009 01:45:58 -0700 (PDT)
    78	Organization: http://groups.google.com
    79	Lines: 12
    80	Message-ID: <[email protected]>
    81	NNTP-Posting-Host: 222.156.254.82
    82	Mime-Version: 1.0
    83	Content-Type: text/plain; charset=ISO-8859-1
    84	Content-Transfer-Encoding: 7bit
    85	X-Trace: posting.google.com 1236847558 11995 127.0.0.1 (12 Mar 2009 08:45:58 GMT)
    86	X-Complaints-To: [email protected]
    87	NNTP-Posting-Date: Thu, 12 Mar 2009 08:45:58 +0000 (UTC)
    88	Complaints-To: [email protected]
    89	Injection-Info: v5g2000prm.googlegroups.com; posting-host=222.156.254.82; 
    90		posting-account=9gbkqQoAAADvAJ4sl0ZjWbAQlDU695Gk
    91	User-Agent: G2/1.0
    92	X-HTTP-Via: 1.1 TNS0402
    93	X-HTTP-UserAgent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 
    94		1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; 
    95		.NET CLR 3.5.21022; MS-RTC LM 8),gzip(gfe),gzip(gfe)
    96	Bytes: 1504
    97	Xref: number1.nntp.dca.giganews.com comp.os.linux.embedded:21275
    98	.
    99	HEAD 21276
   100	221 21276 <[email protected]>
   101	Path: border1.nntp.dca.giganews.com!nntp.giganews.com!newsfeed00.sul.t-online.de!t-online.de!feeder.news-service.com!cyclone03.ams2.highwinds-media.com!news.highwinds-media.com!npeersf02.ams.highwinds-media.com!newsfe17.ams2.POSTED!7564ea0f!not-for-mail
   102	Newsgroups: comp.os.linux.embedded
   103	From: Kristof Provost <[email protected]>
[a bunch more HEAD commands]
  1680	.


-- 
Grant Edwards                   grante             Yow! An Italian is COMBING
                                  at               his hair in suburban DES
                               visi.com            MOINES!


------------------------------------------------------------------------------