Re: Populating a local message store AND multithreading

On Mon, Apr 12, 2004 at 10:19:24AM -0500, Gerald V. Livingston II wrote:
> Hello all,

Hi! I don't have answers for everything, so I'll just reply partially..

> I telneted into the news server and switched to the group. Start/finish
> article numbers indicated about 30,000 articles total. I pulled the header
> for the lowest number article and it was about 3 days old.

Snfetch seems to use "XHDR Message-Id low-" Perhaps your newsserver
gives different data for XHDR?

> 2. Speed and parallel connections. 
 
> This makes it appear as if the -c and -p flags are having no effect. Can
> someone explain them (especially -c) more clearly?

-c is the command pipeline. Normally in NNTP, you ask for an article,
wait for the reply, and then ask for another article. With a pipeline, 
you don't wait for the complete arrival of one message before asking for
the next.  So with a queue of 3, you ask for three articles, and once
the first article has arrived completely, you ask for another article
already. 

This way, the server always has some commands from you waiting, and it
can send the next message as fast as it can, improving your download
speed. 

The effect of this is larger if you have a very slow upload rate. I have
no idea what a sane setting would be, but if you're already having the
full (capped) dl-speed, it should matter much.

The -p is for parallel download, which in your case should get you
multiple connections to your newsserver. Note however, that the manpage
says "Attempt to fetch for nparallel newsgroups at once". If you have
only two newsgroups you download, anything more than '-p 2' will never
be used. A single newsgroup will never use multiple connections to your
server. 

> 3. File locking
> 
> The 2 groups I want to pull are VERY active. In order to stay current I'd
> like to set a cron job to run snget every 15 or 30 minutes to try to make
> sure I get all of the articles. If one of the groups has received a LOT of
> articles and the previous run has not finished when cron calls for a new
> run will it be handled in a clean manner?

I believe it will. There is some locking, but I'm not sure about what
it's all used for.

> 4. Symlinked groups.
> 

The README says:
sn is a small news system for small sites serving perhaps a few
dozen newsgroups, and with a slow connection to the internet.

With 20 Gigs of news in 3 days it seems to me you're not really the
target audience of 'sn'. I think a lot of news software will make
assumptions about the newsspool being on a single disk. 

Have you looked into lvm or something like that to make multiple disks
appear as one?


> 5. Article expiration.
> 
> Is there any way to force article expiration based on something other than
> date? I've set up sn to pull messages while I'm not home (where the fast
> connection is located). When I get home I want to pull those articles I'm
> interested in then expire the whole lot to recover the space. I'll want to
> expire everything up to the last download.

Run snexpire -1h news.group by hand?


Wouldn't you be better of with some binary harvester program? Seems more
applicable in your situation. 

Greetings,
Chris Niekel
-- 
    I've been down so long, if I'd cheer up, I'd still be depressed.
            - Lisa Simpson, Moanin' Lisa Blues.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFAgtOFgQ3S2XWlk0sRAtEXAKCm6qAXxZpaZi0Yv/w1yuaFHNqaKwCfVh5F
HVQh99EXb7XGd85SnzSGQWc=
=0K9c
-----END PGP SIGNATURE-----