Re: Re: Populating a local message store AND multithreading

On Sun, 18 Apr 2004 21:14:14 +0200 Chris Niekel <[email protected]> wrote:

> On Mon, Apr 12, 2004 at 10:19:24AM -0500, Gerald V. Livingston II wrote:
> > Hello all,
> 
> Hi! I don't have answers for everything, so I'll just reply partially..
> 
> > I telneted into the news server and switched to the group. Start/finish
> > article numbers indicated about 30,000 articles total. I pulled the
> > header for the lowest number article and it was about 3 days old.
> 
> Snfetch seems to use "XHDR Message-Id low-" Perhaps your newsserver
> gives different data for XHDR?

I managed to get caught up by massaging the .serial file a few times.
Luckily sn does handle articles that already exist locally very well. ;-)

> > 2. Speed and parallel connections. 
>  
> > This makes it appear as if the -c and -p flags are having no effect.
> > Can someone explain them (especially -c) more clearly?
> 
> -c is the command pipeline. Normally in NNTP, you ask for an article,
> wait for the reply, and then ask for another article. With a pipeline, 
> you don't wait for the complete arrival of one message before asking for
> the next.  So with a queue of 3, you ask for three articles, and once
> the first article has arrived completely, you ask for another article
> already. 
> 
> This way, the server always has some commands from you waiting, and it
> can send the next message as fast as it can, improving your download
> speed. 
> 
> The effect of this is larger if you have a very slow upload rate. I have
> no idea what a sane setting would be, but if you're already having the
> full (capped) dl-speed, it should matter much.
> 
> The -p is for parallel download, which in your case should get you
> multiple connections to your newsserver. Note however, that the manpage
> says "Attempt to fetch for nparallel newsgroups at once". If you have
> only two newsgroups you download, anything more than '-p 2' will never
> be used. A single newsgroup will never use multiple connections to your
> server. 

I'm using -c 4 -p 4 now and I apear to be getting max dl even with only one
group being pulled in. Either I was mistaken about the cap-per-connection
or the news server I'm connecting to is sending across the parallel (-p)
connections based on the concurent (-c) article requests -- and sn is
accepting the multiple sends. I'll play with it next week when i have a
chance and drop it to -c 1 -p 1 and see if it limits my speed or still
pulls at full pipe through a single connection.

> > 3. File locking
> > 
> > The 2 groups I want to pull are VERY active. In order to stay current
> > I'd like to set a cron job to run snget every 15 or 30 minutes to try
> > to make sure I get all of the articles. If one of the groups has
> > received a LOT of articles and the previous run has not finished when
> > cron calls for a new run will it be handled in a clean manner?
> 
> I believe it will. There is some locking, but I'm not sure about what
> it's all used for.

Work got in my way so I'm still pulling by hand, waiting for one invocation
to finish before starting another. I just keep an ssh session open to my
home system. I'm running screen so it just keeps plugging along even if my
connection gets dropped and I can pick it back up when I ssh back in.

> > 4. Symlinked groups.
> 
> The README says:
> sn is a small news system for small sites serving perhaps a few
> dozen newsgroups, and with a slow connection to the internet.
> 
> With 20 Gigs of news in 3 days it seems to me you're not really the
> target audience of 'sn'. I think a lot of news software will make
> assumptions about the newsspool being on a single disk. 
> 
> Have you looked into lvm or something like that to make multiple disks
> appear as one?

My original plan for this machine was to have a small boot/system drive and
a larger drive holding /home and /var. I ended up with two mid-size drives
though. Having lived through a system crash in the past I prefer to keep
physical disks completely separate as far as file systems go -- so
sometimes I symlink like crazy (had a system with 4 small drives installed
once that was REALLY ugly as I added drives and linked out parts of /usr,
/var, and /home when I started running out of room).

I ended up just scripting the call to snget. Wrote a bash script that
specifies the groups to dl and the switches I want used. I do think that
symlinked directories should be handled though. One example might be a
setup where an admin allows the use of sn to create local spools in
/home/username but want to allow the news groups disk quota to be separate
from the rest of the user's /home quota. A symlinked private spool
directory on another file system would be the way to go.

> > 5. Article expiration.
> > 
> > Is there any way to force article expiration based on something other
> > than date? I've set up sn to pull messages while I'm not home (where
> > the fast connection is located). When I get home I want to pull those
> > articles I'm interested in then expire the whole lot to recover the
> > space. I'll want to expire everything up to the last download.
> 
> Run snexpire -1h news.group by hand?

That's what I'm doing now and it works OK except in cases where I'm on an
"extended" work week. I just worked a 10 day stretch and it got ...
interesting. It will be less critical when I add a large spool drive later.

> Wouldn't you be better of with some binary harvester program? Seems more
> applicable in your situation. 

Yes, got one that will run on the command line in linux? Preferably with a
Debian install package? How would one select the articles in a CLI based
app? 

That's what I use to pull the articles I want from the linux box to the Win
box when I get home. Really nice grabbing articles at over 1M/sec <grin>. I
use a program called "Grabit" on the Win box. I'm setting up PAN to replace
Grabit so I have decent text article reading capabilities as well. I'm glad
PAN has a Win port.

It seems I'm slowly migrating all of my "regular use" programs (email and
news) to Win ports of software initially written for linux. PAN for nntp
and Mahogany for mail I'll set up a linux box with X again at some point.
Have to keep the Win box for now because of work.

Gerald

--
Gerald V. Livingston II

Configure your Email to send TEXT ONLY -- See the following page:
http://expita.com/nomime.html

'74 Kombi (to be renamed - LifeSaver doesn't 'feel' right)
'69 Type 1 - AirBall the Rolling Basket Case
http://www.phorce1.com
http://www.buskatiers.org