multiple nntp connections to same server in ubhdb

Hello again,

I was curious if any more thought or development had been made on
enabling multiple, concurrent connections to the same nntp servers in
ubhdb.  

I was considering how to implement this and wanted to get some
feedback on the approach.

Currently, in Ubh::ServerConnections::connect_to_servers, we take an
arrayref of hashrefs containing server connection properties and stash
the Ubh::NNTP object in the nntp element. 

Then in freshen_headers(), we iterate through each newsgroup, then
each server, fetch the range of articles available, delete from the
database anything expired from that server, figure out the range of
articles we do have and what new ranges we should retrieve.  Finally,
we split that into full sips, retrieve, then fetch the final partial
sip (if any).

I am considering modifying Ubh::ServerConnections::connect_to_servers
to create an arrayref of Ubh::NNTP objects.  The servers table would
be modified to hold a maxconns field indicating the maximum number of
concurrent connections permitted and perhaps another field to indicate
how many of those connections to actually utilize.  

In the sipping logic of freshen_headers(), we would then fork and call
socketpair to establish pipes for bidirectional communication with
each child.  The child would block on reading the parent pipe for a
range to sip or command to exit.  The parent, after handing sips to
the children, would select on the pipes to see when they have
completed that sip, and either send another sip or instruct the child
to exit.  When all sips have been exhausted, the parent does a waitpid
on each child to avoid zombies. 

That sounds kind of complex but I've written some test programs to
verify how it would all work and it actually doesn't seem too bad,
Granted I buried my face in the the manpages for select, vec,
socketpair, perlipc, IO::Handle, waitpid, and friends for a while.

However, it still leaves the case where you have 5 servers defined in
your servers table and 1 of them only permits one connection at 10kbps
or some similarly slow rate.  Your other servers would be blocked
until that server finished freshening it's headers, though I suppose
you could use the priority field to make it update last?

Otherwise, I had thought we could also fork each main server 
definition and iterate through the newsgroups within each child, 
applying the above described sipping logic in each of those children.

I don't know if any of this would work under Windows.  It also adds a
lot of memory and filehandle consumption.  But I think it would
increase article fetching speed quite a bit.

I've probably left out some details that are in my head.  Does this
sound crazy?  Is there an easier way I'm overlooking?  Maybe manually
running additional ubh processes and making ubhdb store it's state in
the database so other processes can not conflict?

Ben





 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/ubh/

<*> To unsubscribe from this group, send an email to:
    [email protected]

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/