Re: multiple nntp connections to same server in ubhdb

what ben wants to do is have multiple connections during the ubhdb
(header downloading) phase.  that's different from having multiple
instances of during the ubh (article downloading) phase.



--- sean roehnelt <[email protected]> wrote:
> I'm not sure if this solves what your trying to do, but I will run 
> several concurrent ubh processes sometimes, and the effect is multiple 
> concurrent connections. - Although, I only connect to a single nntp 
> server.
> 
> On Mar 18, 2005, at 1:11 AM, Ben wrote:
> 
> >  Hello again,
> >
> >  I was curious if any more thought or development had been made on
> >  enabling multiple, concurrent connections to the same nntp servers in
> >  ubhdb. 
> >
> >  I was considering how to implement this and wanted to get some
> >  feedback on the approach.
> >
> >  Currently, in Ubh::ServerConnections::connect_to_servers, we take an
> >  arrayref of hashrefs containing server connection properties and stash
> >  the Ubh::NNTP object in the nntp element.
> >
> >  Then in freshen_headers(), we iterate through each newsgroup, then
> >  each server, fetch the range of articles available, delete from the
> >  database anything expired from that server, figure out the range of
> >  articles we do have and what new ranges we should retrieve.  Finally,
> >  we split that into full sips, retrieve, then fetch the final partial
> >  sip (if any).
> >
> >  I am considering modifying Ubh::ServerConnections::connect_to_servers
> >  to create an arrayref of Ubh::NNTP objects.  The servers table would
> >  be modified to hold a maxconns field indicating the maximum number of
> >  concurrent connections permitted and perhaps another field to indicate
> >  how many of those connections to actually utilize. 
> >
> >  In the sipping logic of freshen_headers(), we would then fork and call
> >  socketpair to establish pipes for bidirectional communication with
> >  each child.  The child would block on reading the parent pipe for a
> >  range to sip or command to exit.  The parent, after handing sips to
> >  the children, would select on the pipes to see when they have
> >  completed that sip, and either send another sip or instruct the child
> >  to exit.  When all sips have been exhausted, the parent does a waitpid
> >  on each child to avoid zombies.
> >
> >  That sounds kind of complex but I've written some test programs to
> >  verify how it would all work and it actually doesn't seem too bad,
> >  Granted I buried my face in the the manpages for select, vec,
> >  socketpair, perlipc, IO::Handle, waitpid, and friends for a while.
> >
> >  However, it still leaves the case where you have 5 servers defined in
> >  your servers table and 1 of them only permits one connection at 10kbps
> >  or some similarly slow rate.  Your other servers would be blocked
> >  until that server finished freshening it's headers, though I suppose
> >  you could use the priority field to make it update last?
> >
> >  Otherwise, I had thought we could also fork each main server
> >  definition and iterate through the newsgroups within each child,
> >  applying the above described sipping logic in each of those children.
> >
> >  I don't know if any of this would work under Windows.  It also adds a
> >  lot of memory and filehandle consumption.  But I think it would
> >  increase article fetching speed quite a bit.
> >
> >  I've probably left out some details that are in my head.  Does this
> >  sound crazy?  Is there an easier way I'm overlooking?  Maybe manually
> >  running additional ubh processes and making ubhdb store it's state in
> >  the database so other processes can not conflict?
> >
> >  Ben
> 
> 


 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/ubh/

<*> To unsubscribe from this group, send an email to:
    [email protected]

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/