Re: multiple nntp connections to same server in ubhdb

sounds interesting, but it would have to work equally well on windows
as it does elsewhere for me to consider it.

a smaller task would be to investigate how orthogonal the group processing
is.  the main thing to look out for is how it keeps track of cross posts.
the objective is to determine if you could run multiple copies of ubhdb
simultaneously, each copy runs on its own group.  so long as each instance
of ubhdb isn't interfering with the processing of other groups (most likely
due to cross postings, as i said), then it might be worthwhile to see
if or how this might work out.  just an idea.

actually, now that i think about it, this issue of cross posts would 
be an issue to investigate with your approach as well.


-gerard

--- Ben <[email protected]> wrote:
> 
> Hello again,
> 
> I was curious if any more thought or development had been made on
> enabling multiple, concurrent connections to the same nntp servers in
> ubhdb.  
> 
> I was considering how to implement this and wanted to get some
> feedback on the approach.
> 
> Currently, in Ubh::ServerConnections::connect_to_servers, we take an
> arrayref of hashrefs containing server connection properties and stash
> the Ubh::NNTP object in the nntp element. 
> 
> Then in freshen_headers(), we iterate through each newsgroup, then
> each server, fetch the range of articles available, delete from the
> database anything expired from that server, figure out the range of
> articles we do have and what new ranges we should retrieve.  Finally,
> we split that into full sips, retrieve, then fetch the final partial
> sip (if any).
> 
> I am considering modifying Ubh::ServerConnections::connect_to_servers
> to create an arrayref of Ubh::NNTP objects.  The servers table would
> be modified to hold a maxconns field indicating the maximum number of
> concurrent connections permitted and perhaps another field to indicate
> how many of those connections to actually utilize.  
> 
> In the sipping logic of freshen_headers(), we would then fork and call
> socketpair to establish pipes for bidirectional communication with
> each child.  The child would block on reading the parent pipe for a
> range to sip or command to exit.  The parent, after handing sips to
> the children, would select on the pipes to see when they have
> completed that sip, and either send another sip or instruct the child
> to exit.  When all sips have been exhausted, the parent does a waitpid
> on each child to avoid zombies. 
> 
> That sounds kind of complex but I've written some test programs to
> verify how it would all work and it actually doesn't seem too bad,
> Granted I buried my face in the the manpages for select, vec,
> socketpair, perlipc, IO::Handle, waitpid, and friends for a while.
> 
> However, it still leaves the case where you have 5 servers defined in
> your servers table and 1 of them only permits one connection at 10kbps
> or some similarly slow rate.  Your other servers would be blocked
> until that server finished freshening it's headers, though I suppose
> you could use the priority field to make it update last?
> 
> Otherwise, I had thought we could also fork each main server 
> definition and iterate through the newsgroups within each child, 
> applying the above described sipping logic in each of those children.
> 
> I don't know if any of this would work under Windows.  It also adds a
> lot of memory and filehandle consumption.  But I think it would
> increase article fetching speed quite a bit.
> 
> I've probably left out some details that are in my head.  Does this
> sound crazy?  Is there an easier way I'm overlooking?  Maybe manually
> running additional ubh processes and making ubhdb store it's state in
> the database so other processes can not conflict?
> 
> Ben
> 
> 
> 
> 


 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/ubh/

<*> To unsubscribe from this group, send an email to:
    [email protected]

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/