Re: troll performance questions

Zitat von Eric Rostetter <[email protected]>:

> Okay, here to resurrect an old thread.  From the archives we have:
> 
> > Quoting Jan Schneider <[email protected]>:
> >
> > > If I understand Tree.php right we get at startup the first _and_ the
> > > complete second level of the newsgroups hierarchy from the server.
> That
> > > takes a lot of time and leads to a timeout on my poor little server.
> It
> > > is possible to only get one level and the information if there is
> > > another level below (I tested this), and to retrieve the next level
> of
> > > the selected branch only on request.
> >
> > This follows the logic from IMP, where we need information on the level
> > below to make sure we can accurately tell which folders have
> subfolders. I
> > might be able to make it work the other way; no idea when I could get
> to
> > it, though.
> 
> The problem is real.  I have a pretty good sized feed.  When I enter
> troll
> for the first time in the session, the building of the tree takes about
> 4 minutes, which is too long for most users to wait.  During those 4
> minutes
> it keeps one CPU (on a multi-cpu machine) running at 99%.  This means
> that just a few users hitting troll in the same time period can cause a
> denial of service on my whole Horde site -- Not a good thing!

Funny that this comes up on several places at the same time. We talked about
it shortly on IRC yesterday and I added a task to Nag on dev.horde.org.
 
> So, here are some ideas, put here for discussion.
> 
> * We could do as said above.  Of the 4 minutes, about 2 minutes are spent
> in the imap_getmailboxes() functions (called twice, once for % and once
> for %.%).  If we could do this in one call, we would cut the hit by 25%
> (from 4 minutes down to 3 minutes).

As I said in the quoted thread but perhaps didn't make clear it is possible
to _only_ fetch the top level hierarchy. The list we get already contains
the information if there are more levels below one node. That should
decrease the time even more as the search for %.% is completely dropped.
 
> * Maybe we could cache the newsgroup lists locally?  So the first person
> to
> hit it (cache empty or expired) would have to wait the 4 minutes, but
> then
> everyone else would use the cached values?  After all, the group listing
> won't
> change that often, and isn't real critical, and won't take up much space
> to
> cache...  Would this work?  Not sure how this would work with
> subscriptions...

That sounds like a very good idea. It won't help in my case as I am the only
user on my server, but ... ;-)
Subscriptions shouldn't be a problem because they have to be handled locally
if cache is used. That means that the first user always gets the whole list,
no matter if he turned on subscriptions, but we show him only his subscribed
groups.

> * Could this be something in c-client or php that someone could optimize
> there?  Seems strange to me that it consumes so much cpu just to get a
> list
> of newsgroups and its attributes...

Don't know about this one, but another thought. All newsgroup clients that I
know of are capable of retrieving the list of newsgroups that have been
added to the server since the last time. (That means that they do exactly
the sort of caching we are thinking about)
Is there any functionality inside the c-client to support such a task? Does
anybody know how other news clients handle this?

Jan.

--
http://www.horde.org - The Horde Project
http://www.ammma.de - discover your knowledge
http://www.tip4all.de - Deine private Tippgemeinschaft

-- 
Troll mailing list
Frequently Asked Questions: http://horde.org/faq/
To unsubscribe, mail: [email protected]