Re: [viewvc-dev] viewvc repository listing very slow

------=_Part_15347_1030996246.1445606768781
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Wolfgang,

Thanks for the well-written and detailed email. I can certainly understand
your situation and why a repeated parsing of the svn authz file would be
troublesome.  It never occurred to me that of all the places to find a
performance bottleneck, group parsing would be an issue for anyone.

The complicated part of this is, as you already noted, persistence -- too
little hurts performance, while too much hurts agility.

Using a global variable as you did solves your re-parsing problem, and
would probably work out okay for ViewVC when hosted as a CGI application
and where all the Subversion repositories use the same authz file.

Unfortunately, for a mod_python- or mod_wsgi-based instance, this creates a
circumstance where the Apache web server has to be restarted in order to
enact a change to the authz file. Subversion itself deems this an
unacceptable scenario, and so re-parses the authz file *at least* as often
as with each new Subversion-related network connection made to the Apache
server.

Additionally, ViewVC supports repos-relative authz paths (just as
Subversion itself does).  In such configurations, there are many authz
files (most commonly, one per repos) and so you have to be sure that your
cache doesn't bleed data where it shouldn't.  One authz file's group
definitions shouldn't be applied to a different authz file.

In answer to your question, though: yes, I am absolutely interested in
making ViewVC perform as well and as quickly as possible, and open to all
ideas as to how to do this without compromising data integrity, ideally
without making its already-obnoxious configuration file even larger. If you
(or anyone else) has some bright ideas as to how to make this happen,
please do share them!

-- Mike

On Fri, Oct 23, 2015 at 4:59 AM, Wolfgang Friebel <[email protected]>
wrote:

> We do have an svn server with about 400 repositories and a single authz
> access file containing about 600 group definitions. The file itself has
> about 8000 lines.
>
> When doing a repo listing the '_get_paths_for_root' method in
> viewvc/lib/vcauth/svnauthz/__init__.py is called for each repo. One call
> takes in our environment about 0.8 seconds. A user would have to wait
> therefore for the repo listing about 5 minutes if he is lucky, if not, the
> connection would time out.
>
> It would be very beneficial to reorganize the svnauthz plugin and parse
> the file only once keeping the parsed results in memory. As a quick and
> dirty hack and proof of concept I did cache the parsed group definitions
> (for version 1.1.24):
>
> --- __init__.py.orig    2015-10-22 20:42:19.457665386 +0200
> +++ __init__.py 2015-10-22 20:54:15.259056248 +0200
> @@ -78,8 +78,12 @@
>            aliases.append(alias)
>
>      # Figure out which groups USERNAME has a part of.
> -    groups = []
> -    if cp.has_section('groups'):
> +    global groups
> +    myuser = '__user__:' + (self.username or '')
> +    if not globals().has_key('groups') or groups[0] != myuser:
> +      groups = [None]
> +    if cp.has_section('groups') and groups[0] != myuser:
> +      groups = [myuser]
>        all_groups = []
>
>        def _process_group(groupname):
>
> That brought the time already down from about 5 minutes to 20 seconds.
> Of course using a 'global' definition and storing the username in the
> first element of groups is bad, but that was the quickest solution for me.
>
> Instead of repeatedly calling the function for all repository roots it
> would also be much more efficient to call it only once for all repo roots.
>
> Could something like that be foreseen in some future version of viewvc?
>
> ------------------------------------------------------
>
> http://viewvc.tigris.org/ds/viewMessage.do?dsForumId=4251&dsMessageId=3143575
>
> To unsubscribe from this discussion, e-mail: [
> [email protected]].
>

------------------------------------------------------
http://viewvc.tigris.org/ds/viewMessage.do?dsForumId=4251&dsMessageId=3143615

To unsubscribe from this discussion, e-mail: [[email protected]].
------=_Part_15347_1030996246.1445606768781
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Wolfgang,<div><br></div><div>Thanks for the well-written a=
nd detailed email. I can certainly understand your situation and why a repe=
ated parsing of the svn authz file would be troublesome.=C2=A0 It never occ=
urred to me that of all the places to find a performance bottleneck, group =
parsing would be an issue for anyone.<br></div><div><br></div><div>The comp=
licated part of this is, as you already noted, persistence -- too little hu=
rts performance, while too much hurts agility.</div><div><br></div><div>Usi=
ng a global variable as you did solves your re-parsing problem, and would p=
robably work out okay for ViewVC when hosted as a CGI application and where=
 all the Subversion repositories use the same authz file.</div><div><br></d=
iv><div>Unfortunately, for a mod_python- or mod_wsgi-based instance, this c=
reates a circumstance where the Apache web server has to be restarted in or=
der to enact a change to the authz file. Subversion itself deems this an un=
acceptable scenario, and so re-parses the authz file <i>at least</i> as oft=
en as with each new Subversion-related network connection made to the Apach=
e server.</div><div><br></div><div>Additionally, ViewVC supports repos-rela=
tive authz paths (just as Subversion itself does).=C2=A0 In such configurat=
ions, there are many authz files (most commonly, one per repos) and so you =
have to be sure that your cache doesn&#39;t bleed data where it shouldn&#39=
;t.=C2=A0 One authz file&#39;s group definitions shouldn&#39;t be applied t=
o a different authz file.</div><div><br></div><div>In answer to your questi=
on, though: yes, I am absolutely interested in making ViewVC perform as wel=
l and as quickly as possible, and open to all ideas as to how to do this wi=
thout compromising data integrity, ideally without making its already-obnox=
ious configuration file even larger. If you (or anyone else) has some brigh=
t ideas as to how to make this happen, please do share them!</div><div><br>=
</div><div>-- Mike</div></div><div class=3D"gmail_extra"><br><div class=3D"=
gmail_quote">On Fri, Oct 23, 2015 at 4:59 AM, Wolfgang Friebel <span dir=3D=
"ltr">&lt;<a href=3D"mailto:[email protected]" target=3D"_blank">wol=
[email protected]</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_q=
uote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1e=
x">We do have an svn server with about 400 repositories and a single authz =
access file containing about 600 group definitions. The file itself has abo=
ut 8000 lines.<br>
<br>
When doing a repo listing the &#39;_get_paths_for_root&#39; method in viewv=
c/lib/vcauth/svnauthz/__init__.py is called for each repo. One call takes i=
n our environment about 0.8 seconds. A user would have to wait therefore fo=
r the repo listing about 5 minutes if he is lucky, if not, the connection w=
ould time out.<br>
<br>
It would be very beneficial to reorganize the svnauthz plugin and parse the=
 file only once keeping the parsed results in memory. As a quick and dirty =
hack and proof of concept I did cache the parsed group definitions (for ver=
sion 1.1.24):<br>
<br>
--- __init__.py.orig=C2=A0 =C2=A0 2015-10-22 20:42:19.457665386 +0200<br>
+++ __init__.py 2015-10-22 20:54:15.259056248 +0200<br>
@@ -78,8 +78,12 @@<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0aliases.append(alias)<br>
<br>
=C2=A0 =C2=A0 =C2=A0# Figure out which groups USERNAME has a part of.<br>
-=C2=A0 =C2=A0 groups =3D []<br>
-=C2=A0 =C2=A0 if cp.has_section(&#39;groups&#39;):<br>
+=C2=A0 =C2=A0 global groups<br>
+=C2=A0 =C2=A0 myuser =3D &#39;__user__:&#39; + (self.username or &#39;&#39=
;)<br>
+=C2=A0 =C2=A0 if not globals().has_key(&#39;groups&#39;) or groups[0] !=3D=
 myuser:<br>
+=C2=A0 =C2=A0 =C2=A0 groups =3D [None]<br>
+=C2=A0 =C2=A0 if cp.has_section(&#39;groups&#39;) and groups[0] !=3D myuse=
r:<br>
+=C2=A0 =C2=A0 =C2=A0 groups =3D [myuser]<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0all_groups =3D []<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0def _process_group(groupname):<br>
<br>
That brought the time already down from about 5 minutes to 20 seconds.<br>
Of course using a &#39;global&#39; definition and storing the username in t=
he first element of groups is bad, but that was the quickest solution for m=
e.<br>
<br>
Instead of repeatedly calling the function for all repository roots it woul=
d also be much more efficient to call it only once for all repo roots.<br>
<br>
Could something like that be foreseen in some future version of viewvc?<br>
<br>
------------------------------------------------------<br>
<a href=3D"http://viewvc.tigris.org/ds/viewMessage.do?dsForumId=3D4251&amp;=
dsMessageId=3D3143575" rel=3D"noreferrer" target=3D"_blank">http://viewvc.t=
igris.org/ds/viewMessage.do?dsForumId=3D4251&amp;dsMessageId=3D3143575</a><=
br>
<br>
To unsubscribe from this discussion, e-mail: [<a href=3D"mailto:dev-unsubsc=
[email protected]">[email protected]</a>].<br>
</blockquote></div><br></div>

------=_Part_15347_1030996246.1445606768781--