Re: TODO 157

--===============5360544299095567940==
Content-Type: multipart/signed; micalg=pgp-sha256;
	boundary="Sig_/df0CvXPU6lLFD3iyxq.h0UZ";
	protocol="application/pgp-signature"

--Sig_/df0CvXPU6lLFD3iyxq.h0UZ
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable

CC and Reply-To set to privoxy-devel@ as some people have
no ijbswa-developers@ subscription (anymore).

Lee <[email protected]> wrote on ijbswa-developers@:

> > Add #157: Add a directive to control the currently hardcoded
> > receive-buffer size =20
>=20
> Doesn't setting the receive buffer requires a setsockopt call with
> either SO_SNDBUF or SO_RCVBUF?  I don't see it.
> Or do you mean the
>   char buf[BUFFER_SIZE];
> in jcc.c:handle_established_connection changes ?  If that's it,

I mean the latter.
=20
> - what do you hope to gain making it a user-controlled size

The "best" size depends on the environment Privoxy runs in
and on the Privoxy admin's requirements so hard-coding it
complicates tuning.

=46rom the commit message in my local tree (which will change
before I commit it):

    Add a receive-buffer-size directive
   =20
    ... that can be used to set the size of the previously statically
    allocated buffer in handle_established_connection().
   =20
    Increasing the buffer size increases Privoxy's memory usage but
    can lower the number of context switches and thereby reduce the
    cpu usage and potentially increase the throughput.
   =20
    This is mostly relevant for fast network connections and
    large downloads that don't require filtering.
   =20
    Currently BUFFER_SIZE is kept as default and lower limit.
   =20
    We should probably change the default to 16384, though,
    while some users may further increase it to 32768 or 65536.
   =20
    A dtrace command like:
    sudo dtrace -n 'syscall::read:return /execname =3D=3D "privoxy"/ { @[ex=
ecname] =3D llquantize(arg0, 10, 0, 5, 20); @m =3D max(arg0)}'
    can be used to properly tune the receive-buffer-size.
   =20
    If the buffer is too large it will increase Privoxy's memory
    footprint without any benefit. As the memory is (currently)
    cleared before using it, a buffer that is too large can
    actually reduce the throughput.
   =20
    Things could be improved further by upwards scaling the buffer
    dynamically based on how much of the previous allocation
    was actually used.
   =20
    Additionally the buffer should be referenced through
    csp and also be used for other receive-related functions.
   =20
    Measured throughput when using four connections to
    constantly request a 10 MB file:
   =20
    ~320 MB/s with the default
    ~400 MB/s with "receive-buffer-size    8192"
    ~490 MB/s with "receive-buffer-size   16384"
    ~610 MB/s with "receive-buffer-size   32768"
    ~700 MB/s with "receive-buffer-size   65536"
    ~755 MB/s with "receive-buffer-size  131072"
    ~795 MB/s with "receive-buffer-size  262144"
    ~804 MB/s with "receive-buffer-size  524288"
    ~798 MB/s with "receive-buffer-size 1048576"
    ~780 MB/s with "receive-buffer-size 2097152"
   =20
    Sponsored by: Robert Klemme

> - how do you feel about using a GNU extension .. at least initially,
> to see if the change is worth the effort?

I already know that the change is worth the (quite moderate) effort.

> https://www.gnu.org/software/gnu-c-manual/gnu-c-manual.html#Declaring-Arr=
ays
>   Another GNU extension allows you to declare an array size using
> variables, rather than only constants.

I don't like GNU extensions in general but in this case a stack-allocated
array is also inappropriate as the "best" buffer size for many environments
would likely result in Privoxy reaching the stack size limit when there are
more than a couple of threads at the same time.

Fabian

--Sig_/df0CvXPU6lLFD3iyxq.h0UZ
Content-Type: application/pgp-signature
Content-Description: OpenPGP digital signature

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEETzbBfzgWkTZUoehQaRgikYuiNxwFAlklTuMACgkQaRgikYui
NxyNCQ/9Hb50GTJ4mI36YP6zJBRrLrtH+7gHXbKmkbI9HiUwswZ1Vdevtf8EaCER
O6wRXZCtedeIcnAfdGs3VI+RxJZrnTs6ncnTLbpW0hJJDXgs1HHg1xUr7fYgPwK+
nIfXC6O8xfDtDeykhVi+reC/QWPtXammRIFphx3Xxe38+nD5NxHLDS74jFJpvpp7
vhWVsf75GD7SM9M1P3rmSO2+uabEjlVQNAbU3OOinK3wbzfSXDhx7GBq/T3BJFKq
KQYJpO8IaR6uca7ugwiXo5tGVHHymKi/5uoqH5fnpB9FzZw69xpw8/cpLZTwhR/f
ZwyGECLOixMpS77XDnFCKORk/qDsyIoLkkGKENbhpbg6BXuGs04YxOYunZ+B21K4
dFtDJhKUmrN+4WtROl2M0fWAJQjR6BuJGm//X5CZrfz1iQ2a6Ui7/9hqOkbnTad2
MJDZcWnBH5DnmpWwmbOrOzWVRuvdbVka0cZa9mMEOjm5D9ThXDxL1mwXY2WFusFr
kvYL8fdfw4O14XXtYMs5c6daw3Xmjkk4Je+BBhfWt3BXXXY6B2HH5Cp56USy64Pi
ARufFcegH8Iz1TfdeN3VVu+6Q6rZTy1Ktg+qcQR5pX5atuMRJLWXp+zDLtrROlbY
2M8WdUKovHTk96k7aA8EGtbFYsjoV+mv6OiHMw/6oZH1fyNIXrs=
=jAx4
-----END PGP SIGNATURE-----

--Sig_/df0CvXPU6lLFD3iyxq.h0UZ--


--===============5360544299095567940==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
--===============5360544299095567940==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Ijbswa-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ijbswa-developers

--===============5360544299095567940==--