Fwd: [squid-users] Scaling concurrent TCP sessions beyond ephemeral port range

Hi,

Moving this thread to the squid-dev list as the PR below has been opened to
propose a fix to the ephemeral port exhaustion problem we had experienced
with ver 5.5.
https://github.com/squid-cache/squid/pull/1115

Thanks to Alex and others for providing pointers in this regards. Apologies
if I have missed out any requirements on the PR, please let me know and I
can update as needed. The fix proposed is based on the post below from
Cloudflare that had experienced this issue with their own applications.
https://blog.cloudflare.com/how-to-stop-running-out-of-ephemeral-ports-and-start-to-love-long-lived-connections/

We were hitting the errors stated on the PR as soon as the system exhausted
1x ip_local_port_range concurrent TCP sessions. With this fix, we have been
able to scale well beyond that ( > 100k concurrent outbound TCP sessions).
This patch has been running in our production environment for close to 2
months without any new issues.

Please let me know if you have any questions, and again apologize if I
forgot to include something before or with the PR.

Thanks
Praveen

---------- Forwarded message ---------
From: Alex Rousskov <[email protected]>
Date: Tue, Jun 21, 2022 at 2:11 PM
Subject: Re: [squid-users] Scaling concurrent TCP sessions beyond ephemeral
port range
To: <[email protected]>


On 6/19/22 12:48, Praveen Ponakanti wrote:

> What is the process to have this code patch upstreamed for future squid
> versions?

In short, just post a quality pull request on GitHub (or find somebody
who can guide your code towards official acceptance for you). For
details, please see https://wiki.squid-cache.org/MergeProcedure


Thank you,

Alex.


> On Fri, May 20, 2022 at 9:31 PM Amos Jeffries <[email protected]
> wrote:
>
>     On 20/05/22 19:44, Praveen Ponakanti wrote:
>      > Hi Alex,
>      >
>      > Thanks for going through several steps to help mitigate src port
>      > exhaustion. We are looking to achieve 400-500% more
>      > concurrent connections if we could :) as there is a
>     significant buffer
>      > on the available CPU.
>
>     Then you require at least 4, maybe 5, IP addresses to handle that many
>     concurrent connections with Squid.
>
>
> We would like to investigate going beyond the ephemeral port range for
> some specific destination IP:PORT addresses. For that it appears squid
> does not round-robin requests if we use multiple tcp_outgoing_addresses.
> We could use ACL’s to pick a different outbound IP based on the clients
> source IP, however that is not very ideal in our environment as our
> clients aren’t always equally split by subnet. However, if we could
> split by the client’s source port that might help achieve this. For
> example something like:
>
>
> acl pool1 clientport 0-32768
>
> acl pool2 clientport 32769-65536
>
>
> tcp_outgoing_address 10.1.0.1 pool1
>
> tcp_outgoing_address 10.1.0.2 pool2
>
>
> Squid's ACLs currently do not allow filtering by the client's source
> port. We could look into a separate patch to add this functionality to
> squid’s ACL code if that makes sense. Or is there a better way to
> achieve this?
>
>
> Thanks
>
> Praveen
>
>
>      > The option to use multiple tcp_outoing_addresses appears to be
>     promising
>      > along with some tweaks to the TCP timeouts. I guess we could use
>     ACLs to
>      > pick a different outbound IP based on the requesting client's
>     prefix. We
>      > had not considered that option as the ephemeral ports were no
longer
>      > available to other applications when squid uses most of them with a
>      > single outbound IP configured. We are also looking to modify the
>     code to
>      > use the IP_BIND_ADDRESS_NO_PORT sockopt as that could help delay
>     port
>      > assignment with the bind() call on the outbound TCP sessions (to
>      > hopefully allow access to the 4-tuple on the socket).
>
>     Patches welcome.
>
>     However, please be aware that use of the 4-tuple is often no different
>     from the 3-tuple since the dst-port is typically identical for all
>     outgoing traffic to a given dst-IP.
>
>
>     Cheers
>     Amos
>     _______________________________________________
>     squid-users mailing list
>     [email protected]
>     <mailto:[email protected]>
>     http://lists.squid-cache.org/listinfo/squid-users
>     <http://lists.squid-cache.org/listinfo/squid-users>
>
>
> _______________________________________________
> squid-users mailing list
> [email protected]
> http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[email protected]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-dev mailing list
[email protected]
http://lists.squid-cache.org/listinfo/squid-dev