Re: Single destination, multiple tcp connections?

On Fri, Mar 24, 2023, 18:05 Steve Bernacki <[email protected]> wrote:

> I am running syslog-ng 3.38.1.
>
> I have a scenario where I am receiving logs from hundreds devices over a
> number of different protocols (tcp, udp, tcp/TLS). I perform some
> filtering on these logs and then send them to one or more destinations
> for further processing. Due to the volume of logs being received, the
> destinations are becoming saturated and logs begin queuing up on my end.
> I have memory and disk buffering enabled, but the receiving end isn't
> able to pull logs off quickly enough since they are being funneled
> through a single TCP connection. I'd like to be able to establish some
> number of concurrent tcp connections to a single destination ip:port,
> and balance all of the incoming logs through those connections.
>
> I'm aware of techniques to load balance to destinations using multiple
> channels in a single destination and filtering traffic by the R_MSEC,
> but this technique quickly causes simple configs to balloon to hundreds
> of lines. I suppose this would work even if the destinations are all the
> same (I'd need to use unique persist-name() labels), but is there an
> easier / more straightforward way of accomplishing this?
>


Just one idea to throw around, you can generate syslog-ng config using
confgen (using a Shell script in earlier versions or a python function
since 4.0). This would mean that your primary config would only contain a
single destination and behind that  you can have the entire load balancing
logic.

Here's a sample for a python based confgen from the 4.0 announcement.

https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.0.1

Bazsi

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq