Re: TCP orig_log_iw_size='3', why?

--===============6064721889320263677==
Content-Type: multipart/alternative; boundary="0000000000006208fa062f5b1a73"

--0000000000006208fa062f5b1a73
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi,

The default behaviour for window sizing is to divide the window into equal
sized chunks for each of your possible connections.

if your max-connections() is set to 100 and your log-iw-size() is set to
300, then each of your potential log connection would end up having 3 slots
in their window. This can easily cause performance issues, therefore
syslog-ng increases the window size to the value specified by
min-iw-size-per-reader(), which defaults to 100.

The warning message basically means that your log-iw-size() seems low
compared to your max-connections() and syslog-ng is increasing the window
size, so that each connection has 100 slots in their window.

I'd recommend simply sizing max-connections() appropriately and then use a
window size proportionate to that. A window size of 100 per connection
should be good enough for moderate traffic.

With that said, here's a breakdown of each of the values in that log
message:
* orig_log_iw_size: the value your max-connections() and log-iw-size()
determines, in case you have max-connections(100) and log-iw-size(300),
that would yield 3 in this field (300/100)
* new_log_iw_size: the result of the clamping, e.g. since 3 is too small,
syslog-ng upped the per-connection log-iw-size() to 100 (which happens to
be the value of min_iw_size_per_reader() option)
* min_iw_size_per_reader: this is a configuration option that determines
the threshold of this warning, what size window is considered too small
* min_log_fifo_size: this is the amount of messages that can be in flight
with these settings, if all your connections (of which there's 300) fill
their own window allocation (100 each), you can have 30000 messages
in-flight.

This should be less than the log-fifo-size() of your destination (each of
them, if you are routing to multiple destinations), otherwise you could
start losing data.

The only downside of a higher number of in-flight messages is queue memory
(or disk) usage. So as long as you have enough of these, you can just
increase the log-iw-size() as recommended.

NOTE: There were a few related changes in syslog-ng after this change was
added:
* flow-controlled log paths never drop messages, even if their queues would
become full. They will over-subscribe their queues instead (e.g.
log-fifo-size() is ignored if the message is being forwarded along a flow
controlled log path), this means that log-fifo-size(), is only considered
for non-flow-controlled log paths.
* a dynamic-window() sizing feature was added, which allocates the window
to individual connections in a less strict manner.


Balazs

On Sat, Mar 1, 2025 at 8:40=E2=80=AFAM <[email protected]> wrote:

> Hello syslog-ng community,
>
> I get this warning at syslog-ng start (or restart) when running journalct=
l
>
> [...]
> Feb 27 02:46:57 xxxsplunksyslog1 syslog-ng[2998304]:
> [2025-02-27T02:46:57.330923] WARNING: window sizing for tcp sources were
> changed in syslog-ng 3.3, the configuration value was divided by the valu=
e
> of max-connections(). The result was too small, clamping to value of
> min_iw_size_per_reader. Ensure you have a proper log_fifo_size setting to
> avoid message loss.; orig_log_iw_size=3D'3', new_log_iw_size=3D'100',
> min_iw_size_per_reader=3D'100', min_log_fifo_size=3D'30000'
> Feb 27 02:46:57 xxxsplunksyslog1 systemd[1]: Started System Logger Daemon=
.
>
> I thought the orig_log_iw_size value was coming from the formula
> log_iw_size/max_connections which should be 100/300=3D1/3
> What is this orig_log_iw_size=3D'3' ?
>
> Thanks.
>
> _________________________________________________________________________=
_____
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=3Dsyslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>

--=20
Bazsi

--0000000000006208fa062f5b1a73
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi,</div><div><br></div><div>The default behaviour fo=
r window sizing is to divide the window into equal sized chunks for each of=
 your possible connections.=C2=A0</div><div><br></div><div>if your max-conn=
ections() is set to 100 and your log-iw-size() is set to 300, then each of =
your potential log connection would end up having 3 slots in their window. =
This can easily cause performance issues, therefore syslog-ng increases the=
 window size to the value specified by min-iw-size-per-reader(), which defa=
ults to 100.</div><div><br></div><div>The warning message basically means t=
hat  your log-iw-size() seems low compared to your max-connections() and sy=
slog-ng is increasing the window size, so that each connection has 100 slot=
s in their window.=C2=A0</div><br><div>I&#39;d recommend simply sizing max-=
connections() appropriately and then use a window size proportionate to tha=
t. A window size of 100 per connection should be good enough for moderate t=
raffic.</div><div><br></div><div>With that said, here&#39;s a breakdown of =
each of the values in that log message:</div><div>* orig_log_iw_size: the v=
alue your max-connections() and log-iw-size() determines, in case you have =
max-connections(100) and log-iw-size(300), that would yield 3 in this field=
 (300/100)</div><div>* new_log_iw_size: the result of the clamping, e.g. si=
nce 3 is too small, syslog-ng upped the per-connection log-iw-size() to 100=
 (which happens to be the value of min_iw_size_per_reader() option)<br></di=
v><div>* min_iw_size_per_reader: this is a configuration option that determ=
ines the threshold of this warning, what size window is considered too smal=
l</div><div>* min_log_fifo_size: this is the amount of messages that can be=
 in flight with these settings, if all your connections (of which there&#39=
;s 300) fill their own window allocation (100 each), you can have 30000 mes=
sages in-flight.</div><div><br></div><div>This should be less than the log-=
fifo-size() of your destination (each of them, if you are routing to multip=
le destinations), otherwise you could start losing data.</div><div><br></di=
v><div>The only downside of a higher number of in-flight messages is queue =
memory (or disk) usage. So as long as you have enough of these, you can jus=
t increase the log-iw-size() as recommended.</div><div><br></div><div>NOTE:=
 There were a few related changes in syslog-ng after this change was added:=
</div><div>* flow-controlled log paths never drop messages, even if their q=
ueues would become full. They will over-subscribe their queues instead (e.g=
. log-fifo-size() is ignored if the message is being forwarded along a flow=
 controlled log path), this means that log-fifo-size(), is only considered =
for non-flow-controlled log paths.</div><div>* a dynamic-window() sizing fe=
ature was added, which allocates the window to individual connections in a =
less strict manner.</div><div><br></div><div><br></div><div>Balazs</div></d=
iv><br><div class=3D"gmail_quote gmail_quote_container"><div dir=3D"ltr" cl=
ass=3D"gmail_attr">On Sat, Mar 1, 2025 at 8:40=E2=80=AFAM &lt;<a href=3D"ma=
ilto:[email protected]">[email protected]</a>&gt; wrote:<br></div><blockquote=
 class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px so=
lid rgb(204,204,204);padding-left:1ex"><div style=3D"font-size:10pt;font-fa=
mily:Verdana,Geneva,sans-serif">
<p>Hello syslog-ng community,<br> <br> I get this warning at syslog-ng star=
t (or restart) when running journalctl<br> <br> [...]<br> Feb 27 02:46:57 x=
xxsplunksyslog1 syslog-ng[2998304]: [2025-02-27T02:46:57.330923] WARNING: w=
indow sizing for tcp sources were changed in syslog-ng 3.3, the configurati=
on value was divided by the value of max-connections(). The result was too =
small, clamping to value of min_iw_size_per_reader. Ensure you have a prope=
r log_fifo_size setting to avoid message loss.; orig_log_iw_size=3D&#39;3&#=
39;, new_log_iw_size=3D&#39;100&#39;, min_iw_size_per_reader=3D&#39;100&#39=
;, min_log_fifo_size=3D&#39;30000&#39;<br> Feb 27 02:46:57 xxxsplunksyslog1=
 systemd[1]: Started System Logger Daemon.<br> <br> I thought the orig_log_=
iw_size value was coming from the formula log_iw_size/max_connections which=
 should be 100/300=3D1/3<br> What is this orig_log_iw_size=3D&#39;3&#39; ?<=
br> <br> Thanks.</p>
</div>
___________________________________________________________________________=
___<br>
Member info: <a href=3D"https://lists.balabit.hu/mailman/listinfo/syslog-ng=
" rel=3D"noreferrer" target=3D"_blank">https://lists.balabit.hu/mailman/lis=
tinfo/syslog-ng</a><br>
Documentation: <a href=3D"http://www.balabit.com/support/documentation/?pro=
duct=3Dsyslog-ng" rel=3D"noreferrer" target=3D"_blank">http://www.balabit.c=
om/support/documentation/?product=3Dsyslog-ng</a><br>
FAQ: <a href=3D"http://www.balabit.com/wiki/syslog-ng-faq" rel=3D"noreferre=
r" target=3D"_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div><div><br clear=3D"all"></div><br><span class=3D"gmail_si=
gnature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature">Ba=
zsi</div>

--0000000000006208fa062f5b1a73--

--===============6064721889320263677==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq


--===============6064721889320263677==--