Re: TCP orig_log_iw_size='3', why?
[email protected] Sun, 02 Mar 2025 14:40:55 +0100
| Newsgroups | gmane.comp.syslog-ng |
|---|---|
| Message-ID | <[email protected]> |
--===============8134942796657565920==
Content-Type: multipart/alternative;
boundary="=_7fa6514690398cb715b3eb2e194e921e"
--=_7fa6514690398cb715b3eb2e194e921e
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8
Hello Balazs,
thanks for your in depth answer, I sincerely appreciate your analysis
and the time you spent in it. Really thanks.
I have one doubt though, in my configuration I have max-connections(300)
defined in this source:
source s_nas {
network(ip(X.X.X.X) transport(tcp) port(514));
network(ip(X.X.X.X) transport(udp) port(514) so-rcvbuf(26214400));
tcp(ip(X.X.X.X) port(44514) max-connections(300) tls(
key_file("/etc/syslog-ng/key.d/priv.key")
cert_file("/etc/syslog-ng/cert.d/cert.pem")
peer_verify(optional-untrusted))
);
and log-iw-size is not specified so it should be using the default
(100?). I tried to get info on the default value for log-iw-size but in
the administration manual the default value is left blank (am I looking
in the wrong place? maybe).
Anyway, on 3 different webinars available online I saw the log-iw-size
defaults to 100 hence I would expect that
orig_log_iw_size = log-iw-size / max-connections = 100/300 = 1/3
that should (could) be approximated with 0 or maybe 1 but instead I get
orig_log_iw_size=3 as the startup log shows this:
Feb 27 02:46:57 xxxsplunksyslog1 syslog-ng[2998304]:
[2025-02-27T02:46:57.330923] WARNING: window sizing for tcp sources were
changed in syslog-ng 3.3, the configuration value was divided by the
value of max-connections(). The result was too small, clamping to value
of min_iw_size_per_reader. Ensure you have a proper log_fifo_size
setting to avoid message loss.; orig_log_iw_size='3',
new_log_iw_size='100', min_iw_size_per_reader='100',
min_log_fifo_size='30000'
Can you please explain?
Thanks.
Claudio
Il 2025-03-02 13:22 Balazs Scheidler ha scritto:
> Hi,
>
> The default behaviour for window sizing is to divide the window into equal sized chunks for each of your possible connections.
>
> if your max-connections() is set to 100 and your log-iw-size() is set to 300, then each of your potential log connection would end up having 3 slots in their window. This can easily cause performance issues, therefore syslog-ng increases the window size to the value specified by min-iw-size-per-reader(), which defaults to 100.
>
> The warning message basically means that your log-iw-size() seems low compared to your max-connections() and syslog-ng is increasing the window size, so that each connection has 100 slots in their window.
>
> I'd recommend simply sizing max-connections() appropriately and then use a window size proportionate to that. A window size of 100 per connection should be good enough for moderate traffic.
>
> With that said, here's a breakdown of each of the values in that log message:
> * orig_log_iw_size: the value your max-connections() and log-iw-size() determines, in case you have max-connections(100) and log-iw-size(300), that would yield 3 in this field (300/100)
> * new_log_iw_size: the result of the clamping, e.g. since 3 is too small, syslog-ng upped the per-connection log-iw-size() to 100 (which happens to be the value of min_iw_size_per_reader() option)
> * min_iw_size_per_reader: this is a configuration option that determines the threshold of this warning, what size window is considered too small
> * min_log_fifo_size: this is the amount of messages that can be in flight with these settings, if all your connections (of which there's 300) fill their own window allocation (100 each), you can have 30000 messages in-flight.
>
> This should be less than the log-fifo-size() of your destination (each of them, if you are routing to multiple destinations), otherwise you could start losing data.
>
> The only downside of a higher number of in-flight messages is queue memory (or disk) usage. So as long as you have enough of these, you can just increase the log-iw-size() as recommended.
>
> NOTE: There were a few related changes in syslog-ng after this change was added:
> * flow-controlled log paths never drop messages, even if their queues would become full. They will over-subscribe their queues instead (e.g. log-fifo-size() is ignored if the message is being forwarded along a flow controlled log path), this means that log-fifo-size(), is only considered for non-flow-controlled log paths.
> * a dynamic-window() sizing feature was added, which allocates the window to individual connections in a less strict manner.
>
> Balazs
>
> On Sat, Mar 1, 2025 at 8:40 AM <[email protected]> wrote:
>
>> Hello syslog-ng community,
>>
>> I get this warning at syslog-ng start (or restart) when running journalctl
>>
>> [...]
>> Feb 27 02:46:57 xxxsplunksyslog1 syslog-ng[2998304]: [2025-02-27T02:46:57.330923] WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to value of min_iw_size_per_reader. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='3', new_log_iw_size='100', min_iw_size_per_reader='100', min_log_fifo_size='30000'
>> Feb 27 02:46:57 xxxsplunksyslog1 systemd[1]: Started System Logger Daemon.
>>
>> I thought the orig_log_iw_size value was coming from the formula log_iw_size/max_connections which should be 100/300=1/3
>> What is this orig_log_iw_size='3' ?
>>
>> Thanks. ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng [1]
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng [2]
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq [3]
>
> --
>
> Bazsi
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng [1]
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng [2]
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq [3]
Links:
------
[1] https://lists.balabit.hu/mailman/listinfo/syslog-ng
[2] http://www.balabit.com/support/documentation/?product=syslog-ng
[3] http://www.balabit.com/wiki/syslog-ng-faq
--=_7fa6514690398cb715b3eb2e194e921e
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset=UTF-8
<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; charset=
=3DUTF-8" /></head><body style=3D'font-size: 10pt; font-family: Verdana,Gen=
eva,sans-serif'>
<p>Hello Balazs,</p>
<p>thanks for your in depth answer, I sincerely appreciate your analysis an=
d the time you spent in it. Really thanks.</p>
<p>I have one doubt though, in my configuration I have max-connections(300)=
defined in this source:</p>
<pre><span style=3D"font-family: courier new, courier, monospace; font-size=
: 10pt;">source s_nas {
network(ip(X.X.X.X) transport(tcp) port(514));
network(ip(X.X.X.X) transport(udp) port(514) so-rcvbuf(26214400));
tcp(ip(X.X.X.X) port(44514) max-connections(300) tls(
key_file("/etc/syslog-ng/key.d/priv.key")
cert_file("/etc/syslog-ng/cert.d/cert.pem")
peer_verify(optional-untrusted))
); </span></pre>
<p>and log-iw-size is not specified so it should be using the default (100?=
). I tried to get info on the default value for log-iw-size but in the admi=
nistration manual the default value is left blank (am I looking in the wron=
g place? maybe).</p>
<p>Anyway, on 3 different webinars available online I saw the log-iw-size d=
efaults to 100 hence I would expect that </p>
<p><span style=3D"font-family: courier new, courier, monospace;">orig_log_i=
w_size =3D log-iw-size / max-connections =3D 100/300 =3D 1/3</span> </=
p>
<p>that should (could) be approximated with 0 or maybe 1 but instead I get =
orig_log_iw_size=3D3 as the startup log shows this:</p>
<pre><span style=3D"font-family: courier new, courier, monospace; font-size=
: 10pt;">Feb 27 02:46:57 xxxsplunksyslog1 syslog-ng[2998304]: [2025-02-27T0=
2:46:57.330923] WARNING: window sizing for tcp sources were changed in sysl=
og-ng 3.3, the configuration value was divided by the value of max-connecti=
ons(). The result was too small, clamping to value of min_iw_size_per_reade=
r. Ensure you have a proper log_fifo_size setting to avoid message loss.; o=
rig_log_iw_size=3D'3', new_log_iw_size=3D'100', min_iw_size_per_reader=3D'1=
00', min_log_fifo_size=3D'30000'</span></pre>
<p>Can you please explain?</p>
<p>Thanks.</p>
<p>Claudio</p>
<div> </div>
<p> </p>
<p>Il 2025-03-02 13:22 Balazs Scheidler ha scritto:</p>
<blockquote type=3D"cite" style=3D"padding: 0 0.4em; border-left: #1010ff 2=
px solid; margin: 0"><!-- html ignored --><!-- head ignored --><!-- meta ig=
nored -->
<div dir=3D"ltr">
<div>Hi,</div>
<div> </div>
<div>The default behaviour for window sizing is to divide the window into e=
qual sized chunks for each of your possible connections. </div>
<div> </div>
<div>if your max-connections() is set to 100 and your log-iw-size() is set =
to 300, then each of your potential log connection would end up having 3 sl=
ots in their window. This can easily cause performance issues, therefore sy=
slog-ng increases the window size to the value specified by min-iw-size-per=
-reader(), which defaults to 100.</div>
<div> </div>
<div>The warning message basically means that your log-iw-size() seems low =
compared to your max-connections() and syslog-ng is increasing the window s=
ize, so that each connection has 100 slots in their window. </div>
<br />
<div>I'd recommend simply sizing max-connections() appropriately and then u=
se a window size proportionate to that. A window size of 100 per connection=
should be good enough for moderate traffic.</div>
<div> </div>
<div>With that said, here's a breakdown of each of the values in that log m=
essage:</div>
<div>* orig_log_iw_size: the value your max-connections() and log-iw-size()=
determines, in case you have max-connections(100) and log-iw-size(300), th=
at would yield 3 in this field (300/100)</div>
<div>* new_log_iw_size: the result of the clamping, e.g. since 3 is too sma=
ll, syslog-ng upped the per-connection log-iw-size() to 100 (which happens =
to be the value of min_iw_size_per_reader() option)</div>
<div>* min_iw_size_per_reader: this is a configuration option that determin=
es the threshold of this warning, what size window is considered too small<=
/div>
<div>* min_log_fifo_size: this is the amount of messages that can be in fli=
ght with these settings, if all your connections (of which there's 300) fil=
l their own window allocation (100 each), you can have 30000 messages in-fl=
ight.</div>
<div> </div>
<div>This should be less than the log-fifo-size() of your destination (each=
of them, if you are routing to multiple destinations), otherwise you could=
start losing data.</div>
<div> </div>
<div>The only downside of a higher number of in-flight messages is queue me=
mory (or disk) usage. So as long as you have enough of these, you can just =
increase the log-iw-size() as recommended.</div>
<div> </div>
<div>NOTE: There were a few related changes in syslog-ng after this change =
was added:</div>
<div>* flow-controlled log paths never drop messages, even if their queues =
would become full. They will over-subscribe their queues instead (e.g. log-=
fifo-size() is ignored if the message is being forwarded along a flow contr=
olled log path), this means that log-fifo-size(), is only considered for no=
n-flow-controlled log paths.</div>
<div>* a dynamic-window() sizing feature was added, which allocates the win=
dow to individual connections in a less strict manner.</div>
<div> </div>
<div> </div>
<div>Balazs</div>
</div>
<br />
<div class=3D"gmail_quote gmail_quote_container">
<div class=3D"gmail_attr" dir=3D"ltr">On Sat, Mar 1, 2025 at 8:40=E2=80=AFA=
M <<a href=3D"mailto:[email protected]">[email protected]</a>> wrote:</=
div>
<blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px 0px 0.8ex; borde=
r-left: 1px solid #cccccc; padding-left: 1ex;">
<div style=3D"font-size: 10pt; font-family: Verdana,Geneva,sans-serif;">
<p>Hello syslog-ng community,<br /><br /> I get this warning at syslog-ng s=
tart (or restart) when running journalctl<br /><br /> [...]<br /> Feb 27 02=
:46:57 xxxsplunksyslog1 syslog-ng[2998304]: [2025-02-27T02:46:57.330923] WA=
RNING: window sizing for tcp sources were changed in syslog-ng 3.3, the con=
figuration value was divided by the value of max-connections(). The result =
was too small, clamping to value of min_iw_size_per_reader. Ensure you have=
a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size=3D=
'3', new_log_iw_size=3D'100', min_iw_size_per_reader=3D'100', min_log_fifo_=
size=3D'30000'<br /> Feb 27 02:46:57 xxxsplunksyslog1 systemd[1]: Started S=
ystem Logger Daemon.<br /><br /> I thought the orig_log_iw_size value was c=
oming from the formula log_iw_size/max_connections which should be 100/300=
=3D1/3<br /> What is this orig_log_iw_size=3D'3' ?<br /><br /> Thanks.</p>
</div>
___________________________________________________________________________=
___<br /> Member info: <a href=3D"https://lists.balabit.hu/mailman/listinfo=
/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br /> D=
ocumentation: <a href=3D"http://www.balabit.com/support/documentation/?prod=
uct=3Dsyslog-ng">http://www.balabit.com/support/documentation/?product=3Dsy=
slog-ng</a><br /> FAQ: <a href=3D"http://www.balabit.com/wiki/syslog-ng-faq=
">http://www.balabit.com/wiki/syslog-ng-faq</a><br /><br /></blockquote>
</div>
<div> </div>
<br /><span class=3D"gmail_signature_prefix">-- </span><br />
<div class=3D"gmail_signature" dir=3D"ltr">Bazsi</div>
<br />
<div class=3D"pre" style=3D"margin: 0; padding: 0; font-family: monospace">=
___________________________________________________________________________=
___<br /> Member info: <a href=3D"https://lists.balabit.hu/mailman/listinfo=
/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br /> D=
ocumentation: <a href=3D"http://www.balabit.com/support/documentation/?prod=
uct=3Dsyslog-ng">http://www.balabit.com/support/documentation/?product=3Dsy=
slog-ng</a><br /> FAQ: <a href=3D"http://www.balabit.com/wiki/syslog-ng-faq=
">http://www.balabit.com/wiki/syslog-ng-faq</a><br /><br /></div>
</blockquote>
</body></html>
--=_7fa6514690398cb715b3eb2e194e921e--
--===============8134942796657565920==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq
--===============8134942796657565920==--