[PATCH net-next RFC 0/6] netconsole: Support messages ratelimit-ing

Breno Leitao <[email protected]>
Newsgroups org.kernel.vger.linux-doc,org.kernel.vger.linux-kernel,org.kernel.vger.linux-kselftest,org.kernel.vger.netdev
Message-ID <[email protected]>
netconsole hands every console message to every enabled target, with no
bound on the rate.

At Meta, a few hosts caused some DoS and netconsd was OOM killed in
some regions, fed by 3 billion RCU messages from thousand hosts and by
a swap fault that retries forever, which logged 6 million copies of one
line on a single host in a few hours.

Both floods are being fixed where they are printed, by Paula and me for RCU
Tasks [1][2][3] and by me for swap faul fix [4], but that only ever
covers the flood already found, not new issues that might be happening.

Use the ratelimit in the kernel to ratelimit messages from netconsole.

Give each target a token bucket, consulted once per message so that a
message split into several ncfrag packets is sent whole or dropped
whole.

It starts with a zero interval, which struct ratelimit_state treats as
unlimited, so nothing changes until ratelimit_interval_ms and
ratelimit_burst are written.

What the bucket discards is reported on the wire, so the receiver sees
"netconsole: 45 messages dropped by rate limit" rather than an
unexplained gap. This is what is captured on netconsd, when this patch
is applied:

	7.2.0-rc7-01460-g75848acaf136,13,628,252392647,-;netconsole selftest: netcons_AGujw 1
	7.2.0-rc7-01460-g75848acaf136,13,629,252392782,-;netconsole selftest: netcons_AGujw 2
	....
	7.2.0-rc7-01460-g75848acaf136,4,0,254742176,-;netconsole: 45 messages dropped by rate limit
	7.2.0-rc7-01460-g75848acaf136,13,678,254742125,-;netconsole selftest: netcons_AGujw 1

Crash output is exempt. The bucket is skipped while oops_in_progress is
set, so a limit configured for steady-state logging cannot cost a target
part of an oops or a panic.

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://lore.kernel.org/all/[email protected]/

Signed-off-by: Breno Leitao <[email protected]>
---
Breno Leitao (6):
      netconsole: add a per-target message rate limit
      netconsole: allow configuring the rate limit interval through configfs
      netconsole: allow configuring the rate limit burst through configfs
      netconsole: tell the target when the rate limit drops messages
      docs: netconsole: document rate limit feature
      selftests: netcons: test the per-target rate limit

 Documentation/networking/netconsole.rst            |  43 ++++++
 drivers/net/netconsole.c                           | 145 +++++++++++++++++++
 .../selftests/drivers/net/netconsole/Makefile      |   1 +
 .../drivers/net/netconsole/netcons_ratelimit.sh    | 160 +++++++++++++++++++++
 4 files changed, 349 insertions(+)
---
base-commit: e6a5d573d24cd375e09d24f136523cb3cc85c9d3
change-id: 20260817-netcons_ratelimit-629b04a73c57

Best regards,
--  
Breno Leitao <[email protected]>
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.