Cannot disable kernel's audit system via auditctl

--===============8792915135381254125==
Content-Type: multipart/alternative; boundary="000000000000c0b3c305ff4c4c45"

--000000000000c0b3c305ff4c4c45
Content-Type: text/plain; charset="UTF-8"

Hi linux-audit,

I'm running a fleet of Linux hosts with Red Canary Linux EDR (Endpoint
Detection and Response) which uses eBPF for gathering telemetry in service `
cfsvcd.service`. In an older configuration, it gathered data from the
kernel's audit system and everything was fine. However, when we switched
cfsvcd to gathering data from eBPF instead, we noticed that the kernel ring
buffer was flooded with audit messages. This is because cfsvcd.service now
stops auditd.service, but leaves the kernel audit system enabled.

I've mitigated this issue by manually running `# auditctl -e 0` on our
hosts (via Puppet). However, I'm running into a strange issue where _some_
hosts (~0.5%) are still logging all audit events to the kernel ring buffer
even after I have disabled the audit system via `# auditctl -e 0`. A `#
auditctl -s` run shows `enabled 0`, yet audit logs continue to flood the
kernel ring buffer.

I'm running Linux kernel 5.4.0-1063-aws on Ubuntu 18.04 with auditctl
v2.8.2. `systemd-journald-audit.socket` is masked & inactive, `
auditd.service` is disabled & inactive.

I cannot entirely disable the audit system via a kernel parameter because
Red Canary Linux EDR fails to start cfsvcd.service as it fails to run some
auditctl command due to no audit support in the kernel:

Jun 28 20:41:04 systemd[1]: Started Canary Forwarder Service.
> Jun 28 20:41:04 cfsvcd[105781]: Found config file at path
> '"/opt/redcanary/config.json"', continuing execution...
> Jun 28 20:41:12 cfsvcd[105781]: component: Sentry
> Jun 28 20:41:12 cfsvcd[105781]:  Jun 28 20:41:12.055 INFO Initializing
> Sentry Config { version: "1.4.17.release.[...]", https_proxy: None,
> extra_tags: [("telemetry.source", "eBPF")], logging_enabled: true,
> metrics_enabled: true, [...]}
> Jun 28 20:41:12 cfsvcd[105781]: Failed to setup or configure host system:
> Linux Audit watcher failure: Netlink Error: IO Error: Protocol not
> supported (os error 93)
> Jun 28 20:41:12 cfsvcd[105781]: Received stop. Exiting
> unix::wait_for_signal. Signal: 0, done.is_signaled(): true
> Jun 28 20:41:12 cfsvcd[105781]: Joining async_agent_thread
> Jun 28 20:41:13 cfsvcd[105781]: Attempting to stop subscriber (Ebpf).
> Jun 28 20:41:13 cfsvcd[105781]: Shutting down due to termination signal
> [...]
> Jun 28 20:41:15 cfsvcd[105781]: Subscriber stopped.
> Jun 28 20:41:15 auditctl[106733]: Error - audit support not in kernel
> Jun 28 20:41:15 auditctl[106733]: Cannot open netlink audit socket
> Jun 28 20:41:15 systemd[1]: cfsvcd.service: Service hold-off time over,
> scheduling restart.
> Jun 28 20:41:15 systemd[1]: cfsvcd.service: Scheduled restart job, restart
> counter is at 301.
> Jun 28 20:41:15 systemd[1]: Stopped Canary Forwarder Service.
>

Here's the unit file for cfsvcd.service:

> [Unit]
> Description=Canary Forwarder Service
>
> DefaultDependencies=no
> After=local-fs.target systemd-tmpfiles-setup.service auditd.service
> Before=sysinit.target shutdown.target
> # Replace the auditd service if it is running
> Conflicts=auditd.service shutdown.target
>
> [Service]
> WorkingDirectory=/opt/redcanary
> ExecStart=/opt/redcanary/cfsvcd
> ExecStopPost=-/sbin/auditctl -D
> Restart=always
> TimeoutSec=15
>
> [Install]
> WantedBy=multi-user.target
>

Is this a known issue? Is there a workaround to stop the logging to the
kernel ring buffer? Is there any more information I can provide to help
debug?

Thanks,
Samuel Bahr
Pinterest Sr. Site Reliability Engineer

--000000000000c0b3c305ff4c4c45
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi linux-audit,<div><br></div><div>I&#39;m running a fleet=
 of Linux hosts with Red Canary Linux EDR (Endpoint Detection and Response)=
 which uses eBPF for gathering telemetry in=C2=A0service `<font face=3D"mon=
ospace" style=3D"background-color:rgb(238,238,238)">cfsvcd.service</font>`.=
 In an older configuration, it gathered data from the kernel&#39;s audit sy=
stem and everything was fine. However, when we switched cfsvcd to gathering=
 data from eBPF instead, we noticed that the kernel ring buffer was flooded=
 with audit messages. This is because cfsvcd.service now stops auditd.servi=
ce, but leaves the kernel audit system enabled.</div><div><br></div><div>I&=
#39;ve=C2=A0mitigated this issue by manually running `<font face=3D"monospa=
ce" style=3D"background-color:rgb(238,238,238)"># auditctl -e 0</font>` on =
our hosts (via Puppet). However, I&#39;m running into a strange issue where=
 _some_ hosts (~0.5%) are still logging all audit events to the kernel ring=
 buffer even after I have disabled the audit system via `<font face=3D"mono=
space" style=3D"background-color:rgb(238,238,238)"># auditctl -e 0</font>`.=
 A `<font face=3D"monospace" style=3D"background-color:rgb(238,238,238)"># =
auditctl -s</font>` run shows `<font face=3D"monospace" style=3D"background=
-color:rgb(238,238,238)">enabled 0</font>`, yet audit logs continue to floo=
d the kernel ring buffer.</div><div><br></div><div>I&#39;m running Linux ke=
rnel 5.4.0-1063-aws on Ubuntu 18.04 with auditctl v2.8.2. `<font face=3D"mo=
nospace" style=3D"background-color:rgb(238,238,238)">systemd-journald-audit=
.socket</font>` is masked &amp; inactive, `<font face=3D"monospace" style=
=3D"background-color:rgb(238,238,238)">auditd.service</font>` is disabled &=
amp; inactive.</div><div><br></div><div>I cannot entirely disable the audit=
 system via a kernel parameter because Red Canary Linux EDR fails to start =
cfsvcd.service as it fails to run some auditctl command due to no audit sup=
port in the kernel:</div><div><br></div><blockquote class=3D"gmail_quote" s=
tyle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);pad=
ding-left:1ex"><font face=3D"monospace">Jun 28 20:41:04 systemd[1]: Started=
 Canary Forwarder Service.<br>Jun 28 20:41:04 cfsvcd[105781]: Found config =
file at path &#39;&quot;/opt/redcanary/config.json&quot;&#39;, continuing e=
xecution...<br>Jun 28 20:41:12 cfsvcd[105781]: component: Sentry<br>Jun 28 =
20:41:12 cfsvcd[105781]: =C2=A0Jun 28 20:41:12.055 INFO Initializing Sentry=
 Config { version: &quot;1.4.17.release.[...]&quot;, https_proxy: None, ext=
ra_tags: [(&quot;telemetry.source&quot;, &quot;eBPF&quot;)], logging_enable=
d: true, metrics_enabled: true, [...]}<br>Jun 28 20:41:12 cfsvcd[105781]: F=
ailed to setup or configure host system: Linux Audit watcher failure: Netli=
nk Error: IO Error: Protocol not supported (os error 93)<br>Jun 28 20:41:12=
 cfsvcd[105781]: Received stop. Exiting unix::wait_for_signal. Signal: 0, d=
one.is_signaled(): true<br>Jun 28 20:41:12 cfsvcd[105781]: Joining async_ag=
ent_thread<br>Jun 28 20:41:13 cfsvcd[105781]: Attempting to stop subscriber=
 (Ebpf).<br>Jun 28 20:41:13 cfsvcd[105781]: Shutting down due to terminatio=
n signal<br>[...]<br>Jun 28 20:41:15 cfsvcd[105781]: Subscriber stopped.<br=
>Jun 28 20:41:15 auditctl[106733]: Error - audit support not in kernel<br>J=
un 28 20:41:15 auditctl[106733]: Cannot open netlink audit socket<br>Jun 28=
 20:41:15 systemd[1]: cfsvcd.service: Service hold-off time over, schedulin=
g restart.<br>Jun 28 20:41:15 systemd[1]: cfsvcd.service: Scheduled restart=
 job, restart counter is at 301.<br>Jun 28 20:41:15 systemd[1]: Stopped Can=
ary Forwarder Service.</font><br></blockquote><div><br></div><div>Here&#39;=
s the unit file for cfsvcd.service:</div><blockquote class=3D"gmail_quote" =
style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);pa=
dding-left:1ex"><font face=3D"monospace">[Unit]<br>Description=3DCanary For=
warder Service<br><br>DefaultDependencies=3Dno<br>After=3Dlocal-fs.target s=
ystemd-tmpfiles-setup.service auditd.service<br>Before=3Dsysinit.target shu=
tdown.target<br># Replace the auditd service if it is running<br>Conflicts=
=3Dauditd.service shutdown.target<br><br>[Service]<br>WorkingDirectory=3D/o=
pt/redcanary<br>ExecStart=3D/opt/redcanary/cfsvcd<br>ExecStopPost=3D-/sbin/=
auditctl -D<br>Restart=3Dalways<br>TimeoutSec=3D15<br><br>[Install]<br>Want=
edBy=3Dmulti-user.target</font><br></blockquote><div><br></div><div>Is this=
 a known issue? Is there a workaround to stop the logging to the kernel rin=
g buffer? Is there any more information I can provide to help debug?</div><=
div><br></div><div>Thanks,</div><div>Samuel Bahr</div><div>Pinterest Sr. Si=
te Reliability Engineer</div></div>

--000000000000c0b3c305ff4c4c45--

--===============8792915135381254125==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

--
Linux-audit mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-audit

--===============8792915135381254125==--