RE: Re: OpenAFS 1.8.7 on Linux systems running Crowdstrike falcon-sensor

This is exactly the issue referenced in the LKML thread, and the one that
Jeffrey Altman analyzed, so it seems the issue has not yet been fixed.
Unfortunately, I don't think the patches dealing with credentials will fix
this because dentry_open calls security_file_open regardless of credentials,
and security_file_open expects process context.

 

I believe the way to fix this is that OpenAFS needs to open this file in a
way that does not go through the LSM, because there is no process context
here.

 

From: Jonathan Billings <[email protected]> 
Sent: Tuesday, March 9, 2021 5:46 AM
To: [email protected]
Cc: Martin Kelly <[email protected]>
Subject: [External] Re: [OpenAFS] OpenAFS 1.8.7 on Linux systems running
Crowdstrike falcon-sensor

 

Basically, this is what I'm running:

 

# git describe --abbrev=4 openafs-stable-1_8_x 
openafs-stable-1_8_7-109-gb7bdd

# rxdebug localhost 7001 -version
Trying 127.0.0.1 [127.0.0.1]
<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1&d=DwQFaQ&c=08
AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M&r=wsraf2W9gLu3Kwvcq4lteczxlOggIsFy
oEwCrqmJ1gE&m=YBSO0k1d9jL782APR0Fvy6pws5z_ZZ2yXwqqX2r4Tzk&s=za_1x9waWw1p_UYf
sz-uz1ZXNQCsoJwjTXFBN_tLnSc&e=>  (port 7001):
AFS version: OpenAFS 1.8.7-109-gb7bdd 2021-03-08 mockbuild@

 

With this kmod and the latest RHEL7 kernel, this is the kernel backtrace:

 

 

[  170.503804] BUG: unable to handle kernel NULL pointer dereference at
0000000000000004
[  170.506260] IP: [<ffffffff8238a6ec>] _raw_spin_lock+0xc/0x30
[  170.507074] PGD 0 
[  170.507824] Oops: 0002 [#1] SMP 
[  170.508596] Modules linked in: cts rpcsec_gss_krb5 nfsv4 dns_resolver nfs
lockd grace falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE)
falcon_kal(E) falcon_lsm_pinned_11308(E) nf_log_ipv4 nf_log_common xt_LOG
ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4
xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat
nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle
ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack
libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter
ip6_tables iptable_filter vmw_vsock_vmci_transport vsock cachefiles fscache
sb_edac ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel vmw_balloon
aesni_intel lrw gf128mul glue_helper
[  170.512708]  ablk_helper cryptd pcspkr joydev sg vmw_vmci i2c_piix4
parport_pc parport binfmt_misc openafs(POE) auth_rpcgss sunrpc ip_tables
ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif
crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm mptsas ata_piix scsi_transport_sas nfit drm crct10dif_pclmul
mptscsih crct10dif_common libata serio_raw crc32c_intel libnvdimm mptbase
vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log
dm_mod fuse
[  170.516344] CPU: 6 PID: 2782 Comm: llvmpipe-8 Kdump: loaded Tainted: P
OE  ------------   3.10.0-1160.15.2.el7.x86_64 #1
[  170.517353] Hardware name: VMware, Inc. VMware Virtual Platform/440BX
Desktop Reference Platform, BIOS 6.00 12/12/2018
[  170.518320] task: ffff9c0127fe0000 ti: ffff9c0556be0000 task.ti:
ffff9c0556be0000
[  170.519315] RIP: 0010:[<ffffffff8238a6ec>]  [<ffffffff8238a6ec>]
_raw_spin_lock+0xc/0x30
[  170.520311] RSP: 0018:ffff9c0556be3710  EFLAGS: 00010246
[  170.521317] RAX: 0000000000000000 RBX: 0000000000000004 RCX:
0000000000000000
[  170.522310] RDX: 0000000000000001 RSI: ffff9c013eb85d80 RDI:
0000000000000004
[  170.523301] RBP: ffff9c0556be3730 R08: 000000000001f060 R09:
ffff9c013efc67a0
[  170.524292] R10: ffff9c0073e7aa80 R11: ffff9c013c7ae440 R12:
ffff9c0556be3740
[  170.525298] R13: 0000000000000000 R14: 0000000000000000 R15:
ffff9c050c867810
[  170.526303] FS:  00007f5b1dd69700(0000) GS:ffff9c0569300000(0000)
knlGS:0000000000000000
[  170.527315] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  170.528324] CR2: 0000000000000004 CR3: 00000004da610000 CR4:
00000000001607e0
[  170.529415] Call Trace:
[  170.530459]  [<ffffffffc0af7d9d>] ? 0xffffffffc0af7d9c
[  170.531482]  [<ffffffffc0b56ab8>] _ZdlPv+0x48088/0x484f0
[falcon_lsm_serviceable]
[  170.532523]  [<ffffffffc0b5725b>]
cshook_security_file_permission+0x9b/0x8c0 [falcon_lsm_serviceable]
[  170.533579]  [<ffffffffc0b5ea9e>] cshook_security_file_open+0xe/0x10
[falcon_lsm_serviceable]
[  170.534637]  [<ffffffffc0ae4873>] pinnedhook_security_file_open+0x43/0x70
[falcon_lsm_pinned_11308]
[  170.535719]  [<ffffffff81f08f12>] security_file_open+0x22/0x70
[  170.536789]  [<ffffffff81e4b339>] do_dentry_open+0xc9/0x2d0
[  170.537850]  [<ffffffff81e4b5da>] vfs_open+0x5a/0xb0
[  170.538876]  [<ffffffff81e4b679>] dentry_open+0x49/0xc0
[  170.540008]  [<ffffffffc06cdb0f>] afs_linux_raw_open+0x8f/0x140 [openafs]
[  170.541094]  [<ffffffffc06cdc74>] osi_UFSOpen+0xb4/0x1a0 [openafs]
[  170.542140]  [<ffffffffc06567f1>] DRead+0x341/0x520 [openafs]
[  170.543225]  [<ffffffffc0669a7e>] FindItem+0x5e/0x1f0 [openafs]
[  170.544306]  [<ffffffffc066a183>] afs_dir_Delete+0x33/0x1a0 [openafs]
[  170.545409]  [<ffffffffc06a6d17>] ? RXAFS_RemoveFile+0x67/0x120 [openafs]
[  170.546510]  [<ffffffffc068e1ad>] ? afs_LocalHero+0x11d/0x200 [openafs]
[  170.547616]  [<ffffffffc069abc9>] afsremove+0x489/0x7d0 [openafs]
[  170.548705]  [<ffffffffc069beca>] afs_remunlink+0x24a/0x2f0 [openafs]
[  170.549798]  [<ffffffffc068489d>] afs_InactiveVCache+0x7d/0x80 [openafs]
[  170.550891]  [<ffffffffc06d28e8>] afs_dentry_iput+0x58/0x140 [openafs]
[  170.551931]  [<ffffffff81e670ca>] __dentry_kill+0x13a/0x1d0
[  170.553003]  [<ffffffff81e67785>] dput+0xb5/0x1a0
[  170.554060]  [<ffffffff81e500cd>] __fput+0x18d/0x230
[  170.555118]  [<ffffffff81e5025e>] ____fput+0xe/0x10
[  170.556169]  [<ffffffff81cc294b>] task_work_run+0xbb/0xe0
[  170.557191]  [<ffffffff81ca1894>] do_exit+0x2d4/0xa30
[  170.558162]  [<ffffffff81d1317f>] ? futex_wait+0x11f/0x280
[  170.559126]  [<ffffffff81c63d03>] ? x2apic_send_IPI_mask+0x13/0x20
[  170.560048]  [<ffffffff81ca206f>] do_group_exit+0x3f/0xa0
[  170.560939]  [<ffffffff81cb323e>] get_signal_to_deliver+0x1ce/0x5e0
[  170.561802]  [<ffffffff81c2c527>] do_signal+0x57/0x6f0
[  170.562626]  [<ffffffff81d14f46>] ? do_futex+0x106/0x5a0
[  170.563420]  [<ffffffff81c2cc32>] do_notify_resume+0x72/0xc0
[  170.564184]  [<ffffffff823952ef>] int_signal+0x12/0x17
[  170.564878] Code: 5d c3 0f 1f 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66
0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00
<f0> 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 ea 18 ff ff 5d 
[  170.566509] RIP  [<ffffffff8238a6ec>] _raw_spin_lock+0xc/0x30
[  170.567274]  RSP <ffff9c0556be3710>
[  170.568018] CR2: 0000000000000004

 

 

On Mon, Mar 8, 2021 at 8:22 PM Jeffrey E Altman <[email protected]
<mailto:[email protected]> > wrote:

On 3/8/2021 7:20 PM, Benjamin Kaduk ([email protected] <mailto:[email protected]> )
wrote:
> On Mon, Mar 08, 2021 at 07:35:19PM +0000, Martin Kelly wrote:
>> Below is the LKML LSM thread regarding this. Please let me know if you
have any other questions:
>>
>> https://www.spinics.net/lists/linux-security-module/msg39081.html
[spinics.net]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.spinics.net_lists_
linux-2Dsecurity-2Dmodule_msg39081.html&d=DwMFaQ&c=08AGY6txKsvMOP6lYkHQpPMRA
1U6kqhAwGa8-0QCg3M&r=wsraf2W9gLu3Kwvcq4lteczxlOggIsFyoEwCrqmJ1gE&m=YBSO0k1d9
jL782APR0Fvy6pws5z_ZZ2yXwqqX2r4Tzk&s=1KnxijRcsGp8k_dt6qTga6LJh5Nu53ja1ZuNA6x
9AQc&e=> 
>> https://www.spinics.net/lists/linux-security-module/msg39083.html
[spinics.net]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.spinics.net_lists_
linux-2Dsecurity-2Dmodule_msg39083.html&d=DwMFaQ&c=08AGY6txKsvMOP6lYkHQpPMRA
1U6kqhAwGa8-0QCg3M&r=wsraf2W9gLu3Kwvcq4lteczxlOggIsFyoEwCrqmJ1gE&m=YBSO0k1d9
jL782APR0Fvy6pws5z_ZZ2yXwqqX2r4Tzk&s=DWivoWdmUWYhc2fPQ040MVvE8ryH7ncDdVo9vQV
V1A8&e=> 
> 
> Thanks for spotting this thread and the quick follow-up.

This is the same thread that Yadav discussed with the openafs-release 
team on 11 Dec 2020.

> I suspect that the changes at https://gerrit.openafs.org/#/c/13751/
[gerrit.openafs.org]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__gerrit.openafs.org_-23
_c_13751_&d=DwMFaQ&c=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M&r=wsraf2W9g
Lu3Kwvcq4lteczxlOggIsFyoEwCrqmJ1gE&m=YBSO0k1d9jL782APR0Fvy6pws5z_ZZ2yXwqqX2r
4Tzk&s=IzzLrxc0PFa4NClPYSdo20jhIRaCdMK_JWp58-1aroQ&e=>  are
> going to be relevant in this space, but without seeing the stack trace of
> the crash in question it's hard to be sure.  Can you speak to whether this
> is touching the "right" part of the code with respect to the crashes you
> were investigating?

The suggested change was cherry-picked to openafs-stable-1_8_x as
https://gerrit.openafs.org/14082 [gerrit.openafs.org]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__gerrit.openafs.org_140
82&d=DwMFaQ&c=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M&r=wsraf2W9gLu3Kwvc
q4lteczxlOggIsFyoEwCrqmJ1gE&m=YBSO0k1d9jL782APR0Fvy6pws5z_ZZ2yXwqqX2r4Tzk&s=
lHcFqdLdUXfADi3w5wJ7Ig4CBNsRsa9dqLCuSp2rJ10&e=>  and merged as 
ee578e92d9f810d93659a9805d0c12084fe2bb95.

As Jonathan wrote to IRC OpenAFS:

 > (4:53:15 PM) billings: I built openafs from the latest commit in
 > 1_8_x and crowdstrike still panics, so it doesnt look like any
 > merged commits there fix my issue.

Martin's e-mail describes the call pattern:

 > - A process exits, calling task_exit().

I think Martin meant do_exit().

 > - exit_fs() is called, setting current->fs = NULL.

task_struct field struct fs_struct *fs;

 > - Next, exit_task_work() is called,

exit_task_work() calls task_work_run() which flushes any pending works.

 > which calls fput().

which must have been called by a pending work.

 > - In response to the fput(), the filesystem opens a file

disk cache

 >   to update some metadata, calling dentry_open().

dentry_open() in turn will trigger a call to any configured LSM.
If task_struct.fs is NULL, Kaboom!!!

Jeffrey Altman



-- 

Jonathan Billings <[email protected] <mailto:[email protected]> > (he/his)
College of Engineering - CAEN - Linux Support