[PATCH RFC 0/7] coredump, files: exit files on request

We've had quite a few proposal for exiting files before generating the
coredump (see [1]-[8]). The various proposed solution were quite
unacceptable. So I took some time to look what a _remotely_ acceptable
version of this could look like. Here it is.

The first patch is somewhat unrelated but without it the code would have
to become so ugly to the point of being unacceptable so it's a stylistic
prerequisite imho.

In the first patch we make exit_files() synchronous and stop shunting
all files onto task work only to run task work right after. For tasks
exiting with a very large file descriptor table this means getting rid
of _a lot_ of cmpxchg()s and hammering on task->pi_lock. The cost of
this was mentioned in various threads over the years. I found at least
recent comments by Oleg and Neil.

So let close_files() close right away. No locks are held, the path
already sleeps, close() does the same thing already. Kernel threads can
just keep deferring. ->flush() and ->release() now run before exit_fs()
and disassociate_ctty() which should be fine.

The other half is adding support for coredumps to switch to an empty
file descriptor table before generating the coredump. That requires a
bit of synchronization but I think I got the basics down.

A new COREDUMP_CLOSE_FILES extension that allows a coredump server to
request that the thread-group close all files before generating the
coredump.

The coredumping thread allocates a new empty file descriptor table. It
then wakes all threads in the thread-group and tells them to switch to
the empty file descriptor table and get rid of the old one. They report
back once they're done.

So that handles most cases where locks would be held for an unreasonable
time until the coredump is generated but since files can be shared
between completely unrelated processes that's not a guarantee and we
can't give one. But it solves the reported issue without resorting to
even grosser hacks.

Link: https://lore.kernel.org/[email protected] [1]
Link: https://lore.kernel.org/[email protected] [2]
Link: https://lore.kernel.org/[email protected] [3]
Link: https://lore.kernel.org/[email protected] [4]
Link: https://lore.kernel.org/[email protected] [5]
Link: https://lore.kernel.org/[email protected] [6]
Link: https://lore.kernel.org/[email protected] [7]
Link: https://lore.kernel.org/[email protected] [8]

Signed-off-by: Christian Brauner (Amutable) <[email protected]>
---
Christian Brauner (7):
      fs: add filp_close_sync()
      fs: make close_files() synchronous
      fs: add switch_files_struct()
      coredump: s/startup/done/g
      coredump: add COREDUMP_CLOSE_FILES
      tools: sync coredump.h header
      selftests/coredump: test COREDUMP_CLOSE_FILES

 fs/coredump.c                                      |  18 +-
 fs/file.c                                          | 152 +++++-
 fs/file_table.c                                    |  53 +-
 fs/internal.h                                      |   3 +-
 fs/open.c                                          |  39 +-
 include/linux/fdtable.h                            |   5 +
 include/linux/sched/signal.h                       |   5 +-
 include/uapi/linux/coredump.h                      |   8 +
 kernel/exit.c                                      |   7 +-
 tools/include/uapi/linux/coredump.h                |   8 +
 tools/testing/selftests/coredump/Makefile          |   4 +-
 .../selftests/coredump/coredump_close_files_test.c | 592 +++++++++++++++++++++
 .../coredump/coredump_socket_protocol_test.c       |   6 +
 .../selftests/coredump/coredump_test_helpers.c     |   3 +-
 14 files changed, 832 insertions(+), 71 deletions(-)
---
base-commit: 5870ce43c013ad564b3b83bae330b160e056df67
change-id: 20260824-work-coredump-unlock-self-63a898912870