https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=297777
Bug ID: 297777
Summary: geom_part: access leaked when a resized table is
spoiled, wedging all USB enumeration
Product: Base System
Version: 15.1-RELEASE
Hardware: amd64
OS: Any
Status: New
Severity: Affects Some People
Priority: ---
Component: kern
Assignee: [email protected]
Reporter: [email protected]
Created attachment 274011
--> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=274011&action=edit
Fix: release the access in g_part_spoiled(), against main
ENVIRONMENT
FreeBSD 15.1-RELEASE-p1 amd64, GENERIC, and reproduced against
release/15.1.0-p2 (aadd58dddcbc). Single xHCI controller
(xhci0@pci0:0:20:0, Intel 100/C230 series). A USB card reader behind a
GenesysLogic hub, holding a card whose partition table describes a smaller
disk than the medium.
AFFECTED VERSIONS
Verified present in main (16.0-CURRENT), stable/15 (15.1-STABLE), releng/15.1
(15.1-RELEASE-p2), stable/14 (14.5-STABLE) and releng/14.4 (14.4-RELEASE-p8).
Older branches were not checked, but the hold was introduced in 2013 by
884c8e4feacd ("Add an automatic resize support to the GEOM_PART class."),
which added the g_access() in g_part_resize() without a matching release on
the spoil path, so branches predating 14.4 are likely affected too.
g_part_spoiled() is unchanged in main as of b54dcb897a5f, and the attached
patch applies there cleanly.
SUMMARY
When GEOM_PART automatically resizes a partition table it takes
g_access(cp, 1, 1, 1) on the disk and holds it until the change is committed
or reverted. g_part_spoiled() withers the geom without giving that access
back, and g_part_wither() clears gp->softc, which is where the gpt_opened
flag guarding every release lives. After that the access can never be
released by anything.
The consequences reach past GEOM:
- the DISK provider can never be destroyed
- the da peripheral stays open (OPEN in kern.cam.da.0.flags)
- the CAM bus keeps its reference on the SIM, so sim->refcount never
reaches zero
- cam_sim_free() sleeps on simfree with no timeout, inside umass_detach()
- that detach is running on the bus's single USB_BUS_EXPLORE_PROC thread,
which serialises every attach and detach on the bus
So one leaked GEOM reference permanently disables USB enumeration for an
entire controller. On a machine with one xHCI controller, as here, that is
every USB port in the box, including the motherboard's own. Devices already
attached keep working and the controller keeps taking interrupts normally,
which makes it look like a device problem rather than a kernel one. Only a
reboot recovers.
REPRODUCTION
Insert a card whose table does not match the medium size, so an automatic
resize occurs; remove it; reinsert it. Reproduced twice out of two attempts
on the hardware above. A reader that spontaneously re-enumerates makes it
easier to hit.
EVIDENCE
dtrace on the partition class during the failure. The resize takes the
access:
00:37:09 RESIZE da0 acr=0 acw=0 ace=0 flags=0x30
00:37:09 RESIZE da0 acr=1 acw=1 ace=1 flags=0x30
Five seconds later, with the access still held, the geom is withered from
the spoil path, which does not release it:
00:37:14 WITHER da0 softc=fffff8004472d400 error=6
kernel`g_spoil_event+0x47
kernel`g_run_events+0x11e
kernel`fork_exit+0x7b
A later media-change event finds the softc already cleared, so it cannot
help either:
00:37:17 WITHER da0 softc=0 error=6
kernel`g_media_changed_event+0x67
kernel`g_run_events+0x11e
Note there is no orphan in this sequence: the reader stayed on the bus and
only the medium changed, so g_part_orphan() -- which would have released the
access -- never ran.
The resulting state, identical on both occurrences:
class=PART geom=da0 wither=True consumer mode=r1w1e1
class=DISK geom=da0 provider da0 mode=r1w1e1 wither=True
kern.cam.da.0.flags: 0x3359<PACK_INVALID,PACK_REMOVABLE,ROTATING,
WAS_OTAG,OPEN,SCTX_INIT,DIRTY,ANNOUNCED>
No /dev/da0 exists, and no process holds it open (fstat empty). The
consumer is held entirely inside GEOM.
The blocked thread:
14 100097 usb usbus0
mi_switch+0xbc _sleep+0x19e
cam_sim_free+0x77
umass_detach+0x136
device_detach+0x180
device_delete_child+0x18
usb_detach_device+0x17f
usb_unconfigure+0x83
usb_free_device+0x111
uhub_explore+0x3fc
uhub_explore+0xa75
usb_bus_explore+0x116
usb_process+0xee
Everything else is idle, not blocked: g_event, g_up, g_down and all four CAM
threads are in their normal waits, and irq128: xhci0 is in plain
ithread_loop still taking about 69 interrupts a second. The system is
quiescent; nothing is pending that could ever release the reference.
Raw captures are attached (see ATTACHMENTS).
THE DEFECT
sys/geom/part/g_part.c. g_part_resize() takes the access:
if (table->gpt_opened == 0) {
if (g_access(cp, 1, 1, 1) != 0)
return;
table->gpt_opened = 1;
}
g_part_wither() begins with gp->softc = NULL, so any release attempted
afterwards is skipped, because every release tests table->gpt_opened.
g_part_orphan() releases before withering:
table = cp->geom->softc;
if (table != NULL && table->gpt_opened)
g_access(cp, -1, -1, -1);
g_part_wither(cp->geom, pp->error);
g_part_spoiled() does not:
cp->flags |= G_CF_ORPHAN;
g_part_wither(cp->geom, ENXIO);
g_part_spoiled() is not a rare path. It runs whenever anything opens a disk
for writing, via g_spoil() from g_access(). It is harmless in the ordinary
case only because gpt_opened is 0 then.
FIX
The attached patch makes g_part_spoiled() release the access exactly as
g_part_orphan() does. It applies to main, passes
tools/build/checkstyle9.pl with no errors or warnings, and is offered under
the same licence as the file it modifies.
A broader fix would move the release into g_part_wither() so no future
caller can leak it. That needs care: several existing callers already
release before calling it without clearing gpt_opened (g_part.c:931, 1085,
1534), so they would have to be audited first to avoid a double release.
The narrow fix was chosen for that reason.
TESTING
A GENERIC kernel built from release/15.1.0-p2 with this patch was installed
on the affected machine. The sequence that had wedged it twice out of two
attempts no longer wedges it.
The same kernel, the same reader, the same card, with the tracer running.
The resize takes the access on reinsertion:
01:18:51 ACCESS da0 dcr=1 dcw=1 dce=1 (now 0/0/0)
01:18:51 RESIZE da0 acr=1 acw=1 ace=1
and when the card is pulled, the spoil path now gives it back before
withering:
01:21:44 ACCESS da0 dcr=-1 dcw=-1 dce=-1 (now 1/1/1)
01:21:44 WITHER da0 softc=fffff8012d89bc00 error=6
kernel`g_spoil_event+0x47
kernel`g_run_events+0x11e
The unpatched kernel, given the same sequence, produced the resize and then
the same g_spoil_event wither with no release in between, and wedged.
Nothing is left holding the disk afterwards and the bus keeps enumerating.
TWO OBSERVATIONS, NOT PART OF THE FIX
cam_sim_free() sleeps with no timeout and prints nothing. A bounded wait
that logged the outstanding refcount would have made this diagnosable
without tracing the kernel.
g_spoil() contains KASSERT(cp2->ace == 0, ...). INVARIANTS is off on
RELEASE kernels, so spoiling a consumer that holds exclusive access proceeds
silently rather than panicking, which hides exactly this case on the kernels
users actually run.
IMPACT
Denial of service of an entire USB controller until reboot, reachable from
ordinary use of removable media, with no privilege beyond inserting a card.
ATTACHMENTS
0001-geom_part-fix-access-leak-when-a-resized-table-is-spoiled.patch
the fix, against main, git am -able
gpart-trace-failing.log
dtrace of the partition class across both failures. The sequence at
00:37:09-00:37:17 is the one quoted above.
gpart-trace-fixed.log
the same trace on the patched kernel, showing the release
procstat-usb-wedged.txt
the usb process while wedged: the explore thread asleep in
cam_sim_free inside umass_detach
procstat-geom-cam-idle.txt
the geom and cam processes at the same moment, all idle, showing the
deadlock is quiescent
gpart-trace.d
the dtrace script used to produce the traces above
A full automatic capture taken at the second occurrence (thread list, full
kern.geom.confxml, kern.cam sysctls, dmesg, messages, interrupt counts) is
available on request; in its confxml the leaked reference is the PART
consumer on da0 at mode r1w1e1, on a geom already marked wither.
--
You are receiving this mail because:
You are the assignee for the bug.
lmpx.com only provides a reader for public news (NNTP) servers. It is not
affiliated with the servers or forums shown here and is not responsible for
the content of articles, which is written by their respective authors.