Experience report: bhyve virtio-blk & passthru hangs, & nvme works
Mathias Picker <[email protected]>
| Newsgroups | gmane.os.freebsd.questions |
|---|---|
| Message-ID | <[email protected]> |
Hi all, just to have this somewhere searchable: I just configured a machine with a Supermicro X11DPI-NT motherboard, two Xeon Silver 4114, 128Gb RAM and two RTX 3090. It's for experimenting with LLMs, using bhyve with the GraKa passed through. This installed relatively easy, but on downloading large models from hugingface the system always stopped, showing multiple errors like this on the console: ixl0: ixl_process_adminq: Unable to allocate memory for Admin Queue event! ixl1: ixl_process_adminq: Unable to allocate memory for Admin Queue event! This lead me in the wrong direction, the culprit was the disk. I used a sparse zvol and virtuio-blk. With bhyve started AND ONLY WITH ACTIVE PASSTHRU this lead to hanging path from the zvol to the host, e.g. with the VΜ w. passthru up running $ mdo dd if=/dev/zvol/zroot/vm/llm/disk0 of=/dev/null bs=1M status=progress on the FreeBSD host (!) would hang after 7-11Gb. That was puzzling. the same dd with /dev/nda0 worked just fine. So I just tried to change the vm-bhyve config for this vm from virtio-blk to nvme and voila, everything seems to work fine now. dd works for 200Gb and the download from huggingface is not at 50Gb. Hope this helps someone, Mathias -- Mathias Picker Geschäftsführer [email protected] virtual earth Gesellschaft für Wissens re/prä sentation mbH http://www.virtual-earth.de/ HRB126870 [email protected] Westendstr. 142 089 / 1250 3943