Re: [Sbcl-bugs] [BUG] WITH-OPEN-FILE registers redundant auto-close finalizer (~31% of OPEN allocation)

Solved in a different way. Thanks for the idea.


On Wed, Mar 11, 2026 at 6:41 PM John Mallery <[email protected]> wrote:
>
> SBCL version: 2.6.1 (also affects current git HEAD)
> Platform: all
>
> DESCRIPTION
>
> Every call to OPEN registers an auto-close finalizer on the fd-stream
> via FINALIZE (which calls SB-LOCKLESS:SO-INSERT to insert into the
> finalizer hash table).  This allocates a closure and a hash table
> node.  When OPEN is called from WITH-OPEN-FILE, the finalizer is
> redundant because the UNWIND-PROTECT in WITH-OPEN-FILE guarantees
> CLOSE is called on both normal and abnormal exit.
>
> Profiling with sb-sprof in :alloc mode shows SB-LOCKLESS:SO-INSERT
> accounting for ~31% of OPEN's total allocation.  In a web server
> that opens/closes files thousands of times per second, this is a
> significant source of GC pressure.
>
> HOW TO REPRODUCE
>
>  (require :sb-sprof)
>
>  (let ((path (namestring *load-truename*)))  ; any existing file
>    (dotimes (i 50) (close (open path)))      ; warm buffer pool
>
>    ;; Profile allocation in OPEN
>    (sb-sprof:start-profiling :mode :alloc :sample-interval 1)
>    (dotimes (i 5000) (close (open path)))
>    (sb-sprof:stop-profiling)
>    (sb-sprof:report :type :flat :max 10))
>
>  ;; Output shows SB-LOCKLESS:SO-INSERT at ~31% of samples
>
> FIX
>
> Two files changed: src/code/fd-stream.lisp and src/code/macros.lisp.
>
> 1. Add a dynamic variable *SUPPRESS-STREAM-AUTO-CLOSE* that gates
>   the finalizer registration in MAKE-FD-STREAM:
>
> --- a/src/code/fd-stream.lisp
> +++ b/src/code/fd-stream.lisp
> @@ -2475,6 +2475,12 @@
> ;;;
> ;;; NAME is used to identify the stream when printed.
> ;;;
> +;;; When true, MAKE-FD-STREAM skips the auto-close finalizer registration.
> +;;; Bound by WITH-OPEN-FILE, where the UNWIND-PROTECT guarantees CLOSE,
> +;;; making the finalizer redundant.  Eliminates ~31% of OPEN allocation
> +;;; (finalizer closure + SO-INSERT hash table node).
> +(defvar *suppress-stream-auto-close* nil)
> +
> ;;; If SERVE-EVENTS is true, SERVE-EVENT machinery is used to
> ;;; handle blocking IO on the stream.
> (defun make-fd-stream (fd
> @@ -2544,7 +2550,7 @@
>       (set-fd-stream-routines stream element-type ...)
> -      (when auto-close
> +      (when (and auto-close (not *suppress-stream-auto-close*))
>         (finalize stream
>                   (lambda ()
>                     (sb-unix:unix-close fd)
>
> 2. Modify WITH-OPEN-FILE to bind *SUPPRESS-STREAM-AUTO-CLOSE* to T.
>   The stream variable is initialized to NIL and set inside the
>   UNWIND-PROTECT scope so the suppression is active during OPEN:
>
> --- a/src/code/macros.lisp
> +++ b/src/code/macros.lisp
> @@ -1437,12 +1437,18 @@
>   (multiple-value-bind (forms decls) (parse-body body nil)
>     (let ((abortp (gensym)))
> -      `(let ((,stream (open ,filespec ,@options))
> -             (,abortp t))
> +      ;; Suppress the auto-close finalizer -- UNWIND-PROTECT guarantees
> +      ;; CLOSE, making the finalizer redundant.
> +      `(let ((,stream)
> +             (,abortp t)
> +             (sb-impl::*suppress-stream-auto-close* t))
>          ,@decls
>          (unwind-protect
>               (multiple-value-prog1
> -                  (progn ,@forms)
> +                  (progn
> +                    (setf ,stream (open ,filespec ,@options))
> +                    ,@forms)
>                 (setq ,abortp nil))
>            (when ,stream
>              (close ,stream :abort ,abortp)))))))
>
> SAFETY
>
> The auto-close finalizer is ONLY suppressed within WITH-OPEN-FILE,
> where the UNWIND-PROTECT guarantees CLOSE.  Bare OPEN calls outside
> WITH-OPEN-FILE still register the finalizer as a safety net.
>
> If WITH-OPEN-FILE's OPEN succeeds but the thread is destroyed before
> UNWIND-PROTECT runs (e.g., SB-THREAD:TERMINATE-THREAD), the fd would
> leak.  However, this is already the case for any resource acquired
> inside an UNWIND-PROTECT -- thread termination can prevent cleanup
> forms from running regardless of finalizers.
>
> IMPACT
>
> Any I/O-intensive application that opens files via WITH-OPEN-FILE
> benefits.  Web servers, compilers, and build systems are typical
> examples.  The fix eliminates ~31% of OPEN's allocation with zero
> behavioral change for correct programs.
>
>
> _______________________________________________
> Sbcl-bugs mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/sbcl-bugs


_______________________________________________
Sbcl-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/sbcl-devel