Re: Conduit missing closing file handles

On Thu, Apr 30, 2026 at 12:07:08PM +0200, Henning Thielemann wrote:
> 'Zip.getEntrySource selector' generates a conduit for reading a file from
> the ZIP archive. It opens the archive anew for every accessed file, but it
> calls Conduit.Binary.sourceIOHandle, which promises to close a file promptly
> after reading. [2]

You have to be a bit careful about what "immediately after reading"
means.  Actually, I think it means "immediately after the file handle
indicates it has reached end of file".

`getEntrySource`[1] calls `sourceEntry`[2] which runs

    source .| CB.isolate (fromIntegral edCompressedSize) .| decompress

Now, source is

    source = CB.sourceIOHandle $ do
      ...

But `sourceIOHandle` will only close its handle after it determines
that it has reached EOF.  If you look at the argument to
`sourceIOHandle` in [2] you'll see that it just seeks to the start
position of the target file in the archive.  It's CB.isolate which is
used to limit reading to the extent of the target file only. It brings
down the pipeline once it is done but because EOF is never reached in
the handle (i.e. the final bytes of the entire .zip file are never
read) the cleanup action doesn't trigger at this point.

The cleanup action will nonetheless trigger at the end of the
runConduitRes block, but that's too late. You've already opened too
many files.

I have a simpler reproducer for this below.

Maybe this is salvagable by putting a ResourceT block within
the definition of `sourceEntry`, maybe not. I don't have the finer
details of ResourceT in working memory.  However, there are numerous
problems with Conduit and Pipes bracketing that make it very difficult
to use them safely.  My effect system Bluefin solves this problem, as
I elaborated in my article "Bluefin streams finalize promptly".

https://h2.jaguarpaw.co.uk/posts/bluefin-streams-finalize-promptly/

However, there is sadly not a large ecosystem around Bluefin, and no
Zip library.  I'm happy to help you implement what you need if you're
interested!

Tom





[1] https://hackage-content.haskell.org/package/zip-2.2.1/docs/src/Codec.Archive.Zip.html#getEntrySource

[2] https://hackage-content.haskell.org/package/zip-2.2.1/docs/src/Codec.Archive.Zip.Internal.html#sourceEntry



{- cabal:
build-depends: conduit, base
-}
{-# LANGUAGE GHC2021 #-}

import Conduit (liftIO, mapM_C, runConduitRes, (.|))
import Control.Monad (forever, when)
import Data.Conduit.Combinators (sourceIOHandle)
import Data.Conduit.List qualified as CB
import Data.IORef (modifyIORef', newIORef, readIORef, writeIORef)
import System.IO (IOMode (ReadMode), openFile)

-- cabal-script-test27.hs: /dev/zero: openFile: resource exhausted (Too
-- many open files)
main :: IO ()
main = do
  ref <- newIORef @Int 0
  runConduitRes $
    forever $ do
      sourceIOHandle (openFile "/dev/zero" ReadMode)
        .| CB.isolate 1
        .| mapM_C
          ( \_ -> liftIO $ do
              modifyIORef' ref (+ 1)
              n <- readIORef ref
              when (n `mod` 20 == 0) $ do
                Prelude.print n
          )
_______________________________________________
Haskell-Cafe mailing list -- [email protected]
To (un)subscribe, modify options or view archives go to:
Only members subscribed via the mailman list are allowed to post.