Conduit missing closing file handles
Henning Thielemann <[email protected]> Thu, 30 Apr 2026 12:07:08 +0200 (CEST)
| Newsgroups | gmane.comp.lang.haskell.cafe |
|---|---|
| Message-ID | <[email protected]> |
I have a pretty simple program for converting ZIP to TAR archives using
the 'zip' and 'tar-conduit' packages, both using Conduit for streaming
data. In principle the program works, but if the ZIP archive contains a
non-trivial number of files, the program breaks because of exceeded limit
of open file handles:
$ zip2tar /usr/share/libreoffice/share/config/images_colibre.zip >/tmp/images_colibre.tar
zip2tar: Uncaught exception
ghc-internal:GHC.Internal.IO.Exception.IOException:
/usr/share/libreoffice/share/config/images_colibre.zip: openFile: resource
exhausted (Too many open files)
While handling /usr/share/libreoffice/share/config/images_colibre.zip:
openFile: resource exhausted (Too many open files)
HasCallStack backtrace:
throwIO, called at ./Control/Monad/Trans/Resource.hs:195:13 in
resourcet-1.3.0-8b355f9483371034fb75ff40adb2e07157502721c941fbf8a6f607df6dd81546:Control.Monad.Trans.Resource
The main function is essentially this one: [1]
convertZipToTar ::
(Zip.EntrySelector -> Zip.EntryDescription -> CTar.FileInfo) ->
Sort -> FilePath -> IO ()
convertZipToTar fileInfoFromZipEntry_ sortMode zipPath =
Zip.withArchive zipPath $ do
entries <- Map.toList <$> Zip.getEntries
files <-
for entries $ \(selector, descr) ->
flip fmap (Zip.getEntrySource selector) $ \src ->
src
.|
(C.yield (Left $ fileInfoFromZipEntry_ selector descr)
>>
C.mapC Right)
C.liftIO $ runConduitRes $
sequence_ files
.|
void CTar.tar
.|
C.stdoutC
'Zip.getEntrySource selector' generates a conduit for reading a file from
the ZIP archive. It opens the archive anew for every accessed file, but it
calls Conduit.Binary.sourceIOHandle, which promises to close a file
promptly after reading. [2]
It seems that files are not immediately closed after reading, but I cannot
spot the cause. I suspected that "sequence_ files" might concatenate the
Conduits while accumulating open file handles, but I think it should not
and I did not find a matching alternative to sequence_.
[1] https://hackage-content.haskell.org/package/zip2tar-0.0/src/src/Main.hs
[2] https://hackage-content.haskell.org/package/zip-2.2.1/docs/src/Codec.Archive.Zip.Internal.html#sourceEntry
_______________________________________________
Haskell-Cafe mailing list -- [email protected]
To (un)subscribe, modify options or view archives go to:
Only members subscribed via the mailman list are allowed to post.