[PATCH v2] Cygwin: Ensure unused fd available for open()

The existing logic for open() assumes an fd is always available in
the fdtable for a created file.  This leads to a situation where, if
there is no fd available due to the OPEN_MAX limit being hit, the
file is created but cannot be referenced by a Cygwin fd.

Move the fd reservation code to an earlier location within open().

Reported-by: Christian Franke <[email protected]>
Addresses: https://cygwin.com/pipermail/cygwin/2026-May/259664.html
Signed-off-by: Mark Geisert <[email protected]>
Fixes: e859706578ba (* autoload.cc (NtCreateFile): Add.)

---
 winsup/cygwin/syscalls.cc | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/winsup/cygwin/syscalls.cc b/winsup/cygwin/syscalls.cc
index 7a8e5d4fd..2bea79768 100644
--- a/winsup/cygwin/syscalls.cc
+++ b/winsup/cygwin/syscalls.cc
@@ -1547,6 +1547,13 @@ open (const char *unix_path, int flags, ...)
 	  fh = fh_file;
 	}
 
+      /* Reserve an fdtable entry here, before calling open_with_arch() below.
+         Otherwise there's a tiny chance of hitting OPEN_MAX further on which
+         could create a new file without any way for Cygwin to refer to it. */
+      cygheap_fdnew fd;
+      if (fd < 0)
+        __leave;		/* errno already set */
+
       if (fh->dev () == FH_PROCESSFD && fh->pc.follow_fd_symlink ())
 	{
 	  /* Reopen file by descriptor */
@@ -1573,14 +1580,6 @@ open (const char *unix_path, int flags, ...)
 	try_to_bin (fh->pc, fh->get_handle (), DELETE,
 		    FILE_OPEN_FOR_BACKUP_INTENT);
 
-      cygheap_fdnew fd;
-
-      if (fd < 0)
-	{
-	  fh->close();
-	  __leave;		/* errno already set */
-	}
-
       fd = fh;
       if (fd <= 2)
 	set_std_handle (fd);
-- 
2.51.0