[newlib-cygwin/cygwin-3_6-branch] Cygwin: pty: Return EMFILE when too meny ptmx are opened

https://sourceware.org/git/gitweb.cgi?p=newlib-cygwin.git;h=00e7a8938e8b78a0d984175b37dd843e475d9841

commit 00e7a8938e8b78a0d984175b37dd843e475d9841
Author: Takashi Yano <[email protected]>
Date:   Mon Mar 31 23:01:50 2025 +0900

    Cygwin: pty: Return EMFILE when too meny ptmx are opened
    
    Previously, opening /dev/ptmx fails without setting errno when it
    is opened too many times. With this patch, return EMFILE in that
    situation.
    
    Addresses: https://cygwin.com/pipermail/cygwin/2025-March/257786.html
    Fixes: 09738c30627c ("Cygwin: pty: setup new pty on opening the master, not in constructor")
    Reported-by: Christian Franke <[email protected]>
    Signed-off-by: Takashi Yano <[email protected]>
    (cherry picked from commit 7fc7d8b1d4e323a531ab2e71581263a0675072d8)

Diff:
---
 winsup/cygwin/fhandler/pty.cc | 3 +++
 winsup/cygwin/release/3.6.1   | 3 +++
 winsup/cygwin/tty.cc          | 2 +-
 3 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/winsup/cygwin/fhandler/pty.cc b/winsup/cygwin/fhandler/pty.cc
index e61a1c89b..3128b92da 100644
--- a/winsup/cygwin/fhandler/pty.cc
+++ b/winsup/cygwin/fhandler/pty.cc
@@ -1933,7 +1933,10 @@ int
 fhandler_pty_master::open (int flags, mode_t)
 {
   if (!setup ())
+    {
+      set_errno (EMFILE);
       return 0;
+    }
   set_open_status ();
   dwProcessId = GetCurrentProcessId ();
   return 1;
diff --git a/winsup/cygwin/release/3.6.1 b/winsup/cygwin/release/3.6.1
index 254cfdd9c..491d7dcb9 100644
--- a/winsup/cygwin/release/3.6.1
+++ b/winsup/cygwin/release/3.6.1
@@ -27,3 +27,6 @@ Fixes:
 - Accommodate a change in Windows exception handling affecting software
   generated exceptions.
   Addresses: https://cygwin.com/pipermail/cygwin/2025-March/257808.html
+
+- Return EMFILE when opening /dev/ptmx too many times.
+  Addresses: https://cygwin.com/pipermail/cygwin/2025-March/257786.html
diff --git a/winsup/cygwin/tty.cc b/winsup/cygwin/tty.cc
index 2cd4ae6ed..a4b716721 100644
--- a/winsup/cygwin/tty.cc
+++ b/winsup/cygwin/tty.cc
@@ -147,7 +147,7 @@ tty_list::allocate (HANDLE& r, HANDLE& w)
     termios_printf ("pty%d allocated", freetty);
   else
     {
-      system_printf ("No pty allocated");
+      termios_printf ("No pty allocated");
       r = w = NULL;
     }