[PATCH v2 1/3] Cygwin: pty: detect pcon-backed pty for non-Cygwin-spawned children

From: Johannes Schindelin <[email protected]>

When a Cygwin process (e.g. `bash` under MinTTY) spawns a native
Win32 child (e.g. `git.exe`) with pseudo console support enabled,
the child gets a pseudo console that bridges the pty. If that native
child then spawns a Cygwin grandchild (e.g. `vim`, `less`), the
grandchild inherits the pseudo console's console handles. In
`init_std_file_from_handle()`, the grandchild's msys2-runtime sees
`GetConsoleScreenBufferInfo()` succeed on those handles and, with
no valid `ctty` set, falls back to `FH_CONSOLE` and gives the
process `cons0` instead of connecting to the pty.

This causes scrollback clobbering in MinTTY because alternate screen
sequences (`ESC[?1049h` / `ESC[?1049l`) are handled by
`fhandler_console`'s `save_restore()` against the pseudo
console's buffer, which has no correspondence to MinTTY's scrollback.

Fix this in the existing console branch of
`init_std_file_from_handle()`: when there is no valid `ctty` and
we are about to fall back to `FH_CONSOLE`, first scan the shared
tty table for an entry whose `pcon_activated` is set and whose
`nat_pipe_owner_pid` is in our console's process list (via
`GetConsoleProcessList`). If found, parse the device as that pty
slave instead of as a real console. The handle is closed in either
fallback path, matching the existing `FH_CONSOLE` behavior.
`myself->ctty` is left untouched; the regular
`fhandler_pty_slave::open_setup()` path will set it via
`myself->set_ctty()` when the pty slave is opened.

The structure of `find_pcon_pty()` matters and is easy to get
wrong in case a keen developer would like to refactor this code in
the future. This code runs on every Cygwin process startup whose
parent is non-Cygwin, so the common path (no pty with an active
pseudo console) must remain free of expensive operations. Two
pitfalls to avoid: filtering tty entries with `tty::exists()`
looks correct but creates and destroys a named pipe per entry
(128 entries on every call), and hoisting the
`GetConsoleProcessList()` call out of the loop pays the
cross-process cost even when no candidate exists. The current
shape, a cheap shared-memory boolean check first and a lazily
fetched process list only on the first candidate, keeps the
common case at a handful of pointer reads.

Reported downstream at https://github.com/git-for-windows/git/issues/5303
and bisected to a Git for Windows release that upgraded the bundled
msys2-runtime from 3.3.6 (no pseudo console code) to 3.4.6 (the new
pseudo console architecture).

Fixes: bb4285206207 ("Cygwin: pty: Implement new pseudo console support.")
Assisted-by: Claude Opus 4.7 (1M context)
Signed-off-by: Johannes Schindelin <[email protected]>
Reviewed-by: Takashi Yano <[email protected]>
---
 winsup/cygwin/dtable.cc            | 14 ++++++++++-
 winsup/cygwin/local_includes/tty.h |  5 ++++
 winsup/cygwin/tty.cc               | 37 ++++++++++++++++++++++++++++++
 3 files changed, 55 insertions(+), 1 deletion(-)

diff --git a/winsup/cygwin/dtable.cc b/winsup/cygwin/dtable.cc
index f1832a169..e4d1cdf8f 100644
--- a/winsup/cygwin/dtable.cc
+++ b/winsup/cygwin/dtable.cc
@@ -326,7 +326,19 @@ dtable::init_std_file_from_handle (int fd, HANDLE handle)
       if (CTTY_IS_VALID (myself->ctty))
 	dev.parse (myself->ctty);
       else
-	dev.parse (FH_CONSOLE);
+	{
+	  /* Check whether the inherited console is actually a pseudo
+	     console bridging a pty.  This happens when our non-Cygwin
+	     parent was itself spawned by a Cygwin process from a pty
+	     (e.g. bash spawning git.exe which then spawns vim).  In
+	     that case, connect to the pty slave instead of treating
+	     the handle as a real console. */
+	  int pcon_minor = cygwin_shared->tty.find_pcon_pty ();
+	  if (pcon_minor >= 0)
+	    dev.parse (FHDEV (DEV_PTYS_MAJOR, pcon_minor));
+	  else
+	    dev.parse (FH_CONSOLE);
+	}
     }
   else if (GetCommState (handle, &dcb))
     /* FIXME: Not right - assumes ttyS0 */
diff --git a/winsup/cygwin/local_includes/tty.h b/winsup/cygwin/local_includes/tty.h
index 0adad03e6..507f7772e 100644
--- a/winsup/cygwin/local_includes/tty.h
+++ b/winsup/cygwin/local_includes/tty.h
@@ -181,6 +181,10 @@ public:
   void wait_fwd ();
   bool pty_input_state_eq (xfer_dir x) { return pty_input_state == x; }
   bool nat_fg (pid_t pgid);
+  bool has_active_pcon () const
+    { return pcon_activated && switch_to_nat_pipe; }
+  bool has_pcon_and_owner (DWORD pid) const
+    { return pcon_activated && switch_to_nat_pipe && nat_pipe_owner_pid == pid; }
   friend class fhandler_pty_common;
   friend class fhandler_pty_master;
   friend class fhandler_pty_slave;
@@ -199,6 +203,7 @@ public:
   int connect (int);
   void init ();
   tty_min *get_cttyp ();
+  int find_pcon_pty ();
   int attach (int n);
   static void init_session ();
   friend class lock_ttys;
diff --git a/winsup/cygwin/tty.cc b/winsup/cygwin/tty.cc
index e8083dc1f..667aa2682 100644
--- a/winsup/cygwin/tty.cc
+++ b/winsup/cygwin/tty.cc
@@ -123,6 +123,43 @@ tty_list::init ()
     }
 }
 
+/* Search for a pty whose pseudo console owns our console.
+   Return tty minor number or -1 if not found.
+   Called from init_std_file_from_handle() for processes started by
+   non-Cygwin parents to detect that inherited console handles are
+   from a pcon-backed pty.
+
+   The cheap precondition (any tty with pcon active in shared memory)
+   short-circuits the common case where no pty has a pseudo console
+   active, avoiding the GetConsoleProcessList() LPC call entirely. */
+int
+tty_list::find_pcon_pty ()
+{
+  DWORD pids[128];
+  DWORD count = 0;
+  bool got_pids = false;
+
+  for (int i = 0; i < NTTYS; i++)
+    {
+      if (!ttys[i].has_active_pcon ())
+	continue;
+
+      /* Fetch the console process list lazily, only on first candidate. */
+      if (!got_pids)
+	{
+	  count = GetConsoleProcessList (pids, 128);
+	  if (!count)
+	    return -1;
+	  got_pids = true;
+	}
+
+      for (DWORD j = 0; j < count; j++)
+	if (ttys[i].has_pcon_and_owner (pids[j]))
+	  return i;
+    }
+  return -1;
+}
+
 /* Search for a free tty and allocate it.
    Return tty number or -1 if error.
  */
-- 
2.51.0