Re: [PATCH v2] Cygwin: Ensure unused fd available for open()

Hi Jon,

On 6/1/2026 6:30 AM, Jon Turney wrote:
> On 28/05/2026 06:42, Mark Geisert wrote:
>> The existing logic for open() assumes an fd is always available in
>> the fdtable for a created file.  This leads to a situation where, if
>> there is no fd available due to the OPEN_MAX limit being hit, the
>> file is created but cannot be referenced by a Cygwin fd.
>>
>> Move the fd reservation code to an earlier location within open().
> 
> Hmm... the more I stare at cygheap_fdnew, the less sure I am I 
> understand what's going on.
> 
> I'm sure you considered this, but just so I can tell myself I've done 
> due diligence, perhaps you can briefly explain why this doesn't create 
> the opposite leak? (i.e. the reserved fd is released if actually opening 
> the file fails).

Sure.  What happens is that cygheap_fdnew doesn't mark the chosen fd 
reserved (i.e. the fdtable is not updated at all, yet), it's that the 
calling thread has locked the fdtable and knows where the first unused 
fd in fdtable is.

All the validations of open() parameters are done and eventually a file, 
pipe, device, socket, whatever open attempt at Windows level is done. 
If that succeeds, fdtable[fd] is updated with a pointer to the 
fhandler_XXX stuff being carried along in variable fh.  The fdtable is 
unlocked at the end of the __try block by a dtor (see below).

If that Windows-level open attempt fails, a __leave is performed to exit 
the enclosing __try block.  The destructor for cygheap_fdmanip, 
superclass of cygheap_fdnew, unlocks the fdtable.  fdtable[fd] is left 
as it was, NULL.

That's my story and I'm sticking to it, but I'm at the limits of my C++ 
knowledge.  The overloading of "fd" really makes it difficult to follow 
things.. but I have to admit this seems like tight code to me.  H/T to 
CGF warranted.

Feel free to ask more questions.
Thanks & Regards,

..mark