Tentative "login:" loop fix

Jody Bruchon <[email protected]> Tue, 03 Mar 2015 23:07:47 -0500
Newsgroups org.kernel.vger.linux-8086
Message-ID <[email protected]>
There's this piece of code with no comments in arch/i86/mm/malloc.c:

if (currentp->t_begstack > currentp->t_endbrk)
  if(len > currentp->t_endseg - 0x1000) {
     return -ENOMEM;
}

I don't understand why t_endseg is being reduced by 0x1000 before 
checking and can't find any justification for it anywhere else in the 
kernel code; fs/exec.c:sys_execve() sets t_endseg and only says "needed 
for sys_brk()" which is of no help. Documentation/text/bin_formats.txt 
mentions 0x1000 as part of an example while discussing the ELKS 
executable file format but it doesn't quite seem to be related. From 
what I can gather, ELKS uses 0x1000 as a data segment offset in some 
places but I can't see any of those being related to what's going on in 
sys_brk()...

One thing is for certain: this check is causing the frustrating problem 
with the "login:" prompt that never ends. Removing the offset in the 
check fixes the login loop problem (which has plagued ELKS for a long 
time) and I can't seem to trigger any bad behavior in any random 
userspace programs with the check removed.

The t_endseg check code was originally introduced when ELKS executable 
format support was put into CVS in 2003 or so. If anyone knows why the 
0x1000 reduction was there, please explain it to me. Otherwise I'm 
leaving the change I've already committed in place which fixes this 
problem for now so we can worry about other things.

-Jody