Re: Tentative "login:" loop fix

On Tue, 03 Mar 2015 23:07:47 -0500
Jody Bruchon <[email protected]> wrote:

> There's this piece of code with no comments in arch/i86/mm/malloc.c:
> 
> if (currentp->t_begstack > currentp->t_endbrk)
>   if(len > currentp->t_endseg - 0x1000) {
>      return -ENOMEM;
> }
> 
> I don't understand why t_endseg is being reduced by 0x1000 before 
> checking and can't find any justification for it anywhere else in the 
> kernel code; fs/exec.c:sys_execve() sets t_endseg and only says "needed 
> for sys_brk()" which is of no help. Documentation/text/bin_formats.txt 
> mentions 0x1000 as part of an example while discussing the ELKS 
> executable file format but it doesn't quite seem to be related. From 
> what I can gather, ELKS uses 0x1000 as a data segment offset in some 
> places but I can't see any of those being related to what's going on in 
> sys_brk()...
> 
> One thing is for certain: this check is causing the frustrating problem 
> with the "login:" prompt that never ends. Removing the offset in the 
> check fixes the login loop problem (which has plagued ELKS for a long 
> time) and I can't seem to trigger any bad behavior in any random 
> userspace programs with the check removed.

Why does it cause the login prompt problem ?

Removing the check fixes it but you don't say why that cures it, just
that it does.

Alan