Re: Input on PRs 99/100

I have some further analysis from Jakub in regards to this issue.

On Fri, 2005-09-23 at 13:48 +0100, Andrew Josey wrote:
> I don't think it should be allowed.  The relevant part of the
> description of the mprotect() ENOMEM error in the standard is
> "Addresses in the range [addr,addr+len) are invalid for the address
> space of a process", so clearly the check for (this type of) ENOMEM
> condition has to be done against the address space of the calling
> process.  Since 32-bit and 64-bit processes have different address
> spaces the check needs to be done differently for 32-bit and 64-bit
> processes in order for ENOMEM to be correctly detected in both cases.
> 
The problem is not that ENOMEM is not correctly detected in both cases,
it is correctly detected. However, mprotect also does what it is allowed
to do by:

"When mprotect() fails for reasons other than [EINVAL], the protections
on some  of the pages in the range [addr,addr+len) may have been
changed."  

And the crash is caused by the changed permissions.

As minimal testcase, try e.g.
#include <sys/mman.h>

int main (void)
{
  mprotect ((void *) 0x08048000, 0xffff0000, PROT_NONE);
  return 0;
}
on x86_64 and i?86, as a -m32 binary (assuming the binary starts at
0x8048000, which is the default).
On x86-64 kernel, this results in:
mprotect(0x8048000, 4294901760, PROT_NONE) = -1 ENOMEM (Cannot allocate
memory)
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
while on i?86 kernel:
mprotect(0x8048000, 4294901760, PROT_NONE) = -1 ENOMEM (Cannot allocate
memory)
exit_group(0)                           = ?

Both behaviours are conforming.

> Even if it could be argued that detecting ENOMEM should not be
> required in this case, the only allowed alternative behaviours are
> for mprotect() to succeed or to return a different error indication.  
> It should not hang, as the application has not done anything that
> the standard says is undefined behaviour.

1) mprotect did not hang, the hang is caused by wrong protection flags
of the SIGSEGV signal handler, so when kernel sends the signal, it
immediately crashes again
2) "has not done anything that the standard says is undefined behaviour"
Changing protection of pages not under application control, but which
contain the implementation's code/data are undefined behaviour
definitely.