Re: MLton and Valgrind

Hi,

I've been cutting down my example code some more,  trying to get it to
the smallest I can come up with that still reproduces the problem.

If you compile the attached with
  mlton -codegen c -link-opt -ldl test.mlb

and then execute "valgrind ./test 0",  no errors are reported.
Executing "valgrind ./test 1" shows errors similar to the ones I reported
in my previous email.  Note that apparently the .so file does not even
need to exist for the errors to come up.  I'm more and more getting the
feeling that this is just some misunderstanding between Valgrind and
MLton-generated code but it is curious nonetheless since I can execute
much larger MLton-compiled SML programs without any errors reported by
Valgrind.  So it doesn't seem like there is a fundamental problem.

Here's some info about the system where this is happening.  I suspect
that this behavior will be very system specific and not reproducible on
other systems.  Let me know if you would like more info.

> uname -a
Linux 2.6.32-32-generic #62-Ubuntu SMP Wed Apr 20 21:54:21 UTC 2011 i686 GNU/Linux
> gcc --version
gcc (Ubuntu 4.4.3-4ubuntu5) 4.4.3
> mlton
MLton 20100608 (built Tue Jun  8 13:48:45 EDT 2010 on fenrirvm-ubuntu-10-04-i386)
> valgrind --version
valgrind-3.6.0.SVN-Debian

David

On 3/3/2012 12:17 PM, Matthew Fluet wrote:
> On Thu, Mar 1, 2012 at 4:24 PM, David Hansel
> <[email protected]> wrote:
>> I've been trying to use Valgrind (valgrind.org) to try to find out if
>> there are any memory errors in FFI code called by our MLton program
>> (this is on Ubuntu 10.04).
> 
> I've not tried Valgrind with MLton; and while I've heard the
> 1-sentence summary of Valgrind, I've never really worked with the
> tools.
> 
>> Running any MLton compiled program under Valgrind at first gave me a
>> large number of errors, until I tried compiling with the C codegen
>> (instead of the native).  After changing that the errors seem to be
>> gone.
> 
> That makes some sense, as the native codegen doesn't follow C
> conventions, either statically or dynamically.  Statically, that means
> that dumping the referenced labels in native codegen object file may
> not be interpretable by Valgrind as functions with well-defined
> boundaries.  Dynamically, that means that if Valgrind is inspecting a
> running process, then $esp isn't pointing to the top of the C stack
> when executing within a native codegen object file.  On the other
> hand, if Valgrind only inspects the program state at calls to C
> functions (e.g., malloc/free/...), then the dynamic program state
> (e.g., $esp) should look reasonably like a C program at those points.
> 
>> However, after adding FFI code to the MLton program I do occasionally
>> get memory errors reported by Valgrind, presumably during garbage
>> collection.  Note that the program runs fine and does not crash (with or
>> without Valgrind). The errors reported look like this:
>>
>> ==19049== Invalid write of size 1
>> ==19049==    at 0x4027448: memset (mc_replace_strmem.c:602)
>> ==19049==    by 0x805157C: setCardMapAndCrossMap (in /home/hansel/test1)
>> ==19049==  Address 0x467b000 is not stack'd, malloc'd or (recently) free'd
>>
>> ==19049== Invalid read of size 1
>> ==19049==    at 0x4026970: memcpy (mc_replace_strmem.c:497)
>> ==19049==    by 0x805183D: forwardObjptr (in /home/hansel/test1)
>> ==19049==  Address 0x470a5ce is not stack'd, malloc'd or (recently) free'd
> 
> I agree with Wesley that these are true statements.  The memory
> obtained for the ML heap (which includes the card map and cross map)
> is obtained my mmap (and resized by mremap and released by unmap).
> So, it is true that any manipulation of addresses within the ML heap
> is "no stack'd, malloc'd or (recently) free'd".  On the other hand, I
> would imagine that mmap is a sufficiently widespread idiom for
> obtaining large blocks of memory that Valgrind would recognize it; but
> I may be wrong.  Of course, even if Valgrind recognized mmap obtained
> memory, it is quite true that the GC does mem* operations within the
> heap long after it was initially obtained.
> 
>> I'm not an expert with Valgrind and I guess it is possible that these
>> are somehow wrongfully detected and there is no problem with MLton
>> itself.  Has anybody here tried using Valgrind with MLton?  Is there
>> something that I should be aware of?  Should I just ignore these errors
>> or are they maybe an indication of something going wrong in MLton?
>>
>> I can provide some example code which produces the errors mentioned
>> when running under Valgrind.  For the same code the errors go away if
>> I remove the FFI functions from the .mlb file (the FFI functions are
>> not actually called by the code,  it seems that just loading the .so
>> file creates the problem).
> 
> That is a little bit strange.  I suppose the shared library could have
> some initialization code that triggers something in Valgrind, but it
> is curious that it manifests itself with warnings about the ML heap.
> The shared library ought not to have any reference to the ML heap.
> 
>> If you are interested, please let me know where I can upload
>> the (2KB) .zip file.
> 
> You can upload files too large for the mailing list to:
>   http://mlton.org/TemporaryUpload

------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing 
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/

_______________________________________________
MLton-devel mailing list
[email protected]; [email protected]
https://lists.sourceforge.net/lists/listinfo/mlton-devel

local ann "allowFFI true" in

  $(SML_LIB)/basis/basis.mlb
  $(SML_LIB)/basis/mlton.mlb
  $(SML_LIB)/basis/c-types.mlb
  test.sml

end 
in
end

val RTLD_GLOBAL = 0wx100
val RTLD_LAZY   = 0wx1
val RTLD_LOCAL  = 0wx0
val RTLD_NOW    = 0wx2
val dlopen      = _import "dlopen" : string * C_UInt.word -> MLton.Pointer.t;

fun loop (s, i) =
    if   i>0
    then s^(loop (s, i-1))
    else s

val _ = 
    (if   0 <> ((valOf (Int.fromString (hd (CommandLine.arguments ())))) handle _ => 1)
     then (dlopen ("foo.so\000", C_UInt.orb (RTLD_LAZY, RTLD_LOCAL)); ())
     else ();
     loop ("x", 1000))