Re: How to instrument a custom allocator properly?

On 8/6/19 11:47 AM, Ruurd Beerstra wrote:
> Hi,
> 
> We have a custom allocator which I have instrumented with VALGRIND 
> macro’s so we can run our software under valgrind and detect memory 
> abuse and leaks.
> 
> Works great, and has done so for many years now.
> 
> That allocator used to use sbrk() to get the raw memory in large chunks, 
> and then uses VALGRIND_MALLOCLIKE_BLOCK when it doled out smaller chunks 
> to the applications.
> 
> But sbrk() does not work well when a single application uses multiple 
> allocators, and our software now links with 3^rd party code which has 
> its own ideas about allocation.
> 
> That caused problems and instability.
> 
> So, our custom allocator now uses the system ‘malloc()’ instead of 
> sbrk() to allocate large chunks, which solves the interference problems 
> between allocators.
> That stabilized the software.
> 
> However, now valgrind gets confused: It sees overlapping mallocs() (the 
> smaller VALGRIND_MALLOCLIKE_BLOCKS are within the large chunks allocated 
> with malloc()).
> 
> I’m now struggling with the instrumentation in our allocator but I can’t 
> get it to work.
> 
> The error reports I get are just plain wrong: It says stuff like:
> 
> 2019-08-06 00:57:20: ==30298== Invalid read of size 1
> 
> 2019-08-06 00:57:20: ==30298==    at 0x4059CF: main (valgrindTest.c:126)
> 
> 2019-08-06 00:57:20: ==30298==  Address 0x5e43878 is 72 bytes inside a 
> block of size 65,536 alloc'd
> 
> 2019-08-06 00:57:20: ==30298==    at 0x4C29DDB: malloc 
> (vg_replace_malloc.c:299)
> 
> 2019-08-06 00:57:20: ==30298==    by 0x42DE98: mal_own_sbrk 
> (al_alloc.c:4251)
> 
> 2019-08-06 00:57:20: ==30298==    by 0x42E7E5: mal_inc_space 
> (al_alloc.c:4442)
> 
> 2019-08-06 00:57:20: ==30298==    by 0x4290E3: _mal_allocater 
> (al_alloc.c:1915)
> 
> 2019-08-06 00:57:20: ==30298==    by 0x428235: mal_allocater_c 
> (al_alloc.c:1608)
> 
> 2019-08-06 00:57:20: ==30298==    by 0x426C10: vmal_reserve_id 
> (al_alloc.c:1131)
> 
> 2019-08-06 00:57:20: ==30298==    by 0x42690B: mal_reserve_id 
> (al_alloc.c:999)
> 
> 2019-08-06 00:57:20: ==30298==    by 0x405495: main (valgrindTest.c:46)
> 
> So the offending byte is always in one of the “outer” malloced block of 
> 64KB, which is not what I want.
> The stack is that of the allocator itself, also not what I want (the 
> function is still called mal_own_sbrk, but that is  a misnomer now).
> 

I'm not sure what the problem is (I've seen these "invalid read" reports 
in various situations, e.g. when accessing "padding" space in a struct, 
or when accessing areas marked as VALGRIND_MAKE_MEM_NOACCESS).

But maybe take a look at this:

https://github.com/postgres/postgres/blob/master/src/backend/utils/mmgr/aset.c

That's a custom allocator used by PostgreSQL, using pretty much exactly 
the same design like you do (based on malloc etc.), and it's heavily 
valgrind-enabled. And it works quite fine. So maybe you could look at 
what we're doing there and copy the instrumentation ...

> Another example:
> 
> ==16897==  Uninitialised value was created by a stack allocation
> 
> ==16897==    at 0x492748: evt_get_3glLong_event (evt_fun.c:388)
> 
> ==16897==
> 
> The memory in question here is allocated using our allocator (and not on 
> the stack, so the reports is just wrong).
> Secondly, the rest of call stack is missing. Seems valgrind is confused 
> here.
> 

If I had to guess, I'd say this is complaining about the local variable 
not being initialized before use, not the memory allocated on heap.

regards
T.