Re: unable to read core generated by valgrind in gdb / aarch64

On 9/4/22 13:18, Philippe Waroquiers wrote:
> On Sun, 2022-09-04 at 00:14 +0200, Tomas Vondra wrote:
>>
>> Clearly, this is not an issue valgrind is meant to detect (like invalid
>> memory access, etc.) but an application issue. I've tried reproducing it
>> without valgrind, but it only ever happens with valgrind - my theory is
>> it's some sort of race condition, and valgrind changes the timing in a
>> way that makes it much more likely to hit. I need to analyze the core to
>> inspect the state more closely, etc.
>>
>> Any ideas what I might be doing wrong? Or how do I load the core file?
> 
> Rather than have the core dump and analyse it, you might interactively debug
> your program under valgrind.
> E.g. you might put a breakpoint on the assert or at some interesting points
> before the assert.
> 
> See https://valgrind.org/docs/manual/manual-core-adv.html#manual-core-adv.gdbserver
> for more info.
> 

I know, and I've used vgdbserver before. But sometimes that's not very 
practical for a number of reasons:

1) Our tests run mostly unattended, possibly even on CI machines that we 
don't have access to. And we don't want the machine to just sit there 
and wait for someone to debug it interactively, it's better to report 
the failure. Being able to inspect the core later would be helpful, though.

2) The error may be quite rare and/or hard to trigger - we regularly see 
race conditions that happen 1 in 1000 runs. True, I could automate that 
using a gdb script.

3) I'd bet it's not so simple in multi-process system that forks various 
processes that can trigger the issue. I'd have to do attach a gdb to 
each of those.

4) It's already pretty slow under valgrind, I'd bet it'll be even worse 
with gdb, but maybe it's not that bad. rpi4 is very constrained, though.

5) Race conditions are often very sensitive to change in timing. For 
example I've never seen this particular issue without valgrind. I can 
easily imagine gdb changing the timing just enough for the race 
condition to not happen.

regards
Tomas