[RFC] Running Wine under AddressSanitizer

Hi all,

I've been working on this for a while and the list of changes are
getting long. So I think this is a good time as ever to announce it,
and get some feedback to make sure I haven't gone too far off the
rails.

This email is a bit long.

What is this about?
===================

AddressSanitizer (ASan for short) is one of a class of tools that help
with discovering common coding mistakes. ASan specifically detects
memory problems, like buffer overflows, use after frees, memory leaks,
etc. It has been extremely successful in the past decade, and the list
of bugs it helped discover is very very long.

It has even already seen successes in the wine project as well.
@bernhardu was able to get the PE side of wine to work with ASan (with
some exceptions) using llvm-mingw, after he fixed [1] a handful of
incompatibility problems in llvm compiler-rt. And with that many bugs
[2] were then discovered and fixed.

And my plan is to take this a step further.

[1]: https://github.com/llvm/llvm-project/pulls?q=is%3Apr+is%3Aclosed+author%3Abernhardu+asan
[2]: https://gitlab.winehq.org/bernhardu/wine/-/blob/asan-pe_2025-04-26_wine-10.7/README-ASan.md?ref_type=heads#known-commits-fixing-issues-found-by-asan-pe-or-related

Goals
=====

I want this to be a quick and easy to use tool, the first thing one
would reach for when seeing something that might be a memory related
problem. e.g. I spend 5 minutes reproducing the bug with an
ASan-enabled wine build, if it catches it, then I am done; otherwise I
move on with my usual debugging routine, without having wasted too
much time. I also want it to be a first line of defence, which runs in
CI to catch occasional bugs that slip through reviews.

In my mind, this means 1) it must be easy to set up, ideally just a
flag passed to `configure`, no fuss. 2) it should cover as much of
wine as possible, both PE and unix, and ideally all PE DLLs.

Current status
==============

Wine builds and runs. A majority of test cases pass, but some failures
seem to be ASan related, which I still need to investigate.

Both stack and heap problems are detected and reported. Stack traces
are captured, but only on the PE side, and are not symbolized yet - I
need to pipe the numbers into llvm-symbolizer. Stack traces aren't
being saved for heap allocations and frees, so for use-after-frees the
reports aren't very useful yet. Fiber support is TODO. Leak detection
is TODO as well.

Works done so far
================

You can find my branch here: [3]. To build it, you need either clang,
or a slightly patched [4] gcc, and you just run configure with
`--enable-sanitize`.

I tried to put into the commit messages as comprehensive a list of
what I did as possible, but I'll also provide an overview here. It's
very abbreviated to not make this email even longer. If you need to
know of how ASan works, there is a quick explanation here [5].

Since no existing ASan runtimes can support the unix and the PE side
at the same time, I chose to write my own ASan runtime. The runtime is
made part of ntdll, since that's the first DLL that loads. A
non-standard shadow offset was chosen, so in the new WoW64 mode the
32-bit and the 64-bit side can share a common shadow. Shadow memory is
mapped very early inside `virtual_init` to make sure we can enable
ASan for as much code as possible (though doing it inside the loader
is also an option). Some care needed to be taken to make sure no false
positives show up after stack manipulations (e.g. unwinding).
Redzoning and quarantining is added to ntdll's heap allocator, though
I don't like that.

[3]: https://gitlab.winehq.org/yshui/wine/-/commits/address-sanitizer-presentable
[4]: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=119926
[5]: https://github.com/google/sanitizers/wiki/AddressSanitizerAlgorithm

Request for help
================

I intend to continue working on this, and I don't really see any
technical challenges that would make this fundamentally undoable. But
I do worry that certain things I had/will have to do might make this
not upstreamable. So, I think I need some guidance. (I will open a
draft MR on winehq gitlab shortly, if you prefer commenting there).

Bad things I did to get it working
----------------------------------

There are certain things I did that didn't feel good when I did them,
so I am listing them here to see if anyone has a better way to do
them.

- ASan runtime as part of ntdll. I can't think of a better place to
put it, since the runtime needs to be available before PE code runs.
This also means all these functions are exported from ntdll, and I
have yet to find a way to not export them when ASan is disabled.

- dinput/tests and ntoskrnl.exe/tests need to be linked with ntdll,
since they need to get their ASan runtime from ntdll.

- Things I added to winecrt0. ASan uses some global variables for
information such as where the shadow was mapped, and dynamically
linked libraries need this information too. On the unix side the
dynamic linker will nicely find them for you, but the same doesn't
work with PE. So ASan duplicates these variables for each DLLs. In the
vanilla ASan runtime, this comes from linking with a "dynamic runtime
thunk"; in wine, I felt winecrt0 is the best place for them? These
variables also need to be initialized, which means calling an ASan
interface function which is exported from ntdll, from
`DllMainCRTStartup`. This meant I had to add `-Wl,--{start,end}-group`
for starters, because now libntdll.a and libwinecrt0.a have circular
dependencies. Additionally though this works for other DLLs, it
becomes a problem when winecrt0 is linked into ntdll, which creates a
importing-from-self situation. I added a `DllMainCRTStartup` to ntdll,
so the one from libwinecrt0.a is not pulled in, thus breaking the
cycle. But it definitely feels wrong. (Vanilla ASan does this
initialization by putting functions into the `.CRT$XIB` section, which
the CRT will call on DLL load. I was told wine CRT doesn't implement
this).

- `__WINE_FRAME`. I don't fully understand this, but IIUC this is a
hacky exception/unwinding mechanism, for when the compiler doesn't
support native __try/__except? These are put on the stack which later
the unwinder will look for to find the correct exception handler. This
doesn't work since ASan uses fake stack frames, so the unwinder can't
find these `__WINE_FRAME`s and thus fails. I get this somewhat working
by translating between fake and real stack addresses, ew. I don't know
much about SEH, is it really not possible to use it for these
__try/__excepts as well?

Things I need advices on
-----------------------

- Heap. I added ASan redzoning directly to the ntdll heap allocator,
which I don't like since it added a lot of complexity to it, and it's
not available on the unix side. I want to write an ASan specific
allocator, since performance and memory efficiency aren't major
concerns here, this allocator can be made simple. But I am not 100%
sure if this is a good idea.

- Stack trace. What's the best way of capturing stack traces from PE
_and_ unix, if we are in a syscall/unix call for example?

- Shadow offset. This is a bit complicated. clang supports dynamic
shadow offset but mingw-gcc doesn't (it doesn't officially have ASan
support), so for max compatibility we want a fixed shadow offset. If
we want a single shadow to be shared between 32 and 64 bit in the case
of new WoW64, the shadow offset needs to be below 4G, which basically
precludes non-relocatable 64-bit PE from being loaded (the shadow
takes 16 TiB); or we choose a high shadow offset for 64-bit to avoid
conflicts, but that means we need to mirror this shadow into the lower
4G of the address space for 32-bit, which complicates the
initialization phase.

(If you made it to the end, thank you!)

--

Regards
Yuxuan Shui