set sysroot and debug-file-directory don't work together?

Greetings everyone,

I have a dummy question regarding using "set sysroot LOCAL_DIR" and
"set debug-file-directory SOME_OTHER_DIR" together to find symbols for
a debug session. I was under the impression, gdb is able to fallback
to debug-file-directory and do build-id style lookup if symbols cannot
be found in "sysroot", but it does not seem to be doing that so I am
curious if it's I am doing something wrong or this is not "supported".

A little context, we are doing embedded device debugging for Linux
based platform hence, the target device will not have sufficient space
to host all debug symbols. We have most of the debug symbols in the
"sysroot", and for something that is built outside, we want to export
additional symbols look up through debug-file-directory and debug over
gdb+gdbserver.

I will share my gdbinit as well. It isn't terribly sophisticated :)
and I am on GNU gdb (GDB) 11.2
```
set verbose on
set pagination off
set sysroot /path/to/rootfs
set debug-file-directory /path/to/more-debug/
set substitute-path /usr/src/debug /path/to/rootfs/usr/src/debug
set detach-on-fork on
set follow-fork-mode child
target remote :5039
where
```

I am trying to debug a binary whose symbols do not exist in the
"sysroot", so I expect the symbols to be picked up from
debug-file-directory, but above isn't working as it seems if sysroot
is specified, it will not fallback to debug-file-directory. If I
comment out sysroot line, GDB will attempt to load the debug symbols
through the local build-id/ mechanism as I would expect, but then it
will not have additional libs debug provided by the "sysroot". So yap,
this is what I am struggling with currently and would like to know if
I am not configuring something right or this is expected behavior and
how to get it to work the way I intend to use this.

Thanks,
Mike