[Turnip/Adreno 702] Vulkan compute MUL_MAT gives wrong results for,small batch sizes (n=1..~8), independent of data type

Hi all,

I'm seeing incorrect numerical results from Vulkan compute matrix
multiplication on a Turnip-driven Adreno 702 (Qualcomm QRB2210, Arduino
Uno Q board), reproducible with ggml's upstream test-backend-ops suite.
Writing this up because the pattern is very consistent and I've been able
to rule out several likely causes already.

=2D-----------------------------------------------------------------------=
=2D-
Hardware / software
=2D-----------------------------------------------------------------------=
=2D-
GPU:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Turnip Adreno (TM) 702
vendorID:=C2=A0 =C2=A0 =C2=A0 =C2=A00x5143 (Qualcomm)
Mesa:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0built from git main, driverV=
ersion reported as 26.2.99
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (also reproduces =
on the distro-shipped Mesa version)
maxComputeSharedMemorySize: 16384 bytes
Test tool:=C2=A0 =C2=A0 =C2=A0 ggml's test-backend-ops (from ggml-org/llam=
a.cpp), Vulkan
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 backend, built na=
tively/cross-compiled aarch64


MUL_MAT (general matrix x matrix / matrix x vector multiply) produces
numerically wrong output when the batch dimension n is small (n=3D1 throug=
h
roughly 8, depending on k), across every plain floating point type tested
(f32xf32, f16xf32, bf16xf32 - no quantization involved). Larger n (>=3D9,
or sufficiently large n*k) produces correct results in the same test run,
with the same operands.

This is significant in practice because n=3D1 is the standard shape for
single-token autoregressive LLM decoding, i.e. this hits ggml/llama.cpp's
most common real-world workload, not an edge case.

=2D-----------------------------------------------------------------------=
=2D-
Reproduction
=2D-----------------------------------------------------------------------=
=2D-
Build ggml/llama.cpp's test-backend-ops with the Vulkan backend enabled
and run:

 =C2=A0 =C2=A0 ./test-backend-ops test -o MUL_MAT

Representative output (trimmed, full log available on request):

 =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D1,k=3D256,...): FAIL =
ERR=3D1.214182167
 =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D2,k=3D256,...): FAIL =
ERR=3D0.738095533
 =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D3,k=3D256,...): FAIL =
ERR=3D0.964137248
 =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D4,k=3D256,...): FAIL =
ERR=3D1.433633782
 =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D5,k=3D256,...): FAIL =
ERR=3D1.523953343
 =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D6,k=3D256,...): FAIL =
ERR=3D0.961054406
 =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D7,k=3D256,...): FAIL =
ERR=3D0.725742246
 =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D8,k=3D256,...): FAIL =
ERR=3D0.977231894
 =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D9,k=3D256,...): OK

Same pattern for f16 and bf16 operands. Tolerance in these tests is
0.0005; observed errors are ~0.6-1.5, i.e. not float rounding noise, but
substantially wrong values.

The threshold is not a flat "n<9" rule - it interacts with total work
size. For example:

 =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D1,k=3D1024,bs=3D[3,2]=
,...): FAIL
 =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D8,k=3D1024,bs=3D[3,2]=
,...): OK
 =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D16,k=3D1024,bs=3D[3,2=
],...): OK

but

 =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D8,k=3D256,...): FAIL

suggesting the selection of a particular compute tile/pipeline variant
(rather than n alone) determines whether the result is correct - larger
total workloads appear to route through a different, working code path.

n=3D1 specifically fails almost unconditionally across every m/k/batch/
permutation combination tested in the suite (dozens of distinct shapes),
e.g.:

 =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D1056,n=3D1,k=3D128,...): FAI=
L
 =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D128,n=3D1,k=3D1056,...): FAI=
L
 =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D1057,n=3D1,k=3D129,...): FAI=
L
 =C2=A0 (and many more shapes/batch/permutation variants, all n=3D1, all F=
AIL)

=2D-----------------------------------------------------------------------=
=2D-
What I've already ruled out
=2D-----------------------------------------------------------------------=
=2D-
- Not quantization-specific: reproduces identically on plain f32xf32 with
 =C2=A0 no quantized types involved at all.
- Not an issue with a single dispatch path: reproduces whether the
 =C2=A0 mmvq quantized-matmul path is force-enabled or force-disabled via
 =C2=A0 ggml's GGML_VK_DISABLE_MMVQ (tested on a related quantized-MUL_MAT
 =C2=A0 investigation before broadening to this type-independent repro).
- Not fixed by forcing the smallest matmul tile size (mul_mat_s) via a
 =C2=A0 local patch adding a Qualcomm-specific override alongside the exis=
ting
 =C2=A0 Honeykrisp special-case in ggml-vulkan.cpp's device setup - the
 =C2=A0 numerical corruption persisted with the smallest tile forced.
- Not fixed on latest Mesa main (driverVersion 26.2.99 locally built),
 =C2=A0 reproduces identically to the distro-shipped Mesa version.
- A from-scratch, standalone Vulkan compute program (no ggml/llama.cpp
 =C2=A0 code at all) implementing the same dequant/dot-product math scalar=
,
 =C2=A0 with parallel shared-memory reduction, at batch sizes up to n=3D25=
6 and
 =C2=A0 k=3D4096, across f32/f16/q8_0/q4_0 (including a byte-faithful ggml
 =C2=A0 block_q4_0 layout with real fp16-stored scale) all produced CORREC=
T
 =C2=A0 results. This suggests the bug is specific to ggml's actual genera=
ted
 =C2=A0 shaders/dispatch parameters (tile selection, workgroup sizing, etc=
.)
 =C2=A0 rather than a blanket compute-correctness problem with basic
 =C2=A0 arithmetic, barriers, or shared memory on this device.

=2D-----------------------------------------------------------------------=
=2D-
Separately noticed (may be unrelated, mentioning for completeness)
=2D-----------------------------------------------------------------------=
=2D-
During the same test-backend-ops run, MUL_MAT invocations with bf16 and
quantized type_a operands also emit:

 =C2=A0 SPIR-V WARNING:
 =C2=A0 =C2=A0 In file ../src/compiler/spirv/spirv_to_nir.c:5651
 =C2=A0 =C2=A0 Unsupported SPIR-V capability: SpvCapabilityStorageBuffer8B=
itAccess=20
(4448)
 =C2=A0 =C2=A0 52 bytes into the SPIR-V binary

This is logged as a non-fatal warning rather than a hard failure, so
execution continues, but it suggests glslc/the driver's SPIR-V->NIR
consumer doesn't fully support a capability the shader declares
(VK_KHR_8bit_storage). Not sure if this is a contributing cause of the
n=3D1..8 MUL_MAT corruption above or a separate issue - flagging it since
it appears in the same log and involves the same driver/shader pipeline.

The same test run additionally showed CONV_2D producing large errors for
inputs with height>1, and GET_ROWS/SET_ROWS failing specifically for
q4_0/q8_0 types, plus a crash (uncaught exception in ggml_vk_submit)
partway through the CONV_2D tests. Happy to provide full logs for these
if useful, but keeping this report focused on the clearest, most
type-independent finding (MUL_MAT small-n).

Full test-backend-ops log, the ggml-vulkan.cpp patch I tried, and the
standalone Vulkan reproducer programs are all available on request. I'm
happy to test patches/build with extra debug output, or run the
IR3_SHADER_OVERRIDE_PATH / RenderDoc capture workflow described in the
freedreno docs if a maintainer can point me toward which shader variant
is selected for small-n MUL_MAT dispatches - I wasn't able to conclusively
identify that from the ggml-vulkan.cpp tile-selection code on my own.

Thanks,
Stef