Re: [Turnip/Adreno 702] Vulkan compute MUL_MAT gives wrong results for,small batch sizes (n=1..~8), independent of data type

Hello Stefan,

I tested MUL_MAT on A750 and got just these failures:
MUL_MAT(type_a=f16,type_b=f32,m=64,n=45,k=128,bs=[8,1],nr=[4,1],per=[0,1,2,3],k_v=0,o=1)
MUL_MAT(type_a=f16,type_b=f32,m=128,n=45,k=64,bs=[8,1],nr=[4,1],per=[0,1,2,3],k_v=0,o=1)
MUL_MAT(type_a=f32,type_b=f32,m=64,n=77,k=77,bs=[12,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1)
MUL_MAT(type_a=q4_0,type_b=f32,m=576,n=512,k=576,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1)

So this could be something isolated to the A702.

Regarding the `SPIR-V WARNING`. A702 doesn't support KHR_8bit_storage and
storageBuffer8BitAccess (A750 does have support for this), and some of 
the tests
might be trying to compile a shader requiring that.

In ggml-vulkan.cpp I don't see VK_KHR_8bit_storage being enabled or 
checked for (I can see
16bit is) and I can see mul_mat_vec_base.glsl requires 
GL_EXT_shader_8bit_storage, so the
issue here could be that the tests aren't adhering to the Vulkan spec 
and end up with broken
result. The way to check this would be to use Vulkan Validation Layers.

Anyway, it would be good if you could file the issue here instead:
https://gitlab.freedesktop.org/mesa/mesa/-/work_items

Regards,
Karmjit


On 21/07/2026 03:10, Stefan Rinass wrote:
> Hi all,
>
> I'm seeing incorrect numerical results from Vulkan compute matrix
> multiplication on a Turnip-driven Adreno 702 (Qualcomm QRB2210, Arduino
> Uno Q board), reproducible with ggml's upstream test-backend-ops suite.
> Writing this up because the pattern is very consistent and I've been able
> to rule out several likely causes already.
>
> -------------------------------------------------------------------------- 
>
> Hardware / software
> -------------------------------------------------------------------------- 
>
> GPU:            Turnip Adreno (TM) 702
> vendorID:       0x5143 (Qualcomm)
> Mesa:           built from git main, driverVersion reported as 26.2.99
>                 (also reproduces on the distro-shipped Mesa version)
> maxComputeSharedMemorySize: 16384 bytes
> Test tool:      ggml's test-backend-ops (from ggml-org/llama.cpp), Vulkan
>                 backend, built natively/cross-compiled aarch64
>
>
> MUL_MAT (general matrix x matrix / matrix x vector multiply) produces
> numerically wrong output when the batch dimension n is small (n=1 through
> roughly 8, depending on k), across every plain floating point type tested
> (f32xf32, f16xf32, bf16xf32 - no quantization involved). Larger n (>=9,
> or sufficiently large n*k) produces correct results in the same test run,
> with the same operands.
>
> This is significant in practice because n=1 is the standard shape for
> single-token autoregressive LLM decoding, i.e. this hits ggml/llama.cpp's
> most common real-world workload, not an edge case.
>
> -------------------------------------------------------------------------- 
>
> Reproduction
> -------------------------------------------------------------------------- 
>
> Build ggml/llama.cpp's test-backend-ops with the Vulkan backend enabled
> and run:
>
>     ./test-backend-ops test -o MUL_MAT
>
> Representative output (trimmed, full log available on request):
>
>   MUL_MAT(type_a=f32,type_b=f32,m=16,n=1,k=256,...): FAIL ERR=1.214182167
>   MUL_MAT(type_a=f32,type_b=f32,m=16,n=2,k=256,...): FAIL ERR=0.738095533
>   MUL_MAT(type_a=f32,type_b=f32,m=16,n=3,k=256,...): FAIL ERR=0.964137248
>   MUL_MAT(type_a=f32,type_b=f32,m=16,n=4,k=256,...): FAIL ERR=1.433633782
>   MUL_MAT(type_a=f32,type_b=f32,m=16,n=5,k=256,...): FAIL ERR=1.523953343
>   MUL_MAT(type_a=f32,type_b=f32,m=16,n=6,k=256,...): FAIL ERR=0.961054406
>   MUL_MAT(type_a=f32,type_b=f32,m=16,n=7,k=256,...): FAIL ERR=0.725742246
>   MUL_MAT(type_a=f32,type_b=f32,m=16,n=8,k=256,...): FAIL ERR=0.977231894
>   MUL_MAT(type_a=f32,type_b=f32,m=16,n=9,k=256,...): OK
>
> Same pattern for f16 and bf16 operands. Tolerance in these tests is
> 0.0005; observed errors are ~0.6-1.5, i.e. not float rounding noise, but
> substantially wrong values.
>
> The threshold is not a flat "n<9" rule - it interacts with total work
> size. For example:
>
>   MUL_MAT(type_a=f32,type_b=f32,m=16,n=1,k=1024,bs=[3,2],...): FAIL
>   MUL_MAT(type_a=f32,type_b=f32,m=16,n=8,k=1024,bs=[3,2],...): OK
>   MUL_MAT(type_a=f32,type_b=f32,m=16,n=16,k=1024,bs=[3,2],...): OK
>
> but
>
>   MUL_MAT(type_a=f32,type_b=f32,m=16,n=8,k=256,...): FAIL
>
> suggesting the selection of a particular compute tile/pipeline variant
> (rather than n alone) determines whether the result is correct - larger
> total workloads appear to route through a different, working code path.
>
> n=1 specifically fails almost unconditionally across every m/k/batch/
> permutation combination tested in the suite (dozens of distinct shapes),
> e.g.:
>
>   MUL_MAT(type_a=f32,type_b=f32,m=1056,n=1,k=128,...): FAIL
>   MUL_MAT(type_a=f32,type_b=f32,m=128,n=1,k=1056,...): FAIL
>   MUL_MAT(type_a=f32,type_b=f32,m=1057,n=1,k=129,...): FAIL
>   (and many more shapes/batch/permutation variants, all n=1, all FAIL)
>
> -------------------------------------------------------------------------- 
>
> What I've already ruled out
> -------------------------------------------------------------------------- 
>
> - Not quantization-specific: reproduces identically on plain f32xf32 with
>   no quantized types involved at all.
> - Not an issue with a single dispatch path: reproduces whether the
>   mmvq quantized-matmul path is force-enabled or force-disabled via
>   ggml's GGML_VK_DISABLE_MMVQ (tested on a related quantized-MUL_MAT
>   investigation before broadening to this type-independent repro).
> - Not fixed by forcing the smallest matmul tile size (mul_mat_s) via a
>   local patch adding a Qualcomm-specific override alongside the existing
>   Honeykrisp special-case in ggml-vulkan.cpp's device setup - the
>   numerical corruption persisted with the smallest tile forced.
> - Not fixed on latest Mesa main (driverVersion 26.2.99 locally built),
>   reproduces identically to the distro-shipped Mesa version.
> - A from-scratch, standalone Vulkan compute program (no ggml/llama.cpp
>   code at all) implementing the same dequant/dot-product math scalar,
>   with parallel shared-memory reduction, at batch sizes up to n=256 and
>   k=4096, across f32/f16/q8_0/q4_0 (including a byte-faithful ggml
>   block_q4_0 layout with real fp16-stored scale) all produced CORRECT
>   results. This suggests the bug is specific to ggml's actual generated
>   shaders/dispatch parameters (tile selection, workgroup sizing, etc.)
>   rather than a blanket compute-correctness problem with basic
>   arithmetic, barriers, or shared memory on this device.
>
> -------------------------------------------------------------------------- 
>
> Separately noticed (may be unrelated, mentioning for completeness)
> -------------------------------------------------------------------------- 
>
> During the same test-backend-ops run, MUL_MAT invocations with bf16 and
> quantized type_a operands also emit:
>
>   SPIR-V WARNING:
>     In file ../src/compiler/spirv/spirv_to_nir.c:5651
>     Unsupported SPIR-V capability: 
> SpvCapabilityStorageBuffer8BitAccess (4448)
>     52 bytes into the SPIR-V binary
>
> This is logged as a non-fatal warning rather than a hard failure, so
> execution continues, but it suggests glslc/the driver's SPIR-V->NIR
> consumer doesn't fully support a capability the shader declares
> (VK_KHR_8bit_storage). Not sure if this is a contributing cause of the
> n=1..8 MUL_MAT corruption above or a separate issue - flagging it since
> it appears in the same log and involves the same driver/shader pipeline.
>
> The same test run additionally showed CONV_2D producing large errors for
> inputs with height>1, and GET_ROWS/SET_ROWS failing specifically for
> q4_0/q8_0 types, plus a crash (uncaught exception in ggml_vk_submit)
> partway through the CONV_2D tests. Happy to provide full logs for these
> if useful, but keeping this report focused on the clearest, most
> type-independent finding (MUL_MAT small-n).
>
> Full test-backend-ops log, the ggml-vulkan.cpp patch I tried, and the
> standalone Vulkan reproducer programs are all available on request. I'm
> happy to test patches/build with extra debug output, or run the
> IR3_SHADER_OVERRIDE_PATH / RenderDoc capture workflow described in the
> freedreno docs if a maintainer can point me toward which shader variant
> is selected for small-n MUL_MAT dispatches - I wasn't able to 
> conclusively
> identify that from the ggml-vulkan.cpp tile-selection code on my own.
>
> Thanks,
> Stef