Re: [Turnip/Adreno 702] Vulkan compute MUL_MAT gives wrong results for,small batch sizes (n=1..~8), independent of data type

Hi Karmjit,

Thanks for testing on A750 and for the pointer on 8bit_storage - that
narrowed things down a lot faster than I expected.

I ran with VK_LAYER_KHRONOS_validation enabled as you suggested, and it
turned up something that goes a bit further than the 8bit_storage
mismatch: the validation layer reports that ggml is creating the
VkDevice itself in violation of the spec on this hardware, independent
of any specific shader.

The Adreno 702 only reports Vulkan 1.0.354 (no 1.1+, altrough being=20
capable according
to Wikipedia). Relevant validation errors:

 =C2=A0 Missing extension required by the device extension VK_KHR_16bit_st=
orage:
 =C2=A0 VK_KHR_storage_buffer_storage_class.

 =C2=A0 vkCreateBuffer(): pCreateInfo->usage has VkBufferUsageFlagBits val=
ues
 =C2=A0 (... VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT) that requires the
 =C2=A0 extensions VK_KHR_buffer_device_address or VK_EXT_buffer_device_ad=
dress.

 =C2=A0 vkAllocateMemory(): pAllocateInfo->pNext<VkMemoryAllocateFlagsInfo=
>.flags
 =C2=A0 has VkMemoryAllocateFlagBits values=20
(VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_BIT)
 =C2=A0 that requires the extensions VK_KHR_buffer_device_address.

My reading: on Vulkan 1.1/1.2+ devices, storage_buffer_storage_class and
buffer-device-address behavior are either promoted to core or commonly
enabled together with their prerequisites without issue, so this likely
never surfaces on more modern hardware (probably including the A750).

Thanks again for the quick and helpful response.

Regards,
Stef

Am 21.07.26 um 13:53 schrieb Karmjit Mahil:
> Hello Stefan,
>
> I tested MUL_MAT on A750 and got just these failures:
> MUL_MAT(type_a=3Df16,type_b=3Df32,m=3D64,n=3D45,k=3D128,bs=3D[8,1],nr=3D=
[4,1],per=3D[0,1,2,3],k_v=3D0,o=3D1)=20
>
> MUL_MAT(type_a=3Df16,type_b=3Df32,m=3D128,n=3D45,k=3D64,bs=3D[8,1],nr=3D=
[4,1],per=3D[0,1,2,3],k_v=3D0,o=3D1)=20
>
> MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D64,n=3D77,k=3D77,bs=3D[12,1],nr=3D=
[1,1],per=3D[0,1,2,3],k_v=3D0,o=3D1)=20
>
> MUL_MAT(type_a=3Dq4_0,type_b=3Df32,m=3D576,n=3D512,k=3D576,bs=3D[1,1],nr=
=3D[1,1],per=3D[0,1,2,3],k_v=3D0,o=3D1)=20
>
>
> So this could be something isolated to the A702.
>
> Regarding the `SPIR-V WARNING`. A702 doesn't support KHR_8bit_storage an=
d
> storageBuffer8BitAccess (A750 does have support for this), and some of=
=20
> the tests
> might be trying to compile a shader requiring that.
>
> In ggml-vulkan.cpp I don't see VK_KHR_8bit_storage being enabled or=20
> checked for (I can see
> 16bit is) and I can see mul_mat_vec_base.glsl requires=20
> GL_EXT_shader_8bit_storage, so the
> issue here could be that the tests aren't adhering to the Vulkan spec=20
> and end up with broken
> result. The way to check this would be to use Vulkan Validation Layers.
>
> Anyway, it would be good if you could file the issue here instead:
> https://gitlab.freedesktop.org/mesa/mesa/-/work_items
>
> Regards,
> Karmjit
>
>
> On 21/07/2026 03:10, Stefan Rinass wrote:
>> Hi all,
>>
>> I'm seeing incorrect numerical results from Vulkan compute matrix
>> multiplication on a Turnip-driven Adreno 702 (Qualcomm QRB2210, Arduino
>> Uno Q board), reproducible with ggml's upstream test-backend-ops suite.
>> Writing this up because the pattern is very consistent and I've been=20
>> able
>> to rule out several likely causes already.
>>
>> -----------------------------------------------------------------------=
=2D--=20
>>
>> Hardware / software
>> -----------------------------------------------------------------------=
=2D--=20
>>
>> GPU:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Turnip Adreno (TM) 702
>> vendorID:=C2=A0 =C2=A0 =C2=A0 =C2=A00x5143 (Qualcomm)
>> Mesa:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0built from git main, driv=
erVersion reported as 26.2.99
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (also reproduce=
s on the distro-shipped Mesa version)
>> maxComputeSharedMemorySize: 16384 bytes
>> Test tool:=C2=A0 =C2=A0 =C2=A0 ggml's test-backend-ops (from ggml-org/l=
lama.cpp),=20
>> Vulkan
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 backend, built =
natively/cross-compiled aarch64
>>
>>
>> MUL_MAT (general matrix x matrix / matrix x vector multiply) produces
>> numerically wrong output when the batch dimension n is small (n=3D1=20
>> through
>> roughly 8, depending on k), across every plain floating point type=20
>> tested
>> (f32xf32, f16xf32, bf16xf32 - no quantization involved). Larger n (>=3D=
9,
>> or sufficiently large n*k) produces correct results in the same test=20
>> run,
>> with the same operands.
>>
>> This is significant in practice because n=3D1 is the standard shape for
>> single-token autoregressive LLM decoding, i.e. this hits=20
>> ggml/llama.cpp's
>> most common real-world workload, not an edge case.
>>
>> -----------------------------------------------------------------------=
=2D--=20
>>
>> Reproduction
>> -----------------------------------------------------------------------=
=2D--=20
>>
>> Build ggml/llama.cpp's test-backend-ops with the Vulkan backend enabled
>> and run:
>>
>> =C2=A0 =C2=A0 ./test-backend-ops test -o MUL_MAT
>>
>> Representative output (trimmed, full log available on request):
>>
>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D1,k=3D256,...): FAI=
L=20
>> ERR=3D1.214182167
>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D2,k=3D256,...): FAI=
L=20
>> ERR=3D0.738095533
>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D3,k=3D256,...): FAI=
L=20
>> ERR=3D0.964137248
>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D4,k=3D256,...): FAI=
L=20
>> ERR=3D1.433633782
>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D5,k=3D256,...): FAI=
L=20
>> ERR=3D1.523953343
>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D6,k=3D256,...): FAI=
L=20
>> ERR=3D0.961054406
>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D7,k=3D256,...): FAI=
L=20
>> ERR=3D0.725742246
>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D8,k=3D256,...): FAI=
L=20
>> ERR=3D0.977231894
>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D9,k=3D256,...): OK
>>
>> Same pattern for f16 and bf16 operands. Tolerance in these tests is
>> 0.0005; observed errors are ~0.6-1.5, i.e. not float rounding noise, bu=
t
>> substantially wrong values.
>>
>> The threshold is not a flat "n<9" rule - it interacts with total work
>> size. For example:
>>
>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D1,k=3D1024,bs=3D[3,=
2],...): FAIL
>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D8,k=3D1024,bs=3D[3,=
2],...): OK
>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D16,k=3D1024,bs=3D[3=
,2],...): OK
>>
>> but
>>
>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D8,k=3D256,...): FAI=
L
>>
>> suggesting the selection of a particular compute tile/pipeline variant
>> (rather than n alone) determines whether the result is correct - larger
>> total workloads appear to route through a different, working code path.
>>
>> n=3D1 specifically fails almost unconditionally across every m/k/batch/
>> permutation combination tested in the suite (dozens of distinct shapes)=
,
>> e.g.:
>>
>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D1056,n=3D1,k=3D128,...): F=
AIL
>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D128,n=3D1,k=3D1056,...): F=
AIL
>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D1057,n=3D1,k=3D129,...): F=
AIL
>> =C2=A0 (and many more shapes/batch/permutation variants, all n=3D1, all=
 FAIL)
>>
>> -----------------------------------------------------------------------=
=2D--=20
>>
>> What I've already ruled out
>> -----------------------------------------------------------------------=
=2D--=20
>>
>> - Not quantization-specific: reproduces identically on plain f32xf32=20
>> with
>> =C2=A0 no quantized types involved at all.
>> - Not an issue with a single dispatch path: reproduces whether the
>> =C2=A0 mmvq quantized-matmul path is force-enabled or force-disabled vi=
a
>> =C2=A0 ggml's GGML_VK_DISABLE_MMVQ (tested on a related quantized-MUL_M=
AT
>> =C2=A0 investigation before broadening to this type-independent repro).
>> - Not fixed by forcing the smallest matmul tile size (mul_mat_s) via a
>> =C2=A0 local patch adding a Qualcomm-specific override alongside the ex=
isting
>> =C2=A0 Honeykrisp special-case in ggml-vulkan.cpp's device setup - the
>> =C2=A0 numerical corruption persisted with the smallest tile forced.
>> - Not fixed on latest Mesa main (driverVersion 26.2.99 locally built),
>> =C2=A0 reproduces identically to the distro-shipped Mesa version.
>> - A from-scratch, standalone Vulkan compute program (no ggml/llama.cpp
>> =C2=A0 code at all) implementing the same dequant/dot-product math scal=
ar,
>> =C2=A0 with parallel shared-memory reduction, at batch sizes up to n=3D=
256 and
>> =C2=A0 k=3D4096, across f32/f16/q8_0/q4_0 (including a byte-faithful gg=
ml
>> =C2=A0 block_q4_0 layout with real fp16-stored scale) all produced CORR=
ECT
>> =C2=A0 results. This suggests the bug is specific to ggml's actual gene=
rated
>> =C2=A0 shaders/dispatch parameters (tile selection, workgroup sizing, e=
tc.)
>> =C2=A0 rather than a blanket compute-correctness problem with basic
>> =C2=A0 arithmetic, barriers, or shared memory on this device.
>>
>> -----------------------------------------------------------------------=
=2D--=20
>>
>> Separately noticed (may be unrelated, mentioning for completeness)
>> -----------------------------------------------------------------------=
=2D--=20
>>
>> During the same test-backend-ops run, MUL_MAT invocations with bf16 and
>> quantized type_a operands also emit:
>>
>> =C2=A0 SPIR-V WARNING:
>> =C2=A0 =C2=A0 In file ../src/compiler/spirv/spirv_to_nir.c:5651
>> =C2=A0 =C2=A0 Unsupported SPIR-V capability:=20
>> SpvCapabilityStorageBuffer8BitAccess (4448)
>> =C2=A0 =C2=A0 52 bytes into the SPIR-V binary
>>
>> This is logged as a non-fatal warning rather than a hard failure, so
>> execution continues, but it suggests glslc/the driver's SPIR-V->NIR
>> consumer doesn't fully support a capability the shader declares
>> (VK_KHR_8bit_storage). Not sure if this is a contributing cause of the
>> n=3D1..8 MUL_MAT corruption above or a separate issue - flagging it sin=
ce
>> it appears in the same log and involves the same driver/shader pipeline=
.
>>
>> The same test run additionally showed CONV_2D producing large errors fo=
r
>> inputs with height>1, and GET_ROWS/SET_ROWS failing specifically for
>> q4_0/q8_0 types, plus a crash (uncaught exception in ggml_vk_submit)
>> partway through the CONV_2D tests. Happy to provide full logs for these
>> if useful, but keeping this report focused on the clearest, most
>> type-independent finding (MUL_MAT small-n).
>>
>> Full test-backend-ops log, the ggml-vulkan.cpp patch I tried, and the
>> standalone Vulkan reproducer programs are all available on request. I'm
>> happy to test patches/build with extra debug output, or run the
>> IR3_SHADER_OVERRIDE_PATH / RenderDoc capture workflow described in the
>> freedreno docs if a maintainer can point me toward which shader variant
>> is selected for small-n MUL_MAT dispatches - I wasn't able to=20
>> conclusively
>> identify that from the ggml-vulkan.cpp tile-selection code on my own.
>>
>> Thanks,
>> Stef