Re: [Turnip/Adreno 702] Vulkan compute MUL_MAT gives wrong results for,small batch sizes (n=1..~8), independent of data type

Hi,

seems to be a deeper issue related to subgroupAdd().

I filed a report here:

https://gitlab.freedesktop.org/mesa/mesa/-/work_items/15920

Best regards
Stef


Am 21.07.26 um 18:29 schrieb Stefan Rinass:
> Hi Karmjit,
>
> Thanks for testing on A750 and for the pointer on 8bit_storage - that
> narrowed things down a lot faster than I expected.
>
> I ran with VK_LAYER_KHRONOS_validation enabled as you suggested, and it
> turned up something that goes a bit further than the 8bit_storage
> mismatch: the validation layer reports that ggml is creating the
> VkDevice itself in violation of the spec on this hardware, independent
> of any specific shader.
>
> The Adreno 702 only reports Vulkan 1.0.354 (no 1.1+, altrough being=20
> capable according
> to Wikipedia). Relevant validation errors:
>
> =C2=A0 Missing extension required by the device extension=20
> VK_KHR_16bit_storage:
> =C2=A0 VK_KHR_storage_buffer_storage_class.
>
> =C2=A0 vkCreateBuffer(): pCreateInfo->usage has VkBufferUsageFlagBits va=
lues
> =C2=A0 (... VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT) that requires the
> =C2=A0 extensions VK_KHR_buffer_device_address or=20
> VK_EXT_buffer_device_address.
>
> =C2=A0 vkAllocateMemory():=20
> pAllocateInfo->pNext<VkMemoryAllocateFlagsInfo>.flags
> =C2=A0 has VkMemoryAllocateFlagBits values=20
> (VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_BIT)
> =C2=A0 that requires the extensions VK_KHR_buffer_device_address.
>
> My reading: on Vulkan 1.1/1.2+ devices, storage_buffer_storage_class and
> buffer-device-address behavior are either promoted to core or commonly
> enabled together with their prerequisites without issue, so this likely
> never surfaces on more modern hardware (probably including the A750).
>
> Thanks again for the quick and helpful response.
>
> Regards,
> Stef
>
> Am 21.07.26 um 13:53 schrieb Karmjit Mahil:
>> Hello Stefan,
>>
>> I tested MUL_MAT on A750 and got just these failures:
>> MUL_MAT(type_a=3Df16,type_b=3Df32,m=3D64,n=3D45,k=3D128,bs=3D[8,1],nr=
=3D[4,1],per=3D[0,1,2,3],k_v=3D0,o=3D1)=20
>>
>> MUL_MAT(type_a=3Df16,type_b=3Df32,m=3D128,n=3D45,k=3D64,bs=3D[8,1],nr=
=3D[4,1],per=3D[0,1,2,3],k_v=3D0,o=3D1)=20
>>
>> MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D64,n=3D77,k=3D77,bs=3D[12,1],nr=
=3D[1,1],per=3D[0,1,2,3],k_v=3D0,o=3D1)=20
>>
>> MUL_MAT(type_a=3Dq4_0,type_b=3Df32,m=3D576,n=3D512,k=3D576,bs=3D[1,1],n=
r=3D[1,1],per=3D[0,1,2,3],k_v=3D0,o=3D1)=20
>>
>>
>> So this could be something isolated to the A702.
>>
>> Regarding the `SPIR-V WARNING`. A702 doesn't support KHR_8bit_storage=
=20
>> and
>> storageBuffer8BitAccess (A750 does have support for this), and some=20
>> of the tests
>> might be trying to compile a shader requiring that.
>>
>> In ggml-vulkan.cpp I don't see VK_KHR_8bit_storage being enabled or=20
>> checked for (I can see
>> 16bit is) and I can see mul_mat_vec_base.glsl requires=20
>> GL_EXT_shader_8bit_storage, so the
>> issue here could be that the tests aren't adhering to the Vulkan spec=
=20
>> and end up with broken
>> result. The way to check this would be to use Vulkan Validation Layers.
>>
>> Anyway, it would be good if you could file the issue here instead:
>> https://gitlab.freedesktop.org/mesa/mesa/-/work_items
>>
>> Regards,
>> Karmjit
>>
>>
>> On 21/07/2026 03:10, Stefan Rinass wrote:
>>> Hi all,
>>>
>>> I'm seeing incorrect numerical results from Vulkan compute matrix
>>> multiplication on a Turnip-driven Adreno 702 (Qualcomm QRB2210, Arduin=
o
>>> Uno Q board), reproducible with ggml's upstream test-backend-ops suite=
.
>>> Writing this up because the pattern is very consistent and I've been=
=20
>>> able
>>> to rule out several likely causes already.
>>>
>>> ----------------------------------------------------------------------=
=2D---=20
>>>
>>> Hardware / software
>>> ----------------------------------------------------------------------=
=2D---=20
>>>
>>> GPU:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Turnip Adreno (TM) 702
>>> vendorID:=C2=A0 =C2=A0 =C2=A0 =C2=A00x5143 (Qualcomm)
>>> Mesa:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0built from git main, dri=
verVersion reported as 26.2.99
>>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (also reproduc=
es on the distro-shipped Mesa version)
>>> maxComputeSharedMemorySize: 16384 bytes
>>> Test tool:=C2=A0 =C2=A0 =C2=A0 ggml's test-backend-ops (from ggml-org/=
llama.cpp),=20
>>> Vulkan
>>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 backend, built=
 natively/cross-compiled aarch64
>>>
>>>
>>> MUL_MAT (general matrix x matrix / matrix x vector multiply) produces
>>> numerically wrong output when the batch dimension n is small (n=3D1=20
>>> through
>>> roughly 8, depending on k), across every plain floating point type=20
>>> tested
>>> (f32xf32, f16xf32, bf16xf32 - no quantization involved). Larger n (>=
=3D9,
>>> or sufficiently large n*k) produces correct results in the same test=
=20
>>> run,
>>> with the same operands.
>>>
>>> This is significant in practice because n=3D1 is the standard shape fo=
r
>>> single-token autoregressive LLM decoding, i.e. this hits=20
>>> ggml/llama.cpp's
>>> most common real-world workload, not an edge case.
>>>
>>> ----------------------------------------------------------------------=
=2D---=20
>>>
>>> Reproduction
>>> ----------------------------------------------------------------------=
=2D---=20
>>>
>>> Build ggml/llama.cpp's test-backend-ops with the Vulkan backend enable=
d
>>> and run:
>>>
>>> =C2=A0 =C2=A0 ./test-backend-ops test -o MUL_MAT
>>>
>>> Representative output (trimmed, full log available on request):
>>>
>>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D1,k=3D256,...): FA=
IL=20
>>> ERR=3D1.214182167
>>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D2,k=3D256,...): FA=
IL=20
>>> ERR=3D0.738095533
>>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D3,k=3D256,...): FA=
IL=20
>>> ERR=3D0.964137248
>>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D4,k=3D256,...): FA=
IL=20
>>> ERR=3D1.433633782
>>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D5,k=3D256,...): FA=
IL=20
>>> ERR=3D1.523953343
>>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D6,k=3D256,...): FA=
IL=20
>>> ERR=3D0.961054406
>>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D7,k=3D256,...): FA=
IL=20
>>> ERR=3D0.725742246
>>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D8,k=3D256,...): FA=
IL=20
>>> ERR=3D0.977231894
>>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D9,k=3D256,...): OK
>>>
>>> Same pattern for f16 and bf16 operands. Tolerance in these tests is
>>> 0.0005; observed errors are ~0.6-1.5, i.e. not float rounding noise,=
=20
>>> but
>>> substantially wrong values.
>>>
>>> The threshold is not a flat "n<9" rule - it interacts with total work
>>> size. For example:
>>>
>>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D1,k=3D1024,bs=3D[3=
,2],...): FAIL
>>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D8,k=3D1024,bs=3D[3=
,2],...): OK
>>> MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D16,k=3D1024,bs=3D[3,2],..=
.): OK
>>>
>>> but
>>>
>>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D16,n=3D8,k=3D256,...): FA=
IL
>>>
>>> suggesting the selection of a particular compute tile/pipeline variant
>>> (rather than n alone) determines whether the result is correct - large=
r
>>> total workloads appear to route through a different, working code path=
.
>>>
>>> n=3D1 specifically fails almost unconditionally across every m/k/batch=
/
>>> permutation combination tested in the suite (dozens of distinct=20
>>> shapes),
>>> e.g.:
>>>
>>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D1056,n=3D1,k=3D128,...): =
FAIL
>>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D128,n=3D1,k=3D1056,...): =
FAIL
>>> =C2=A0 MUL_MAT(type_a=3Df32,type_b=3Df32,m=3D1057,n=3D1,k=3D129,...): =
FAIL
>>> =C2=A0 (and many more shapes/batch/permutation variants, all n=3D1, al=
l FAIL)
>>>
>>> ----------------------------------------------------------------------=
=2D---=20
>>>
>>> What I've already ruled out
>>> ----------------------------------------------------------------------=
=2D---=20
>>>
>>> - Not quantization-specific: reproduces identically on plain f32xf32=
=20
>>> with
>>> =C2=A0 no quantized types involved at all.
>>> - Not an issue with a single dispatch path: reproduces whether the
>>> =C2=A0 mmvq quantized-matmul path is force-enabled or force-disabled v=
ia
>>> =C2=A0 ggml's GGML_VK_DISABLE_MMVQ (tested on a related quantized-MUL_=
MAT
>>> =C2=A0 investigation before broadening to this type-independent repro)=
.
>>> - Not fixed by forcing the smallest matmul tile size (mul_mat_s) via a
>>> =C2=A0 local patch adding a Qualcomm-specific override alongside the=
=20
>>> existing
>>> =C2=A0 Honeykrisp special-case in ggml-vulkan.cpp's device setup - the
>>> =C2=A0 numerical corruption persisted with the smallest tile forced.
>>> - Not fixed on latest Mesa main (driverVersion 26.2.99 locally built),
>>> =C2=A0 reproduces identically to the distro-shipped Mesa version.
>>> - A from-scratch, standalone Vulkan compute program (no ggml/llama.cpp
>>> =C2=A0 code at all) implementing the same dequant/dot-product math sca=
lar,
>>> =C2=A0 with parallel shared-memory reduction, at batch sizes up to n=
=3D256 and
>>> =C2=A0 k=3D4096, across f32/f16/q8_0/q4_0 (including a byte-faithful g=
gml
>>> =C2=A0 block_q4_0 layout with real fp16-stored scale) all produced COR=
RECT
>>> =C2=A0 results. This suggests the bug is specific to ggml's actual gen=
erated
>>> =C2=A0 shaders/dispatch parameters (tile selection, workgroup sizing, =
etc.)
>>> =C2=A0 rather than a blanket compute-correctness problem with basic
>>> =C2=A0 arithmetic, barriers, or shared memory on this device.
>>>
>>> ----------------------------------------------------------------------=
=2D---=20
>>>
>>> Separately noticed (may be unrelated, mentioning for completeness)
>>> ----------------------------------------------------------------------=
=2D---=20
>>>
>>> During the same test-backend-ops run, MUL_MAT invocations with bf16 an=
d
>>> quantized type_a operands also emit:
>>>
>>> =C2=A0 SPIR-V WARNING:
>>> =C2=A0 =C2=A0 In file ../src/compiler/spirv/spirv_to_nir.c:5651
>>> =C2=A0 =C2=A0 Unsupported SPIR-V capability:=20
>>> SpvCapabilityStorageBuffer8BitAccess (4448)
>>> =C2=A0 =C2=A0 52 bytes into the SPIR-V binary
>>>
>>> This is logged as a non-fatal warning rather than a hard failure, so
>>> execution continues, but it suggests glslc/the driver's SPIR-V->NIR
>>> consumer doesn't fully support a capability the shader declares
>>> (VK_KHR_8bit_storage). Not sure if this is a contributing cause of the
>>> n=3D1..8 MUL_MAT corruption above or a separate issue - flagging it si=
nce
>>> it appears in the same log and involves the same driver/shader=20
>>> pipeline.
>>>
>>> The same test run additionally showed CONV_2D producing large errors=
=20
>>> for
>>> inputs with height>1, and GET_ROWS/SET_ROWS failing specifically for
>>> q4_0/q8_0 types, plus a crash (uncaught exception in ggml_vk_submit)
>>> partway through the CONV_2D tests. Happy to provide full logs for thes=
e
>>> if useful, but keeping this report focused on the clearest, most
>>> type-independent finding (MUL_MAT small-n).
>>>
>>> Full test-backend-ops log, the ggml-vulkan.cpp patch I tried, and the
>>> standalone Vulkan reproducer programs are all available on request. I'=
m
>>> happy to test patches/build with extra debug output, or run the
>>> IR3_SHADER_OVERRIDE_PATH / RenderDoc capture workflow described in the
>>> freedreno docs if a maintainer can point me toward which shader varian=
t
>>> is selected for small-n MUL_MAT dispatches - I wasn't able to=20
>>> conclusively
>>> identify that from the ggml-vulkan.cpp tile-selection code on my own.
>>>
>>> Thanks,
>>> Stef
>