Re: How to triage an "Invalid Key" error message?

Hi Christian,

On 1/21/25 6:24 AM, Christian Hewitt wrote:
>> On 21 Jan 2025, at 5:50 pm, James Prestwood <[email protected]> wrote:
>>
>> Hi Christian,
>>
>> On 1/20/25 7:36 AM, Christian Hewitt wrote:
>>>> On 18 Jan 2025, at 8:38 pm, KeithG <[email protected]> wrote:
>>>>
>>>> Christian,
>>>>
>>>> Is this, perhaps, on devices with the brcmfmac chip? I have seen this
>>>> recently with on my Pis with wpa2 and wpa3 APs.
>>> The majority of reports are from RPi 4/5 users which have brcmfmac
>>> chips, but 85% of our distro installed base runs an RPi board so
>>> that mostly reflects our demographics. I’ve also seen reports on
>>> Amlogic and Rockchip SoC devices (also using brcmfmac); but to
>>> offset theories on it being brcmfmac specific, we’ve also had
>>> reports from users with Intel iwlwifi hardware and someone using
>>> an RTL8723BS chip. So it’s more likely to be an interaction with
>>> common code not driver code.
>>>
>>> At one point we suspected an issue with WPA3, but that was triaged
>>> with RPi developers and we now disable SAE offload which allows
>>> boards to connect to mixed WPA2/WPA3 networks, see:
>>>
>>> https://forum.libreelec.tv/thread/29155-rpi5-network-disappeared
>>> https://github.com/LibreELEC/LibreELEC.tv/pull/9477/files
>>>
>>> In the following forum and GitHub issue threads. There’s lots of
>>> assumptions, guesswork, and false conclusions, but among the noise
>>> there are some snippets of iwd debug log:
>>>
>>> https://github.com/LibreELEC/LibreELEC.tv/issues/7166
>> This one actually has some iwmon logs. Looks like the kernel is sending a failure within ~100ms of the connection which is odd. IWD waits for a successful connect event before starting the 4-way handshake so its not even getting to that point:
>>
>> < Request: Connect (0x2e) len 172 [ack]                                5.644853
>> ...
>>> Event: Connect (0x2e) len 36                                         5.735838
>>      Wiphy: 0 (0x00000000)
>>      Interface Index: 3 (0x00000003)
>>      MAC Address 00:00:00:00:00:00
>>      Status Code: 16 (0x0010)
>>
>> There is also a core dump attached, but the back trace isn't of much use.
>>
>>> https://github.com/LibreELEC/LibreELEC.tv/issues/8731
>> This issue _appears_ as if wpa_supplicant is also running along side IWD. That is generally why you see this log:
>>
>> Mar 16 21:10:35 pineh64-b iwd[1046]: Unexpected connection related event -- is another supplicant running?
> 100% guaranteed wpa is not installed at the same time. LibreELEC has embedded style packaging with a read-only filesystem and no package manager. The iwd/wpa selection is baked into the image at distro image creation time and can’t be changed afterwards.
Ok, this must be some unforeseen interaction/bug between IWD and 
brcmfmac. Debug logs will help here for sure.
>
>>> https://github.com/LibreELEC/LibreELEC.tv/issues/8760
>> This seems the same as #7166.
>>> https://github.com/LibreELEC/LibreELEC.tv/issues/8762
>> Not a whole lot in here, but apparently resolved.
>>> I’ll need to rig a distro image for greater debug (as per James’
>>> instructions) to see if we can obtain more log output.
>>>
>>> Christian
>>>
>> So overall the "invalid-key" issue seems to be due to brcmfmac signalling a failure very early on. In a perfect world it would be great if we could capture debug/iwmon logs of both IWD trying to connect and wpa_supplicant, so we can compare the iwmon logs and see what each supplicant is sending to the kernel. The messages _looked_ ok from IWD but if wpa_supplicant works fine there is obviously a difference somewhere. As Keith knows, brcmfmac is kinda a pain to support as its the odd-ball as far as drivers go. IWD doesn't get much contributions from wifi vendors so any odd quirks only get solved by trial and error.
> It’s been seen on other chips too, but if it’s timing related it could simply be that brcmfmac trips the issue more frequently and that plus distro demographics skews the volume of reports.
>
> I’ll prepare some end-user debug instructions around debug/iwmon and then attempt to recruit some affected users in a GitHub thread. Let’s see where things take us. Thx :)

Thanks, lets see what comes of that.

- James

>
> Christian
>