Windows error SEC_E_ILLEGAL_MESSAGE (0x80090326)

Environment:
Windows 11 (5 out of 500 machines affected)
Curl 8.0.1 and 8.0.4 both affected
See details below about windows updates

Error:
curl: (35) schannel: next InitializeSecurityContext failed:
SEC_E_ILLEGAL_MESSAGE (0x80090326) .

Affected websites:
https://www.python.org/ftp/python
Several internal services

I have to admit in advance that I know this might not be CURL
team/community's problem to solve, but raising here because:
- I think people will keep reporting this issue to CURL project over time
(albeit incorrectly)
- I think a simple set of registry keys can be found to resolve this issue
- I think the Windows CURL experts here are in a unique position to
identify those keys

I support some scripts which use Windows native C:\Windows\System32\curl.exe

In the past 6 months,  ~5 out of ~500 user machines have hit this error
when connecting to python.org and several internal websites.  The error
only starts after they install Windows updates, it seems to be the ones in
March/April 2023 which trigger it.  I've identified that powershell's
"invoke-webrequest" is also broken on these systems. So, the problem is
very clearly the schannel backend.

After several debug sessions on these 5 machines the only fix has been to
re-install windows 11.  We tried removing all the updates which got applied
and the problem remained.  We also tried applying a bunch of other TLS
related hotfixes, but the problem remained.

Here are the following interesting facts I'm hoping someone here will be
willing to consider and potentially look into:

- We think it's only machines that went through Windows 10 to 11 Upgrade
process
- From-scratch Windows 10 / Windows 11 machines have never exhibited

I believe that some Windows 10 service packs added some registry keys which
changes some default TLS. behaviors with schannel, and those keys are
getting lost/overwritten in the Windows 11 upgrade process.  Meanwhile,
clean Windows 11 machines get the correct settings. I saw one article which
explained how this happened with Windows 7 to Windows 8 upgrade in the past
(specifically a Default TLS setting), but can't find that article now.

So, if anyone here is expert-enough with Windows SChannel/TLS and CURL to
identify the list of registry keys which can affect CURL, and then also
identify which values are required to make CURL work properly with
SChannel, then I think publishing that information on the Windows
troubleshooting page for CURL, AND sharing it with CURL's contacts at
microsoft, COULD prevent many future discussions about this super annoying
topic.  Microsoft might even recognize the issue when presented with the
data and create an update pack which ensures the SChannel registry keys are
in a valid state for the future.


Gerald R. Wiltse
[email protected]

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html