Re: Windows error SEC_E_ILLEGAL_MESSAGE (0x80090326)

On 12/4/2023 4:12 PM, Gerald Wiltse via curl-users wrote:
> Environment:
> Windows 11 (5 out of 500 machines affected)
> Curl 8.0.1 and 8.0.4 both affected
> See details below about windows updates
>
> Error:
> curl: (35) schannel: next InitializeSecurityContext failed: 
> SEC_E_ILLEGAL_MESSAGE (0x80090326) .
>
> Affected websites:
> https://www.python.org/ftp/python
> Several internal services
>
> I have to admit in advance that I know this might not be 
> CURLteam/community's problem to solve, but raising here because:
> - I think people will keep reporting this issue to CURL project over 
> time (albeit incorrectly)
> - I think a simple set of registry keys can be found to resolve this 
> issue
> - I think the Windows CURL experts here are in a unique position to 
> identify those keys
>
> I support some scripts which use Windows native 
> C:\Windows\System32\curl.exe
>
> In the past 6 months,  ~5 out of ~500 user machines have hit this 
> error when connecting to python.org <http://python.org> and several 
> internal websites.  The error only starts after they install Windows 
> updates, it seems to be the ones in March/April 2023 which trigger it. 
> I've identified that powershell's "invoke-webrequest" is also broken 
> on these systems. So, the problem is very clearly the schannel backend.
>
> After several debug sessions on these 5 machines the only fix has been 
> to re-installwindows 11.  We tried removing all the updates which got 
> applied and the problem remained.  We also tried applying a bunch of 
> other TLS related hotfixes, but the problem remained.
>
> Here are the following interesting facts I'm hoping someone here will 
> be willing to consider and potentially look into:
>
> - We think it's only machines that went through Windows 10 to 11 
> Upgrade process
> - From-scratch Windows 10 / Windows 11 machines have never exhibited
>
> I believe that some Windows 10 service packs added some registry keys 
> which changes some default TLS. behaviors with schannel, and those 
> keys are getting lost/overwritten in the Windows 11 upgrade process.  
> Meanwhile, clean Windows 11 machines get the correct settings. I saw 
> one article which explained how this happened with Windows 7 to 
> Windows 8 upgrade in the past (specifically a Default TLS setting), 
> but can't find that article now.
>
> So, if anyone here is expert-enough with Windows SChannel/TLS and CURL 
> to identify the list of registry keys which can affect CURL, and then 
> also identify which values are required to make CURL work properly 
> with SChannel, then I think publishing that information on the Windows 
> troubleshooting page for CURL, AND sharing it with CURL's contacts at 
> microsoft, COULD prevent many future discussions about this super 
> annoying topic. Microsoft might even recognize the issue when 
> presented with the data and create an update pack which ensures the 
> SChannel registry keys are in a valid state for the future.


For some time we've had an error message for SEC_E_ILLEGAL_MESSAGE that 
explains it's usually caused by a fatal TLS alert from the server. I 
notice that the full error message is omitted, did you do that or did 
curl? Here's what the error message should say:

curl: (35) schannel: next InitializeSecurityContext failed: 
SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a 
fatal SSL/TLS alert is received (e.g. handshake failed). More detail may 
be available in the Windows System event log.

Sometimes Schannel errors are in the system event log but often not. I 
just tried a site that sends a handshake failure alert (which causes 
SEC_E_ILLEGAL_MESSAGE) and there's nothing in the event log. I suggest 
see what is being sent in the handshake. Use Wireshark to record the 
client hello and the server hello or alert. Compare known good and bad 
client hello.

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html