[openpgp] Re: Analysis document

Hi Daniel

Am 14.07.25 um 11:43 schrieb Daniel Huigens:
> Hi Falko,
>
> Andrew asked a very specific question, namely:
>
> On Thursday, July 10th, 2025 at 17:25, Andrew Gallagher wrote:
>> if I sign over a UTF-8 document with a text signature, and it turns 
>> out that there is an invalid UTF-8 character in it (say, a mishandled 
>> surrogate), is the signature invalid?
>
> Regardless of your interpretation of RFC9580, if an implementation 
> hashes and signs over some data that turns out not to be valid UTF-8, 
> then it hasn't followed the spec (in your interpretation, it should've 
> re-encoded the data, in my interpretation, it should've made a 
> signature with type=binary or returned an error, but for this specific 
> question it doesn't matter). That's what I meant by saying "basically 
> it's already forbidden".
>
> On Monday, July 14th, 2025 at 08:40, Falko Strenzke 
> <[email protected]> wrote:
>>
>> I don’t read quote like that. As I wrote before: The UTF-8 
>> re-encoding is formulated exactly analogously to the line-ending 
>> conversion. Both are required to be performed/before hashing/. And 
>> that means: before hashing for signature generation and also before 
>> hashing for the signature/verification/. The encoded data in the LIT 
>> packet is/not changed/in this course.
>>
> This is not necessarily true, also for the case of line ending 
> normalization. RFC9580 says, in section 5.9 (Literal Data Packet):
>
>     Text data MUST be encoded with UTF-8 (see [RFC3629]) and stored
>     with <CR><LF> text endings (that is, network-normal line endings).
>
It is true that the sender has this requirement (and also that for 
encoding the line endings as you quote below). However, it seems clear 
to me that RFC 9580 at the same time defines the verification such that 
it includes a re-encoding operation:

Section 5.2.4: “For text document signatures (type ID 0x01), the 
implementation MUST first canonicalize
the document by converting line endings to <CR><LF> and encoding it in 
UTF-8 (see [RFC 3629]).
The resulting UTF-8 byte stream is hashed.”

It seems that RFC 9580 applies the well established paradigm of "be 
strict in what you send, be liberal in what you accept".

>
> Similarly, RFC4880 says:
>
>     Text data is stored with <CR><LF> text endings (i.e.,
>     network-normal line endings).
>
>
> In other words, the literal data packet contents are also supposed to 
> be normalized when signing text with non-canonical line endings.
>
> In practice, this is not always the case for data in the wild, so it's 
> true that for interoperability, it's necessary to normalize line 
> endings when verifying data stored in literal data packets. But I 
> consider this as a quirk to work around misbehaving implementations. 
> The fact that GnuPG and RNP also do so doesn't change what the spec 
> says. And, hopefully we don't have to replicate this quirk for UTF-8 
> re-encoding (/knocks on wood/).
>
> However, there remains an open question about what to do when creating 
> a detached signature. Because there, it's possible to have a file on 
> disk that's stored with non-canonical line endings and non-UTF-8 
> encoding. In that case, I think the best strategy is to create a 
> signature with type=binary, but it's true that the spec doesn't 
> require this very explicitly.
>
> Maybe, let's take a step back and say: we agree (hopefully) that the 
> signing implementation is responsible for making sure that the data 
> that's input into the hash for a type=text signature is UTF-8. It has 
> 4 possible strategies for that:
>
>  1. Accept a Unicode string in the API, and encode it into UTF-8 itself
>  2. Check that the input is valid UTF-8, and if not, use type=binary
>     instead, or return an error
>  3. Re-encode the data to be valid UTF-8 (assuming it knows the
>     original encoding somehow), /and/ put the re-encoded data in the
>     literal data packet (or update the file in case of a detached
>     signature?)
>  4. Re-encode the data to be valid UTF-8(assuming it knows the
>     original encoding somehow), /but/put the original data in the
>     literal data packet (or leave the data as is in case of a detached
>     signature)
>
>
> Out of all of these strategies, 1-3 don't lead to any problems for the 
> verifier. Option 4 does potentially lead to a problem for the 
> verifier, but in the case of an inline (non-detached) signature it's 
> forbidden by the text in section 5.9.
> In the case of detached signatures, it's not explicitly forbidden, so 
> there we have a potential problem. All I'm really proposing is that we 
> forbid that option and suggest that implementations choose one of the 
> other options (1-3) instead.
> (It also doesn't lead to a problem if the verifier somehow knows the 
> original encoding. I'm not sure how it would know, but then again I'm 
> not sure how the signer would know either, so also there I wouldn't 
> choose option 3 or 4. If we all agree on that, there's never any issue.)

I think for text signatures, assuming the LIT data is in UTF-8 encoding 
and this is known or a deviating encoding is known correctly by the 
verifier, no problem can arise. If there the encoding is interpreted 
differently on both sides, I myself couldn't say whether or not targeted 
manipulations to the message might become possible.

Best regards,
Falko

>
> Best,
> Daniel
>
>
> _______________________________________________
> openpgp mailing list [email protected]
> To unsubscribe send an email [email protected]
-- 

*MTG AG*
Dr. Falko Strenzke

Phone: +49 6151 8000 24
E-Mail: [email protected]
Web: mtg.de <https://www.mtg.de>

------------------------------------------------------------------------

MTG AG - Dolivostr. 11 - 64293 Darmstadt, Germany
Commercial register: HRB 8901
Register Court: Amtsgericht Darmstadt
Management Board: Jürgen Ruf (CEO), Tamer Kemeröz
Chairman of the Supervisory Board: Dr. Thomas Milde

This email may contain confidential and/or privileged information. If 
you are not the correct recipient or have received this email in error,
please inform the sender immediately and delete this email.Unauthorised 
copying or distribution of this email is not permitted.

Data protection information: Privacy policy 
<https://www.mtg.de/en/privacy-policy>

_______________________________________________
openpgp mailing list -- [email protected]
To unsubscribe send an email to [email protected]