Re: msdosfs: Lookup of some existing files fails when mounting with long names

Am 22.05.26 um 18:10 schrieb Stefan Esser:
> Am 21.05.26 um 23:56 schrieb [email protected]:
>> Fabian sent me the relevant part of the image, and the name
>> that is not correctly displayed contains the following UTF-16
>> sequence:
>>
>>      U+D83E U+DD10 ===> U+1F910 ===> "🤐"
>>
>> Seems that the conversion of these combined UTF-16 sequences
>> (surrogates) is not correct. This will probably affect all
>> file names that contain Unicode characters that cannot be
>> encoded in a single UTF-16 value.
>>
>> I'm going to further debug this issue tomorrow; it should be
>> easy to reproduce and fix ...
> 
> I had time to look at the sources and found that the function
> win2unixchr() does not support surrogates.
> 
> The problem with surrogates in file names is that they can be
> split between VFAT directory entry elements (i.e., last symbol
> in one group of 13, and first symbol in the preceding group of
> 13 symbols). But each 13 symbols are currently processed with
> no state kept between them.
> 
> I'll come up with a patch for review when I have decided how
> to keep the first surrogate symbol and to present it when
> the second surrogate is processed (possibly only in the next
> invocation of win2unixfn(), which causes this problem).

The patch that adds surrogate pair support to MSDOSFS is now
available for review:

	https://reviews.freebsd.org/D57313

I have performed extensive tests, but the directory reading
and writing has been complex before, is significantly more
complex with these changes. I'd rather rewrite all directory
operations to use a buffer holding the UTF-16 coded long
filename, instead of operating on directory slots that hold
up to 13 UTF-16 symbols max. and that are laid out in reverse
order (end of file name first) in the directory.

The changed code uses 32 bit variables to hold up to 2 UTF-16
symbols (i.e., the surrogate pairs). If a surrogate pair has
to be stored, the high surrogate is placed in the upper half
of the "code" variable.

While directory writing has been implemented in a way that
allows processing from the start of the file name to its end,
reading the long file name starts at the end and proceeds
towards the start of the name. Especially complex it the case
of a surrogate pair split between directory slots, since the
second half will be found as the first value in a directory
entry, but cannot be added to the result buffer yet. Only
when the following (logically preceding directory slot is
processed, the last value is the first half of the surrogate
pair and only then can the converted UTF-8 sequence be added
to the result buffer. The high surrogate must be stored in
the context of the calling function and provided on the next
iteration (also true for writing pairs over slot bondaries).
Surrogate pairs within a directory slot can easily be
converted, though.

There are already functions that have up to 1KB of local
variables, and if such a buffer was accepted to hold the
full UTF-16 encoded name, the code could be significantly
simplified.

Regards, STefan