Re: xapian-core and Windows non-ASCII paths

Olly Betts writes:
 > On Thu, Jun 04, 2020 at 12:49:58PM +0200, Jean-Francois Dockes wrote=
:
 > > I am attaching a patch against the xapian-core 1.4 branch.
 >=20
 > Patches need to go to git master first (unless they're only relevant=
 to
 > 1.4.x, which this clearly isn't).

Understood. This was intended mostly as a proof of concept, to show how=
 the
file interface calls would need to be changed for general Unicode path
access to work, and not intended for direct merging, as I mentionned.

 > > The idea of the patch is that a conversion to a Windows Unicode wi=
de char
 > > string is attempted prior to performing a relevant system call. If=
 the
 > > conversion succeeds, the wide version of the call is used, else, t=
he
 > > previous narrow call is used. This should ensure that  existing
 > > applications are undisturbed, and provides a way to tunnel a Unico=
de path
 > > by using utf-8.
 >=20
 > I think this needs input from people with deeper knowledge of this
 > platform.

Sure. That's also why I first asked if somebody on the list had an idea=
 of
the right approach. When nobody answered, I just applied an equivalent =
of
the changes which were needed in Recoll.=20

 > The approach of patching every affected call site doesn't really see=
m
 > workable to me - the maintenance and development overhead just seems=
 too
 > high.  We do need platform-specific code for some things, but no oth=
er
 > platform needs platform-specific code for something as pervasive as
 > working on a filename.  We'll just end up fighting an ongoing battle=

 > against newly introduced places that also need this special handling=
,
 > and because it works fine without for common uses such issues can to=
o
 > easily go undetected for a long time (yours is the first report of t=
his
 > problem, but it's always been there).

 > If it's really necessary to use these wide-character variants of
 > everything which takes a filename, I think the only way to sensibly
 > deal with that is to have a set of wrappers which present them as
 > the non-wide variants to the rest of the code - that way this at
 > least only needs addressing once per such function (though even that=

 > is a maintenance pain as a patch making use of a currently-unwrapped=
 C
 > library function taking a filename would require a new wrapper).

After reading a bit in this area, I have the impression that most
experienced people think that the only sane approach to non-ASCII file
names on Windows is to use the wide interfaces.

I would be very happy if someone could indicate another approach.

The problem has surfaced with Recoll, because it is an end-user tool, s=
o I
have to make their life easy and work with their home directory as it
is. When Xapian is used with a WEB site or some other configuration set=
 up
by professionals, obviously, they can avoid storing stuff in C:/users/=EC=
=8B=9C=EB=83=87=EB=AC=BC

A set of wrappers is how it is implemented in Recoll, and, yes, you end=
 up
wrapping the whole file name interface from directory reading to openin=
g
files to unlink etc., and you keep reintroducing bare calls which fail,=
 and
the whole thing is a PITA. The Recoll wrappers  are not pretty enough t=
o be
reused, which is why I just changed the call sites in the patch.

This said, Xapian does a lot less file manipulations than Recoll, as
demonstrated by the limited amount of changes in the patch, so I don't
think that this should be really unworkable. As another consequence, it=

should be easy for me to maintain the thing for my own use.

It all depends on how fully you want to support the platform.

I guess that there is a nice library somewhere to do this, and actually=
 I
think it's std::filesystem, which was unfortunately not really there wh=
en I
needed it.

 > In terms of workarounds, simply changing directory to where the data=
base
 > lives and then using a relative non-wide path should work.

It quite probably would, assuming that Xapian never computes an
absolute path (which you know, but I don't without scanning the code), =
and
also that there are no getcwd/chdir Windows pitfalls waiting for me...

Cheers,

jf