Re: New features for xerces-c 4.0.0?

Hi Francesco,


To clarify a few points in your message. Xerces-C++ has always been 
effectively UTF-16-only.  Previously the 16-bit type used to represent 
XMLCh was one of several possible types, including char16_t, wchar_t, 
uint16_t, unsigned short or unsigned int or possibly others, depending 
upon your platform.  The change I proposed for the 4.0.0 was to switch 
to C++11/14 and use char16_t.  I.e. standardising upon the standard type 
for a UTF-16 codepoint.  It doesn't change any assumption about UTF-16 
usage internally: those assumptions were already in place.  The purpose 
of the change is as Scott stated: it's to improve interoperability and 
allow for the use of UTF-16 character and string literals, and to remove 
platform-specific variation in favour of a single type that's consistent 
across platforms.


Personally, I would absolutely prefer for Xerces-C++ to use UTF-8 in its 
external interfaces and its internal representation.  Like most people, 
I have input in UTF-8, output in UTF-8, and all of the parameters I want 
to pass into Xerces-C++ like element and attribute names, text content 
etc. are UTF-8.  All of this needs transcoding to and from UTF-16.  This 
bears a hugely significant fraction (~50%) of CPU utilisation when I've 
profiled it in the past, and it makes using Xerces-C++ unnecessarily 
painful.  But Xerces-C++ is a product of its time.  Back then, before 
widespread UTF-8 adoption, it was likely seen as forward-looking.  ICU 
and other libraries of the same era are all using UTF-16, or 
Unicode/UCS-2 as it was then.


However, such a change would be massively breaking.  I don't have the 
time or resources to do the work, and even if I did it would be hard to 
justify such a breaking change unless it could be introduced without 
breaking compatibility with the UTF-16 interfaces.  If such a change 
could be made compatibly, I would be in favour of it, but I doubt I 
could spare the required time and effort to do it myself.  I no longer 
get paid to work on Xerces-C++-related projects, and I have a full-time 
job to do which is of much greater priority.  What time I can contribute 
as part of Xerces-C++-using open source project work is limited and as 
such I need to make sure that the work I do is tightly-focussed and 
realistic in its objectives.  The above char16_t work is an example of 
such a change.  It takes great care to avoid a compatibility break (you 
can already opt into it with Xerces-C++ 3.2.x).


If you would like to investigate the changes which would be needed to 
change the internal representation from UTF-16 to UTF-8 and/or 
supplement the external interfaces with UTF-8 alternatives to the UTF-16 
interfaces we have at present, I'm sure we would all be very interested 
to hear your proposals.  As a long-term project goal, I think it would 
be beneficial.  For myself, the question isn't whether the change is 
desirable, it's whether it's realistic and achievable while not breaking 
all the existing projects which have invested time and money into using 
Xerces-C++.  Xerces-C++ has a long history at this point, and breaking 
changes are not something which I think any of us would countenance.  
(The current interfaces do expose some internal details; maybe 
hiding/changing some of them might be justifiable; but certainly not the 
core user-facing APIs.)


Kind regards,

Roger


On 16/07/2020 14:15, Francesco Pretto wrote:
> Migrating XMLChar to char16_t basically means setting in stone and
> forever that xerces-c is an utf-16 only library so it's going in a
> radically different direction than I was suggesting, so I'm not very
> happy to hear about it. I think it can be safely stated that this move
> actually closes more doors than it opens. While simplifying the code
> base and test grid is great, as I'm reading in [1], I would have
> chosen to drop support for wchar_t and int16_t but keep XMLChar for
> future possibility to support utf8 for internal encoding. Are you really
> sure you want to pursue this direction?
>
> [1] https://issues.apache.org/jira/browse/XERCESC-2206
>
>
>
> On Thu, 16 Jul 2020 at 14:22, Cantor, Scott <[email protected]> wrote:
>> On 7/16/20, 8:07 AM, "Francesco Pretto" <[email protected]> wrote:
>>
>>>     Thank you, and thank you for frankness! Probably of the two the utf-8
>>>     for internal encoding would be more oriented towards c++ modernization
>>>     changes, as you said, but probably a big change touching all the code
>>>     base.
>> It's massive. What Roger is doing is making XMLChar work as char16_t. That eliminates a lot of problems with literals and STL integration, but it doesn’t change the fact that virtually every other C or older C++ library still won't integrate well.
>>
>> -- Scott
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [email protected]
>> For additional commands, e-mail: [email protected]
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>