Expat occasionaly cropping multibyte character strings

Juraj Ivančić <[email protected]> Sat, 15 May 2010 13:30:54 +0200

Newsgroups	gmane.text.xml.expat.bugs
Message-ID	<[email protected]>

Expat does not handle multibyte characters correctly.
Steps to reproduce this behaviour:

1) You need an input XML in e.g. UTF8 encoding which
contains some multibyte characters (e.g. cyrillic characters)

2) Create an XML parser and feed it input file, but ensure
that buffer breaks somewhere in the middle of a multibyte string.
(To make sure - feed the parser one byte at a time).

Say input file contains:
'... <element>Соме валуе</element> ...'

and it gets buffered like this:

Buffer1: '... <element>Соме '
Buffer2: 'валуе</element> ...'

Expat parser will, when completing parsing Buffer1, invoke character
data handler containing only partial ('Соме ') data, instead of waiting
for the rest of the input. I think this is a bug as it only manifests
when multibyte characters appear.

I managed to correct this behaviour. Patch attached. This made my input 
file parse correctly, but note that my use case only had multibyte
characters as element values, not as element names or attribute
names/values. I'd imagine that some of these might also suffer from a 
similar issue.

I also created a test case which demonstrates this behaviour.

Please let me know if more information is needed.

HTH,
	Juraj Ivančić

P.S. Patch is made against wxWidgets fork of the Expat project.
I did however verify with Expat CVS - the issue is still there.

P.P.S. I also filed this issue to wxWidgets TRAC
http://trac.wxwidgets.org/ticket/12053

_______________________________________________
Expat-bugs mailing list
[email protected]
http://mail.libexpat.org/mailman/listinfo/expat-bugs

expat_mb_fix.diff (text/plain, 636 B)

Index: src/expat/lib/xmltok_impl.c
===================================================================
--- src/expat/lib/xmltok_impl.c	(revision 64305)
+++ src/expat/lib/xmltok_impl.c	(working copy)
@@ -834,8 +834,7 @@
 #define LEAD_CASE(n) \
     case BT_LEAD ## n: \
       if (end - ptr < n || IS_INVALID_CHAR(enc, ptr, n)) { \
-        *nextTokPtr = ptr; \
-        return XML_TOK_DATA_CHARS; \
+        return XML_TOK_PARTIAL; \
       } \
       ptr += n; \
       break;
@@ -871,6 +870,8 @@
       break;
     }
   }
+  if( ptr == end )
+      return XML_TOK_PARTIAL;
   *nextTokPtr = ptr;
   return XML_TOK_DATA_CHARS;
 }

expatbug.cpp (text/plain, 1.8 KB)

#include <iostream>
#include <string>

#include "lib/expat.h"

// The following is a UTF8 with bom encoded data with the following contents:
// <?xml version="1.0" encoding="UTF-8"?>
// <test>ТХИС ИС А ТЕСТ</test>
char xmlDocument[] =
{
        0xef, 0xbb, 0xbf, 0x3c, 0x3f, 0x78, 0x6d, 0x6c, 0x20, 0x76,
        0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x3d, 0x22, 0x31, 0x2e,
        0x30, 0x22, 0x20, 0x65, 0x6e, 0x63, 0x6f, 0x64, 0x69, 0x6e,
        0x67, 0x3d, 0x22, 0x55, 0x54, 0x46, 0x2d, 0x38, 0x22, 0x3f,
        0x3e, 0x0d, 0x0a, 0x3c, 0x74, 0x65, 0x73, 0x74, 0x3e,
                                                              0xd0,
        0xa2, 0xd0, 0xa5, 0xd0, 0x98, 0xd0, 0xa1, 0x20, 0xd0, 0x98,
        0xd0, 0xa1, 0x20, 0xd0, 0x90, 0x20, 0xd0, 0xa2, 0xd0, 0x95,
        0xd0, 0xa1, 0xd0, 0xa2,
                                0x3c, 0x2f, 0x74, 0x65, 0x73, 0x74,
        0x3e, 0x0d, 0x0a
};

size_t const documentLen = sizeof( xmlDocument );

void handleCharData( void *, XML_Char const * s, int length )
{
    if ( length != 25 ) // 3 spaces + 11 cyrillic chars (2 bytes each).
    {
        std::cout << "Invalid length: " << length << ".\n";
        return;
    }
    int result = std::memcmp( s, xmlDocument + 49, length );
    std::cout << ( result ? "Failure." : "Success." ) << '\n';
}


int main()
{
    XML_Parser parser = XML_ParserCreate( 0 );
    XML_SetCharacterDataHandler( parser, &handleCharData );
    
    for ( size_t index = 0; index < documentLen; ++index )
    {
        int parseResult = XML_Parse( parser, xmlDocument + index, 1, index == documentLen - 1 );
        if ( parseResult == XML_STATUS_ERROR )
            std::cout << "Parse error.\n";
    }
    std::cout << "Done.";
    XML_ParserFree( parser );
}