Fwd: Patch to fix ICU flush and pivot buffer
Joel Hockey <[email protected]>
| Newsgroups | gmane.comp.gnome.lib.xml.general |
|---|---|
| Message-ID | <CAJJNyZYLAgAG2mM5XmdJi6sVPKzyUDZ1seMv+fEfdt+w5SENXQ@mail.gmail.com> |
Sending again, since I don't think this email made it to the libxml mailing list since I was not subscribed. ---------- Forwarded message ---------- From: Joel Hockey <[email protected]> Date: Wed, Jan 3, 2018 at 5:01 PM Subject: Re: [xml] Patch to fix ICU flush and pivot buffer To: "Jungshik Shin (신정식, 申政湜)" <[email protected]> Cc: Nick Wellnhofer <[email protected]>, Markus Scherer < [email protected]>, "[email protected]" <[email protected]>, Markus Scherer < [email protected]> Nick, I have another patch for some additional call sites where flush is being incorrectly set on the non-final read. This was found by the chromium fuzzing tests. https://bugs.chromium.org/p/chromium/issues/detail?id=790944 I have included a test case for this which uses UTF8 and only works with icu. I saw that you were able to create a testcase with EUC-JP last time which worked with icu and iconv. I've tried quite a bit to do something similar, but I can't replicate the error condition with that encoding. I don't expect that you would want to check in this testcase, but I've included for you to run locally if you like. On Thu, Nov 9, 2017 at 11:36 AM, Joel Hockey <[email protected]> wrote: > Yes, I will update chromium with this as per https://cs.chromium.org/ch > romium/src/third_party/libxml/chromium/roll.py > > On Thu, Nov 9, 2017 at 10:35 AM, Jungshik Shin (신정식, 申政湜) < > [email protected]> wrote: > >> Thank you, Joel and Nick ! >> >> Joel: I guess you're gonna roll libxml in the Chromium tree to a version >> including these changes. >> >> Jungshik >> >> 2017-11-08 15:22 GMT-08:00 Joel Hockey <[email protected]>: >> >>> Thanks Nick. Nice work with the test. >>> >>> >>> >>> On Sun, Nov 5, 2017 at 2:04 AM, Nick Wellnhofer <[email protected]> >>> wrote: >>> >>>> On 26/10/2017 03:17, Joel Hockey wrote: >>>> >>>>> I've updated the patch using git format-patch. >>>>> >>>> >>>> Thanks for the updated patch. Applied here: >>>> https://git.gnome.org/browse/libxml2/commit/?id=0b19f236a263 >>>> a7b0acacd4ea84dc7237303ee3d9 >>>> >>>> The original bug found by fuzzer only relates to UTF8 decoding, so >>>>> using Shift-JIS or anything else wont help. >>>>> >>>> >>>> Why not? My reasoning was that ICU uses the same code path for all >>>> variable-width encodings. I simply converted your test file to EUC-JP and >>>> it turns out that this triggers the bug as well: >>>> >>>> https://git.gnome.org/browse/libxml2/commit/?id=72182550926d >>>> 31ad17357bd3ed69e49d7e69df02 >>>> >>>> Nick >>>> >>> >>> >> > _______________________________________________ xml mailing list, project page http://xmlsoft.org/ [email protected] https://mail.gnome.org/mailman/listinfo/xml
0001-Change-calls-to-xmlCharEncInput-to-set-flush-false-w.patch
(text/x-patch, 2.3 KB)
From 441e1e413a8f67c0813fa0e04b19dfea76e5ece6 Mon Sep 17 00:00:00 2001 From: Joel Hockey <[email protected]> Date: Tue, 2 Jan 2018 21:47:35 -0800 Subject: [PATCH] Change calls to xmlCharEncInput to set flush false when not final call. Having flush incorrectly set to true causes errors for ICU. --- HTMLparser.c | 2 +- parserInternals.c | 2 +- xmlIO.c | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/HTMLparser.c b/HTMLparser.c index 7e243e60..9adeb174 100644 --- a/HTMLparser.c +++ b/HTMLparser.c @@ -3635,7 +3635,7 @@ htmlCheckEncodingDirect(htmlParserCtxtPtr ctxt, const xmlChar *encoding) { */ processed = ctxt->input->cur - ctxt->input->base; xmlBufShrink(ctxt->input->buf->buffer, processed); - nbchars = xmlCharEncInput(ctxt->input->buf, 1); + nbchars = xmlCharEncInput(ctxt->input->buf, 0); if (nbchars < 0) { htmlParseErr(ctxt, XML_ERR_INVALID_ENCODING, "htmlCheckEncoding: encoder error\n", diff --git a/parserInternals.c b/parserInternals.c index 09876ab4..8c0cd57a 100644 --- a/parserInternals.c +++ b/parserInternals.c @@ -1214,7 +1214,7 @@ xmlSwitchInputEncodingInt(xmlParserCtxtPtr ctxt, xmlParserInputPtr input, /* * convert as much as possible of the buffer */ - nbchars = xmlCharEncInput(input->buf, 1); + nbchars = xmlCharEncInput(input->buf, 0); } else { /* * convert just enough to get diff --git a/xmlIO.c b/xmlIO.c index f61dd05a..82543477 100644 --- a/xmlIO.c +++ b/xmlIO.c @@ -3157,7 +3157,7 @@ xmlParserInputBufferPush(xmlParserInputBufferPtr in, * convert as much as possible to the parser reading buffer. */ use = xmlBufUse(in->raw); - nbchars = xmlCharEncInput(in, 1); + nbchars = xmlCharEncInput(in, 0); if (nbchars < 0) { xmlIOErr(XML_IO_ENCODER, NULL); in->error = XML_IO_ENCODER; @@ -3273,7 +3273,7 @@ xmlParserInputBufferGrow(xmlParserInputBufferPtr in, int len) { * convert as much as possible to the parser reading buffer. */ use = xmlBufUse(in->raw); - nbchars = xmlCharEncInput(in, 1); + nbchars = xmlCharEncInput(in, 0); if (nbchars < 0) { xmlIOErr(XML_IO_ENCODER, NULL); in->error = XML_IO_ENCODER; -- 2.15.1.620.gb9897f4670-goog
icu_parse_test.utf8.xml
(text/xml, 535 B)
<?xml version="1.0" encoding="UTF8-"?> <foo> Text with UTF8 chars at position 214 (0xd6) and 513 (0x201) ______ _______________ _______________ _______________ _______________ _______________ _______________ ____£_____ _______________ _______________ _______________ _______________ _______________ _______________ _______________ _______________ _______________ _______________ _______________ _______________ _______________ _______________ _______________ _______________ _______________ _______________ _£________ </foo>