[Patch] Improved UTF-8 conformance
"Nikita V. Borodikhin" <[email protected]> Tue, 6 Apr 2004 18:39:13 +0700
| Newsgroups | gmane.comp.web.aspseek.devel |
|---|---|
| Organization | sysattack.com |
| Message-ID | <[email protected]> |
This is a multi-part message in MIME format. --Multipart=_Tue__6_Apr_2004_18_39_13_+0700_Sy+Zi8053CavHQmW Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Hello, everybody! There are some problems in ASPSeek when parsing UTF-8 pages, it doesn't understand some chars, assumes they are zeros and stops parsing page. There is an example where page parsing is ended at Unicode 'EM DASH' symbol: http://forum.aspseek.org/index.php?t=msg&th=1080&start=0& There is actually problem in include/ucharset.h, where UTF->unicode algorithm doesn't conform to Unicode standard. I attached a patch for both latest 1.2.10 and CVS versions here. Nikita --Multipart=_Tue__6_Apr_2004_18_39_13_+0700_Sy+Zi8053CavHQmW Content-Type: application/octet-stream; name="aspseek-1.2.10-utf_fix.patch" Content-Disposition: attachment; filename="aspseek-1.2.10-utf_fix.patch" Content-Transfer-Encoding: base64 ZGlmZiAtcHJ1TiBhc3BzZWVrLTEuMi4xMC9pbmNsdWRlL3VjaGFyc2V0LmggYXNwc2Vlay0xLjIu MTAtdXRmX2ZpeC9pbmNsdWRlL3VjaGFyc2V0LmgKLS0tIGFzcHNlZWstMS4yLjEwL2luY2x1ZGUv dWNoYXJzZXQuaAkyMDAyLTA2LTA4IDIyOjUyOjQxLjAwMDAwMDAwMCArMDcwMAorKysgYXNwc2Vl ay0xLjIuMTAtdXRmX2ZpeC9pbmNsdWRlL3VjaGFyc2V0LmgJMjAwNC0wNC0wNiAxODoxNzoxMi4w MDAwMDAwMDAgKzA3MDAKQEAgLTY1MCw5ICs2NTAsMTYgQEAgaW5saW5lIFdPUkQgVUNvZGUoY29u c3QgQllURSomIHNyYykKIAl7CiAJCXJldHVybiAqc3JjID8gKnNyYysrIDogMDsKIAl9Ci0JaWYg KHNyY1swXSA8IDB4RTApCisJZWxzZQorCWlmIChzcmNbMF0gPCAweEMwKQorCXsKKwkJc3JjKys7 CisJCXJldHVybiAwOworCX0KKwllbHNlCisJaWYgKChzcmNbMF0gJiAweEUwKSA9PSAweEMwKQog CXsKLQkJaWYgKHNyY1sxXSAmJiAoc3JjWzBdICYgMHgxRikpCisJCWlmICgoc3JjWzFdICYgMHhD MCkgPT0gMHg4MCkKIAkJewogCQkJdWNvZGUgPSAoKHNyY1swXSAmIDB4MUYpIDw8IDYpIHwgKHNy Y1sxXSAmIDB4M0YpOwogCQkJc3JjICs9IDI7CkBAIC02NjQsMTIgKzY3MSwxNSBAQCBpbmxpbmUg V09SRCBVQ29kZShjb25zdCBCWVRFKiYgc3JjKQogCQl9CiAJfQogCWVsc2UKKyAJaWYgKChzcmNb MF0gJiAweGYwKSA9PSAweEUwKQogCXsKLQkJaWYgKHNyY1sxXSAmJiAoc3JjWzBdICYgMHhGKSkK KyAJCWlmICgoc3JjWzFdICYgMHhDMCkgPT0gMHg4MCkKIAkJewotCQkJaWYgKHNyY1syXSAmIChz cmNbMV0gJiAweDNGKSkKKyAJCQlpZiAoKHNyY1syXSAmIDB4QzApID09IDB4ODApCiAJCQl7Ci0J CQkJdWNvZGUgPSAoKHNyY1swXSAmIDB4RikgPDwgMTIpIHwgKChzcmNbMV0gJiAweDNGKSA8PCA2 KSB8IChzcmNbMl0gJiAweDNGKTsKKyAJCQkJdWNvZGUgPSAoKHNyY1swXSAmIDB4MEYpIDw8IDEy KSB8CisgCQkJCQkoKHNyY1sxXSAmIDB4M0YpIDw8IDYpIHwKKyAJCQkJCShzcmNbMl0gJiAweDNG KTsKIAkJCQlzcmMgKz0gMzsKIAkJCX0KIAkJCWVsc2UKQEAgLTY4NCw2ICs2OTQsMzkgQEAgaW5s aW5lIFdPUkQgVUNvZGUoY29uc3QgQllURSomIHNyYykKIAkJCXJldHVybiAwOwogCQl9CiAJfQor CWVsc2UKKwlpZiAoKHNyY1swXSAmIDB4RjgpID09IDB4RjApCisJeworCQlpZiAoKHNyY1sxXSAm IDB4QzApID09IDB4ODApCisJCXsKKwkJCWlmICgoc3JjWzJdICYgMHhDMCkgPT0gMHg4MCkKKwkJ CXsKKwkJCQlpZiAoKHNyY1szXSAmIDB4QzApID09IDB4ODApCisJCQkJeworCQkJCQl1Y29kZSA9 ICgoc3JjWzBdICYgMHgwNykgPDwgMTgpIHwKKwkJCQkJCSgoc3JjWzFdICYgMHgzRikgPDwgMTIp IHwKKwkJCQkJCSgoc3JjWzJdICYgMHgzRikgPDwgNikgfAorCQkJCQkJKHNyY1szXSAmIDB4M0Yp OworCQkJCQlzcmMgKz0gNDsKKwkJCQl9CisJCQkJZWxzZQorCQkJCXsKKwkJCQkJc3JjICs9IDM7 CisJCQkJCXJldHVybiAwOworCQkJCX0KKwkJCX0KKwkJCWVsc2UKKwkJCXsKKwkJCQlzcmMgKz0g MjsKKwkJCQlyZXR1cm4gMDsKKwkJCX0KKwkJfQorCQllbHNlCisJCXsKKwkJCXNyYysrOworCQkJ cmV0dXJuIDA7CisJCX0KKwl9CiAJcmV0dXJuIHVjb2RlOwogfQogCg== --Multipart=_Tue__6_Apr_2004_18_39_13_+0700_Sy+Zi8053CavHQmW Content-Type: application/octet-stream; name="aspseek-cvs-utf_fix.patch" Content-Disposition: attachment; filename="aspseek-cvs-utf_fix.patch" Content-Transfer-Encoding: base64 ZGlmZiAtcHJ1TiBhc3BzZWVrLWN2cy9pbmNsdWRlL3VjaGFyc2V0LmggYXNwc2Vlay11dGZfZml4 L2luY2x1ZGUvdWNoYXJzZXQuaAotLS0gYXNwc2Vlay1jdnMvaW5jbHVkZS91Y2hhcnNldC5oCTIw MDItMTItMTkgMDA6Mzg6NTMuMDAwMDAwMDAwICswNjAwCisrKyBhc3BzZWVrLXV0Zl9maXgvaW5j bHVkZS91Y2hhcnNldC5oCTIwMDQtMDQtMDYgMTc6NTY6MDkuMDAwMDAwMDAwICswNzAwCkBAIC02 NDQsOSArNjQ0LDE2IEBAIGlubGluZSBXT1JEIFVDb2RlKGNvbnN0IEJZVEUqJiBzcmMpCiAJewog CQlyZXR1cm4gKnNyYyA/ICpzcmMrKyA6IDA7CiAJfQotCWlmIChzcmNbMF0gPCAweEUwKQorCWVs c2UKKwlpZiAoc3JjWzBdIDwgMHhDMCkKKwl7CisJCXNyYysrOworCQlyZXR1cm4gMDsKKwl9CisJ ZWxzZQorCWlmICgoc3JjWzBdICYgMHhFMCkgPT0gMHhDMCkKIAl7Ci0JCWlmIChzcmNbMV0gJiYg KHNyY1swXSAmIDB4MUYpKQorCQlpZiAoKHNyY1sxXSAmIDB4QzApID09IDB4ODApCiAJCXsKIAkJ CXVjb2RlID0gKChzcmNbMF0gJiAweDFGKSA8PCA2KSB8IChzcmNbMV0gJiAweDNGKTsKIAkJCXNy YyArPSAyOwpAQCAtNjU4LDEyICs2NjUsMTUgQEAgaW5saW5lIFdPUkQgVUNvZGUoY29uc3QgQllU RSomIHNyYykKIAkJfQogCX0KIAllbHNlCisJaWYgKChzcmNbMF0gJiAweGYwKSA9PSAweEUwKQog CXsKLQkJaWYgKHNyY1sxXSAmJiAoc3JjWzBdICYgMHhGKSkKKwkJaWYgKChzcmNbMV0gJiAweEMw KSA9PSAweDgwKQogCQl7Ci0JCQlpZiAoc3JjWzJdICYmIChzcmNbMV0gJiAweDNGKSkKKwkJCWlm ICgoc3JjWzJdICYgMHhDMCkgPT0gMHg4MCkKIAkJCXsKLQkJCQl1Y29kZSA9ICgoc3JjWzBdICYg MHhGKSA8PCAxMikgfCAoKHNyY1sxXSAmIDB4M0YpIDw8IDYpIHwgKHNyY1syXSAmIDB4M0YpOwor CQkJCXVjb2RlID0gKChzcmNbMF0gJiAweDBGKSA8PCAxMikgfAorCQkJCQkoKHNyY1sxXSAmIDB4 M0YpIDw8IDYpIHwKKwkJCQkJKHNyY1syXSAmIDB4M0YpOwogCQkJCXNyYyArPSAzOwogCQkJfQog CQkJZWxzZQpAQCAtNjc4LDYgKzY4OCwzOSBAQCBpbmxpbmUgV09SRCBVQ29kZShjb25zdCBCWVRF KiYgc3JjKQogCQkJcmV0dXJuIDA7CiAJCX0KIAl9CisJZWxzZQorCWlmICgoc3JjWzBdICYgMHhG OCkgPT0gMHhGMCkKKwl7CisJCWlmICgoc3JjWzFdICYgMHhDMCkgPT0gMHg4MCkKKwkJeworCQkJ aWYgKChzcmNbMl0gJiAweEMwKSA9PSAweDgwKQorCQkJeworCQkJCWlmICgoc3JjWzNdICYgMHhD MCkgPT0gMHg4MCkKKwkJCQl7CisJCQkJCXVjb2RlID0gKChzcmNbMF0gJiAweDA3KSA8PCAxOCkg fAorCQkJCQkJKChzcmNbMV0gJiAweDNGKSA8PCAxMikgfAorCQkJCQkJKChzcmNbMl0gJiAweDNG KSA8PCA2KSB8CisJCQkJCQkoc3JjWzNdICYgMHgzRik7CisJCQkJCXNyYyArPSA0OworCQkJCX0K KwkJCQllbHNlCisJCQkJeworCQkJCQlzcmMgKz0gMzsKKwkJCQkJcmV0dXJuIDA7CisJCQkJfQor CQkJfQorCQkJZWxzZQorCQkJeworCQkJCXNyYyArPSAyOworCQkJCXJldHVybiAwOworCQkJfQor CQl9CisJCWVsc2UKKwkJeworCQkJc3JjKys7CisJCQlyZXR1cm4gMDsKKwkJfQorCX0KIAlyZXR1 cm4gdWNvZGU7CiB9CiAK --Multipart=_Tue__6_Apr_2004_18_39_13_+0700_Sy+Zi8053CavHQmW--