Revised PATCH: [perl #75350] revamp ibcmp_utf8() for efficiency, clarity
[email protected] (karl williamson)
| Newsgroups | perl.perl5.porters |
|---|---|
| Message-ID | <[email protected]> |
Attached is a revised patch that meets Yves' objections. A later commit in it also changes all the functions whose names begin with ibcmp to foldEQ, which is the complement of the existing ones. I think the latest plan is to deliver the affected generated files, like embed.h. So they are included here. I did not change PPP stuff because I really don't know what to do there, if anything, as the old names remain valid
0001-Clarify-some-documentation.patch
(text/x-patch, 1.3 KB)
From 82bb7afbb7090de9e317380f4940aff9c0042dda Mon Sep 17 00:00:00 2001 From: Karl Williamson <khw@khw-desktop.(none)> Date: Tue, 25 May 2010 11:17:22 -0600 Subject: [PATCH] Clarify some documentation --- utf8.c | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-) diff --git a/utf8.c b/utf8.c index eeea158..6c522d0 100644 --- a/utf8.c +++ b/utf8.c @@ -33,7 +33,7 @@ #include "perl.h" #ifndef EBCDIC -/* Separate prototypes needed because in ASCII systems these +/* Separate prototypes needed because in ASCII systems these are * usually macros but they still are compiled as code, too. */ PERL_CALLCONV UV Perl_utf8n_to_uvchr(pTHX_ const U8 *s, STRLEN curlen, STRLEN *retlen, U32 flags); PERL_CALLCONV U8* Perl_uvchr_to_utf8(pTHX_ U8 *d, UV uv); @@ -57,8 +57,10 @@ within non-zero characters. /* =for apidoc is_ascii_string -Returns true if first C<len> bytes of the given string are ASCII (i.e. none -of them even raise the question of UTF-8-ness). +Returns true if the first C<len> bytes of the given string are the same whether +or not the string is encoded in UTF-8 (or UTF-EBCDIC on EBCDIC machines). That +is, if they are invariant. On ASCII-ish machines, only ASCII characters +fit this definition, hence the function's name. See also is_utf8_string(), is_utf8_string_loclen(), and is_utf8_string_loc(). -- 1.5.6.3
0002-Revamp-ibcmp_utf8-for-performance-and-clarity.patch
(text/x-patch, 10.4 KB)
From a018ccf058864c4a0c8485a800ef3b3319c1336c Mon Sep 17 00:00:00 2001
From: Karl Williamson <khw@khw-desktop.(none)>
Date: Tue, 25 May 2010 11:18:42 -0600
Subject: [PATCH] Revamp ibcmp_utf8 for performance and clarity
I had a hard time understanding how this routine worked; there were no
comments. In figuring it out, I discovered it could be made more
efficient. This routine is called over and over in the innermost loops
in regex matching, so efficiency is a concern.
Setup is done once before the main while loop so that it now has two
conditions instead of eight. The loop was rearranged slightly to be
smaller and a couple of unneeded assignments to temporaries were
removed, and recomputation of some values was avoided. Several other
small efficiency changes were made.
Several asserts had been commented out, saying that they make tests
fail. But they no longer do, at least on my platform. There was a
reason that they were asserts to begin with, and that is they denoted an
insane or trivial condition. Apparently there have been fixes to the
other code calling this, so I re-enabled them.
The names of several variables were changed to be less confusing; hence
f1 means the fold buffer for string 1 whereas it used to mean its goal,
which is now g1.
The leading indent was changed from 5 to 4 blanks. I made enough
other changes that I didn't submit this as a separate commit
---
utf8.c | 260 +++++++++++++++++++++++++++++++++++++---------------------------
1 files changed, 152 insertions(+), 108 deletions(-)
diff --git a/utf8.c b/utf8.c
index 6c522d0..8fd5db9 100644
--- a/utf8.c
+++ b/utf8.c
@@ -2504,22 +2504,33 @@ Perl_sv_uni_display(pTHX_ SV *dsv, SV *ssv, STRLEN pvlim, UV flags)
/*
=for apidoc ibcmp_utf8
-Return true if the strings s1 and s2 differ case-insensitively, false
-if not (if they are equal case-insensitively). If u1 is true, the
-string s1 is assumed to be in UTF-8-encoded Unicode. If u2 is true,
-the string s2 is assumed to be in UTF-8-encoded Unicode. If u1 or u2
-are false, the respective string is assumed to be in native 8-bit
-encoding.
-
-If the pe1 and pe2 are non-NULL, the scanning pointers will be copied
-in there (they will point at the beginning of the I<next> character).
-If the pointers behind pe1 or pe2 are non-NULL, they are the end
-pointers beyond which scanning will not continue under any
-circumstances. If the byte lengths l1 and l2 are non-zero, s1+l1 and
-s2+l2 will be used as goal end pointers that will also stop the scan,
-and which qualify towards defining a successful match: all the scans
-that define an explicit length must reach their goal pointers for
-a match to succeed).
+Returns true if the strings s1 and s2 differ case-insensitively, false
+if they are equal case-insensitively. Note that this is the complement of what
+you might expect (perhaps it would have been better to name it C<ibncmp_utf8>).
+
+If u1 is true, the string s1 is assumed to be in UTF-8-encoded Unicode;
+otherwise it is assumed to be in native 8-bit encoding. Correspondingly for u2
+with respect to s2.
+
+If the byte length l1 is non-zero, s1+l1 will be used as a goal to reach. The
+scan will not be considered to be a match unless the goal is reached, and
+scanning won't continue past that goal. Correspondingly for l2 with respect to
+s2.
+
+If pe1 is non-NULL and the pointer it points to is not NULL, that pointer is
+considered an end pointer beyond which scanning of s1 will not continue under
+any circumstances. This means that if both l1 and pe1 are specified, and pe1
+is less than s1+l1, the match will never be successful because it can never
+get as far as its goal. Correspondingly for pe2 with respect to s2.
+
+At least one of s1 and s2 must have a goal, and if both do, both have to be
+reached for a successful match. Also, if the fold of a character is multiple
+characters, all of them must be matched (see tr21 reference below for
+'folding').
+
+Upon a successful match (when the routine returns false), if pe1 is non-NULL,
+it will be set to point to the beginning of the I<next> character of s1 beyond
+what was matched. Correspondingly for pe2 and s2.
For case-insensitiveness, the "casefolding" of Unicode is used
instead of upper/lowercasing both the characters, see
@@ -2529,98 +2540,131 @@ http://www.unicode.org/unicode/reports/tr21/ (Case Mappings).
I32
Perl_ibcmp_utf8(pTHX_ const char *s1, char **pe1, register UV l1, bool u1, const char *s2, char **pe2, register UV l2, bool u2)
{
- dVAR;
- register const U8 *p1 = (const U8*)s1;
- register const U8 *p2 = (const U8*)s2;
- register const U8 *f1 = NULL;
- register const U8 *f2 = NULL;
- register U8 *e1 = NULL;
- register U8 *q1 = NULL;
- register U8 *e2 = NULL;
- register U8 *q2 = NULL;
- STRLEN n1 = 0, n2 = 0;
- U8 foldbuf1[UTF8_MAXBYTES_CASE+1];
- U8 foldbuf2[UTF8_MAXBYTES_CASE+1];
- U8 natbuf[1+1];
- STRLEN foldlen1, foldlen2;
- bool match;
-
- PERL_ARGS_ASSERT_IBCMP_UTF8;
-
- if (pe1)
- e1 = *(U8**)pe1;
- /* assert(e1 || l1); */
- if (e1 == 0 || (l1 && l1 < (UV)(e1 - (const U8*)s1)))
- f1 = (const U8*)s1 + l1;
- if (pe2)
- e2 = *(U8**)pe2;
- /* assert(e2 || l2); */
- if (e2 == 0 || (l2 && l2 < (UV)(e2 - (const U8*)s2)))
- f2 = (const U8*)s2 + l2;
-
- /* This shouldn't happen. However, putting an assert() there makes some
- * tests fail. */
- /* assert((e1 == 0 && f1 == 0) || (e2 == 0 && f2 == 0) || (f1 == 0 && f2 == 0)); */
- if ((e1 == 0 && f1 == 0) || (e2 == 0 && f2 == 0) || (f1 == 0 && f2 == 0))
- return 1; /* mismatch; possible infinite loop or false positive */
-
- if (!u1 || !u2)
- natbuf[1] = 0; /* Need to terminate the buffer. */
-
- while ((e1 == 0 || p1 < e1) &&
- (f1 == 0 || p1 < f1) &&
- (e2 == 0 || p2 < e2) &&
- (f2 == 0 || p2 < f2)) {
- if (n1 == 0) {
- if (u1)
- to_utf8_fold(p1, foldbuf1, &foldlen1);
- else {
- uvuni_to_utf8(natbuf, (UV) NATIVE_TO_UNI(((UV)*p1)));
- to_utf8_fold(natbuf, foldbuf1, &foldlen1);
- }
- q1 = foldbuf1;
- n1 = foldlen1;
- }
- if (n2 == 0) {
- if (u2)
- to_utf8_fold(p2, foldbuf2, &foldlen2);
- else {
- uvuni_to_utf8(natbuf, (UV) NATIVE_TO_UNI(((UV)*p2)));
- to_utf8_fold(natbuf, foldbuf2, &foldlen2);
- }
- q2 = foldbuf2;
- n2 = foldlen2;
- }
- while (n1 && n2) {
- if ( UTF8SKIP(q1) != UTF8SKIP(q2) ||
- (UTF8SKIP(q1) == 1 && *q1 != *q2) ||
- memNE((char*)q1, (char*)q2, UTF8SKIP(q1)) )
- return 1; /* mismatch */
- n1 -= UTF8SKIP(q1);
- q1 += UTF8SKIP(q1);
- n2 -= UTF8SKIP(q2);
- q2 += UTF8SKIP(q2);
- }
- if (n1 == 0)
- p1 += u1 ? UTF8SKIP(p1) : 1;
- if (n2 == 0)
- p2 += u2 ? UTF8SKIP(p2) : 1;
-
- }
-
- /* A match is defined by all the scans that specified
- * an explicit length reaching their final goals. */
- match = (n1 == 0 && n2 == 0 /* Must not match partial char; Bug #72998 */
- && (f1 == 0 || p1 == f1) && (f2 == 0 || p2 == f2));
-
- if (match) {
- if (pe1)
- *pe1 = (char*)p1;
- if (pe2)
- *pe2 = (char*)p2;
- }
-
- return match ? 0 : 1; /* 0 match, 1 mismatch */
+ dVAR;
+ register const U8 *p1 = (const U8*)s1; /* Point to current char */
+ register const U8 *p2 = (const U8*)s2;
+ register const U8 *g1 = NULL; /* goal for s1 */
+ register const U8 *g2 = NULL;
+ register const U8 *e1 = NULL; /* Don't scan s1 past this */
+ register U8 *f1 = NULL; /* Point to current folded */
+ register const U8 *e2 = NULL;
+ register U8 *f2 = NULL;
+ STRLEN n1 = 0, n2 = 0; /* Number of bytes in current char */
+ U8 foldbuf1[UTF8_MAXBYTES_CASE+1];
+ U8 foldbuf2[UTF8_MAXBYTES_CASE+1];
+ U8 natbuf[2]; /* Holds native 8-bit char converted to utf8;
+ these always fit in 2 bytes */
+
+ PERL_ARGS_ASSERT_IBCMP_UTF8;
+
+ if (pe1) {
+ e1 = *(U8**)pe1;
+ }
+
+ if (l1) {
+ g1 = (const U8*)s1 + l1;
+ }
+
+ if (pe2) {
+ e2 = *(U8**)pe2;
+ }
+
+ if (l2) {
+ g2 = (const U8*)s2 + l2;
+ }
+
+ /* Must have at least one goal */
+ assert(g1 || g2);
+
+ if (g1) {
+
+ /* Will never match if goal is out-of-bounds */
+ assert(! e1 || e1 >= g1);
+
+ /* Here, there isn't an end pointer, or it is beyond the goal. We
+ * only go as far as the goal */
+ e1 = g1;
+ }
+ else assert(e1); /* Must have an end for looking at s1 */
+
+ /* Same for goal for s2 */
+ if (g2) {
+ assert(! e2 || e2 >= g2);
+ e2 = g2;
+ }
+ else assert(e2);
+
+ /* Look through both strings, a character at a time */
+ while (p1 < e1 && p2 < e2) {
+
+ /* If at the beginning of a new character in s1, get its fold to use */
+ if (n1 == 0) {
+ if (u1) {
+ to_utf8_fold(p1, foldbuf1, &n1);
+ }
+ else { /* Not utf8, convert to it first and then get fold */
+ uvuni_to_utf8(natbuf, (UV) NATIVE_TO_UNI(((UV)*p1)));
+ to_utf8_fold(natbuf, foldbuf1, &n1);
+ }
+ f1 = foldbuf1;
+ }
+
+ if (n2 == 0) { /* Same for s2 */
+ if (u2) {
+ to_utf8_fold(p2, foldbuf2, &n2);
+ }
+ else {
+ uvuni_to_utf8(natbuf, (UV) NATIVE_TO_UNI(((UV)*p2)));
+ to_utf8_fold(natbuf, foldbuf2, &n2);
+ }
+ f2 = foldbuf2;
+ }
+
+ /* While there is more to look for in both folds, see if they
+ * continue to match */
+ while (n1 && n2) {
+ U8 fold_length = UTF8SKIP(f1);
+ if (fold_length != UTF8SKIP(f2)
+ || (fold_length == 1 && *f1 != *f2) /* Short circuit memNE
+ function call for single
+ character */
+ || memNE((char*)f1, (char*)f2, fold_length))
+ {
+ return 1; /* mismatch */
+ }
+
+ /* Here, they matched, advance past them */
+ n1 -= fold_length;
+ f1 += fold_length;
+ n2 -= fold_length;
+ f2 += fold_length;
+ }
+
+ /* When reach the end of any fold, advance the input past it */
+ if (n1 == 0) {
+ p1 += u1 ? UTF8SKIP(p1) : 1;
+ }
+ if (n2 == 0) {
+ p2 += u2 ? UTF8SKIP(p2) : 1;
+ }
+ } /* End of loop through both strings */
+
+ /* A match is defined by each scan that specified an explicit length
+ * reaching its final goal, and the other not having matched a partial
+ * character (which can happen when the fold of a character is more than one
+ * character). */
+ if (! ((g1 == 0 || p1 == g1) && (g2 == 0 || p2 == g2)) || n1 || n2) {
+ return 1;
+ }
+
+ /* Successful match. Set output pointers */
+ if (pe1) {
+ *pe1 = (char*)p1;
+ }
+ if (pe2) {
+ *pe2 = (char*)p2;
+ }
+ return 0;
}
/*
--
1.5.6.3
0003-utf8.c-Modify-doc-comment-change-whitespace.patch
(text/x-patch, 8.7 KB)
From aa95f173750f5cda7942cae9d22d7ee0cbf82ccb Mon Sep 17 00:00:00 2001
From: Karl Williamson <khw@khw-desktop.(none)>
Date: Fri, 4 Jun 2010 12:04:45 -0600
Subject: [PATCH] utf8.c: Modify doc comment; change whitespace
This removes the comment about the function name, and converts tabs to
blanks throughout the function, as so much of it is changing already.
It also removes trailing whitespace in other lines of the file.
---
utf8.c | 149 ++++++++++++++++++++++++++++++++--------------------------------
1 files changed, 74 insertions(+), 75 deletions(-)
diff --git a/utf8.c b/utf8.c
index 8fd5db9..668b9fa 100644
--- a/utf8.c
+++ b/utf8.c
@@ -42,7 +42,7 @@ PERL_CALLCONV U8* Perl_uvchr_to_utf8(pTHX_ U8 *d, UV uv);
static const char unees[] =
"Malformed UTF-8 character (unexpected end of string)";
-/*
+/*
=head1 Unicode Support
This file contains various utility functions for manipulating UTF8-encoded
@@ -264,7 +264,7 @@ S_is_utf8_char_slow(const U8 *s, const STRLEN len)
if (!UTF8_IS_CONTINUATION(*s))
return 0;
uv = UTF8_ACCUMULATE(uv, *s);
- if (uv < ouv)
+ if (uv < ouv)
return 0;
ouv = uv;
s++;
@@ -2377,7 +2377,7 @@ Perl_uvchr_to_utf8_flags(pTHX_ U8 *d, UV uv, UV flags)
=for apidoc utf8n_to_uvchr
flags
-Returns the native character value of the first character in the string
+Returns the native character value of the first character in the string
C<s>
which is assumed to be in UTF-8 encoding; C<retlen> will be set to the
length, in bytes, of that character.
@@ -2390,7 +2390,7 @@ Allows length and flags to be passed to low level routine.
a real function in case XS code wants it
*/
UV
-Perl_utf8n_to_uvchr(pTHX_ const U8 *s, STRLEN curlen, STRLEN *retlen,
+Perl_utf8n_to_uvchr(pTHX_ const U8 *s, STRLEN curlen, STRLEN *retlen,
U32 flags)
{
const UV uv = Perl_utf8n_to_uvuni(aTHX_ s, curlen, retlen, flags);
@@ -2475,7 +2475,7 @@ Perl_pv_uni_display(pTHX_ SV *dsv, const U8 *spv, STRLEN len, STRLEN pvlim, UV f
}
if (truncated)
sv_catpvs(dsv, "...");
-
+
return SvPVX(dsv);
}
@@ -2505,8 +2505,7 @@ Perl_sv_uni_display(pTHX_ SV *dsv, SV *ssv, STRLEN pvlim, UV flags)
=for apidoc ibcmp_utf8
Returns true if the strings s1 and s2 differ case-insensitively, false
-if they are equal case-insensitively. Note that this is the complement of what
-you might expect (perhaps it would have been better to name it C<ibncmp_utf8>).
+if they are equal case-insensitively.
If u1 is true, the string s1 is assumed to be in UTF-8-encoded Unicode;
otherwise it is assumed to be in native 8-bit encoding. Correspondingly for u2
@@ -2543,34 +2542,34 @@ Perl_ibcmp_utf8(pTHX_ const char *s1, char **pe1, register UV l1, bool u1, const
dVAR;
register const U8 *p1 = (const U8*)s1; /* Point to current char */
register const U8 *p2 = (const U8*)s2;
- register const U8 *g1 = NULL; /* goal for s1 */
+ register const U8 *g1 = NULL; /* goal for s1 */
register const U8 *g2 = NULL;
- register const U8 *e1 = NULL; /* Don't scan s1 past this */
- register U8 *f1 = NULL; /* Point to current folded */
+ register const U8 *e1 = NULL; /* Don't scan s1 past this */
+ register U8 *f1 = NULL; /* Point to current folded */
register const U8 *e2 = NULL;
register U8 *f2 = NULL;
- STRLEN n1 = 0, n2 = 0; /* Number of bytes in current char */
+ STRLEN n1 = 0, n2 = 0; /* Number of bytes in current char */
U8 foldbuf1[UTF8_MAXBYTES_CASE+1];
U8 foldbuf2[UTF8_MAXBYTES_CASE+1];
- U8 natbuf[2]; /* Holds native 8-bit char converted to utf8;
- these always fit in 2 bytes */
+ U8 natbuf[2]; /* Holds native 8-bit char converted to utf8;
+ these always fit in 2 bytes */
PERL_ARGS_ASSERT_IBCMP_UTF8;
if (pe1) {
- e1 = *(U8**)pe1;
+ e1 = *(U8**)pe1;
}
if (l1) {
- g1 = (const U8*)s1 + l1;
+ g1 = (const U8*)s1 + l1;
}
if (pe2) {
- e2 = *(U8**)pe2;
+ e2 = *(U8**)pe2;
}
if (l2) {
- g2 = (const U8*)s2 + l2;
+ g2 = (const U8*)s2 + l2;
}
/* Must have at least one goal */
@@ -2578,75 +2577,75 @@ Perl_ibcmp_utf8(pTHX_ const char *s1, char **pe1, register UV l1, bool u1, const
if (g1) {
- /* Will never match if goal is out-of-bounds */
- assert(! e1 || e1 >= g1);
+ /* Will never match if goal is out-of-bounds */
+ assert(! e1 || e1 >= g1);
- /* Here, there isn't an end pointer, or it is beyond the goal. We
- * only go as far as the goal */
- e1 = g1;
+ /* Here, there isn't an end pointer, or it is beyond the goal. We
+ * only go as far as the goal */
+ e1 = g1;
}
- else assert(e1); /* Must have an end for looking at s1 */
+ else assert(e1); /* Must have an end for looking at s1 */
/* Same for goal for s2 */
if (g2) {
- assert(! e2 || e2 >= g2);
- e2 = g2;
+ assert(! e2 || e2 >= g2);
+ e2 = g2;
}
else assert(e2);
/* Look through both strings, a character at a time */
while (p1 < e1 && p2 < e2) {
- /* If at the beginning of a new character in s1, get its fold to use */
- if (n1 == 0) {
- if (u1) {
- to_utf8_fold(p1, foldbuf1, &n1);
- }
- else { /* Not utf8, convert to it first and then get fold */
- uvuni_to_utf8(natbuf, (UV) NATIVE_TO_UNI(((UV)*p1)));
- to_utf8_fold(natbuf, foldbuf1, &n1);
- }
- f1 = foldbuf1;
- }
-
- if (n2 == 0) { /* Same for s2 */
- if (u2) {
- to_utf8_fold(p2, foldbuf2, &n2);
- }
- else {
- uvuni_to_utf8(natbuf, (UV) NATIVE_TO_UNI(((UV)*p2)));
- to_utf8_fold(natbuf, foldbuf2, &n2);
- }
- f2 = foldbuf2;
- }
+ /* If at the beginning of a new character in s1, get its fold to use */
+ if (n1 == 0) {
+ if (u1) {
+ to_utf8_fold(p1, foldbuf1, &n1);
+ }
+ else { /* Not utf8, convert to it first and then get fold */
+ uvuni_to_utf8(natbuf, (UV) NATIVE_TO_UNI(((UV)*p1)));
+ to_utf8_fold(natbuf, foldbuf1, &n1);
+ }
+ f1 = foldbuf1;
+ }
- /* While there is more to look for in both folds, see if they
- * continue to match */
- while (n1 && n2) {
- U8 fold_length = UTF8SKIP(f1);
- if (fold_length != UTF8SKIP(f2)
- || (fold_length == 1 && *f1 != *f2) /* Short circuit memNE
- function call for single
- character */
- || memNE((char*)f1, (char*)f2, fold_length))
- {
- return 1; /* mismatch */
- }
+ if (n2 == 0) { /* Same for s2 */
+ if (u2) {
+ to_utf8_fold(p2, foldbuf2, &n2);
+ }
+ else {
+ uvuni_to_utf8(natbuf, (UV) NATIVE_TO_UNI(((UV)*p2)));
+ to_utf8_fold(natbuf, foldbuf2, &n2);
+ }
+ f2 = foldbuf2;
+ }
- /* Here, they matched, advance past them */
- n1 -= fold_length;
- f1 += fold_length;
- n2 -= fold_length;
- f2 += fold_length;
- }
+ /* While there is more to look for in both folds, see if they
+ * continue to match */
+ while (n1 && n2) {
+ U8 fold_length = UTF8SKIP(f1);
+ if (fold_length != UTF8SKIP(f2)
+ || (fold_length == 1 && *f1 != *f2) /* Short circuit memNE
+ function call for single
+ character */
+ || memNE((char*)f1, (char*)f2, fold_length))
+ {
+ return 1; /* mismatch */
+ }
+
+ /* Here, they matched, advance past them */
+ n1 -= fold_length;
+ f1 += fold_length;
+ n2 -= fold_length;
+ f2 += fold_length;
+ }
- /* When reach the end of any fold, advance the input past it */
- if (n1 == 0) {
- p1 += u1 ? UTF8SKIP(p1) : 1;
- }
- if (n2 == 0) {
- p2 += u2 ? UTF8SKIP(p2) : 1;
- }
+ /* When reach the end of any fold, advance the input past it */
+ if (n1 == 0) {
+ p1 += u1 ? UTF8SKIP(p1) : 1;
+ }
+ if (n2 == 0) {
+ p2 += u2 ? UTF8SKIP(p2) : 1;
+ }
} /* End of loop through both strings */
/* A match is defined by each scan that specified an explicit length
@@ -2654,15 +2653,15 @@ Perl_ibcmp_utf8(pTHX_ const char *s1, char **pe1, register UV l1, bool u1, const
* character (which can happen when the fold of a character is more than one
* character). */
if (! ((g1 == 0 || p1 == g1) && (g2 == 0 || p2 == g2)) || n1 || n2) {
- return 1;
+ return 1;
}
/* Successful match. Set output pointers */
if (pe1) {
- *pe1 = (char*)p1;
+ *pe1 = (char*)p1;
}
if (pe2) {
- *pe2 = (char*)p2;
+ *pe2 = (char*)p2;
}
return 0;
}
--
1.5.6.3
0004-utf8.c-further-doc-tweaks.patch
(text/x-patch, 2.7 KB)
From f7bd77ffd6a1ffb6639593e549d21606566d662a Mon Sep 17 00:00:00 2001
From: Karl Williamson <khw@khw-desktop.(none)>
Date: Sat, 5 Jun 2010 11:08:25 -0600
Subject: [PATCH] utf8.c: further doc tweaks
---
utf8.c | 17 +++++++++++------
1 files changed, 11 insertions(+), 6 deletions(-)
diff --git a/utf8.c b/utf8.c
index 668b9fa..d0be794 100644
--- a/utf8.c
+++ b/utf8.c
@@ -2504,14 +2504,16 @@ Perl_sv_uni_display(pTHX_ SV *dsv, SV *ssv, STRLEN pvlim, UV flags)
/*
=for apidoc ibcmp_utf8
-Returns true if the strings s1 and s2 differ case-insensitively, false
-if they are equal case-insensitively.
+Returns true if the leading portions of the strings s1 and s2 (either or both
+of which may be in UTF-8) differ case-insensitively; false otherwise.
+How far into the strings to compare is determined by other input parameters.
If u1 is true, the string s1 is assumed to be in UTF-8-encoded Unicode;
otherwise it is assumed to be in native 8-bit encoding. Correspondingly for u2
with respect to s2.
-If the byte length l1 is non-zero, s1+l1 will be used as a goal to reach. The
+If the byte length l1 is non-zero, it says how far into s1 to check for fold
+equality. In other words, s1+l1 will be used as a goal to reach. The
scan will not be considered to be a match unless the goal is reached, and
scanning won't continue past that goal. Correspondingly for l2 with respect to
s2.
@@ -2520,9 +2522,11 @@ If pe1 is non-NULL and the pointer it points to is not NULL, that pointer is
considered an end pointer beyond which scanning of s1 will not continue under
any circumstances. This means that if both l1 and pe1 are specified, and pe1
is less than s1+l1, the match will never be successful because it can never
-get as far as its goal. Correspondingly for pe2 with respect to s2.
+get as far as its goal (and in fact is asserted against). Correspondingly for
+pe2 with respect to s2.
-At least one of s1 and s2 must have a goal, and if both do, both have to be
+At least one of s1 and s2 must have a goal (at least one of l1 and l2 must be
+non-zero), and if both do, both have to be
reached for a successful match. Also, if the fold of a character is multiple
characters, all of them must be matched (see tr21 reference below for
'folding').
@@ -2596,7 +2600,8 @@ Perl_ibcmp_utf8(pTHX_ const char *s1, char **pe1, register UV l1, bool u1, const
/* Look through both strings, a character at a time */
while (p1 < e1 && p2 < e2) {
- /* If at the beginning of a new character in s1, get its fold to use */
+ /* If at the beginning of a new character in s1, get its fold to use
+ * and the length of the fold */
if (n1 == 0) {
if (u1) {
to_utf8_fold(p1, foldbuf1, &n1);
--
1.5.6.3
0005-Change-name-of-ibcmp-to-foldEQ.patch
(text/x-patch, 12.6 KB)
From d62261930c5e30b5a9ee5f36d49eb0ee669d7e26 Mon Sep 17 00:00:00 2001
From: Karl Williamson <khw@khw-desktop.(none)>
Date: Sat, 5 Jun 2010 11:12:47 -0600
Subject: [PATCH] Change name of ibcmp to foldEQ
As discussed on p5p, ibcmp has different semantics from other cmp
functions in that it is a binary instead of ternary function. It is
less confusing then to have a name that implies true/false.
There are three functions affected: ibcmp, ibcmp_locale and ibcmp_utf8.
ibcmp is actually equivalent to foldNE, but for the same reason that things
like 'unless' and 'until' are cautioned against, I changed the functions
to foldEQ, so that the existing names, like ibcmp_utf8 are defined as
macros as being the complement of foldEQ.
This patch also changes the one file where turning ibcmp into a macro
causes problems. It changes it to use the new name. It also documents
for the first time ibcmp, ibcmp_locale and their new names.
---
embed.fnc | 11 ++++++++---
embed.h | 12 ++++++------
global.sym | 6 +++---
locale.c | 16 ++++++++--------
proto.h | 24 ++++++++++++++++++------
utf8.c | 16 ++++++++--------
utf8.h | 10 ++++++++++
util.c | 37 +++++++++++++++++++++++++++++--------
util.h | 15 +++++++++++++++
9 files changed, 105 insertions(+), 42 deletions(-)
diff --git a/embed.fnc b/embed.fnc
index 01605a9..36d8c1a 100644
--- a/embed.fnc
+++ b/embed.fnc
@@ -499,9 +499,14 @@ Abmd |HE* |hv_store_ent |NULLOK HV *hv|NULLOK SV *key|NULLOK SV *val\
AbmM |SV** |hv_store_flags |NULLOK HV *hv|NULLOK const char *key \
|I32 klen|NULLOK SV *val|U32 hash|int flags
Apd |void |hv_undef |NULLOK HV *hv
-AnpP |I32 |ibcmp |NN const char* a|NN const char* b|I32 len
-AnpP |I32 |ibcmp_locale |NN const char* a|NN const char* b|I32 len
-Apd |I32 |ibcmp_utf8 |NN const char *s1|NULLOK char **pe1|UV l1 \
+Am |I32 |ibcmp |NN const char* a|NN const char* b|I32 len
+AnpP |I32 |foldEQ |NN const char* a|NN const char* b|I32 len
+Am |I32 |ibcmp_locale |NN const char* a|NN const char* b|I32 len
+AnpP |I32 |foldEQ_locale |NN const char* a|NN const char* b|I32 len
+Am |I32 |ibcmp_utf8 |NN const char *s1|NULLOK char **pe1|UV l1 \
+ |bool u1|NN const char *s2|NULLOK char **pe2 \
+ |UV l2|bool u2
+Apd |I32 |foldEQ_utf8 |NN const char *s1|NULLOK char **pe1|UV l1 \
|bool u1|NN const char *s2|NULLOK char **pe2 \
|UV l2|bool u2
#if defined(PERL_IN_DOIO_C) || defined(PERL_DECL_PROT)
diff --git a/embed.h b/embed.h
index 588c50a..80457a2 100644
--- a/embed.h
+++ b/embed.h
@@ -315,9 +315,9 @@
#endif
#endif
#define hv_undef Perl_hv_undef
-#define ibcmp Perl_ibcmp
-#define ibcmp_locale Perl_ibcmp_locale
-#define ibcmp_utf8 Perl_ibcmp_utf8
+#define foldEQ Perl_foldEQ
+#define foldEQ_locale Perl_foldEQ_locale
+#define foldEQ_utf8 Perl_foldEQ_utf8
#if defined(PERL_IN_DOIO_C) || defined(PERL_DECL_PROT)
#ifdef PERL_CORE
#define ingroup S_ingroup
@@ -2752,9 +2752,9 @@
#endif
#endif
#define hv_undef(a) Perl_hv_undef(aTHX_ a)
-#define ibcmp Perl_ibcmp
-#define ibcmp_locale Perl_ibcmp_locale
-#define ibcmp_utf8(a,b,c,d,e,f,g,h) Perl_ibcmp_utf8(aTHX_ a,b,c,d,e,f,g,h)
+#define foldEQ Perl_foldEQ
+#define foldEQ_locale Perl_foldEQ_locale
+#define foldEQ_utf8(a,b,c,d,e,f,g,h) Perl_foldEQ_utf8(aTHX_ a,b,c,d,e,f,g,h)
#if defined(PERL_IN_DOIO_C) || defined(PERL_DECL_PROT)
#ifdef PERL_CORE
#define ingroup(a,b) S_ingroup(aTHX_ a,b)
diff --git a/global.sym b/global.sym
index 5ab0090..459f796 100644
--- a/global.sym
+++ b/global.sym
@@ -185,9 +185,9 @@ Perl_hv_store
Perl_hv_store_ent
Perl_hv_store_flags
Perl_hv_undef
-Perl_ibcmp
-Perl_ibcmp_locale
-Perl_ibcmp_utf8
+Perl_foldEQ
+Perl_foldEQ_locale
+Perl_foldEQ_utf8
Perl_init_stacks
Perl_init_tm
Perl_instr
diff --git a/locale.c b/locale.c
index 16ccce8..bd93cd6 100644
--- a/locale.c
+++ b/locale.c
@@ -522,23 +522,23 @@ Perl_init_i18nl10n(pTHX_ int printwarn)
codeset = nl_langinfo(CODESET);
#endif
if (codeset)
- utf8locale = (ibcmp(codeset, STR_WITH_LEN("UTF-8")) == 0 ||
- ibcmp(codeset, STR_WITH_LEN("UTF8") ) == 0);
+ utf8locale = (foldEQ(codeset, STR_WITH_LEN("UTF-8"))
+ || foldEQ(codeset, STR_WITH_LEN("UTF8") ));
#if defined(USE_LOCALE)
else { /* nl_langinfo(CODESET) is supposed to correctly
* interpret the locale environment variables,
* but just in case it fails, let's do this manually. */
if (lang)
- utf8locale = (ibcmp(lang, STR_WITH_LEN("UTF-8")) == 0 ||
- ibcmp(lang, STR_WITH_LEN("UTF8") ) == 0);
+ utf8locale = (foldEQ(lang, STR_WITH_LEN("UTF-8"))
+ || foldEQ(lang, STR_WITH_LEN("UTF8") ));
#ifdef USE_LOCALE_CTYPE
if (curctype)
- utf8locale = (ibcmp(curctype, STR_WITH_LEN("UTF-8")) == 0 ||
- ibcmp(curctype, STR_WITH_LEN("UTF8") ) == 0);
+ utf8locale = (foldEQ(curctype, STR_WITH_LEN("UTF-8"))
+ || foldEQ(curctype, STR_WITH_LEN("UTF8") ));
#endif
if (lc_all)
- utf8locale = (ibcmp(lc_all, STR_WITH_LEN("UTF-8")) == 0 ||
- ibcmp(lc_all, STR_WITH_LEN("UTF8") ) == 0);
+ utf8locale = (foldEQ(lc_all, STR_WITH_LEN("UTF-8"))
+ || foldEQ(lc_all, STR_WITH_LEN("UTF8") ));
}
#endif /* USE_LOCALE */
if (utf8locale)
diff --git a/proto.h b/proto.h
index 9ce57d5..271107f 100644
--- a/proto.h
+++ b/proto.h
@@ -1109,24 +1109,36 @@ STATIC struct refcounted_he * S_refcounted_he_new_common(pTHX_ struct refcounted
/* PERL_CALLCONV HE* Perl_hv_store_ent(pTHX_ HV *hv, SV *key, SV *val, U32 hash); */
/* PERL_CALLCONV SV** Perl_hv_store_flags(pTHX_ HV *hv, const char *key, I32 klen, SV *val, U32 hash, int flags); */
PERL_CALLCONV void Perl_hv_undef(pTHX_ HV *hv);
-PERL_CALLCONV I32 Perl_ibcmp(const char* a, const char* b, I32 len)
+/* PERL_CALLCONV I32 ibcmp(pTHX_ const char* a, const char* b, I32 len)
+ __attribute__nonnull__(pTHX_1)
+ __attribute__nonnull__(pTHX_2); */
+
+PERL_CALLCONV I32 Perl_foldEQ(const char* a, const char* b, I32 len)
__attribute__pure__
__attribute__nonnull__(1)
__attribute__nonnull__(2);
-#define PERL_ARGS_ASSERT_IBCMP \
+#define PERL_ARGS_ASSERT_FOLDEQ \
assert(a); assert(b)
-PERL_CALLCONV I32 Perl_ibcmp_locale(const char* a, const char* b, I32 len)
+/* PERL_CALLCONV I32 ibcmp_locale(pTHX_ const char* a, const char* b, I32 len)
+ __attribute__nonnull__(pTHX_1)
+ __attribute__nonnull__(pTHX_2); */
+
+PERL_CALLCONV I32 Perl_foldEQ_locale(const char* a, const char* b, I32 len)
__attribute__pure__
__attribute__nonnull__(1)
__attribute__nonnull__(2);
-#define PERL_ARGS_ASSERT_IBCMP_LOCALE \
+#define PERL_ARGS_ASSERT_FOLDEQ_LOCALE \
assert(a); assert(b)
-PERL_CALLCONV I32 Perl_ibcmp_utf8(pTHX_ const char *s1, char **pe1, UV l1, bool u1, const char *s2, char **pe2, UV l2, bool u2)
+/* PERL_CALLCONV I32 ibcmp_utf8(pTHX_ const char *s1, char **pe1, UV l1, bool u1, const char *s2, char **pe2, UV l2, bool u2)
+ __attribute__nonnull__(pTHX_1)
+ __attribute__nonnull__(pTHX_5); */
+
+PERL_CALLCONV I32 Perl_foldEQ_utf8(pTHX_ const char *s1, char **pe1, UV l1, bool u1, const char *s2, char **pe2, UV l2, bool u2)
__attribute__nonnull__(pTHX_1)
__attribute__nonnull__(pTHX_5);
-#define PERL_ARGS_ASSERT_IBCMP_UTF8 \
+#define PERL_ARGS_ASSERT_FOLDEQ_UTF8 \
assert(s1); assert(s2)
#if defined(PERL_IN_DOIO_C) || defined(PERL_DECL_PROT)
diff --git a/utf8.c b/utf8.c
index d0be794..1f4192f 100644
--- a/utf8.c
+++ b/utf8.c
@@ -2502,10 +2502,10 @@ Perl_sv_uni_display(pTHX_ SV *dsv, SV *ssv, STRLEN pvlim, UV flags)
}
/*
-=for apidoc ibcmp_utf8
+=for apidoc foldEQ_utf8
Returns true if the leading portions of the strings s1 and s2 (either or both
-of which may be in UTF-8) differ case-insensitively; false otherwise.
+of which may be in UTF-8) are the same case-insensitively; false otherwise.
How far into the strings to compare is determined by other input parameters.
If u1 is true, the string s1 is assumed to be in UTF-8-encoded Unicode;
@@ -2531,7 +2531,7 @@ reached for a successful match. Also, if the fold of a character is multiple
characters, all of them must be matched (see tr21 reference below for
'folding').
-Upon a successful match (when the routine returns false), if pe1 is non-NULL,
+Upon a successful match, if pe1 is non-NULL,
it will be set to point to the beginning of the I<next> character of s1 beyond
what was matched. Correspondingly for pe2 and s2.
@@ -2541,7 +2541,7 @@ http://www.unicode.org/unicode/reports/tr21/ (Case Mappings).
=cut */
I32
-Perl_ibcmp_utf8(pTHX_ const char *s1, char **pe1, register UV l1, bool u1, const char *s2, char **pe2, register UV l2, bool u2)
+Perl_foldEQ_utf8(pTHX_ const char *s1, char **pe1, register UV l1, bool u1, const char *s2, char **pe2, register UV l2, bool u2)
{
dVAR;
register const U8 *p1 = (const U8*)s1; /* Point to current char */
@@ -2558,7 +2558,7 @@ Perl_ibcmp_utf8(pTHX_ const char *s1, char **pe1, register UV l1, bool u1, const
U8 natbuf[2]; /* Holds native 8-bit char converted to utf8;
these always fit in 2 bytes */
- PERL_ARGS_ASSERT_IBCMP_UTF8;
+ PERL_ARGS_ASSERT_FOLDEQ_UTF8;
if (pe1) {
e1 = *(U8**)pe1;
@@ -2634,7 +2634,7 @@ Perl_ibcmp_utf8(pTHX_ const char *s1, char **pe1, register UV l1, bool u1, const
character */
|| memNE((char*)f1, (char*)f2, fold_length))
{
- return 1; /* mismatch */
+ return 0; /* mismatch */
}
/* Here, they matched, advance past them */
@@ -2658,7 +2658,7 @@ Perl_ibcmp_utf8(pTHX_ const char *s1, char **pe1, register UV l1, bool u1, const
* character (which can happen when the fold of a character is more than one
* character). */
if (! ((g1 == 0 || p1 == g1) && (g2 == 0 || p2 == g2)) || n1 || n2) {
- return 1;
+ return 0;
}
/* Successful match. Set output pointers */
@@ -2668,7 +2668,7 @@ Perl_ibcmp_utf8(pTHX_ const char *s1, char **pe1, register UV l1, bool u1, const
if (pe2) {
*pe2 = (char*)p2;
}
- return 0;
+ return 1;
}
/*
diff --git a/utf8.h b/utf8.h
index b0cfedf..8e6d4e0 100644
--- a/utf8.h
+++ b/utf8.h
@@ -20,6 +20,16 @@
#define uvuni_to_utf8(d, uv) uvuni_to_utf8_flags(d, uv, 0)
#define is_utf8_string_loc(s, len, ep) is_utf8_string_loclen(s, len, ep, 0)
+/*
+=for apidoc ibcmp_utf8
+
+This is a synonym for (! foldEQ_utf8())
+
+=cut
+*/
+#define ibcmp_utf8(s1, pe1, l1, u1, s2, pe2, l2, u2) \
+ cBOOL(! foldEQ_utf8(s1, pe1, l1, u1, s2, pe2, l2, u2))
+
#ifdef EBCDIC
/* The equivalent of these macros but implementing UTF-EBCDIC
are in the following header file:
diff --git a/util.c b/util.c
index 75c4808..d5f0f1f 100644
--- a/util.c
+++ b/util.c
@@ -878,37 +878,58 @@ Perl_screaminstr(pTHX_ SV *bigstr, SV *littlestr, I32 start_shift, I32 end_shift
return NULL;
}
+/*
+=for apidoc foldEQ
+
+Returns true if the leading len bytes of the strings s1 and s2 are the same
+case-insensitively; false otherwise. Uppercase and lowercase ASCII range bytes
+match themselves and their opposite case counterparts. Non-cased and non-ASCII
+range bytes match only themselves.
+
+=cut
+*/
+
+
I32
-Perl_ibcmp(const char *s1, const char *s2, register I32 len)
+Perl_foldEQ(const char *s1, const char *s2, register I32 len)
{
register const U8 *a = (const U8 *)s1;
register const U8 *b = (const U8 *)s2;
- PERL_ARGS_ASSERT_IBCMP;
+ PERL_ARGS_ASSERT_FOLDEQ;
while (len--) {
if (*a != *b && *a != PL_fold[*b])
- return 1;
+ return 0;
a++,b++;
}
- return 0;
+ return 1;
}
+/*
+=for apidoc foldEQ_locale
+
+Returns true if the leading len bytes of the strings s1 and s2 are the same
+case-insensitively in the current locale; false otherwise.
+
+=cut
+*/
+
I32
-Perl_ibcmp_locale(const char *s1, const char *s2, register I32 len)
+Perl_foldEQ_locale(const char *s1, const char *s2, register I32 len)
{
dVAR;
register const U8 *a = (const U8 *)s1;
register const U8 *b = (const U8 *)s2;
- PERL_ARGS_ASSERT_IBCMP_LOCALE;
+ PERL_ARGS_ASSERT_FOLDEQ_LOCALE;
while (len--) {
if (*a != *b && *a != PL_fold_locale[*b])
- return 1;
+ return 0;
a++,b++;
}
- return 0;
+ return 1;
}
/* copy a string to a safe spot */
diff --git a/util.h b/util.h
index 6eab055..3981656 100644
--- a/util.h
+++ b/util.h
@@ -39,6 +39,21 @@
#endif /* VMS */
/*
+=for apidoc ibcmp
+
+This is a synonym for (! foldEQ())
+
+=for apidoc ibcmp_locale
+
+This is a synonym for (! foldEQ_locale())
+
+=cut
+*/
+#define ibcmp(s1, s2, len) cBOOL(! foldEQ(s1, s2, len))
+#define ibcmp_locale(s1, s2, len) cBOOL(! foldEQ_locale(s1, s2, len))
+
+
+/*
* Local variables:
* c-indentation-style: bsd
* c-basic-offset: 4
--
1.5.6.3
0006-Change-regexec.c-to-use-new-foldEQ-functions.patch
(text/x-patch, 3.8 KB)
From 8b4052b43173e58310c328e19e94791e2f768153 Mon Sep 17 00:00:00 2001
From: Karl Williamson <khw@khw-desktop.(none)>
Date: Sat, 5 Jun 2010 11:25:58 -0600
Subject: [PATCH] Change regexec.c to use new foldEQ functions
---
regexec.c | 24 ++++++++++++------------
1 files changed, 12 insertions(+), 12 deletions(-)
diff --git a/regexec.c b/regexec.c
index 307e274..5cdc3cc 100644
--- a/regexec.c
+++ b/regexec.c
@@ -1184,7 +1184,7 @@ uvc, charid, foldlen, foldbuf, uniflags) STMT_START { \
char *my_strend= (char *)strend; \
if ( (CoNd) \
&& (ln == len || \
- !ibcmp_utf8(s, &my_strend, 0, do_utf8, \
+ foldEQ_utf8(s, &my_strend, 0, do_utf8, \
m, NULL, ln, cBOOL(UTF))) \
&& (!reginfo || regtry(reginfo, &s)) ) \
goto got_it; \
@@ -1195,7 +1195,7 @@ uvc, charid, foldlen, foldbuf, uniflags) STMT_START { \
if ( f != c \
&& (f == c1 || f == c2) \
&& (ln == len || \
- !ibcmp_utf8(s, &my_strend, 0, do_utf8,\
+ foldEQ_utf8(s, &my_strend, 0, do_utf8,\
m, NULL, ln, cBOOL(UTF)))\
&& (!reginfo || regtry(reginfo, &s)) ) \
goto got_it; \
@@ -1207,9 +1207,9 @@ s += len
STMT_START { \
while (s <= e) { \
if ( (CoNd) \
- && (ln == 1 || !(OP(c) == EXACTF \
- ? ibcmp(s, m, ln) \
- : ibcmp_locale(s, m, ln))) \
+ && (ln == 1 || (OP(c) == EXACTF \
+ ? foldEQ(s, m, ln) \
+ : foldEQ_locale(s, m, ln))) \
&& (!reginfo || regtry(reginfo, &s)) ) \
goto got_it; \
s++; \
@@ -1416,7 +1416,7 @@ S_find_byclass(pTHX_ regexp * prog, const regnode *c, char *s,
* than just upper and lower: one needs to use
* the so-called folding case for case-insensitive
* matching (called "loose matching" in Unicode).
- * ibcmp_utf8() will do just that. */
+ * foldEQ_utf8() will do just that. */
if (do_utf8 || UTF) {
UV c, f;
@@ -3456,7 +3456,7 @@ S_regmatch(pTHX_ regmatch_info *reginfo, regnode *prog)
const char * const l = locinput;
char *e = PL_regeol;
- if (ibcmp_utf8(s, 0, ln, cBOOL(UTF),
+ if (! foldEQ_utf8(s, 0, ln, cBOOL(UTF),
l, &e, 0, do_utf8)) {
/* One more case for the sharp s:
* pack("U0U*", 0xDF) =~ /ss/i,
@@ -3487,8 +3487,8 @@ S_regmatch(pTHX_ regmatch_info *reginfo, regnode *prog)
if (PL_regeol - locinput < ln)
sayNO;
if (ln > 1 && (OP(scan) == EXACTF
- ? ibcmp(s, locinput, ln)
- : ibcmp_locale(s, locinput, ln)))
+ ? ! foldEQ(s, locinput, ln)
+ : ! foldEQ_locale(s, locinput, ln)))
sayNO;
locinput += ln;
nextchr = UCHARAT(locinput);
@@ -3866,8 +3866,8 @@ S_regmatch(pTHX_ regmatch_info *reginfo, regnode *prog)
if (ln > 1 && (type == REF
? memNE(s, locinput, ln)
: (type == REFF
- ? ibcmp(s, locinput, ln)
- : ibcmp_locale(s, locinput, ln))))
+ ? ! foldEQ(s, locinput, ln)
+ : ! foldEQ_locale(s, locinput, ln))))
sayNO;
locinput += ln;
nextchr = UCHARAT(locinput);
@@ -5363,7 +5363,7 @@ NULL
char *e = PL_regeol;
to_uni_fold(n, folded, &foldlen);
- if (ibcmp_utf8((const char*) folded, 0, foldlen, 1,
+ if (! foldEQ_utf8((const char*) folded, 0, foldlen, 1,
l, &e, 0, do_utf8)) {
sayNO;
}
--
1.5.6.3