should we use PCRE2_MATCH_INVALID_UTF ?

Stephane Chazelas <[email protected]>
Newsgroups gmane.comp.shells.zsh.devel
Message-ID <[email protected]>
Zsh can generally cope with sequences of bytes that can't be
decoded into characters, treating each byte that can't be
decoded into character as if it was a character in most places.

[[ $'\x80' = ? ]] returns true in UTF-8 locales as bytes that
cannot be decoded into characters are still considered as
a character internally. With var=$'é\x80', $# is still 2, $var[2]
that $'\x80' byte.

For =~, that's down to the system's regexp library without
rematchpcre and PCRE2 (formerly PCRE) with rematchpcre.

On a GNU system (here Debian forky)

  [[ $'x\x80' =~ '^.*$' ]]

returns false as "." won't match that $'\x80' which is not a
character.

But [[ $'x\x80' =~ '^.' ]] does return true as x is matched even
if what follows is not valid text in the user's locale.

With rematchpcre enabled:

$ [[ $'x\x80' =~ '^.*$' ]]
zsh: pcre_exec() error [-22]
$ [[ $'x\x80' =~ '^.' ]]
zsh: pcre_exec() error [-22]

Neither match and in both cases a (not very helpful) error is
output.

PCRE2 have a:

> PCRE2_MATCH_INVALID_UTF  Enable support for matching invalid UTF

flag.

That would not make "." match that $'\x80' byte but would align
the behaviour with that of GNU's ERE's at least.

Would it be worth adding? Patch below.

diff --git a/Src/Modules/pcre.c b/Src/Modules/pcre.c
index 98b52f8de..409ee90c2 100644
--- a/Src/Modules/pcre.c
+++ b/Src/Modules/pcre.c
@@ -82,7 +82,7 @@ bin_pcre_compile(char *nam, char **args, Options ops, UNUSED(int func))
     if (OPT_ISSET(ops, 's')) pcre_opts |= PCRE2_DOTALL;
     
     if (zpcre_utf8_enabled())
-	pcre_opts |= PCRE2_UTF;
+	pcre_opts |= PCRE2_UTF|PCRE2_MATCH_INVALID_UTF;
 
     if (pcre_pattern)
 	pcre2_code_free(pcre_pattern);
@@ -431,7 +431,7 @@ cond_pcre_match(char **a, int id)
     int return_value = 0;
 
     if (zpcre_utf8_enabled())
-	pcre_opts |= PCRE2_UTF;
+	pcre_opts |= PCRE2_UTF|PCRE2_MATCH_INVALID_UTF;
     if (isset(REMATCHPCRE) && !isset(CASEMATCH))
 	pcre_opts |= PCRE2_CASELESS;
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.