Re: What should /\08/ match and do?

On 1 June 2010 21:11, karl williamson <[email protected]> wrote:
> demerphq wrote:
>>
>> On 1 June 2010 20:07, karl williamson <[email protected]> wrote:
>>>
>>> demerphq wrote:
>>>>
>>>> On 1 June 2010 19:05, Eric Brine <[email protected]> wrote:
>>>>>
>>>>> On Tue, Jun 1, 2010 at 6:45 AM, demerphq <[email protected]> wrote:
>>>>>>
>>>>>> IMO these days octal, and any other non parenthesized code-like
>>>>>> escapes should be deprecated.
>>>>>
>>>>> A stylistic warning in ambiguous cases (like Abigail's "/\08/
>>>>> interpreted
>>>>> as
>>>>> /\x{00}8/" warning) sounds like a good idea, but not a *deprecation*
>>>>> warning. That would mean we're planning on removing octal support from
>>>>> Perl,
>>>>> and I haven't heard any reasons for doing that. Does supporting octal
>>>>> escapes cause maintenance problems?
>>>>
>>>> Any non-parenthesized escape structure using numeric values is a
>>>> maintenance problem when seen from the point of view of modifying or
>>>> joining snippets of a regex together.
>>>>
>>>> cheers,
>>>> Yves
>>>>
>>>>
>>> I thought this was all settled until this heated up again, so I was about
>>> to
>>> submit patches to change \08, et. al., including the addition of \o{...}
>>>  for an arbitrary length octal character constant.  (I'll wait for things
>>> to
>>> settle down.)
>>
>> When i said the above i meant for the USER of perl, not us perl
>> authors. Yes there is a modest maintenance burden for US, but it pales
>> in comparison to that affecting the user.
>>
>>> What I find a maintenance problem is more that there are three copies of
>>> these, each with slightly different behavior, and have diverged over the
>>> years.  My understanding is that originally the regex and qq were handled
>>> in
>>> common code.  But, for the most part they've been split apart.  I can't
>>> remember why; Yves has said why in the past, and it's convincing.
>>
>> Heh thanks. Some of the reasons:
>>
>> \1 is a backreference in a regex, in a string it is always the same as
>> \001. This holds for \1 to \7 and for \10 (\8 and \9 dont count as
>> octal).
>
> In a replacement, \1 is a backreference, and that is handled in toke.c

Ah, well, I admit I got a bit muddled with the details... Ive had a
pounding headache all day and it kinda shows. What i was trying to
refer to was something like this:

 if ($string=~/(?:$pat)\21/) { ... }

exactly what the \21 refers to depends on the value of $pat. Example:

$ perl -wle'$_="1234567890" x 2; while ($_) { $l= length($_);
$pat="(\\d)" x $l; $s= $_ . substr($_,-1,1); $pat.="\\$l"; print $s,
$s=~/$pat/ ? " matched " : " did not match ",$pat; chop}'
123456789012345678900 matched
(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)\20
12345678901234567899 matched
(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)\19
1234567890123456788 matched
(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)\18
123456789012345677 matched
(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)\17
12345678901234566 matched
(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)\16
1234567890123455 matched
(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)\15
123456789012344 matched
(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)\14
12345678901233 matched (\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)\13
1234567890122 matched (\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)\12
123456789011 matched (\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)\11
12345678900 matched (\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)\10
1234567899 matched (\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)\9
123456788 matched (\d)(\d)(\d)(\d)(\d)(\d)(\d)(\d)\8
12345677 matched (\d)(\d)(\d)(\d)(\d)(\d)(\d)\7
1234566 matched (\d)(\d)(\d)(\d)(\d)(\d)\6
123455 matched (\d)(\d)(\d)(\d)(\d)\5
12344 matched (\d)(\d)(\d)(\d)\4
1233 matched (\d)(\d)(\d)\3
122 matched (\d)(\d)\2
11 matched (\d)\1

>> Another reason is that we want the octal for "." to match "." and not
>> like the "match anything but a newline" type behaviour it normally
>> has. So we cannot convert it to a literal during compilation like we
>> would in a string.
>>
>
> That's what I now remember as convincing.  But this gives me an idea. We
> could do something like we did for \N{...} which was to move it's handling
> back to the parser, which created an intermediate form for regcomp.
>  Couldn't we move the non-ambiguous parts back to the parser, which would
> convert them all to \x{...} ?

Couldnt we do the same trick with converting it to one of the new
\o{...} style escapes?

That would probably be less confusing for someone debugging the pattern....

>Then only the problematic parts, which
> hopefully is just the octal, would have to be maintained separately.

I think it only applies to octal since it is only in octal that there
is context sensitive grammar involved.

> The one problem I can think of for this is the stringification of the regex
> wouldn't be what was input, and that is confusing to the user.  This is a
> problem with the intermediate form of \N{...} now.  But I imagine it is
> solvable.

Its not clear to me really how. Any give \N{} name could be redefined
at compile time to something really bizarre, and im not sure it really
is a benefit to the user to see the exact \N{...} construct. We should
just document what the debug mode for regexes will do in this case and
not worry about it.

Although, hmm. I guess we could do something horribly ugly like

\N{U+1234.1234.1234=WHATEVERNAMETHEYGAVEUS}

but even then, its hard to see how we would explain that its really
the WHATEVERNAMETHEYGAVEUS defined in file blah blah blah on line blah
blah blah... Which is what you would need for the WHATEVERNAME stuff
to really be useful. Although I suppose even just the name would at
least be helpful for debugging.

> In an earlier thread, I mentioned that [\8] is the same thing as [8\000],
> and that clearly is a bug, and the stringification doesn't show the NUL that
> gets generated.  I now believe the reason is that on successful compilation,
> the stringification is as close as possible to the input string.

It *is* the input string, as perceived by the regex engine. The only
thing that is done to it is that when we produce diagnostics we use
"%" as the escape character, except its not consistent, as when we
stringify EXACT like nodes we use "normal" style escaping, which is
very confusing. This needs to be fixed:

$ perl -Mre=debug -e'$s="ba\x{df}"; utf8::upgrade($s); $s=~/\x{df}/i'
Compiling REx "\x{df}"
Final program:
   1: EXACTF <\337> (3)
   3: END (0)
stclass EXACTF <\337> minlen 1
Matching REx "\x{df}" against "ba%337"
UTF-8 string...
Matching stclass EXACTF <\337> against "ba%337" (4 chars)
   2 <ba> <%337>             |  1:EXACTF <\337>(3)
   4 <ba%337> <>             |  3:END(0)
Match successful!
Freeing REx: "\x{df}"


> That could be extended to avoid the intermediate forms.

Hrm... Im not sure about this one...

Yves


-- 
perl -Mre=debug -e "/just|another|perl|hacker/"