Re: various gelex, unicode

Thomas Beale wrote:
> TB> does it make it easier to use normal gelex/yacc classes if the 
> unicode characters are restricted to being inside String expressions, 
> i.e. inside "" markers?

I don't know what you mean by "normal" gelex/yacc classes.

Even if unicode characters are inside "", they are still a
sequence of bytes for the scanner. So you do whatever you
want with them. Treat them as bytes or build back the corresponding
unicode characters using the underlying encoding convention.
If you want to parse the content of your String expressions,
then you can either do it within the scanner using start
conditions (in that case you will probably use byte sequences),
or you can consider the whole string as a token and parse it
later on (in that case you will probably build back the unicode
characters from the bytes and insert them in the unicode string
token).

> <SS>{
>     <<EOF>> {
>         print ("Scanner T_SS%N")
>         last_token := T_SS
>         last_string_value := "Hello"
>         set_start_condition (INITIAL)
>     }
>     (.|\n) {
>         -- Ignore unmatched characters
>     }
> }
> 
> <<EOF>> {
>     print ("Scanner EOF%N")
>     terminate
> }
> 
> (.|\n) {
>     -- Ignore unmatched characters
> }
> 
> 
> TB> This is the bit that works for me - I didn't realise you could nest 
> patterns and states like that in the .l file...

I'm not sure I understand what you mean by "nesting patterns".

> All this stuff about default rule is explained in the doc:
> 
>     http://www.gobosoft.com/eiffel/gobo/gelex/matching_rules.html
> 
> TB> I did read that pretty carefully, but it doesn't explain the ability 
> to nest patterns...

In the code sample that you sent, the problem was about the
new-line character not being matched (and hence sent to the
output by default). That's what 'matching_rules.html'
explains.

Now, if by "nesting patterns" you mean using start conditions
to match tokens only when they appear in a given context,
then it is explained in 'start_conditions.html'. In particular
you have an example to match special characters (e.g. %N, %/code/)
within Eiffel manifest strings (inside ""), and only in manifest
strings.

What is probably missing in the doc (and I don't think it's
related to whatever "nested patterns" are) is that whenever
you reach <<EOF>> you will stay at <<EOF>> as long as
`terminate' is not called. Indeed, if there is no more character
to be read, you're stuck at <<EOF>>. Otherwise, as I already
explained in my previous message, if you want to rescan the
token (or part of it) in another start condition (or in the
same one), you will have to call `less'.

-- 
Eric Bezault
mailto:ericb-D6Qt/9opevxWk0Htik3J/[email protected]
http://www.gobosoft.com




------------------------ Yahoo! Groups Sponsor --------------------~--> 
You can search right from your browser? It's easy and it's free.  See how.
http://us.click.yahoo.com/_7bhrC/NGxNAA/yQLSAA/saFolB/TM
--------------------------------------------------------------------~-> 

To Post a message, send it to:   [email protected]
To Unsubscribe, send a blank message to: [email protected] 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/gobo-eiffel/

<*> To unsubscribe from this group, send an email to:
    [email protected]

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/