Re: gelex Q

Thomas Beale wrote:
> In our gelex/geyacc archetype parser I have a state during which text in 
> a sub-syntax is consumed. The lexer is in that state when the EOF is 
> reached. The problem is that when EOF is reached, two things need to be 
> done:
> * set last_token to a meaningful value, and put the text scanned so far 
> into last_string
> * call terminate, which I think sets last_token to 0
> 
> But I can't do these both in the same fragment of code.
> 
> I tried doing it like this:
> 
> <ss><<EOF>> {
>        -- unread_character(text.item(1))  -- would this work?

I don't think so. If instead of <<EOF>> it was a "real"
token, you could do:

          less (text_count)

But it will not work with <<EOF>>.

>        last_token := xxxx
>        last_string := ....
>        set_start_condition(INITIAL)
>     }
> 
> ....various other rules....
> 
> <<EOF>> terminate
> 
> the problem is that I don't think the final EOF is being reached, and so 
> the token reading is not terminating properly. I also tried putting a 
> unread_character(text.item(1)) line (commented above).

I tried the following:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
%{
class TEST_SCANNER

inherit

	YY_COMPRESSED_SCANNER_SKELETON

	TEST_TOKENS

%}

%x SS
%option outfile="test_scanner.e"

%%

gobo {
	print ("Scanner T_GOBO%N")
	last_token := T_GOBO
	last_string_value := text
	set_start_condition (SS)
}

<SS><<EOF>> {
		print ("Scanner T_SS%N")
		last_token := T_SS
		last_string_value := "Hello"
		set_start_condition (INITIAL)
}

<<EOF>> {
	print ("Scanner EOF%N")
	terminate
}

%%

end
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
%{
class TEST_PARSER

inherit

	YY_PARSER_SKELETON
		rename
			make as make_parser_skeleton
		end

	TEST_SCANNER
		rename
			make as make_scanner
		end

	KL_SHARED_ARGUMENTS

create

	make, execute

%}

%token <STRING> T_GOBO
%token <STRING> T_SS
%type <STRING> Gobo Ss
%start Gobo_ss

%%

Gobo_ss: Gobo Ss
	;

Gobo: T_GOBO
		{ print ("Parser " + $1 + "#%N") }
	;

Ss: T_SS
		{ print ("Parser " + $1 + "#%N") }
	;
%%

feature {NONE} -- Initialization

	make is
			-- Create a new parser.
		do
			make_scanner
			make_parser_skeleton
		end

	execute is
			-- Parse file passed as argument.
		local
			a_filename: STRING
			a_file: KL_TEXT_INPUT_FILE
			n: INTEGER
		do
			make
			n := Arguments.argument_count
			if n /= 1 then
				print ("usage: test_parser filename%N")
			else
				a_filename := Arguments.argument (1)
				create a_file.make (a_filename)
				a_file.open_read
				if a_file.is_open_read then
					reset
					set_input_buffer (new_file_buffer (a_file))
					parse
					a_file.close
					print ("End of program%N")
				else
					print ("test_parser: cannot read %'")
					print (a_filename)
					print ("%'%N")
				end
			end
		end

end
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I passed as input a file containing just "gobo" and I got
the expected result:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Scanner T_GOBO
Parser gobo#
Scanner T_SS
Parser Hello#
Scanner EOF
End of program
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

> The symptom I get is that although the parser does in fact work 
> properly, it is sending a newline (which I think is probably the last 
> newline in the parsed file) to the console (i.e. to stdout). I don't 
> know why that is happening, but in any case, I would like to know the 
> right way to terminate when you still have work to do by the time you 
> hit EOF. I had a previous scheme of adding a final line pattern to the 
> input text and catching that, but it seems clumsy, and forces a copy of 
> the complete input text.

I think that the problem with the new-line is that you don't
have a rule to handle it, and the default is to send it to
stdout. Indeed, if I add a new-line at the end of my input
file, I get:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Scanner T_GOBO
Parser gobo#

Scanner T_SS
Parser Hello#
Scanner EOF
End of program
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In order to disable this default behavior and get a warning by gelex
instead, you need to add the following line in your .l file:

%option nodefault

gelex will then complain with the following message:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Warning, "test_scanner.l": nodefault or -s option given but default rule 
can be matched
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In order to address this problem, you need to write catch-all rules.
The .l file may look like that:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
%{
class TEST_SCANNER

inherit

	YY_COMPRESSED_SCANNER_SKELETON

	TEST_TOKENS

%}

%x SS
%option nodefault
%option outfile="test_scanner.e"

%%

gobo {
	print ("Scanner T_GOBO%N")
	last_token := T_GOBO
	last_string_value := text
	set_start_condition (SS)
}

<SS>{
	<<EOF>> {
		print ("Scanner T_SS%N")
		last_token := T_SS
		last_string_value := "Hello"
		set_start_condition (INITIAL)
	}
	(.|\n) {
		-- Ignore unmatched characters
	}
}

<<EOF>> {
	print ("Scanner EOF%N")
	terminate
}

(.|\n) {
	-- Ignore unmatched characters
}

%%

end
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

and the output, even with a new-line at the end of the file,
now looks again like that:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Scanner T_GOBO
Parser gobo#
Scanner T_SS
Parser Hello#
Scanner EOF
End of program
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

All this stuff about default rule is explained in the doc:

    http://www.gobosoft.com/eiffel/gobo/gelex/matching_rules.html

-- 
Eric Bezault
mailto:ericb-D6Qt/9opevxWk0Htik3J/[email protected]
http://www.gobosoft.com



------------------------ Yahoo! Groups Sponsor --------------------~--> 
Home is just a click away.  Make Yahoo! your home page now.
http://us.click.yahoo.com/DHchtC/3FxNAA/yQLSAA/saFolB/TM
--------------------------------------------------------------------~-> 

To Post a message, send it to:   [email protected]
To Unsubscribe, send a blank message to: gobo-eiffel-unsubscribe-B11MqFFcr06IwRZHo2/[email protected] 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/gobo-eiffel/

<*> To unsubscribe from this group, send an email to:
    [email protected]

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/