Re: Combining different rules in ocamllex

Hi,

You have to call the lexer recursively in the quoted_string rule, if you
want the whole string to be consumed as a single token:

  | _ as c {add_to_string c; quoted_string lexbuf}

On Tue, Jun 23, 2015 at 9:56 AM, [email protected] [ocaml_beginners]
<[email protected]> wrote:

>
>
> When I compile the basic_lexer.mll file below, and type
> Basic_lexer.parse_all   "abc\"def\"ghi";; in the interpreter,
> I am epecting to get
> [Basic_lexer.CHAR 'a'; Basic_lexer.CHAR 'b'; Basic_lexer.CHAR 'c';
>  Basic_lexer.STRING "def";
>  Basic_lexer.CHAR 'g'; Basic_lexer.CHAR 'h'; Basic_lexer.CHAR 'i']
>
>  but instead I get
>
> [Basic_lexer.CHAR 'a'; Basic_lexer.CHAR 'b'; Basic_lexer.CHAR 'c';
>  Basic_lexer.CHAR 'e'; Basic_lexer.CHAR 'f'; Basic_lexer.CHAR 'h';
>  Basic_lexer.CHAR 'i']
>
>
> What did I do wrong ?
>
>
> Contents of basic_lexer.mll file :
>
>
> {
>
> type lexeme=
>     CHAR of char
>    |STRING of string;;
>
> let list_accu=ref[]
> let string_accu=ref""
>
> let add_to_list x=(list_accu:=x::(!list_accu))
> let add_to_string c=(string_accu:=(!string_accu)^(String.make 1 c))
> let finish_quote ()=(add_to_list (STRING(!string_accu)); string_accu:="";
> );;
>
> }
>
>
> rule usual = parse
>   | '"'  { quoted_string lexbuf;}
>   | _ as c {add_to_list (CHAR c);}
>   | eof { raise End_of_file }
> and quoted_string=parse
>   | "\\\"" {add_to_string '\''}
>   | "\\\\" {add_to_string '\\'}
>   | '"' {finish_quote(); usual lexbuf}
>   | _ as c {add_to_string c}
>   |eof {raise End_of_file}
>
> {
>
>
>
> let parse_all s =
>           let _=(list_accu:=[];string_accu:="") in
>           try
>             let lexbuf = Lexing.from_string s in
>             let _=(while true do
>              usual lexbuf
>             done) in
>             []
>           with End_of_file ->
>             List.rev(!list_accu)
>
>
>
> }
>
>
>
>
>
>
>
> 
>