Re: Retrieving components of a regexp with ocamllex

You are trying to do too much in a lexer, and should read up on how to
structure a lexer+parser combination, or look at existing examples. What
lexers do is to split the input text in atomic units called "lexemes" (and
it outputs one token for each, except for whitespace/comment which might
just be skipped). A rule should parse one lexeme, not more. "PAIR" is a
lexeme. "(" is a lexeme. "`xyz`" is a lexeme. Your ("PAIR" [' ']* "("  ['
']* whatever) rule is not trying to recognize a lexeme.

On Sun, May 10, 2015 at 4:10 PM, [email protected] [ocaml_beginners]
<[email protected]> wrote:

>
>
>
>
>  Is it possible to read and « join » two components of a certain data
> structure in the lexical analysis, or must that be postponed to the parser
> phase ?
>    With the example file below, when typing parse_all "PAIR(xyz,765)";; I
> am expecting to obtain
> [PAIR("xyz", "765")]  but the lecture only sees a sequence of chars :
>
> # parse_all "PAIR(`xyz`,`765`)";;
> - : token list =
> [CHAR 'P'; CHAR 'A'; CHAR 'I'; CHAR 'R'; CHAR '('; CHAR '`'; CHAR 'x';
>  CHAR 'y'; CHAR 'z'; CHAR '`'; CHAR ','; CHAR '`'; CHAR '7'; CHAR '6';
>  CHAR '5'; CHAR '`'; CHAR ')']
>
>
>
> Contents of the mll file :
>
> {
>   type token =
>     | CHAR of char
>     | PAIR of string*string
> }
>
>
> let string_interior= ['a'-'z' 'A'-'Z' '0'-'9' '_' '+' '-' ':' '@']*
>
> rule parse_just_one_item = parse
>   | "PAIR" [' ']* "("  [' ']*
>     '`' (string_interior as s1) '`' [' ']*
>     '`' (string_interior as s2) '`' [' ']* ')'
>     {PAIR(s1,s2)}
>   | [' ' '\t' ]    (* eat up whitespace *)
>        { parse_just_one_item lexbuf }
>   | _ as c
>       {
>       CHAR c
>     }
>   | eof
>       { raise End_of_file }
>
> {
>
> let accu=ref([])
>
> let memorize x=(accu:=x::(!accu))
>
> let parse_all_silently s =
>           let _=(accu:=[]) in
>           try
>             let lexbuf = Lexing.from_string s in
>             while true do
>               let result = parse_just_one_item lexbuf in
>                 memorize result;
>             done
>           with End_of_file ->
>             ()
>
> let parse_all s=let _=parse_all_silently s in List.rev(!accu)
>
> }
>
>
>
>
> 
>