Re: Designing Lisp from scratch

On 3/23/07, Jaap Weel <[email protected]> wrote:
> > Well, I never really stated that that's the way I'd prefer to do it. In
> > fact, I've never really understood the rationale of separating the lexer
> > from the parser.
>
> It's faster that way.

Not only that, from my own personal (though admittedly brief)
experience with scannerless parsing, it's a lot easier to write the
kind of parser most languages need if you have a lexer.  Creating a
scannerless parser for a language that is whitespace-insensitive (in
the sense most programming languages are) but which won't break up
identifiers (so that "ab" is distinct from "a b") can be somewhat
tricky and really requires adding lexing-like tricks to the grammar
specification.  Our common notions of syntax really have tokens as
their foundations, not characters.

-- 
Carl Eastlund