announcing synfl, the next generation of fontlock

That should get your attention!

Returning to reason and modesty, this is a rather primitive module
based on an LR parser engine and an SLR parser generator.  The parser
wants tuning and optimization, but is perfectly servicable for
experimentation.  The parser generator probably should be extended to
a LALR parser generator for general use, but that should be reasonably
straightforward (though a little time-consuming) for anybody with a
copy of The Dragon Book to hand.  (Aho, Sethi, and Ullman, _Compilers:
..._)

I probably shouldn't play with it much for the next few weeks, so, now
that it's basically working, here it is:

cvs -d :pserver:[email protected]:/pack/xemacscvs \
    checkout -d synfl XEmacs/packages/unsupported/stephen/synfl

http://turnbull.sk.tsukuba.ac.jp/Tools/XEmacs/synfl.tar.gz

To use it, you should just byte-compile everything and load up.

The Makefile is untested.  It expects to find XEmacs.rules in the
grandparent directory, as usual.  "distribution" in package-info.in is
set to "unsupported", so "make install" is not a good idea; I don't
think it has a chance of working at all.

Features:

There's a built-in trace/debug facility, so the package is of
educational interest.  You can watch the LR parser, and the SLR
parser-generator, as they do their things.  I thought it was pretty
cool, but maybe that's just me.

The token stream implementation used to catch the results of lexical
analysis is probably OK.  The "interface" used to mesh the token
stream, the grammar, and the parser output is an abortion; cleaning
that up is the next task, so my apologies in advance to anybody who
spontaneously combusts from looking at it.  It just didn't want to be
abstracted yet, but I tried to force it....

The font-lock capability looks to be about twice as fast as (uncached,
nonlazy) traditional font-lock already, but all of that is due to not
using complex regular expressions; the syntax analyzer is currently
just along for the ride, wasting a small amount of time in the
interest of making sure it's reasonably efficient.  It is.  The C
grammar used so far is trivial, just a stream of tokens.

Profile data for fontlock+cc-mode, a lock-c grammar, and a lex'n'lock
implementation applied to src/emacs.c are in doc/profile_emacs.c_*.
Tick counts are purely notional, as the XEmacs used for profiling is
fully pessimized.  (Sorry, Hrvoje. :-)  I will say that since so much
of the time for font-lock is spent in regexp matching (all in C), and
so little is for synfl, I'll bet that the advantage for synfl
_increases_ with optimization of XEmacs.

It probably wouldn't be too difficult to arrange to call the parser
recursively on a different grammar (for the same kinds of files that
mmm-mode is used for).  Haven't done that, but I have given a little
thought to avoiding introducing code that would prevent it.

The lex'n'lock strategy is of independent interest.  What's giving the
speed for both the synfl locker and the lex'n'locker is to lex the
tokens and then figure out what the keywords are by looking up the
tokens in a hash table---this can be done without the parser stuff,
and it's at least 5 times as fast as traditional font-lock.  I am not
yet willing to conjecture whether either strategy will handle bogus
syntax or re-font-lock on-the-fly and still maintain efficiency.

Documentation, outside of the sources, is non-existent, of course.
They are pretty extensively commented, cross-referenced to the Dragon
Book, and the docstrings are fairly good, I hope.  You know where to
find me if you have questions.  :-)

-- 
Institute of Policy and Planning Sciences     http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;
              ask what your business can "do for" free software.