rfc: curvbol - a lisp language for processing text

Marco Baringer <[email protected]> Sat, 05 Jun 2004 10:48:44 +0200
Newsgroups gmane.lisp.clump
Message-ID <[email protected]>
i've been doing a lot of processing of html and good old plain text
files, i originally used franz's html parser but at the end of the day
parsing the resulting lhtml was just as complicated as the mess of
regexps, read-lines and buffers it was supposed to replace. so i did
what every good lisper does and i invented a little language for
working with text. i'd like comments, ideas and suggestions regarding
the following mini language from people who use lisp to work with
text, or use lisp but not when they need to work with text.

curvbol has some things in common with non-regular expressions,
recursive decent parsers, and imperative style lisp code. the name is
a tribute to SNOBOL (not the curvbol has much in common with SNOBOL),
the idea of developing a text manipulation language, and most of the
core ops, come from the snowball text manipulation language. a curvbol
program consists of the curvbol code, a program counter, a
continuation stack, and an enviroment which contains:

1) a subject - the text we're analyzing

2) a cursor - the position within the text we're currently looking at
   (like emacs' point)

3) a cursor shadow - the place the cursor was before the last form
   executed, not stricly neccessary, but very usefull.

4) a limit - the point, in the subject, beyond which the cursor can
   not go. usually equal to the length of the subject.

5) a direction - whether we're processing the text from the start to
   the end or from the end to the start.

6) a variable environment - an object the program can use to store
   values.

a curvbol program returns two values, the value returned by the code
and the environment (an alist) generated during the execution of the
program. curvbol consists of a, rather large, set of core operators
and, a, at the moment, rather small, convience library.

the core operators:

(match thing) - returns THING if the section of text in the subject
starting at the cursor is string= to THING otherwise NIL, moves the
cursor past the matched text. I've considered allowing THING to be a
regexp and using cl-ppcre, but I don't know what to do with the
registers THING may bind, and i haven't really felt the need to match
regexps.

(seq &rest seq) - attempts to match each form of SEQ in turn, as soon
as one returns NIL the enviroment is restored to its value upon
entering the SEQ and NIL is returned. If all forms return a true value
the vale of the last form is returned.

(loop (min &optional max greedyp) &rest seq) - Returns T if SEQ
matches at least min times and at most mak times. MAX may be NIL to
specify that there is no limit on the number of matches. Backtracks if
neccessary to ensure a global match. Unlike the repetition ops in
other languages LOOP does _not_ move the cursor, the user has to be
carefull to avoid an infinite loop.

(or &rest clauses) - Returns the first clause which returns a true
value. Backtracks if neccessary to ensure a global match.

(and &rest clauses) - Just like the SEQ op, except the environment is
saved/restored around each clause (except the last one).

(reverse &rest implicit-seq) - Within the forms in IMPLICIT-SEQ
attempt to match _before_ the cursor and, when a match is successful,
move the cursor tomards the start of the subject.

(backwards &rest implicit-seq) - Swap cursor and limit, evaluate
IMPLICIT-SEQ in a reverse. Unitl I can think of a better way to handle
it the BACKWARDS op can not move the cursor or the limit.

(goto &rest implicit-seq) - Evaluate IMPLICIT-SEQ until it returns a
true value, force the cursor forward (which may be "leftward" if we're
in a reverse) after each failure of SEQ. Leaves the cursor where it
was before SEQ returned a true value.

(gopast &rest implicit-seq) - Just like GOTO but the cursor is left
where it was after SEQ returned a true value. 

Note: I'm not sure on the usefullness of the GOTO/GOPAST combo as it
currently stands, maybe it should be: (gopast SEQ) is a non
backtracking, non greedy, cursor incrementing version of (loop (0 nil
nil) SEQ) and GOTO is a non backtracking, non greedy cursor
incermenting version of (loop (0 nil nil) (not SEQ)). Ie GOPAST moves
forward as long as SEQ matches and GOTO moves forward until SEQ
matches.

(not &rest implicit-seq) - If IMPLICIT-SEQ returns NIL then return T,
otherwise the environemnt is restored and return NIL.

(try &rest implicit-seq) - If IMPLICIT-SEQ returns NIL then TRY
returns NIL, otherwise the envorment is restored and the value
returned by SEQ is returned.

(test &rest implicit-seq) - Returns whatever IMPLICIT-SEQ returns,
always restores the environment.

(hop amount) - Moves the cursor forward by AMOUNT chars. If this would
move the cursor beyond the limit then the cursor is not changed and
NIL is returned, otherwise AMOUNT is returned.

(let var value) - binds VAR to whetever VALUE returns. Returns the
value of VALUE.

(make-splice start end) - Returns the subseq of the subject delimeted
by START and END. END may be less than START.

Real soon now it will be possible to use the SET op to change the
value of splices and madify the subject, i haven't needed it yet
though.

the convience library is just a collection of cruvbol functions and
macros which is growing on a case by case basis, so i'm not gonna
go into it.

-- 
-Marco
Ring the bells that still can ring.
Forget your perfect offering.
There is a crack in everything.
That's how the light gets in.
     -Leonard Cohen