Example of ssax custom parser: openoffice document processing

Andy Wingo <[email protected]> Wed, 10 Nov 2004 15:30:24 +0200

Newsgroups	gmane.lisp.scheme.ssax-sxml
Message-ID	<20041110133024.GB16326@lark>

Hey all,

I just remembered something that might be of interest to the readers of
this list. I've been coauthoring a pair of books on dialects of the
Oshiwambo language of northern Namibia. The books were written with
OpenOffice so that people that come after me might have a hope of
modifying them.

Anyway, I wanted to generate glossaries of Oshiwambo words from the
documents. OOo documents were somehow stored as XML files, packed
together with images and zipped up. I wrote the attached program to run
through the documents, chunk them into words, and pick out the words
that did not appear in the dictionary. The word chunking is a bit of a
pain, because OO breaks words at tag boundaries in addition to the
normal whitespace, so taking a string-value isn't quite enough.

Anyway, it's attached. There are some guile-specific things there to
read from a pipe from the unzip program, but the rest should be fairly
portable.

Cheers,

Wingo.

sxw2words (text/plain, 2.2 KB)

#!/usr/bin/guile -s
!#
(use-modules (sxml ssax)
             (os process)
             (ice-9 rdelim)
             (srfi srfi-14))

(or (= (length (program-arguments)) 2)
    (begin
      (display "usage: sxw2words SXW-FILE\n" (current-error-port))
      (exit 1)))

(define sxw-file (cadr (program-arguments)))

(define (get-dict-words)
  (let ((port (open-input-file "/usr/share/dict/words")))
    (let lp ((words '()) (line (read-line port)))
      (if (eof-object? line)
          (sort! (reverse! words) string-ci<?)
          (lp (cons line words) (read-line port))))))

(define (uniq l)
  (let lp ((last-word "") (in l) (out '()))
    (cond ((null? in) (reverse! out))
          ((string-ci=? last-word (car in)) (lp last-word (cdr in) out))
          (else (lp (car in) (cdr in) (cons (car in) out))))))

(define trim-char-set (char-set-complement char-set:letter))
(define (get-sxw-words)
  ((ssax:make-parser
    NEW-LEVEL-SEED 
    (lambda (elem-gi attributes namespaces
                     expected-content seed)
      seed)
    
    FINISH-ELEMENT
    (lambda (elem-gi attributes namespaces parent-seed seed)
      seed)

    CHAR-DATA-HANDLER
    (lambda (string1 string2 seed)
      (let* ((strs (map
                    (lambda (x) (string-trim-both x trim-char-set))
                    (remove!
                     string-null? 
                     (append-map
                      (lambda (x) (string-split x #\space))
                      (string-split string1 #\newline)))))
             (seed (append! strs seed)))
        (if (string-null? string2) seed
            (cons string2 seed)))))
   (cdr (run-with-pipe ; "r" for read-only
         "r" "unzip" "-p" sxw-file "content.xml"))
   '()))

(let lp ((words (uniq (sort! (get-sxw-words) string-ci<?)))
         (dict-words (get-dict-words))
         (out '()))
  (cond
   ((null? words)
    (for-each (lambda (x) (display x) (newline)) (reverse! out)))
   ((string-ci=? (car words) (car dict-words))
    (lp (cdr words) (cdr dict-words) out))
   ((string-ci>? (car words) (car dict-words))
    (lp words (cdr dict-words) out))
   (else
    (lp (cdr words) dict-words (cons (car words) out)))))

;;; arch-tag: 6c2617d3-32a4-4a4d-8914-48c7ee1b5ad8