Re: effective tokenizer for wiki text

    >> So far, I think most of us have bent our input to look like email.  I
    >> think that would be a lot easier than writing and debugging a new
    >> tokenizer.

    Matt> Yes, I think it would be fine to start testing the filter that
    Matt> way, but I figured since the custom tokenizer had been suggested
    Matt> it was worth looking into what would be required and what the
    Matt> advantages might be.

Maybe subclass tokenizer.Tokenizer and override the tokenize method?

Skip