unit testing lexer

Hi,

I wrote a parser for a MiniJava language using ML-Lex and ML-Yacc, and 
I'm trying to create unit tests for the lexer part. The tokens that I'm 
reading have this form:

val it = TOKEN (T 8,(VOID',17,18)) : 
?.MiniJavaLex.UserDeclarations.lexresult

I'm having a hard time breaking this value apart and recreating the 
original token I created in the lexer rule, eg:

boolean        => (Tokens.BOOLEAN(yypos, yypos + 7));

Can someone give me a hint?
Thanks.


------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword