str:tokenize() bug

Christian Höltje <[email protected]> Thu, 02 Nov 2006 12:14:48 -0500
Newsgroups gmane.text.xml.xslt.extensions
Message-ID <[email protected]>
This is a bug in either the definition of str:tokenize() or the
implementations I have seen.

Given this call:
  str:tokenize('a-b--d','-')

I get back a list of nodes containing a,b, and d.  The empty 'c' node is
missing.  The description implies that I should have gotten four nodes,
but all the implementations I played with drop the empty node.

The description seems to imply that the empty node should be there
but implementor seem to think otherwise. :-)

This makes using tokenize() rather difficult for parsing data.  At the
least, adding a 'strict' flag as an option would be useful.

Ciao!