Re: (SPAM?) space-separated tokens (FAQ?)

Scott wrote:
> On Mon, Jun 27, 2005 at 10:19:22AM -0700, Ron Smith wrote:
> 
>>Scott, it is "bad form" to post code that you have not tested.  I copied 
>>the above verbatim into an editor and every line in your test data 
>>causes an "error" message.  Moreover, there is nothing in your grammar 
>>that handles "comments".
> 
> 
> Hmmm. I had tested the code before sending, and it worked fine. I know
> better than to post untested code.

Well, what I did was simple.  I cut and pasted your code into an editor, 
no more, no less.

I got this result:

Error: 1 pu 5f
Error: 1 pu 5tfo
Error: 1 mu 2n pu 5cni
Error: 1 pu 2n mu 5cni
Error: 1mu2npu5cni


> 
>>Second, it seems that what you want to parse is inherently ambiguous 
>>because there is no obvious difference between "n mu" and "nm" when you 
>>discount white space.
> 
> 
> Right....
> 
> 
>>Fundamentally you need to decide if white-space is part of your
>>grammar. 
> 
> 
> As is evident from my question, it is.

No, see that is the point, it is not "evident" as there is more than one 
way to do it, and one of those ways may not really require white space.

> notjustawholebunchofstuffglommedtogetherinonestream;guessIwaswrong.

It is interesting that the above is not ambiguous!  While it may be 
difficult to read, it is clearly not ambiguous.

Starting from left to right, you have:
"n"
but this is not a word.
Next it could be:
"no"
which is a word, so we have a "possibility" here.  But when you accept 
"no" as a word, the remainder of the sentence starting with "tjust..." 
cannot be completely and totally broken into words.  So in the end, a 
production is forced to accept "not" as the first word, simply because 
it is the only way to allow a production to find a second word.  And so 
on.  Parsing the above sentence does *not* require white space even 
though there are specific instances of ambiguity such as "no" vs. "not".

> 
> Thanks for the crumbs,
> Scott.
> 

If you want more than crumbs, post code that you can prove to yourself 
can be cut and pasted into an editor such as emacs and run without any 
modification.  It is hard enough to reverse engineer someone's intent in 
a piece of code when it works, let alone try to figure it out when it 
doesn't.