Re: Documentation tokenization

Jeff Rafter wrote:

>>I have been testing my theories against
>>XMLDistilled (which is extremely useful) and they are not lining up.
> 
> 
> I tested against Jing and it seems to pass all documentation that I can
> throw at it-- even when it seems it should be failing. I suppose I am a
> little confused as to whether or not:
> 
> ## Documentation # comment
> 
> is allowable. From my reading it is not-- but my reading could be wrong. The
> grammar could be saying that when the # is encountered it is the start of a
> separator (i.e., a comment) and is therefore the end of the documentation.

I don't see any way a comment could jump into the middle of 
documentation, which is a named terminal, which is a token.

> For that matter, is:
> 
> # Comment # Comment
> 
> allowable? Again  it seems to me that A.2 says it is not. But Jing and
> XMLDistilled accept it.

Perhaps you should say why you think A.2 says it is not. Comments start 
with # and continue to the end of the line. If a succeeding # is on the 
same line (and not immediately following) why would it be treated specially?

> ...I think that this is not accurate. I misread the " " restOfLine option. I
> believe it was trying to show that in the case of
> 
> ## Some text
> 
> That the resulting documentation would be "Some Text" not " Some Text". A
> single initial space in documentation is discarded.

I think you're right about this.

Bob Foster