Re: [cowiki-dev] The new coWiki parser

--------------040802030402050803080000
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

John (yt) Hogenmiller wrote:

>As far as the two things that you seem most concerned with (WYSIWYG
>editing and embeded image management), I don't think they are the main
>issue quite yet.  
>
I'm interested-- What would you put first?  :-) 

Just to recall, the problems with the parser came up because we wanted 
to enhance it to improve formatting capability.  The Wiki way has been 
to do it with markup because that's the "easiest" thing for a programmer 
to implement (the XP way).  Problem is that it's hard on the user (I 
don't understand the new table formatting thing at *all*) and it's 
gotten hard on the programmer too!  :-) 

Hence my proposal as a "better" solution (albeit not the only one!)

>However, I have been focusing in two directions for
>WYSIWYG.  One started out seeming simple... integrate the LGPL
>licensed fckEditor into cowiki.  The entire markup language is defined
>in xml files and it should work.  The problem is that the xml file
>names element names and the converter classes do the work of
>bracketing and attributing that name into a tag.  Changing this
>behavior became much more work than I had time for, but it still seems
>possible with more study.   
>
There are several OSS editors like this now, and I have seen wiki's that 
used them.  Most seem aimed at generating HTML code, which isn't quite 
what we want but might be usable.  I haven't really looked hard at this 
particular one because it's difficult for me to take code with a 
"hidden" profanity in the name seriously.  :-\  But this line of 
investigation seems promising.

>The second method is not WYSIWYG, but
>rather a preview  function utilizing AJAX.  Basically, there would be
>an add-on script added in the htdocs directory that would take the raw
>coWiki text, run it through the various parsers and then return the
>resulting html code into a new layer (which curiously enough,
>corresponds with the textarea).  
>
This is tempting as a preview tool, but something still has to parse and 
render the result.  At this point the problem is that we can't build an 
adequate parser and renderer based on wiki-text.  Personally, I believe 
the fundamental problem is that wiki-text contains contextual 
assumptions that break down in even moderately complex formatting scenarios.

I'm not a language guru but from some recent reading our current parser 
really needs to have

    * A tokenizer to split apart units of text and standardize
      representation of hot codes
    * A syntax recognizer to understand the context of token and
      recognize the difference context makes in the interpretation of
      tokens (like "*" meaning a bullet point, and "*" meaning the start
      or end of bold.)
    * A language recognizer to build the token stream into a parse tree

Once we have a parse tree, it's "easy" to convert it to XML.  I don't 
think I have the names of the pieces right, but it gives you an idea of 
what I see.

>The main issue here is the fact that
>the final parser that generates the html output accepts a node id as
>input, and pulls that information from the database.  The fix to that
>is adding some code to the parser class that accepts text as an
>argument and bypasses the database pulling entirely before returning
>the html output.  I think I can implement the AJAX preview solution
>easily enough, but the WYSIWYG editor is a much bigger task.
>  
>
IMAO, AJAX has some uses (like validating links or providing a 
contextual list of available choices) but I don't see that rendering is 
necessarily one of them, unless formatting keystrokes make a round trip 
to the server. And that makes for a tremendous bandwidth and latency 
jump for the user...

AJAX could also be used to implement a behind-the-scenes uplink of local 
information (like images and documents) that needs to be placed on the 
server to display properly.

AJAX would make a killer progressive search feature, but the current 
search has been pretty good for me so far.


--------------040802030402050803080000
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
  <title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
John (yt) Hogenmiller wrote:
<blockquote cite="[email protected]"
 type="cite">
  <pre wrap="">As far as the two things that you seem most concerned with (WYSIWYG
editing and embeded image management), I don't think they are the main
issue quite yet.  </pre>
</blockquote>
I'm interested-- What would you put first?&nbsp; :-)&nbsp; <br>
<br>
Just to recall, the problems with the parser came up because we wanted
to enhance it to improve formatting capability.&nbsp; The Wiki way has been
to do it with markup because that's the "easiest" thing for a
programmer to implement (the XP way).&nbsp; Problem is that it's hard on the
user (I don't understand the new table formatting thing at *all*) and
it's gotten hard on the programmer too!&nbsp; :-)&nbsp; <br>
<br>
Hence my proposal as a "better" solution (albeit not the only one!)<br>
<blockquote cite="[email protected]"
 type="cite">
  <pre wrap="">However, I have been focusing in two directions for
WYSIWYG.  One started out seeming simple... integrate the LGPL
licensed fckEditor into cowiki.  The entire markup language is defined
in xml files and it should work.  The problem is that the xml file
names element names and the converter classes do the work of
bracketing and attributing that name into a tag.  Changing this
behavior became much more work than I had time for, but it still seems
possible with more study.   </pre>
</blockquote>
There are several OSS editors like this now, and I have seen wiki's
that used them.&nbsp; Most seem aimed at generating HTML code, which isn't
quite what we want but might be usable.&nbsp; I haven't really looked hard
at this particular one because it's difficult for me to take code with
a "hidden" profanity in the name seriously.&nbsp; :-\&nbsp; But this line of
investigation seems promising.<br>
<blockquote cite="[email protected]"
 type="cite">
  <pre wrap="">The second method is not WYSIWYG, but
rather a preview  function utilizing AJAX.  Basically, there would be
an add-on script added in the htdocs directory that would take the raw
coWiki text, run it through the various parsers and then return the
resulting html code into a new layer (which curiously enough,
corresponds with the textarea).  </pre>
</blockquote>
This is tempting as a preview tool, but something still has to parse
and render the result.&nbsp; At this point the problem is that we can't
build an adequate parser and renderer based on wiki-text.&nbsp; Personally,
I believe the fundamental problem is that wiki-text contains contextual
assumptions that break down in even moderately complex formatting
scenarios.<br>
<br>
I'm not a language guru but from some recent reading our current parser
really needs to have<br>
<ul>
  <li>A tokenizer to split apart units of text and standardize
representation of hot codes</li>
  <li>A syntax recognizer to understand the context of token and
recognize the difference context makes in the interpretation of tokens
(like "*" meaning a bullet point, and "*" meaning the start or end of
bold.)</li>
  <li>A language recognizer to build the token stream into a parse tree</li>
</ul>
Once we have a parse tree, it's "easy" to convert it to XML.&nbsp; I don't
think I have the names of the pieces right, but it gives you an idea of
what I see.<br>
<blockquote cite="[email protected]"
 type="cite">
  <pre wrap="">The main issue here is the fact that
the final parser that generates the html output accepts a node id as
input, and pulls that information from the database.  The fix to that
is adding some code to the parser class that accepts text as an
argument and bypasses the database pulling entirely before returning
the html output.  I think I can implement the AJAX preview solution
easily enough, but the WYSIWYG editor is a much bigger task.
  </pre>
</blockquote>
IMAO, AJAX has some uses (like validating links or providing a
contextual list of available choices) but I don't see that rendering is
necessarily one of them, unless formatting keystrokes make a round trip
to the server. And that makes for a tremendous bandwidth and latency
jump for the user...<br>
<br>
AJAX could also be used to implement a behind-the-scenes uplink of
local information (like images and documents) that needs to be placed
on the server to display properly.<br>
<br>
AJAX would make a killer progressive search feature, but the current
search has been pretty good for me so far.<br>
<br>
</body>
</html>

--------------040802030402050803080000--