Re: parser in Quanta

On Monday 16 January 2006 21:49, Andras Mantia wrote:
> On Monday 16 January 2006 16:40, Jens Herden wrote:
> > Hi Andras,
> > yes in deed, QXmlReader is only parsing valid code. But my suggestion
> > was not to use QXmlReader ;-)
>
> Well, KDOM seems to be able to use two parsing backends: libxml and
> QXml. The latter is using QXmlReader or a derivate class from it.
>
> > I wanted you to look into this to get an idea what kind of parser I
> > want to have and I knew that we can not use QXmlReader. But the whole
> > structure around QXmlReader with reader and builder is worth to copy,
> > I think.
>
> Might make sense. But I think the accent is on copy. Or maybe can be
> used as a source of inspiration.
>
> > So my suggestion was, like you said, to create our own
> > QXmlReader derived class which is very error tolerant and fixes as
> > many problems as possible to get a useable KDOM tree for the
> > renderer.
>
> Still I'm not sure what we can get from it and if it is possible to
> integrate with KDOM in the way they intended or we just have to use
> hack after hack.

You could create a new KDOM parser backend which isn't an XML parser, but a 
tag soup parser. The current backends are libxml2 and Qt. In this way your 
parser could deliver SAX events to KDOM document builder, and KDOM would take 
care of the document building. To me it sounds like a clean separation, you 
would only replace what you need to replace. From what I can tell, that 
parser would accept valid XML, plus invalid XML.

The one who would write that tag soup parser could perhaps base it on khtml(or 
kdenonbeta/khtml2)'s parser, since that one already is a tag soup parser. One 
would add stuff like namespace handling, I guess.

> > >  The real problem is parsing only part of the document and merging
> > > it with the existing DOM tree. Jens asked we if parsing the whole
> > > document always is slow or not. According to my previous testing it
> > > was very slow with the current parser and I don't except much
> > > better results with any new one.
> >
> > I do expect better results :-)
>
> But how much better? I doubt it will be 2x, not talking about the 5-10x
> we would need...
>
> > >  So I think we need partial parsing and merging with the existing
> > > DOM tree and I don't see a solution with QXml or KDOM.
> >
> > Where exactly do you see the problem to feed parts of documents into
> > the parser and merge the resulting tree with an existing one?
>
> I could not see a way with QXmlReader to parse only a part of the
> document.

It depends on what you mean by "document", but one can get a content handler 
to receive events from any source. The KDOM parser backend can receive events 
from both Qt or libxml2: you could add a third. Perhaps 
kdenonbeta/kdom/backends/ are of interest.

> Merging has to be done by us anyway. 

You could use your tag soup parser, but instead of sending the events to 
KDOM's document builder, you could receive the events and from there manually 
merge/sync. But that surely is tricky, no matter how. I wonder how XML 
Spy/Stylus Studio/etc do? Is there any research in this area?

> > >  There is another problem with QXml: it requires an IODevice, a
> > > ByteArray or a QString a a source.
> >
> > AFAIK this is not correct. What we need is a QXmlInputSource and what
> > QT offers uses IODevice or Byte Array. But I think we can create our
> > own QXmlInputSource that operates on a KTextEditor interface somehow.
>
> We might try to subclass it.
>
> > > - when creating/manipulatin nodes, call a helper class to create
> > > nodes - this helper class will create whatever nodes we want (our
> > > own or KDOM nodes)
> >
> > This is what I would call the builder :-)
> > If we could use the existing builder for KDOM we could save some
> > work.
>
> But as KDOM is not yet available and I have some doubts about the
> possibility to fit clearly for our case (unless someone proves me
> wrong), I would suggest to completely write our own code.
>
> > > - write our own code to detect the changed areas and reparse it and
> > > merge the DOM trees
> >
> > This is the point that could become hard from what I know in the
> > moment. But I see no real blocker here.
>
> If it is possible to insert nodes in whatever position you want, or to
> move nodes (and their children), then it is doable.
>
> > So still the question how to proceed? We can not use KDOM2 yet and
> > creating our new parser with our custom dom tree was the last idea.
> > Do you still want to go this way?
>
> Yes, but in a way that our custom DOM tree can be easily replaced by
> KDOM once we switch to Qt/KDE4.

Of course, if you must go forward without Qt/KDE 4.0, there is no other way 
than to not use KDOM. However, when you are using KDE/Qt 4.0, you could use 
KDOM even if it's in kdenonbeta by bringing it into your module with an 
svn:externals property. In that way it would be very similar to when it is in 
kdelibs, and you wouldn't have to bother with an extra module(kdenonbeta).

But I must say, that in your case I would really try to switch to KDOM, 
because there are so many goodies in there for you. Recently WebCore's 
hash-table based way of storing tag names/namespaces was merged(e.g, a string 
isn't stored for every element name), and I think that should help you with 
memory usage. By adding GUI glue you can add features such as XInclude, OASIS 
Catalog, and XQuery/XPath, which commercial IDEs such as Stylus Studio and 
XML Spy has, simply because it's just waiting there in KDOM. Similarly, you 
can't in a clean way add nifty XSL debugging/functionality if you need to 
convert between different trees(your own, and KDOM's). So, I think there are 
many wins in using a unified platform.

Would it be of interest for you if KDOM moved to kdelibs? If so, I can 
investigate.

> > I am not so sure anymore, because I
> > fear that we make a lot of work that will be thrown away when we
> > switch to KDE4.
>
> We don't have to throw that away. KDOM is needed for VPL, but as I
> understood it will be really useful only if KHTML is based on KDOM as
> well (which is currently not).

khtml, not necessarily kdenonbeta/khtml2, will in some way or another be based 
on KDOM code. That is of course not some representative comment on behalf of 
the "KHTML team", but that's what I easily would set my money on.


Cheers,

		Frans