Re: parser in Quanta

Hi,

 I looked at QXmlReader today and tried to figure out what's happening 
in KDOM (I have only an old version from kdenonbeta, and unfortunately 
it is not documented quite well). My first feeling is that both are 
designed to work with complete documents. QXmlReader also has the 
assumption that what is parsed is valid XML or there will be an error 
thrown by the parser. This latter issue might be solved with our own 
QXmlReader derivate class, where we don't throw errors every time, 
instead build elements/nodes in the best way we can from a broken 
document. We might even specify this as a "feature" for the parser.
 The real problem is parsing only part of the document and merging it 
with the existing DOM tree. Jens asked we if parsing the whole document 
always is slow or not. According to my previous testing it was very 
slow with the current parser and I don't except much better results 
with any new one. As an extreme case I have a 770KB long, 12000 line 
HTML file (yes, I got from a user due to an old bugreport). It doesn't 
have any PHP or other parts. On my system (which isn't slow), it takes 
between 500-600ms to parse. If you just modify (add, remove text) the 
document, regardless if it is at the beginning or the end, re-parsing 
takes around 10ms. Now imagine that in case of a slower machine, the 
whole parsing can take at least 3 times as much, while the re-parsing 
is almost around the same time, as the amount to be parsed is small.
Of course, the current system is not too good and in some cases cannot 
figure out the changed area and requires a whole reparse when typing, 
but this is implementation detail.
 Just rung Quanta from the command line and watch the debug output. It 
tells you about parsing times.
 According to my testing, anything above 200ms results in a noticeable 
delay when typing.
 So I think we need partial parsing and merging with the existing DOM 
tree and I don't see a solution with QXml or KDOM.
 There is another problem with QXml: it requires an IODevice, a 
ByteArray or a QString a a source. We usually get the source from the 
KTextEditor. It is possible to get the whole content of the document as 
a QString and pass to the parser, but it is just a waste of memory. 
Instead it is possible to incrementally read from KTextEditor and feed 
the data to the parser. This is what we do now (line by line). The 
current implementation has its limitations though, see below.
 
 What I suggest is:
- write our own parser
- when creating/manipulatin nodes, call a helper class to create nodes
- this helper class will create whatever nodes we want (our own or KDOM 
nodes)
- write our own code to detect the changed areas and reparse it and 
merge the DOM trees

I *am* suggesting to write a new parser because:
- the current one relies on string searching and regular expressions. 
Jens suggested that a char by char parser should be better and faster. 
- the current one has strong ties to KTextEditor. We should have a 
common way to feed data, so it can come from KTextEditor, a QString or 
a file (QIODevice)
- the current code for re-parsing is ugly and not understandable
- there is no automated test written to see if a change breaks the 
parser or not.

Andras

_______________________________________________
quanta-devel mailing list
[email protected]
https://mail.kde.org/mailman/listinfo/quanta-devel

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQBDy2faTQdfac6L/08RAodeAKDFOk0KwUYUATzMmCXu0jaxvFVxbACcCloF
YczEaLMkiGV1BeX887NcLZE=
=uj7l
-----END PGP SIGNATURE-----