Re: What is "large" versus "huge"? Here's the answer ....

On Fri, 25 Mar 2022 at 11:15, Roger L Costello <[email protected]> wrote:

> Hi Folks,
>
> Here's the number that separates large from huge:
>
>         8K
>
> Allow me to explain.
>
> An XML lexer breaks up into tokens the stream of characters comprising an
> XML document. For example, this stream of characters that we recognize as
> an XML document:
>
> <Document>Hello, world</Document>
>
> might be broken up into these tokens:
>
> '<'
> Document
> '>'
> Hello, world
>


er perhaps put perhaps more likely
H
e
l
l
o
,

w
o
r
l
d


In the XML infoset representation of the parse for example each character
generates a character information item
https://www.w3.org/TR/2004/REC-xml-infoset-20040204/#infoitem.character

and in streaming and other implementations you can't wait until you see the
end tag  before starting the lexical analysis.



> '</'
> Document
> '>'
>
> All of those tokens are very short. But there are some XML documents with
> very long tokens.
>
> How long should your tokens be? What are large tokens? What are huge
> tokens?
>
> Practically speaking, if an XML element contains a string, how long should
> you allow the string to be?
>
> The answer is 8K.
>

If using a particular unspecified parsing strategy and a flex generated
parser.


>
> Tokens less than 8K are large. They are of acceptable length and won't
> incur performance penalties in the lexer.
>
> Tokens greater than 8K are huge. They will incur performance penalties in
> the lexer.
>
> Don't create XML documents with elements containing strings longer than 8K.
>

I don't see that follows.

David