Re: perl process is killed while loading large xsd schema in memory

I forgot to cc the list in the previous message. I ran xmllint against the
same XSD and it is not such a memory hog:

procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id
wa st
 2  0     92 1467960 435156 5180300    0    0     0    10    1    2  0  0
100  0  0
 1  0     92 1289672 435156 5180304    0    0     0   232 1071  438  8  6 86
 0  0
                   ^^^^^^^^^^
 0  0     92 1472276 435156 5180304    0    0     0    14 1085 2523 10  2 88
 0  0
 0  0     92 1472404 435156 5180304    0    0     0    16 1072  276  0  0
100  0  0

Thanks,

Paulo

On Thu, Jun 30, 2011 at 12:47 PM, Paulo SantAnna <[email protected]> wrote:

> Hi Robin,
>
> Thanks for your response. Here is a vmstat output during the script
> execution:
>
> procs -----------memory---------- ---swap-- -----io---- --system--
> -----cpu------
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id
> wa st
>  3  0     92 1459748 434904 5137060    0    0     0    10    0    2  0  0
> 100  0  0
>  0  0     92 1459620 434904 5137060    0    0     0    11 1064  485  0  0
> 99  0  0
>  3  0     92  43880 434892 5126952    0    0     0    33 1061  195 13 10 77
>  0  0
>                    ^^^^^^^^
>  0  0     92 1498928 434772 5097832    0    0     0     0 1132 2548 13 13
> 74  0  0
>
> It is really using a lot of memory, even more than I thought... The
> application that actually uses the schema (my application is more a test
> suite ) relies on SAX-based parsers so AFAIK they are fine. They don't need
> to validate the doc as a whole, that is my job :-|
>
> Reducing the size is not really an option, neither switching away from XSD
> (lots of legacy stuff) so I need to validate the whole XSD, and then even
> larger XML files based on it. Is this maybe possible with a SAX parser?
>
> Has anyone had experienced with other tools for XSD validation?
>
> Thanks,
>
> Paulo
>
> On Thu, Jun 30, 2011 at 4:38 AM, Robin Berjon <[email protected]> wrote:
>
>> On Jun 29, 2011, at 20:52 , Paulo SantAnna wrote:
>> > I have this requirement of loading somewhat large XSD files ( >32M) and
>> use them to
>> > validate corresponding XML instance files. I have used * what I think *
>> is a typical LibXML
>> > approach but it runs out of memory on a Linux (Ubuntu 10.10) with 1G
>> RAM. It runs fine in another much more powerful CentOS box with 8G RAM and I
>> could consider trying to find the exact memory requirements, however since I
>> will run multiple of these tests I'm just afraid I'm postponing the
>> inevitable.
>>
>> How much of that 1GB is free when you try to load? I'm surprised that it
>> would take *that* much memory. Loading 32M as a simple DOM would likely
>> require 150-450M of memory (depending on many factors), I'm surprised that
>> the schema layer would add that much (but looking at how much it uses on
>> your larger box would answer that).
>>
>> Have you tried loading the schema just as XML to see if that works on the
>> same machine that currently crashes?
>>
>> Note that 32M+ XSD files isn't "somewhat large", it's bloody humongous! I
>> guess it's not something you can control but have you looked at ways of
>> reducing those? I can't imagine that there exists an implementation out
>> there that won't be a slow memory hog over these. Could you maybe analyse
>> the the XML and load only a subset of the schema that you need? Or maybe use
>> a different validation technology that wouldn't require something this big?.
>>
>> --
>> Robin Berjon - http://berjon.com/ - @robinberjon
>>
>>
>

_______________________________________________
Perl-XML mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs