Re: perl process is killed while loading large xsd schema in memory
Paulo SantAnna <[email protected]> Thu, 30 Jun 2011 13:26:17 -0700
| Newsgroups | gmane.comp.lang.perl.xml |
|---|---|
| Message-ID | <[email protected]> |
There is a related LibXML bug reported here: https://rt.cpan.org/Public/Bug/Display.html?id=63449 Thanks, Paulo On Thu, Jun 30, 2011 at 1:15 PM, Paulo SantAnna <[email protected]> wrote: > I forgot to cc the list in the previous message. I ran xmllint against the > same XSD and it is not such a memory hog: > > procs -----------memory---------- ---swap-- -----io---- --system-- > -----cpu------ > r b swpd free buff cache si so bi bo in cs us sy id > wa st > 2 0 92 1467960 435156 5180300 0 0 0 10 1 2 0 0 > 100 0 0 > 1 0 92 1289672 435156 5180304 0 0 0 232 1071 438 8 6 > 86 0 0 > ^^^^^^^^^^ > 0 0 92 1472276 435156 5180304 0 0 0 14 1085 2523 10 2 > 88 0 0 > 0 0 92 1472404 435156 5180304 0 0 0 16 1072 276 0 0 > 100 0 0 > > Thanks, > > Paulo > > On Thu, Jun 30, 2011 at 12:47 PM, Paulo SantAnna <[email protected]>wrote: > >> Hi Robin, >> >> Thanks for your response. Here is a vmstat output during the script >> execution: >> >> procs -----------memory---------- ---swap-- -----io---- --system-- >> -----cpu------ >> r b swpd free buff cache si so bi bo in cs us sy id >> wa st >> 3 0 92 1459748 434904 5137060 0 0 0 10 0 2 0 0 >> 100 0 0 >> 0 0 92 1459620 434904 5137060 0 0 0 11 1064 485 0 0 >> 99 0 0 >> 3 0 92 43880 434892 5126952 0 0 0 33 1061 195 13 10 >> 77 0 0 >> ^^^^^^^^ >> 0 0 92 1498928 434772 5097832 0 0 0 0 1132 2548 13 13 >> 74 0 0 >> >> It is really using a lot of memory, even more than I thought... The >> application that actually uses the schema (my application is more a test >> suite ) relies on SAX-based parsers so AFAIK they are fine. They don't need >> to validate the doc as a whole, that is my job :-| >> >> Reducing the size is not really an option, neither switching away from XSD >> (lots of legacy stuff) so I need to validate the whole XSD, and then even >> larger XML files based on it. Is this maybe possible with a SAX parser? >> >> Has anyone had experienced with other tools for XSD validation? >> >> Thanks, >> >> Paulo >> >> On Thu, Jun 30, 2011 at 4:38 AM, Robin Berjon <[email protected]> wrote: >> >>> On Jun 29, 2011, at 20:52 , Paulo SantAnna wrote: >>> > I have this requirement of loading somewhat large XSD files ( >32M) and >>> use them to >>> > validate corresponding XML instance files. I have used * what I think * >>> is a typical LibXML >>> > approach but it runs out of memory on a Linux (Ubuntu 10.10) with 1G >>> RAM. It runs fine in another much more powerful CentOS box with 8G RAM and I >>> could consider trying to find the exact memory requirements, however since I >>> will run multiple of these tests I'm just afraid I'm postponing the >>> inevitable. >>> >>> How much of that 1GB is free when you try to load? I'm surprised that it >>> would take *that* much memory. Loading 32M as a simple DOM would likely >>> require 150-450M of memory (depending on many factors), I'm surprised that >>> the schema layer would add that much (but looking at how much it uses on >>> your larger box would answer that). >>> >>> Have you tried loading the schema just as XML to see if that works on the >>> same machine that currently crashes? >>> >>> Note that 32M+ XSD files isn't "somewhat large", it's bloody humongous! I >>> guess it's not something you can control but have you looked at ways of >>> reducing those? I can't imagine that there exists an implementation out >>> there that won't be a slow memory hog over these. Could you maybe analyse >>> the the XML and load only a subset of the schema that you need? Or maybe use >>> a different validation technology that wouldn't require something this big?. >>> >>> -- >>> Robin Berjon - http://berjon.com/ - @robinberjon >>> >>> >> > _______________________________________________ Perl-XML mailing list [email protected] To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs