Re: perl process is killed while loading large xsd schema in memory

On Jun 29, 2011, at 20:52 , Paulo SantAnna wrote:
> I have this requirement of loading somewhat large XSD files ( >32M) and use them to
> validate corresponding XML instance files. I have used * what I think * is a typical LibXML
> approach but it runs out of memory on a Linux (Ubuntu 10.10) with 1G RAM. It runs fine in another much more powerful CentOS box with 8G RAM and I could consider trying to find the exact memory requirements, however since I will run multiple of these tests I'm just afraid I'm postponing the inevitable.

How much of that 1GB is free when you try to load? I'm surprised that it would take *that* much memory. Loading 32M as a simple DOM would likely require 150-450M of memory (depending on many factors), I'm surprised that the schema layer would add that much (but looking at how much it uses on your larger box would answer that).

Have you tried loading the schema just as XML to see if that works on the same machine that currently crashes?

Note that 32M+ XSD files isn't "somewhat large", it's bloody humongous! I guess it's not something you can control but have you looked at ways of reducing those? I can't imagine that there exists an implementation out there that won't be a slow memory hog over these. Could you maybe analyse the the XML and load only a subset of the schema that you need? Or maybe use a different validation technology that wouldn't require something this big?.

-- 
Robin Berjon - http://berjon.com/ - @robinberjon

_______________________________________________
Perl-XML mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs