Re: var too large
Ger Hobbelt <[email protected]>
| Newsgroups | gmane.mail.spam.crm114 |
|---|---|
| Message-ID | <[email protected]> |
On Thu, Feb 26, 2009 at 11:03 AM, Thomas Michael Hagen <[email protected]> wrote: > my program goes through large files. some of them are almost 300Mb. > > i fix this by setting -w 300000000 on the command line when i call my crm > program. > > however, when i run the program on many files, they somehow accumulate in > the program's memory, and after a while exceed the limit. > > how do i clean the files back out of memory after i have used them? er, you can't, really. Though you can do it, in a way, by nil-ling the variables which contained the file content, i.e. 'alter' their content to nada: alter (:var:) // though I'm not sure about the effect of that on the crm114 data store. There's no real gc() in there, though... sorry, a bit vague about this... there's crm_compress_tdw_section() in the crm114 C source, which is triggered, among other things, by stuff like isolate (:var:) // performed on an already existing var so you could try that one as an alternative to the 'alter' suggestion above to get the buffers cleaned. > what happens in my program is that i go through several folders, with many > files in them: > i define an überfolder, and do a syscall ls on that, > then i extract the subfoldernames from that list with a match, and > syscall ls on each one, then > extract the filenames and run the classification program on them Not that it matters regarding your current issue, but you can cut down on syscalls by using UNIX find, e.g. find ./ -type f -print will get you a list of all files (type f) of the current directory and any subdir on any UNIX machine (not just Linux, which doesn't require the -print there). See 'man find' for more options, such as -maxdepth and -mindepth restrictions on dir tree traversal. -- Met vriendelijke groeten / Best regards, Ger Hobbelt -------------------------------------------------- web: http://www.hobbelt.com/ http://www.hebbut.net/ mail: [email protected] mobile: +31-6-11 120 978 -------------------------------------------------- ------------------------------------------------------------------------------ Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise -Strategies to boost innovation and cut costs with open source participation -Receive a $600 discount off the registration fee with the source code: SFAD http://p.sf.net/sfu/XcvMzF8H