Re: var too large

thanks for the 'find'-tip. i'll use that.

i tried fiddling around with my var names, and made sure none of them were
the same between functions

i.e., function A calls function B with [:*:file:] and  function B names it
(:fffile:) or something in stead of just (:file:).

this seems to have solved the accumulation problem! the classifier is now
well into october (ten months of newspaper articles) without complaining
about var size.


On Thu, Feb 26, 2009 at 11:26 AM, Ger Hobbelt <[email protected]> wrote:

> On Thu, Feb 26, 2009 at 11:03 AM, Thomas Michael Hagen
> <[email protected]> wrote:
> > my program goes through large files. some of them are almost 300Mb.
> >
> > i fix this by setting -w 300000000 on the command line when i call my crm
> > program.
> >
> > however, when i run the program on many files, they somehow accumulate in
> > the program's memory, and after a while exceed the limit.
> >
> > how do i clean the files back out of memory after i have used them?
>
> er, you can't, really. Though you can do it, in a way, by nil-ling the
> variables which contained the file content, i.e. 'alter' their content
> to nada:
>
> alter (:var:) //
>
> though I'm not sure about the effect of that on the crm114 data store.
> There's no real gc() in there, though... sorry, a bit vague about
> this...
> there's crm_compress_tdw_section() in the crm114 C source, which is
> triggered, among other things, by stuff like
>
> isolate (:var:) //
>
> performed on an already existing var so you could try that one as an
> alternative to the 'alter' suggestion above to get the buffers
> cleaned.
>
>
>
> > what happens in my program is that i go through several folders, with
> many
> > files in them:
> > i define an überfolder, and do a syscall ls on that,
> > then i extract the subfoldernames from that list with a match, and
> > syscall ls on each one, then
> > extract the filenames and run the classification program on them
>
> Not that it matters regarding your current issue, but you can cut down
> on syscalls by using UNIX find, e.g.
>
> find ./ -type f -print
>
> will get you a list of all files (type f) of the current directory and
> any subdir on any UNIX machine (not just Linux, which doesn't require
> the -print there).
> See 'man find' for more options, such as -maxdepth and -mindepth
> restrictions on dir tree traversal.
>
>
>
>
> --
> Met vriendelijke groeten / Best regards,
>
> Ger Hobbelt
>
> --------------------------------------------------
> web:    http://www.hobbelt.com/
>        http://www.hebbut.net/
> mail:   [email protected]
> mobile: +31-6-11 120 978
> --------------------------------------------------
>

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H

_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general