Re: [Issue 149] too many open files using PyPy

[These conversations should always be CC to the cvs2svn dev list.
Please "reply all".]

On 05/17/2013 08:27 PM, Jan Smets wrote:
> Pass 16 (output pass) is running out of memory after several hours.
> Probably some PyPy effect too. Any idea if there is anything in that
> pass that isn't properly freeing memory?
> I'll see what I can find, but my python knowledge is weak/non-existing.
> 
> Thanks for all your great help!

The CPython garbage-collector is only needed for circular reference
loops, because most unused objects are cleared via reference counting.
cvs2svn is careful not to create any reference loops, so the
garbage-collector was costing a lot of overhead but bringing zero
benefit.  So:

* In main.py:main(), we turn off the CPython garbage-collector by
calling gc.disable().

* But to be sure that we really aren't creating any reference loops, in
pass_manager.py:PassManager.run(), we call check_for_garbage() at the
end of each conversion pass.  If "debug" level logging is turned on, we
also emit a warning if any garbage was found (normally none should be
found).  I occasionally run tests in debug mode to make sure that code
changes haven't led to the creation of reference loops.

Obviously PyPy does garbage collection differently, and one of the big
differences is that the PyPy garbage collector *is* required.  So if
gc.disable() disables the PyPy garbage collector, then that would
explain the problem.

Otherwise, you might try running the last pass under CPython.  As long
as the PyPy pickle format is compatible with that of CPython, this
should work without having to run all of the passes under CPython.  Does
CPython also fail with an out-of-memory error?

If so, maybe it really *is* using too much memory.  Do you have any
gigantic files in your repo (like ISO images etc)?  cvs2svn sometimes
holds the entire text of a file revision in memory, so this could cause
a problem.

Perhaps you are not running out of physical memory but rather out of
virtual address space?  If you are running the conversion in 32-bit
mode, try 64-bit mode.

If you turn up the verbosity of the output, you can see exactly what
revision and/or files cvs2svn is working on when it fails.  Maybe that
will give you a hint.

Michael

-- 
Michael Haggerty
[email protected]
http://softwareswirl.blogspot.com/

------------------------------------------------------
http://cvs2svn.tigris.org/ds/viewMessage.do?dsForumId=1667&dsMessageId=3055840

To unsubscribe from this discussion, e-mail: [[email protected]].