Re: Git import of cvs2git dumpfile

On 08/11/2013 05:01 PM, Fabian Schäfer wrote:
> I have successfully migrated my 13GB cvs repository with cvs2git. But
> now I get an out-of-memory error by using the git fast-import:
> 
> |fatal: Out of memory, malloc failed (tried to allocate **6196691 bytes**)
> fast-import: dumping crash report to fast_import_crash_13097
> error: git-fast-import died of signal 11

"git fast-import" is separate from cvs2git, so if it is failing then you
might have better luck asking on the Git mailing list.  But...

What did the crash report say?

> |I have already tried to unlimit stacksize and file descriptors. 
> Also I have searched for the biggest file in the repository: 130MB.
> |
> 
> |My System has 32 GB RAM and 40GB Swap, the dump file is 72GB and the 
> blob file 10GB big. I am running the import on a RHEL5.3 with 
> Git 1.8.3.4, Python 2.6.8, GCC44.

What command did you use to load the files?  Did it crash when loading
the blob file or the dump file or did you load them both at the same
time?  If you were loading them at the same time, you might try loading
them separately:

    git fast-import --export-marks=FILENAME <git-blob.dat
    git fast-import --import-marks=FILENAME <git-dump.dat

and see which of the processes dies.

The Python version is irrelevant, since it is "git fast-import" that is
failing.

It is funny that RAM + Swap = 72 GB = size of dump file, but I suppose
it is just a coincidence.

Did you monitor the size of the "git fast-import" process before it
died?  Was it slowly eating up the available 72 Gb?

> |I have read in an older post that it is maybe possible 
> to split the dump file into smaller pieces. How is this possible?
> |Can someone please give me an hint?

Hmmm, I suppose it would be possible if you break it at allowed
boundaries and do the --export-marks/--input-marks thing between runs.
See the git-fast-import(1) man page for details about the format.

But if it is the marks table that is filling up your memory, then
splitting up the input files might not help.

Michael

-- 
Michael Haggerty
[email protected]
http://softwareswirl.blogspot.com/

------------------------------------------------------
http://cvs2svn.tigris.org/ds/viewMessage.do?dsForumId=1670&dsMessageId=3062411

To unsubscribe from this discussion, e-mail: [[email protected]].