Re: Git import of cvs2git dumpfile

>What command did you use to load the files?  Did it crash when loading
>the blob file or the dump file or did you load them both at the same
>time?  If you were loading them at the same time, you might try loading
>them separately:
>
>    git fast-import --export-marks=FILENAME <git-blob.dat
>    git fast-import --import-marks=FILENAME <git-dump.dat
>
>and see which of the processes dies.

I tried both. And the blob-file import does work very well. It interrupts
when I try to import the dump file.



>Did you monitor the size of the "git fast-import" process before it
>died?  Was it slowly eating up the available 72 Gb?

Yes, I have monitored the consumption. the process uses less than 10% of
memory. So a swapfile is never used.

Best Regards,
Fabian


2013/8/11 Michael Haggerty <[email protected]>

> On 08/11/2013 05:01 PM, Fabian Schäfer wrote:
> > I have successfully migrated my 13GB cvs repository with cvs2git. But
> > now I get an out-of-memory error by using the git fast-import:
> >
> > |fatal: Out of memory, malloc failed (tried to allocate **6196691
> bytes**)
> > fast-import: dumping crash report to fast_import_crash_13097
> > error: git-fast-import died of signal 11
>
> "git fast-import" is separate from cvs2git, so if it is failing then you
> might have better luck asking on the Git mailing list.  But...
>
> What did the crash report say?
>
> > |I have already tried to unlimit stacksize and file descriptors.
> > Also I have searched for the biggest file in the repository: 130MB.
> > |
> >
> > |My System has 32 GB RAM and 40GB Swap, the dump file is 72GB and the
> > blob file 10GB big. I am running the import on a RHEL5.3 with
> > Git 1.8.3.4, Python 2.6.8, GCC44.
>
> What command did you use to load the files?  Did it crash when loading
> the blob file or the dump file or did you load them both at the same
> time?  If you were loading them at the same time, you might try loading
> them separately:
>
>     git fast-import --export-marks=FILENAME <git-blob.dat
>     git fast-import --import-marks=FILENAME <git-dump.dat
>
> and see which of the processes dies.
>
> The Python version is irrelevant, since it is "git fast-import" that is
> failing.
>
> It is funny that RAM + Swap = 72 GB = size of dump file, but I suppose
> it is just a coincidence.
>
> Did you monitor the size of the "git fast-import" process before it
> died?  Was it slowly eating up the available 72 Gb?
>
> > |I have read in an older post that it is maybe possible
> > to split the dump file into smaller pieces. How is this possible?
> > |Can someone please give me an hint?
>
> Hmmm, I suppose it would be possible if you break it at allowed
> boundaries and do the --export-marks/--input-marks thing between runs.
> See the git-fast-import(1) man page for details about the format.
>
> But if it is the marks table that is filling up your memory, then
> splitting up the input files might not help.
>
> Michael
>
> --
> Michael Haggerty
> [email protected]
> http://softwareswirl.blogspot.com/
>

------------------------------------------------------
http://cvs2svn.tigris.org/ds/viewMessage.do?dsForumId=1670&dsMessageId=3062470

To unsubscribe from this discussion, e-mail: [[email protected]].