Re: hitting the file size limit with cdbmake

On Sat, May 02, 2009 at 05:53:16PM +1000, Anson Parker wrote:
> I have a cdbmake-ready file that's 1964047825 bytes. When I run the command:
> 
> cdbmake data.cdb dtmp < cdb-ready-file.txt
> 
> I am getting a "File size limit exceeded" error. At the point cdbmake fails
> with this error the temp file is 2147483647 bytes.
> 
> I thought my filesize limit for CDB was 4GB?

I ran into the same problem a few years ago.  To create files larger
than 4g on Linux (and likely others?), you need to call open(2) with
O_LARGEFILE.  The following patch makes that small change to
cdb-0.75/open_trunc.c, and also modifies cdb-0.75/conf-cc to pass gcc
the "-D_LARGEFILE64_SOURCE" and "-D_FILE_OFFSET_BITS=64" command line
arguments:

    http://michael.toren.net/code/cdb-0.75-cdbmake-largefile-support.patch

I haven't tested this on any other platforms, but would be curious to
hear back from others.

HTH,
-mct