Re: "unable to read input" errors when testing cdb on Mac OS X

On Fri, 4 Jun 2004, Kevin Murphy wrote:

> This is probably old news, but there's not much out there on the web.

Yes, some of these issues has been discussed on at least 2 other
occasions on this list.  I believe there is an archive at
geocrawler.com, but their site seems to be down.

> I'm trying to install cdb 0.75 on Mac OS X 10.2.8 on a G5, but both the
> [...]
> bash$ ./cdbtest < test.cdb
>
> cdbtest: fatal: unable to read input: truncated file

I have no idea what's in your test.cdb. Is this supposed to be the
test.cdb from rts-tmp?  In that case, how did it end up in the same
directory as cdbtest?

> bash$ cmp rts.out rts.exp
> rts.out rts.exp differ: char 3670, line 202
> bash$ diff rts.exp rts.out
> 202c202
> < cdbmake: fatal: unable to create test.tmp: out of memory
> ---
>  > cdbmake: fatal: unable to read input: bad format

This is a common problem on machine where softlimit -d doesn't actually
work. The problem is that cdbmake input should end with a blank line.
The produced input does not. It shouldn't matter because cdbmake should
run out of memory before then, but it doesn't. The usual reason I have
seen for this on Linux/glibc systems is that malloc() doesn't allocate
in the data segment; instead it creates anonymous mmap()s, which are not
counted against the data limit.

My experience with MacOS is somewhat limited, though it is supposedly
based on a BSD core. I have no idea what libc or allocation scheme they
use. If you want to be completely sure that's what's happening, strace
the cdbmake binary and see if it's calling brk() or mmap() to allocate
memory.

> 216,221c216,217
> < found: 100
> < different record: 0
> < bad length: 0
> < not found: 0
> < untested: 0
> < 0
> ---
>  > cdbtest: fatal: unable to read input: truncated file
>  > 111

This I've never seen before. It looks like your random.cdb is truncated
for some reason (it should be included in the cdb tarball). Make sure
that it matches this md5sum:
  43edab68c706252ad5c7e12f44148e8b
If not, try your tests from a fresh tarball (maybe you were testing for
the other bug and accidentally overwrote it?).

> 223,235c219,220
> < records        100
> < d0              96
> < d1               4
> < d2               0
> < d3               0
> < d4               0
> < d5               0
> < d6               0
> < d7               0
> < d8               0
> < d9               0
> < >9               0
> < 0
> ---
>  > cdbstats: fatal: unable to read input: truncated file
>  > 111

This would be from the same random.cdb corruption as above.

> 238c223
> < found: 5
> ---
>  > found: 1
> 241c226
> < not found: 0
> ---
>  > not found: 767

I've also never seen this before. It might be indicative of a real
error. Can you post the on the web somewhere the cdb created by the
following command:
  ./testzero 5 >test.cdb

Then we can figure out if the problem is in the creation of the cdb, or
the cdbtest program.

-Peff