Re: Htdig 3.1.6, maximum filesize?

On Sun, 5 Sep 2004, Robert Ribnitz wrote:

> What is the maximum filesize htdig (3.1.6) can access?

Out of the box under Linux there is a 2 GB limit.

> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=269831
>
> Any idea how we can acto on such an error?

There are at least a couple issues related to the maximum file size.

The first is ht://Dig's direct handling of on-disk files, which is the one 
in play here. The db.wordlist.work file is a text file directly handled 
with system calls such as fopen(). Since the ht://Dig package isn't 
compiled with largefile support by default, such access is affected by the 
2 GB limit. A couple people have reported that adding '-D_LARGEFILE_SOURCE 
-D_FILE_OFFSET_BITS=64' has successfully provided a workaround for the 
limit on db.wordlist.

The second issue is one of support at the Berkeley DB level. The version 
provided with 3.1.6 doesn't provide support for large files under Linux. 
There is an --enable-bigfile, but its support is limited to AIX, HP/UX, 
and Solaris. Some time ago attempts were made to force the option for a 
Linux build, but it was reported that the attempts were unsuccessful. A 
more recent post implied that the limit at the BDB level is 4 GB.

So by building ht://Dig with largefile support you can most likely bypass 
problem with limits on the size of db.wordlist. However there will still 
be a limit on the size of the actual databases (presumably 4 GB).

Jim


-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
_______________________________________________
ht://Dig Developer mailing list:
[email protected]
List information (subscribe/unsubscribe, etc.)
https://lists.sourceforge.net/lists/listinfo/htdig-dev