Re: xindice_rebuild - file size vastly multiplied

Natalia,
Thanks for offering some hints.

To put some data on the table ...

Below some size quantities for two parts of the database
   (1) a collection with few and small documents
   (2) a collection with many small documents

For each of these I give size of=20
   (A) exported format=20
   (B) xindice 1.0 database
   (C) xindice 1.1 database

Sizes are computed by "du" (running Cygwin).

Summary table of sizes in KB (table more readable in fixed font ;-) ):

.      |   (A)   |   (B)   |   (C)   |
. (1)  |      20 |     250 |   84000 |
. (2)  |    4000 |     450 |   12000 |

The "1.1" vs "1.0" expansion factor is much larger for the case of
collections with few and small files (factor 350) , than for many and small
files (factor 25).=20

But nevertheless, I wonder for what kind of databases 1.1 would beat 1.0
from a *size* point of view. Maybe comparing processing speed of "1.1" vs
"1.0" would give a different picture, but at the moment I have no numbers t=
o
offer in that respect.=20

By the way, I do not have any very large documents in the database, so I
have no idea how "1.1" compares to "1.0" for such beasts.

And the 64,000 $ question is, of course, what means are available to
decrease disk space footprint of the "1.1"  database?

/O


=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D
(1) a set of collections with few and small documents

(A) This is a size listing of (part of) the exported 1.0 database.
Result: total of 20KB contents.=20
Most of these files are small.
(Note: the "meta" collections/files are my own meta info of the application
level, so not Xindice meta info!).

xxx$ du -ba publications
553     publications/rss-common/meta/meta.xml
1065    publications/rss-common/meta
1577    publications/rss-common
257     publications/rss_2_0/meta/meta.xml
769     publications/rss_2_0/meta
1281    publications/rss_2_0
260     publications/rss_0_92/meta/meta.xml
772     publications/rss_0_92/meta
1284    publications/rss_0_92
260     publications/rss_0_91/meta/meta.xml
772     publications/rss_0_91/meta
1284    publications/rss_0_91
438     publications/newsletter/meta/meta.xml
950     publications/newsletter/meta
1462    publications/newsletter
512     publications/archive/meta
1024    publications/archive
429     publications/newsletter-index/meta/meta.xml
941     publications/newsletter-index/meta
1453    publications/newsletter-index
427     publications/news-archive/meta/meta.xml
939     publications/news-archive/meta
1451    publications/news-archive
394     publications/home-page/meta/meta.xml
906     publications/home-page/meta
1418    publications/home-page
257     publications/rss_1_0/meta/meta.xml
769     publications/rss_1_0/meta
1281    publications/rss_1_0
315     publications/home-page-time-stamp/meta/meta.xml
827     publications/home-page-time-stamp/meta
1339    publications/home-page-time-stamp
350     publications/events/meta/meta.xml
862     publications/events/meta
1374    publications/events
1634    publications/meta/meta.xml
2146    publications/meta
18886   publications
xxx$=20

(B) The size of the corresponding 1.0 database files=20
Result: total of 250KB contents.=20

yyy$ du -ba publications/
12288   publications/archive/archive.tbl
12288   publications/archive/meta/meta.tbl
12288   publications/archive/meta
24576   publications/archive
12288   publications/home-page/home-page.tbl
12288   publications/home-page/meta/meta.tbl
12288   publications/home-page/meta
24576   publications/home-page
12288   publications/meta/meta.tbl
12288   publications/meta
12288   publications/news-archive/meta/meta.tbl
12288   publications/news-archive/meta
12288   publications/news-archive/news-archive.tbl
24576   publications/news-archive
12288   publications/newsletter/meta/meta.tbl
12288   publications/newsletter/meta
12288   publications/newsletter/newsletter.tbl
24576   publications/newsletter
12288   publications/publications.tbl
12288   publications/rss-common/meta/meta.tbl
12288   publications/rss-common/meta
12288   publications/rss-common/rss-common.tbl
24576   publications/rss-common
12288   publications/rss_0_91/meta/meta.tbl
12288   publications/rss_0_91/meta
12288   publications/rss_0_91/rss_0_91.tbl
24576   publications/rss_0_91
12288   publications/rss_0_92/meta/meta.tbl
12288   publications/rss_0_92/meta
12288   publications/rss_0_92/rss_0_92.tbl
24576   publications/rss_0_92
12288   publications/rss_1_0/meta/meta.tbl
12288   publications/rss_1_0/meta
12288   publications/rss_1_0/rss_1_0.tbl
24576   publications/rss_1_0
12288   publications/rss_2_0/meta/meta.tbl
12288   publications/rss_2_0/meta
12288   publications/rss_2_0/rss_2_0.tbl
24576   publications/rss_2_0
245760  publications/
yyy$=20

(C) ... and of the corresponding 1.1 database files
Result: total of 84000KB contents.=20


zzz$ du -ba publications/
4202496 publications/archive/archive.tbl
4202496 publications/archive/meta/meta.tbl
4202496 publications/archive/meta
8404992 publications/archive
4202496 publications/home-page/home-page.tbl
4202496 publications/home-page/meta/meta.tbl
4202496 publications/home-page/meta
8404992 publications/home-page
4202496 publications/meta/meta.tbl
4202496 publications/meta
4202496 publications/news-archive/meta/meta.tbl
4202496 publications/news-archive/meta
4202496 publications/news-archive/news-archive.tbl
8404992 publications/news-archive
4202496 publications/newsletter/meta/meta.tbl
4202496 publications/newsletter/meta
4202496 publications/newsletter/newsletter.tbl
8404992 publications/newsletter
4202496 publications/publications.tbl
4202496 publications/rss-common/meta/meta.tbl
4202496 publications/rss-common/meta
4202496 publications/rss-common/rss-common.tbl
8404992 publications/rss-common
4202496 publications/rss_0_91/meta/meta.tbl
4202496 publications/rss_0_91/meta
4202496 publications/rss_0_91/rss_0_91.tbl
8404992 publications/rss_0_91
4202496 publications/rss_0_92/meta/meta.tbl
4202496 publications/rss_0_92/meta
4202496 publications/rss_0_92/rss_0_92.tbl
8404992 publications/rss_0_92
4202496 publications/rss_1_0/meta/meta.tbl
4202496 publications/rss_1_0/meta
4202496 publications/rss_1_0/rss_1_0.tbl
8404992 publications/rss_1_0
4202496 publications/rss_2_0/meta/meta.tbl
4202496 publications/rss_2_0/meta
4202496 publications/rss_2_0/rss_2_0.tbl
8404992 publications/rss_2_0
84049920 publications/
zzz$=20


=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D
(2) a collection with a large set of small documents

(A) This is a size listing of (part of) the exported 1.0 database.
This one contains approx 4000 rather small documents.
Result: total of 4000KB contents (so approx 1KB per document)

xxx$ du -b content/news
3932206 content/news/db
665     content/news/meta
3933383 content/news
xxx$=20

(note: not displaying individual files/documents ... there are too many of
them)

(B) The size of the corresponding 1.0 database files=20
Result: total of 450KB contents.=20

yyy$ du -ba content/news/
430080  content/news/db/db.tbl
430080  content/news/db
12288   content/news/meta/meta.tbl
12288   content/news/meta
12288   content/news/news.tbl
454656  content/news/
yyy$=20

(C) ... and of the corresponding 1.1 database files
Result: total of 12MB contents.=20

zzz$ du -ba content/news/
4202496 content/news/db/db.tbl
4202496 content/news/db
4202496 content/news/meta/meta.tbl
4202496 content/news/meta
4202496 content/news/news.tbl
12607488   content/news/
zzz$=20

=3D=3D=3Dend=3D=3D=3D


Natalia Shilenkova wrote:
>=20
> There was a change in Xindice v1.1 that could possibly be responsible
> for database size increase.
>=20
> Xindice v1.0 failed to correctly allocate initial file space for a
> collection according to its page size and page count parameters, so a
> collection with just a few documents would occupy several Kb on disk
> instead of reserving some space on disk for the collection to grow. It
> was fixed in v1.1 and a collection with default page size (4Kb) and
> page count (1024) parameters now occupies about 4Mb.
>=20
> If you had several small collections in Xindice v1.0 it is possible
> that after rebuilding them for v1.1 the database would take
> considerably more disk space. How many collections do you have in that
> database and how big are the collections? If this indeed is the reason
> for the database increase, it could be fixed by adjusting page count
> parameter for small collections.
>=20
> Also, I think Meta collections were introduced somewhere between 1.0
> and 1.1, I cannot remember now if they are created when rebuilding a
> database, but if they are, it would take some disk space, too. You can
> explore database directories to see if Meta collections are there (in
> system/Metas, I think). Meta collections can be turned off.
>=20
> Natalia
>=20
> On Thu, Apr 16, 2009 at 9:35 AM, OKO <[email protected]> wrote:
>>
>> Ran xindice_rebuild on a smallish existing 1.0 database to get it into
>> =C2=A01.1
>> format.
>> The size was expanded enormously:
>> db 1.0: 2 M
>> db 1.1: 233 M
>>
>> That is a factor of 100 ;-(
>>
>> I have not dared do this on my real 1.0 database, which now occupies mor=
e
>> than 1G of disk space.
>>
>> *Question*:
>> =C2=A0- Is this a feature or a bug?
>>
>> Tool info as presented:
>> $ xindice -h
>> trying to register database
>>
>> Xindice Command Tools v1.1
>>
>> ...etc...
>>
>> /O
>> --
>> View this message in context:
>> http://www.nabble.com/xindice_rebuild---file-size-vastly-multiplied-tp23=
078111p23078111.html
>> Sent from the Xindice - Users mailing list archive at Nabble.com.
>>
>>
>=20
>=20

--=20
View this message in context: http://www.nabble.com/xindice_rebuild---file-=
size-vastly-multiplied-tp23078111p23084822.html
Sent from the Xindice - Users mailing list archive at Nabble.com.