Re: Resizing DB set

Hi Glenn,

Glenn McGurrin wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hello, I have had some problems with changing the number of DBs used by
> the indexer in the past when using the special import export tool to do
> the job in the best possible way.  If I just Dump the whole thing to an
> sql file I crash it due to the file growing over my systems limit for
> the size of a file.  If I try to dump the DBs separately I somehow get
> it to still dump them all, last time I got it to work by piping the
> output through gzip then to a file but that was still over half the size
> of my system limit and my DB set has grown since then.  I am wondering
> how to get the DBs to actually dump separately, I have seen some parts
> of your documentation that suggest subtly different syntax for
> specifying a certain DB for the operation.  It might also work to use
> the more efficient bzip2 compressor but documentation of compressing
> stdin seems to be in short supply compared to the same with gzip or
> decompressing to stdout with bzip2.

The correct syntax is -Dxxx, where xxx is database number,
starting from 1.
For example:

indexer -Edumpdata -D1
indexer -Edumpdata -D2
indexer -Edumpdata -D3


However, -D did not work together with -Edumpdata.

The next release will fix this problem.


You can also apply this patch and recompile:


diff -u -p -r1.419 db.c
--- db.c        14 Feb 2010 20:26:25 -0000      1.419
+++ db.c        28 Apr 2010 12:54:33 -0000
@@ -458,6 +458,10 @@ int UdmURLActionNoLock(UDM_AGENT *A, UDM
    if(cmd == UDM_URL_ACTION_FLUSH)
      return UdmDocUpdate(A, D);

+  /* indexer -Edumpdata -Dxxx */
+  if (cmd == UDM_URL_ACTION_DUMPDATA)
+    dbnum= UdmVarListFindInt(&A->Conf->Vars, "DBLimit", 0) - 1;
+
  #ifdef USE_TRACE
    fprintf(A->TR, "[%d] URLAction: %d\n", A->handle, cmd);
  #endif



>  I am also happy to report that I
> have surpassed the number of threads that was found stable in your docs,
> I now am using 50 crawler threads and when needed up to 15 indexer
> threads (one per DB).  Any help you can give would be appreciated.

Good to know! Thanks.

> 
> Thanks,
> Glenn McGurrin
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.8 (Darwin)
> 
> iEYEARECAAYFAkvUVQUACgkQzSSLW0oTVN6+RQCfViiOoGhAl/Ed8qbN2FmVMaEi
> EJIAn19sA8ehnQKh76ME3bCpF++kQuXC
> =VS36
> -----END PGP SIGNATURE-----
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]