Re: [PATCH] improved store-file

Hi

At 11:57 AM 2004/10/29, Alexander Malysh wrote:
>Hi Kalle,
>
>some comments to your patch...
>
>1) don't use dict if you don't know how many items will be stored (dict
>filled to more as 80% works very slow). Here we should use something like
>red-black-tree.


Or just change the hash function the dict code uses.
I have attached some code for a different hash key.
Under testing with 200,000 items in the dict. Test inserts, gets and then 
deletes all the items.
uuid keys: new hash +-9secs; current 75secs
int keys: new hash +-5secs;  current +-11mins


I can pass on the dict patches to implement this.

Nisan
>2) don't call any functions in signal handler that are not signal safe (you
>call gwthread_wakeup_all())
>3) check for 'file != NULL' in 'int store_save(Msg *msg)'. Because file !=
>NULL is our marker for enabled store-file support (store-file support is
>optional).
>
>
>Kalle Marjola wrote:
>
> > Attached a patch (against CVS) that greatly improves store-file:
> > It uses Dict instead of List to save messages to memory.
> > Moreover, acknowledged messages are immediately cleaned from
> > memory, instead of being done in a background thread (that thread
> > simple dumps current state to file to reduce startup-times).
> > And, above this all, now it is possible to define the frequency
> > how often this 'cleanup' is done..
> >
> > Thus it is far more powerful if queues are created, and I
> > greatly encourage all kannel instances to use it, at least
> > after it has gained adequate extra testing (I have done some
> > tests but more are naturally needed)
> >
> > As part of the patch, smskannel.conf is updated to enable
> > store-file - I know that lots of Kannel installation are based
> > on smskannel.conf which is then modified...
> >
> > Comments?
> > I can put this into CVS immediately after Stipe puts out 1.4.0,..
> >
>
>--
>Thanks,
>Alex


--------------------------------
www.clickatell.com
Any message, anywhere
Phone: +27 21 9487150

#define hashsize(n) ((unsigned long)1<<(n))
#define hashmask(n) (hashsize(n)-1)
#define mix(a,b,c) \
{ \
  a -= b; a -= c; a ^= (c>>13); \
  b -= c; b -= a; b ^= (a<<8); \
  c -= a; c -= b; c ^= (b>>13); \
  a -= b; a -= c; a ^= (c>>12);  \
  b -= c; b -= a; b ^= (a<<16); \
  c -= a; c -= b; c ^= (b>>5); \
  a -= b; a -= c; a ^= (c>>3);  \
  b -= c; b -= a; b ^= (a<<10); \
  c -= a; c -= b; c ^= (b>>15); \
}

unsigned long dict_hash( register unsigned char *k, register unsigned long  length)
{
   register unsigned long a,b,c,len;

   /* Set up the internal state */
   len = length;
   a = b = 0x9e3779b9;  /* the golden ratio; an arbitrary value */
   c = 0;           /* the previous hash value */

   /*---------------------------------------- handle most of the key */
   while (len >= 12)
   {
      a += (k[0] +((unsigned long)k[1]<<8) +((unsigned long)k[2]<<16) +((unsigned long)k[3]<<24));
      b += (k[4] +((unsigned long)k[5]<<8) +((unsigned long)k[6]<<16) +((unsigned long)k[7]<<24));
      c += (k[8] +((unsigned long)k[9]<<8) +((unsigned long)k[10]<<16)+((unsigned long)k[11]<<24));
      mix(a,b,c);
      k += 12; len -= 12;
   }

   /*------------------------------------- handle the last 11 bytes */
   c += length;
   switch(len)              /* all the case statements fall through */
   {
   case 11: c+=((unsigned long)k[10]<<24);
   case 10: c+=((unsigned long)k[9]<<16);
   case 9 : c+=((unsigned long)k[8]<<8);
      /* the first byte of c is reserved for the length */
   case 8 : b+=((unsigned long)k[7]<<24);
   case 7 : b+=((unsigned long)k[6]<<16);
   case 6 : b+=((unsigned long)k[5]<<8);
   case 5 : b+=k[4];
   case 4 : a+=((unsigned long)k[3]<<24);
   case 3 : a+=((unsigned long)k[2]<<16);
   case 2 : a+=((unsigned long)k[1]<<8);
   case 1 : a+=k[0];
     /* case 0: nothing left to add */
   }
   mix(a,b,c);
   /*-------------------------------------------- report the result */
   return c;
}