Webboard: Alternative SOUNDEX for wrdstat for other languages than english possible?

Author: Alexander Barkov
Email: [email protected]
Message:
Hi,



> Hi

> 

> Sure! I'll try implementing it that way and let you know about the progress. If you don't mind, I'd contact you by email about it. Would be very nice to contribute to your engine! :-)



Great. Thank you very much.



Let me give some tips.



As the German soundex function will have to deal with

non-ASCII letters, I suggest to use cs->mb_wc() 

function to scan characters from the source string.

You cab fine some examples how to use cs->mb_wc() in unidata.c,

e.g. in UdmStrToLower().



Also, I've made a template program for your convenience:



#include <stdio.h>

#include <udmsearch.h>

static int

UdmSoundexCologne(UDM_CHARSET *cs, char *dst, const char *src, size_t srclen)

{

  int mblen;

  const char *srcend= src + srclen;

  UDM_UNIDATA *unidata= udm_unidata_default;

  UDM_CONV cnv;

  UdmConvInit(&cnv, cs, &udm_charset_sys_int, 0);

  for ( ; src <  srcend ; src += mblen)

  {

    int wc[16];

    mblen= cs->mb_wc(&cnv, cs, wc,

                     (const unsigned char *) src,

                     (const unsigned char *) srcend);

    if (mblen <= 0)

      break;

    printf("Unicode=%04X len=%d\n", wc[0], mblen);

    switch (wc[0]= UdmUniToLower(unidata, wc[0]))

    {

      case 0xE4: *dst++= 'a'; *dst++= 'e'; break;           

      case 0xF6: *dst++= 'o'; *dst++= 'e'; break;           

      case 0xDC: *dst++= 'u'; *dst++= 'e'; break;           

      case 0xDF: *dst++= 's'; *dst++= 's'; break;

      default: if (wc[0] < 0x80 && wc[0] >= 'a' && wc[0] <='z') *dst++= wc[0];

    }

  }

  *dst= '\0';

}

int main(int ac, char **av)

{

  char src[]= "AB#2" "\xC3\x84\xC3\x96\xC3\x9C\xC3\x9F" "abc"/* AB#2ÄÖÜßabc */;

  UDM_CHARSET *cs= UdmGetCharSet("utf-8");

  char dst[32];

  UdmSoundexCologne(cs, dst, src, sizeof(src) - 1);

  printf("%s\n", dst);

}







Use this Makefile to compile the program on Unix:



BASE=/usr/local/mnogosearch33

CFLAGS+= -I$(BASE)/include -L$(BASE)/lib -lmnogosearch -lmnogocharset

all: soundex



(perhaps you'll need to adjust the BASE value).





This program already does the preparatory step:

- transcripts German umlauts and SZ to ASCII

- removes non-alphabetic characters

- converts everything to lower case

(Note, you'll have to adjust the algorithm to operate with

lower case letters rather than upper case letters)





I hope adding the other steps on top on this template program

should now be easier.





> 

> Bye

> 



Reply: <http://www.mnogosearch.org/board/message.php?id=21252>