bogofilter-0.95.1 - New Current Release

Greetings,

This release fixes some problems with the unicode support in 0.95.0.
Most notably RFC2047 encoded tokens are converted to unicode (when
appropriate) and bogotune problems with unicode/raw wordlists have been
corrected.

Enjoy!

David

########################################################################

Files are available at http://sourceforge.net/projects/bogofilter for
download.

Here are the md5sums for the release:

6294cb374a0ed4fd467fd8131ed0c3e6  bogofilter-0.95.1-1.i586.rpm
b371d40b7f5e76d1f5738128ef296c74  bogofilter-0.95.1-1.src.rpm
362cecc36d9e82e8fd62496cb7231faf  bogofilter-0.95.1.tar.bz2
f58fa919ad9195ef9d8443c46e7b532d  bogofilter-0.95.1.tar.gz
4f7123851091c533538b4375585f6335  bogofilter-static-0.95.1-1.i586.rpm

########################################################################

Read the NEWS file for a more detailed list of changes since 0.95.0.

Below are the combined RELEASE.NOTES for the unicode releases - 0.95.x


### 0.95.1 ###

Some of the 'make check' scripts use '--unicode=3Dno' and/or
'--unicode=3Dyes' options.  These scripts will fail if bogofilter is
configured in a non-default manner, i.e. if --disable-unicode or
--enable-unicode is specified.

### 0.95.0 ###

This release supports unicode (utf-8).  A new meta-token .ENCODING has
been added to the wordlist so that bogofilter can determine if it's
using unicode or not.  A value of 1 indicates raw storage and 2
indicates utf-8 encoded tokens.  Bogofilter checks for this meta-token
and converts incoming text to utf-8 as appropriate. =20

Command line options "--unicode=3Dyes" and "--unicode=3Dno" can be used.

    =B0 With bogofilter, they control encoding of newly created
      databases.

    =B0 With bogoutil, during wordlist maintenance they change wordlist
      to/from unicode.

    =B0 For bogolexer, they viewing parser results in new and
      old modes

./configure options allow bogofilter customization.

    =B0 "./configure --unicode=3Dyes" will _always_ operate in unicode mode

    =B0 "./configure --unicode=3Dno"  will _never_ operate in unicode mode

Wordlists can be converted from raw storage to unicode using:

    bogoutil -d wordlist.db > wordlist.raw.txt
    iconv -f iso-8859-1 -t utf-8 < wordlist.raw.txt > wordlist.utf8.txt
    bogoutil -l wordlist.db.new < wordlist.utf8.txt

or:

    bogoutil --unicode=3Dyes -m wordlist.db

Wordlists can be converted from unicode to raw storage using:

    bogoutil -d wordlist.db > wordlist.utf8.txt
    iconv -f utf-8  -t iso-8859-1 < wordlist.utf8.txt > wordlist.raw.txt
    bogoutil -l wordlist.db.new < wordlist.raw.txt

or:

    bogoutil --unicode=3Dno -m wordlist.db

For a wordlist based on a different charset, for example CP866 or
KOI8-R, use that charset in dump/convert/load command sequences above.

For a wordlist containing tokens from multiple languages, particularly
non-european languages, the conversion methods described above may not
work well for you.  Building a new wordlist (from scratch) will likely
work better as the new wordlist will be based solely on unicode.

_______________________________________________
Bogofilter-announce mailing list
[email protected]
http://www.bogofilter.org/mailman/listinfo/bogofilter-announce