Re: UTF-8 vs ISO-8859-1 (Latin1)

Hi Yusniel,

Yusniel Hidalgo Delgado wrote:
> Hi. I have a problem with my mnogoserach 3.3.9 and I need your help. My 
> database encode is UTF-8. Mnogoserach is configurated with the following 
> settings:
> 
> LocalCharset: UTF-8
> RemoteCharset: UTF-8
> VaryLang: ¨es en¨
> 
> Everything was fine, but recently, I get the following error when 
> indexer -R is running:
> 
> ¨PQexecPrepared: ERROR:  secuencia de bytes no válida para
> codificación «*UTF8*»: 0xf7848c93#012HINT
> 
> I am using PostgreSQL Server 8.3.

There is a small problem in mnoGoSearch: when a remote page claims to be
UTF-8 but it is in fact ISO-8859-1, then mnoGoSearch does not check 
well-formedness of the text and tries to insert it into the database
as is, so as a result error happen on PostgreSQL level.

Can you please send me the URL of the document which made indexer fail?
I will check if this is the case.

> 
> PD: In my sites I have multiples language, English and Spanish for example.
> 
> What charset is the best recomended?. UTF-8 or ISO-8859-1? Thanks for 
> your time.
> 
> 

If you need only English and Spanish, then ISO-8859-1 is enough.

However, if you also index documents with scripts other than Latin,
e.g. Cyrillic, Hebrew, Arabic, Asian, then UTF-8 is the best choice.