Re: locale problem

On Sat, 8 Oct 2011 10:18:34 +0330
"Hamid A. Toussi" <[email protected]> wrote:
> On Thu, Oct 6, 2011 at 5:36 PM,  <[email protected]> wrote:
> > On Tue, Oct 05, 2011, Hamid A. Toussi wrote:
> >> Everything is working fine except in the case that we want to
> >> retrieve or manipulate non-English text (Farsi) in sql statements.
> >
> > 2.  What is the output of locale(1) on the working and non-working
> > systems?
> 
> I don't have locale on ARM

That might be part of the problem, see below.  

> > 3.  What iconv library, if any, are you using?
> 
> ./configure --host=arm-linux --target=arm-linux
> --prefix=/opt/installed/arm/freetds
> --with-odbc-nodm=/opt/installed/arm/unixODBC_2.3 --enable-static
> --enable-msdblib --with-libiconv-prefix=/opt/fara0/iconv/

Good.

> >4.  What encoding is used on the server?  Are these nvarchar
> >(UCS2) data, or varchar?  If varchar, what collation?
> 
> It is text.

OK, but what collation? 

> > 6.  What iconv messages do you see in each machine's TDSDUMP log?
> on ARM:
> iconv.c:330:tds_iconv_open(0x29538, ISO-8859-1)
> iconv.c:337:error: tds_iconv_init() returned 1; could not find a name
> for ISO 88 59-1 that your iconv accepts.

That's a problem.  You need to verify, very carefully, your iconv
installation on ARM, and to make double sure it's being used by
FreeTDS.  The above messages will not occur if iconv(1) works and GNU
libiconv is linked to FreeTDS.  

1.  Determine the encoding on the server.  

2.  Create a file of a sample phrase.  The SQL:

	select top 1 cast(FARSI as varbinary) from tablename

(substituting the right column and table names) will display the data
as hexadecimal.  

The file you create will have Farsi text using the server's encoding,
perhaps Windows-1256.  Use hexdump(1) to verify it, character by
character, against a chart that documents the encoding.  

3.  Test first on i386, where we know iconv works.  

	$ iconv -f FARSI -t UTF8 farsi.txt 

(subtituting GNU's name of the encoding for "FARSI") should display the
phrase, because your locale is UTF-8. You will need to be using a
terminal capable of UTF-8 and have Farsi fonts installed, but I'm
guessing that's already done. By the way 

	$ cat farsi.txt 

will *not* display the phrase because the locale is UTF8.  If it does,
someone's confused, possibly both of us.  

4.  	$ iconv -f FARSI -t UTF8 farsi.txt | hexdump -C 

displays the file in UTF-8 as hex, for later comparison to ARM.  

5.  Copy the file to the ARM box.  Repeat 

	$ iconv -f FARSI -t UTF8 farsi.txt | hexdump -C 

That should produce results identical to i386.  If it doesn't, iconv is
broken.  Come back when it's fixed.  ;-)

6.  How to display farsi.txt on ARM?  You need a locale afaik.
Regardless, if you're aiming at UTF-8, you should be able to send it to
the terminal:

	$ iconv -f FARSI -t UTF8 farsi.txt 

Make sure that works before proceeding with FreeTDS.  

At this point, you know:

- the encoding used on the server
- the encoding used on the client
- libiconv can convert between the two encodings
- the client can display your text 

7.  Now we can ask FreeTDS to use libiconv.  TDSDUMPCONFIG will show
what client charset will be used.  TDSDUMP will show iconv messages (as
you know) just as it does on your i386.  

I would not expect TDSDUMP with tsql to show iconv errors at this
point.  That would indicate some kind of linking problem.  I would use
ldd(1) to verify the binary if possible, else add flags to the linker
command to print the library being used.  

Absent iconv errors in TDSDUMP and assuming your client charset reports
the encoding you displayed successfully in step #6, I would expect tsql
to display your Farsi text.  

HTH.  

--jkl