Re: charset transcoding question

Hi, Stipe

Stipe Tolj wrote:
 > Vjacheslav Chekushin wrote:
 >
 >>There is transcoding staff in wap-appl.c (static void return_reply()) and
 >>wml_compiler.c (some parsing routines, that do charset_convert).
 >>There are some bugs in this transcoding staff (I affraid - many bugs) but
 >>I want waste my time to clean this code.
 >>I just commented out this code from both modules because it seems unneeded to me
 >>at all.
 >>If someone can show me why trancoding is used and what phone doesn't work
 >>without it, I am ready to make patches for this.
 >
 >
 > AFAIS, Bruno did these changes in return_reply(). In the comment he
 > states it is "required" for the Sharp GX20?!

No, I undestood from comment that Sharp works with UTF-8. So
libxml2 do his job perfectly and sharp can work without transcoding.


 >
 > AFAIS, the block in retrun_reply() cares about transcoding the HTTP
 > response body to either UTF-8 (or ISO-Latin1) *if* the charset of the
 > body is *not* accepted by the device (indicated via corresponding HTTP
 > headers in WSP code).
 >
 > This is legitimate IMO, even while my comment above the block states,
 > that the HTTP server (and the corresponding application) should take
 > care of it. This can be considered as fall-back-sanity.
 >
 > So, you indicate there are bugs. Please report and let's get them out
 > of there if you consider there are some.
 >

Some bugs:

1. Let's take that original charset not UTF-8 and not ISO-8859-1 and
     phone supports ISO-8859-1

/* If device doesn't support body's charset but supports UTF-8, this
* block tries to convert body to UTF-8. */
if (octstr_case_compare(charset, octstr_imm("UTF-8")) < 0 &&
      !http_charset_accepted(device_headers, octstr_get_cstr(charset))) {
              /* convert to utf-8 if original charset is not utf-8
               * and device supports it */
	

/* convert to iso-8859-1 if original charset is not iso
               * and device supports it */
} else {
    if (octstr_case_compare(charset, octstr_imm("ISO-8859-1")) < 0 &&
       !http_charset_accepted(device_headers, octstr_get_cstr(charset))) {
    }
}

"else" cause must be processed but it will not be processed.
(First "if" is true,  but phone doesn't support UTF8, so content not converted)


2. Let's take simple file:

<?xml version="1.0" encoding="iso-8859-7"?>
<!DOCTYPE wml PUBLIC "-//WAPFORUM//DTD WML 1.1//EN"
"http://www.wapforum.org/DTD/wml_1.1.xml">
<wml>
<card id="main_card" title="LMT WAP">
          <p><a href="starts.php">LMT</a></p>
</card>
</wml>

(encoding iso-8859-7 doesn't supported by phone)

wapbox.log shows error

2003-10-07 15:58:45 [9] WARNING: WSP: Device doesn't support charset
<iso-8859-7> neither UTF-8
2003-10-07 15:58:45 [9] DEBUG: WSP: Converting from <text/vnd.wap.wml> to
<application/vnd.wap.wmlc>
2003-10-07 15:58:45 [9] INFO: WBXML: No wbxml version given, assuming 1.1

And compiled deck with iso-8859-7 goes to phone and phone shows error:
2003-10-07 15:58:45 [9] DEBUG:   data: 01 04 0a ...
01 - version 1
04 - Public identifier
0a - iso-8859-7
...


 > Stipe
 >

-- 
Vjacheslav Chekushin                                mailto:[email protected]
Latvian Mobile Phone Company                        http://www.lmt.lv