Re: [RFC] wml->wmlc default character mangling

Remember that si, sl and ota would use conversions, too. Si I think that
conversions should be into a separate module. Xml_shared.c ?

Aarno

On 21.3.2005, at 22:28, Paul P Komkoff Jr wrote:

> Replying to Stipe Tolj:
>> Hi list,
>>
>> ok, following the thread frrom Rune Saetre [Msg-Id:
>> <[email protected]>] I come up with
>> following status-quo conclusion while reviewing the code: (HEAD in 
>> CVS):
>
> I was staring at the code for hour or two. This all needs heavy
> cleanup, concentrating all conversion stuff in wap-appl and removing
> it from wml-compile entirely. Including stripping of preamble in case
> of successful recoding. But I wrote on it here at least 2 times
> already.
>
> See attachemnt for how I am doing it now (skip gwlib/ part if you
> don't like it).
>
>
>>   * entering gw/wap-appl.c:return_reply() we receive the HTTP response
>>   (headers and body)
>>   *  if status was OK (HTTP 200) we perform some character convert to 
>> utf-8
>>   for various content-types, see line 842.
>>   * in fact we check via find_charset_encoding() if there has been an
>>   encoding in the XML preamble. Hence in the body of the response. If 
>> there
>> is not, we assume utf-8 as character encoding. (BTW, no implication 
>> from
>> the content-type HTTP header yet).
>>   * if XML preamble gives a non-utf-8 encoding, but the device 
>> indicates
>>   that it supports utf-8, we do convert, line 866.
>>   * we do the same as a "backup" for iso-8859-1 in case utf-8 is not
>>   supported by the device.
>>   * then we run into the conver_content() routine that does the 
>> lookup in
>>   the converter array, bringing us up to convert_wml_to_wmlc() and 
>> hence to
>> wml_compile().
>>   * so far so good
>>   * wml_compile() seems to do the mangling "once again" ?!
>     ^^^^^ double-encoding bug is triggered by passing already
> converted xml with preamble left intact.
>>   * gw/wml_compiler.c:351: if passed charset var is not utf-8 then 
>> call
>> gw/xml_shared.c:set_charset(), which converts to utf-8.
>>   * now we obey the rules: first: pick XML preamble encoding value, 
>> second:
>> pick given HTTP header value, third: fallback to utf-8.
>>   * and we come up to the libxml2 parsing calls.
>>
>> ok, so where are the things in here to be changed (from policy site of
>> view) if any?
>
> -- 
> Paul P 'Stingray' Komkoff Jr // http://stingr.net/key <- my pgp key
>  This message represents the official view of the voices in my head
> <charset>