Re: UnicodeDecodeError - problem identified. How to solve?

Thiago Lima <thiagomadeira <at> gmail.com> writes:

> 
> 
> Hi,
>  
> I have the UnicodeDecodeError in my tmda-cgi for some long time. Everytime a
user complains I delete the message manualy.  After googleing for the problem
I´ve got no solutions for it. Nobody could describe the problem corretly and the
development team could not solve it.
> 
>  
> After analizing many headers with the problem, I think I solved the problem.
TMDA-CGI exit with error when the Subject uses a different encoding than the
Content-Type header or if there´s any problem with the 'charset' field in
Content-Type header.
> 
>  
> Exemples that crashes TMDA-CGI: 
>  
> -- this one says to use utf-8 but uses latin-1 in the subject. 
> 
> Subject: Curso de Introdução ao mercado  de Açoes
> Content-Type: text/html;
>         charset="utf-8"
> 
>  
> -- this one uses utf, but the charset is strange
> 
> 
> Subject:
=?UTF-8--utf?Q?Seu_pedido_(pensomail_antispam_-_confirme_mensagem_para_mb?= 
=?UTF-8--utf?Q?alasso <at> levysalomao.com.br.)_n=C3=A3o_foi_reconhecido!?=
> 
> 
> 
> 
> Content-Type: text/plain; charset="UTF-8--utf" 
>  
> 
>  -- a really don´t know -1252 charset. think it does not exist.
> 
> Content-Type: text/html; charset="-1252"
> 
>  
> Does that make sense ? There´s any way to fix it with those samples? Need more
samples?  I can´t program in python, otherwise I´d look into the code. :(
>  
>  
> 
> 


Hi!

I have not found a solution for this issue yet, so, as a workarround, I'm
catching the UnicodeDecodeError and replacing the malformed text.  To do so, I
edited Unicode.py and replaced this:

      # Mark it and use the fallback
        else:
          return "(%s) %s" % (CharSet,
            TranslateToUTF8(PVars[("General", "CSEncoding")], Str, Errors))
        Decoder = Lib.Codec().decode

with this:

       # Mark it and use the fallback
        else:
          try:
            return "(%s) %s" % (CharSet,
              TranslateToUTF8(PVars[("General", "CSEncoding")], Str, Errors))
          except UnicodeDecodeError:
            return "(Malformed text. Unable to display) [%s]" % (CharSet)
        Decoder = Lib.Codec().decode

Doing so, the malformed text that appears at message's from and/or subject are
replaced by the string "(Malformed text. Unable do display) [charset]".  The
user can still click on this string at the pending to open it, release the
message and do all thinks they are supose to do with a regular message. The only
difference is that the malformed From and/or Subject is replaced at pending list
and message's detail screen.

Let me know if it helped someone else.

Cheers,

Rodolfo





------------------------------------------------------------------------------
Lotusphere 2011
Register now for Lotusphere 2011 and learn how
to connect the dots, take your collaborative environment
to the next level, and enter the era of Social Business.
http://p.sf.net/sfu/lotusphere-d2d
_______________________________________________
tmda-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/tmda-users