Re: Handling funny characters in XML with XML::Simple XMLin()

Mark Knoop wrote:

> I use XML::Simple to read in XML style logs which then get processed. This 
> code standardises the procesing of XML logs from many different places. 
> Infrequently XML logs which are written by a 3rd party application (ie I 
> cannot do anything about the logs at source) contain a non ascii character 
> which breaks XMLin(). I have put this inside an eval so it does not kill the 
> application however I would really like to ignore this character so I can 
> process the rest of the information in the log as normal. I could preprocess 
> all the log files and strip out the offending character but this seems a bit 
> longwinded. Is there a way I can just get XMLin to ignore or handle the 
> character so I don;t have to jump through too many hoops to avoid losing the 
> odd set of data when this character is present? I looked at NumericEscape 
> but it seems this only works for writing out hashes to XML which contain 
> funny characters and cannot be used for handling funny characters on the way 
> in.
> 
> An example of the data is:
> <?xml version="1.0" encoding="UTF-8"?>
> <CDR-ROOT><DATA>
> <CDR  SessionID="200810051223202224961753020" 
> SessionStartTime="20081005112344+01" SessionEndTime="20081005112445+01" 
> SessionDuration="61" SessionStartSystemTime="1223202224" 
> SessionEndSystemTime="1223202285"  UserAgent="
@" />
> </DATA></CDR-ROOT>
> Where the UserAgent attribute contains the offending character.
> 
> The error I get is
> 
> not well-formed (invalid token) at line 3, column 219, byte 277 at 
> C:/Perl/lib/XML/Parser.pm line 187
> 
> I'm hoping there is a trick I am missing...
> 

There is no trick. The character is not valid, so the XML parser cannot process 
it. That's the end of the story as far as the parser is concerned.

Now what happens is probably that the data is not in utf-8, despite what the XML 
declaration is pretending. You have to figure out in which encoding it is, and 
then either tell the parser (there is an XML::Parser option, ProtocolEncoding, 
as in your case XML::Simple is indeed using XML::Parser, to force the encoding, 
probably to ISO-8859-1 or one of the funny Windows encodings), or even better, 
convert the data to utf8, using for example iconv (which should be available on 
Windows).

You are also allowed to yell at the data provider, which seems to send you wrong 
data.

-- 
mirod
_______________________________________________
Perl-XML mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs