Re: Handling funny characters in XML with XML::Simple XMLin()

This is a guess, but the situation may be WORSE than what mirod is saying.
Character sets in HTTP in general is a messy topic, but the specs are not
really exactly clear on some points as to what character set incoming data
is actually using. The user agent for example is coming from an HTTP header.
What character set is that header? In theory it should be ASCII IIRC but who
knows, certain user agents may be poorly designed... Beyond that in general
it is pretty problematic to know what is going on with user agent character
sets. NOWHERE in the HTTP spec for instance is there any way for a user
agent to indicate the character set of an incoming POST! URLs present an
even worse problem. There isn't even theoretically any way to know what
character sets they use except heuristics.

The upshot is that, unfortunately, you will always run into these sorts of
problems when attempting to mix HTTP logs with XML (or in a lot of other
situations too). You just cannot assume at any point that any particular
data coming from one of those logs is 'sane'. So anyone jamming the data
into an XML document has to be real careful.

The upshot being whoever's software is building the XML document needs to
clean their inputs. You could as you say preprocess the data your getting
and just clean out that sort of garbage, but it may be pretty hard to find
all of it under all circumstances, I don't know for sure what you might run
into.

There is another argument for preprocessing the data. Injection. Depending
on what you're going to ultimately do with the data you may find there can
be all sorts of nasty junk in there. bits of SQL injection attempts,
javascript intended by someone to be injected into web apps, etc. I consider
it mandatory to do a thorough cleanup on all web server logs with code
designed to santize them thoroughly before any other processing.

On Tue, Nov 25, 2008 at 6:46 AM, mirod <[email protected]> wrote:

> Mark Knoop wrote:
>
> > I use XML::Simple to read in XML style logs which then get processed.
> This
> > code standardises the procesing of XML logs from many different places.
> > Infrequently XML logs which are written by a 3rd party application (ie I
> > cannot do anything about the logs at source) contain a non ascii
> character
> > which breaks XMLin(). I have put this inside an eval so it does not kill
> the
> > application however I would really like to ignore this character so I can
> > process the rest of the information in the log as normal. I could
> preprocess
> > all the log files and strip out the offending character but this seems a
> bit
> > longwinded. Is there a way I can just get XMLin to ignore or handle the
> > character so I don;t have to jump through too many hoops to avoid losing
> the
> > odd set of data when this character is present? I looked at NumericEscape
> > but it seems this only works for writing out hashes to XML which contain
> > funny characters and cannot be used for handling funny characters on the
> way
> > in.
> >
> > An example of the data is:
> > <?xml version="1.0" encoding="UTF-8"?>
> > <CDR-ROOT><DATA>
> > <CDR  SessionID="200810051223202224961753020"
> > SessionStartTime="20081005112344+01" SessionEndTime="20081005112445+01"
> > SessionDuration="61" SessionStartSystemTime="1223202224"
> > SessionEndSystemTime="1223202285"  UserAgent=" @" />
> > </DATA></CDR-ROOT>
> > Where the UserAgent attribute contains the offending character.
> >
> > The error I get is
> >
> > not well-formed (invalid token) at line 3, column 219, byte 277 at
> > C:/Perl/lib/XML/Parser.pm line 187
> >
> > I'm hoping there is a trick I am missing...
> >
>
> There is no trick. The character is not valid, so the XML parser cannot
> process
> it. That's the end of the story as far as the parser is concerned.
>
> Now what happens is probably that the data is not in utf-8, despite what
> the XML
> declaration is pretending. You have to figure out in which encoding it is,
> and
> then either tell the parser (there is an XML::Parser option,
> ProtocolEncoding,
> as in your case XML::Simple is indeed using XML::Parser, to force the
> encoding,
> probably to ISO-8859-1 or one of the funny Windows encodings), or even
> better,
> convert the data to utf8, using for example iconv (which should be
> available on
> Windows).
>
> You are also allowed to yell at the data provider, which seems to send you
> wrong
> data.
>
> --
> mirod
> _______________________________________________
> Perl-XML mailing list
> [email protected]
> To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
>



-- 
The Wise adapts himself to the world. The Fool adapts the world to himself.
Therefore, all progress depends on the Fool.

_______________________________________________
Perl-XML mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs