Re: enabling utf8 encoding for sax parer XML::SAX::Expat

thanks for the detailed answer, really helped me understand things better.
see inline for my answers.

On Tue, Apr 28, 2009 at 5:10 PM, Michael Ludwig <[email protected]> wrote:

> eyal edri schrieb:
>
> > i'm getting an error while parsing an xml with non-ascii chars: "not
> > well formed xml at line..... "
>
> The document content doesn't match the specified encoding. The specified
> encoding is the one in the XML declaration, like this:
>
>   <?xml version="1.0" encoding="Windows-1252"?>
>
> Where there is no XML declaration, the encoding must be either UTF-8 or
> UTF-16, and when UTF-16, this must be indicated by a BOM (byte-order
> mark).
>
> I guess your parser is correct, and your document is broken.
>
> >    1. i don't have control of the src xml, so i can't make them give
> >    it to me properly.
>
> If it's broken, it's broken. Should be easy to fix at the source.


can it be that since i'm using LWP::UserAgent , perl somehow changes the
encoding while d/l?

$userAgent->request($request, $file) ?

should i add any sort of HTTP header with encoding like the ones returned in
HTTP responses?

my only thought to check this is to run a sniffer and check what really goes
on the wire...

any ideas?

>
>
> Failing that, you could preprocess the file using some non-XML
> technology, like Perl, and hack the missing XML declaration in,
> which may or may not be sufficient to repair the document.
>
> >    - how can i tell the parser to open the xml in a certain encoding
> >    (despite the sender fucked up and didn't add the magic numbers) ?
>
> As explained above: fix the file.
>
> >    - should i use a different sax parser?
>
> You can try others, but I think Expat is okay.
>
> > and should i always use :encoding(UTF-8) when opening/writing files?
>
> When using open() to open a file encoded in UTF-8, yes; else no.
> Check the following, recorded on a Latin1 terminal:
>
> milu@colinux:~/ds > cat Kaese.pl
> use strict;
> use warnings;
> my $filename = shift or die 'Datei!';
> open my $fh, '<encoding(utf-8)', $filename;
> print <$fh>;
> close $fh;
> milu@colinux:~/ds > cat Kaese.iso88591.txt
> Käse
> milu@colinux:~/ds > cat Kaese.utf8.txt
> Käse
> milu@colinux:~/ds > perl Kaese.pl Kaese.utf8.txt
> Käse
> milu@colinux:~/ds > perl Kaese.pl Kaese.iso88591.txt
> utf8 "\xE4" does not map to Unicode at Kaese.pl line 5.
> K\xE4se
>
> Michael Ludwig
> _______________________________________________
> Perl-XML mailing list
> [email protected]
> To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
>



-- 
Eyal Edri

_______________________________________________
Perl-XML mailing list
[email protected]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs