Re: Reading and writing files character encoding

Tony,

if the xml declaration does not contain the encoding, it uses the  
default encoding, which is UTF-8.
Here's a sample for just dumping the stream to a file :

InputStream is = null;
OutputStream os = null;
try {
     is = urlToVisit.openStream()
     os = new FileOutputStream("file.xml");
     final byte[] buf = new byte[1024 * 8];
     for (int justRead=is.read(buf); justRead != -1; justRead=is.read 
(buf)) {
         os.write(buf, 0, justRead);
     }
} finally {
     if (is != null) {try {is.close();} catch(IOException e)  
{e.printStacktrace();}}
     if (os != null) {try {os.close();} catch(IOException e)  
{e.printStacktrace();}}
}

Note that there is no need to decode and then -re-encode the  
characters, since you're staying on the byte level. This is also much  
faster.

To parse the XML file, I would use one of the many available parsers.  
They take care of your encoding problems for you.

Now, your problem lies probably in your last steps, where you  
retrieve data from the DB and display the stuff in a browser. You  
will need to tell your browser, what character set you are using. So  
the question is: how do you write the content that the browser  
displays? Is it HTML, XML or plain text? Also, are you sure your  
mysql instance supports the right character set?

-hendrik

On Jan 8, 2007, at 12:55 , Tony Spencer wrote:

> Thanks to everyone for all the help thus far.  I guess I'm a little  
> closer to the solution.
>
> Here is exactly what I'm doing:
> Downloading a large XML feed nightly and making a local copy to  
> flat file.  Then we parse the local copy of the XML file and write  
> records to mysql. Storing it to flat file is only a courtesy to the  
> feed provider so we don't keep hitting their server each time we  
> need to parse the XML.
>
> The feed didn't specify a encoding in the first line but when I did  
> the following, I get a nice looking copy of the XML file with all  
> the correct spanish characters:
>
> BufferedReader br = new BufferedReader(new InputStreamReader 
> (urlToVisit.openStream(), "ISO-8859-1"));
> .....
> Writer out = new OutputStreamWriter(new FileOutputStream 
> (filepathToStore), "ISO-8859-1");
>
> Example text in the file:  Nueva Andalucía
>
> BUT, when I parse the file and write the content to records in  
> mysql and then retrieve that data and display it in a browser, the  
> spanish characters are mangled. (When I parse the file I again  
> specify "ISO-8859-1" in my inputstreamreader).
>
> Example text seen in browser:  Nueva AndalucÌ?a
>
> So I know the encoding of the feed to be ISO-8859-1 but don't  
> understand how to convert it or otherwise deal with it so that it  
> can be safely written and retrieved in a DB.  I feel like it needs  
> to be converted to unicode but I tried writing my flat file with  
> the following and that didn't solve the problem (maybe this isn't  
> how you convert to unicode?):
>
> Writer out = new OutputStreamWriter(new FileOutputStream 
> (filepathToStore), "UTF-8");
>
>
> On 1/7/07, Hendrik Schreiber <hs-doAduxQln/dWk0Htik3J/[email protected]> wrote: Tony,
>
> if you are just dumping the text to a file, use streams, not reader/
> writers. That way you do not decode and encode the text.
> If you need to access the actual text, you will have to find out the
> character encoding (either in the HTP content-type header or the
> first line of the XML file), then use an InpuStreamReader with the
> charset specified. To write, you will need to use an
> OutputStreamWriter with the charset specified. AFAIR FileWriter
> unfortunately does not let you specify the charset and is therefore
> fairly useless.
>
> -hendrik
>
>
> On Jan 7, 2007, at 17:55 , Tony Spencer wrote:
>
> > Hi,
> > I am attempting to read a simple XML feed over HTTP and write it to
> > my local disk.  The feed has some spanish in it so there are non-US
> > characters present.  Currently I use a BufferedReader and
> > InputStream to read the feed and I write it to disk with
> > FileWriter.  The problem is that the non-US characters (such as the
> > second character in this provice http://en.wikipedia.org/wiki/
> > Malaga) are mangled in the local copy that is written.
> >
> > Can someone explain how to deal with different character encodings
> > in cases such as this?  Do you need to know the encoding of the
> > content you are receiving?
> >
> > Thanks,
> > Tony
> > _______________________________________________
> > Juglist mailing list
> > [email protected]
> > http://trijug.org/mailman/listinfo/juglist_trijug.org
>
>
> _______________________________________________
> Juglist mailing list
> [email protected]
> http://trijug.org/mailman/listinfo/juglist_trijug.org
>
> _______________________________________________
> Juglist mailing list
> [email protected]
> http://trijug.org/mailman/listinfo/juglist_trijug.org