Re: [website-dev] native language webcontent developers please review your filenames for invalid non UTF-8 characters
Bernd Eilers <[email protected]> Tue, 23 Nov 2010 11:33:56 +0100
| Newsgroups | gmane.comp.openoffice.devel.website |
|---|---|
| Message-ID | <[email protected]> |
--------------000206060805060408060300 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 23.11.2010 11:07, Stefan Taxhet wrote: > Hi, > > On 22.11.2010 17:22, Bernd Eilers wrote: >> >> >> Hi WebContent developers and esp. those from native language projects! >> >> I recently stumbled over a few filenames in OpenOffice.org's webcontent >> which have invalid non-UTF-8 characters in their filenames. > > Do you have examples? > Well the problem is that this happens when doing an "svn import" with the content of our web projects but than there the "svn import" brakes at just the very first occurance of such an invalid file, so you do not get all errors by one try you only would get all if you fix and iterate until no error occurs anymore. I already did fix those very first occurances in this case as I could guess there how the files should be named or there have been duplicates using the correct encoding besides them so I could just delete the one using the wrong encoding etc. But when just looking through some randomly picked directories I have seen at least two more occurances where I could not guess which char should have been really there. That was in the fr project fr/www/Documentation/Gallery two directories starting with the letters "fl" and than followed by some invalid char. To fix something like that someone speaking the language is needed, that´s why I asked for review of native lang webcontent developers. > Andrea's pattern seems to be a bit broad. It would be great if we > could point at the problematic files... > Yes of course it is to broad especially when it comes to asian languages which might be using non ASCII but still valid UTF-8 chars. The best test would be to try an "svn import" with the whole project while having LANG=en-US.UTF-8 environment variable set. If that fails there is at least one error if it doesn´t fail everythings fine. In case of an error than fix the error in the CVS Repository and try the "svn import" again. If it fails .... > Greetings > Stefan Greetings, Bernd > > >> The character encoding to be used for webconent checked into >> OpenOffice.org´s webcontent repository is UTF-8. Please make sure to use >> an UTF-8 locale when checking in files with non-us-AscII chars. For >> example if you are in france and are using some Unix OS set >> LANG=fr.UTF-8 and not LANG=fr.ISO8859-15. GUI CVS Clients used on >> Windows often allow to specify the encoding to be used explicitly. >> >> Filenames with other encodings will not work and what is even worse they >> do create a big problem when moving OpenOffice.org to the new kenai >> based infrastructure. >> While CVS does not care much about invalid chars in filenames subversion >> which will be used on the new infrastructure does treat those filenames >> as errornous and as a result will not import the whole project at all. >> >> Could native language projects webcontent developers please review their >> webcontent and change anything that is currently not UTF-8 compliant. >> And that means not only to copy the broken files to new valid ones but >> also deleting the broken filenames from the CVS repository! >> >> For example there are 2 broken directory names in >> fr/www/Documentation/Gallery starting with the letters "fl" and than >> some non UTF-8 encoded char. >> >> Kind regards, >> Bernd Eilers >> >> -- >> >> <http://www.oracle.com/> >> Bernd Eilers | Software Engineer >> Phone: +49 40 23 646 967 >> >> ORACLE Deutschland B.V. & Co. KG | Nagelsweg 55 | 20097 Hamburg >> >> ORACLE Deutschland B.V. & Co. KG >> Hauptverwaltung: Riesstr. 25, D-80992 München >> Registergericht: Amtsgericht München, HRA 95603 >> >> Komplementärin: ORACLE Deutschland Verwaltung B.V. >> Rijnzathe 6, 3454PV De Meern, Niederlande >> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697 >> Geschäftsführer: Jürgen Kunz, Marcel van de Molen, Alexander van der Ven >> >> <http://www.oracle.com/commitment> >> >> >> >> Oracle is committed to developing practices and products that help >> protect the environment >> >> >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > -- <http://www.oracle.com/> Bernd Eilers | Software Engineer Phone: +49 40 23 646 967 ORACLE Deutschland B.V. & Co. KG | Nagelsweg 55 | 20097 Hamburg ORACLE Deutschland B.V. & Co. KG Hauptverwaltung: Riesstr. 25, D-80992 München Registergericht: Amtsgericht München, HRA 95603 Komplementärin: ORACLE Deutschland Verwaltung B.V. Rijnzathe 6, 3454PV De Meern, Niederlande Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697 Geschäftsführer: Jürgen Kunz, Marcel van de Molen, Alexander van der Ven <http://www.oracle.com/commitment> Oracle is committed to developing practices and products that help protect the environment --------------000206060805060408060300 Content-Type: multipart/related; boundary="------------070305030709000006000803" --------------070305030709000006000803 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head> <meta content="text/html;charset=UTF-8" http-equiv="Content-Type"> </head> <body bgcolor="#ffffff" text="#000000"> On 23.11.2010 11:07, Stefan Taxhet wrote: <blockquote cite="mid:[email protected]" type="cite">Hi, <br> <br> On 22.11.2010 17:22, Bernd Eilers wrote: <br> <blockquote type="cite"><br> <br> Hi WebContent developers and esp. those from native language projects! <br> <br> I recently stumbled over a few filenames in OpenOffice.org's webcontent <br> which have invalid non-UTF-8 characters in their filenames. <br> </blockquote> <br> Do you have examples? <br> <br> </blockquote> Well the problem is that this happens when doing an "svn import" with the content of our web projects but than there the "svn import" brakes at just the very first occurance of such an invalid file, so you do not get all errors by one try you only would get all if you fix and iterate until no error occurs anymore.<br> <br> I already did fix those very first occurances in this case as I could guess there how the files should be named or there have been duplicates using the correct encoding besides them so I could just delete the one using the wrong encoding etc.<br> <br> But when just looking through some randomly picked directories I have seen at least two more occurances where I could not guess which char should have been really there.<br> That was in the fr project fr/www/Documentation/Gallery two directories starting with the letters "fl" and than followed by some invalid char.<br> To fix something like that someone speaking the language is needed, that´s why I asked for review of native lang webcontent developers.<br> <br> <blockquote cite="mid:[email protected]" type="cite">Andrea's pattern seems to be a bit broad. It would be great if we could point at the problematic files... <br> <br> </blockquote> Yes of course it is to broad especially when it comes to asian languages which might be using non ASCII but still valid UTF-8 chars.<br> <br> The best test would be to try an "svn import" with the whole project while having LANG=en-US.UTF-8 environment variable set. If that fails there is at least one error if it doesn´t fail everythings fine. In case of an error than fix the error in the CVS Repository and try the "svn import" again. If it fails ....<br> <blockquote cite="mid:[email protected]" type="cite">Greetings <br> Stefan <br> </blockquote> <br> Greetings,<br> Bernd<br> <blockquote cite="mid:[email protected]" type="cite"><br> <br> <blockquote type="cite">The character encoding to be used for webconent checked into <br> OpenOffice.org´s webcontent repository is UTF-8. Please make sure to use <br> an UTF-8 locale when checking in files with non-us-AscII chars. For <br> example if you are in france and are using some Unix OS set <br> LANG=fr.UTF-8 and not LANG=fr.ISO8859-15. GUI CVS Clients used on <br> Windows often allow to specify the encoding to be used explicitly. <br> <br> Filenames with other encodings will not work and what is even worse they <br> do create a big problem when moving OpenOffice.org to the new kenai <br> based infrastructure. <br> While CVS does not care much about invalid chars in filenames subversion <br> which will be used on the new infrastructure does treat those filenames <br> as errornous and as a result will not import the whole project at all. <br> <br> Could native language projects webcontent developers please review their <br> webcontent and change anything that is currently not UTF-8 compliant. <br> And that means not only to copy the broken files to new valid ones but <br> also deleting the broken filenames from the CVS repository! <br> <br> For example there are 2 broken directory names in <br> fr/www/Documentation/Gallery starting with the letters "fl" and than <br> some non UTF-8 encoded char. <br> <br> Kind regards, <br> Bernd Eilers <br> <br> -- <br> <br> <a class="moz-txt-link-rfc2396E" href="http://www.oracle.com/"><http://www.oracle.com/></a> <br> Bernd Eilers | Software Engineer <br> Phone: +49 40 23 646 967 <br> <br> ORACLE Deutschland B.V. & Co. KG | Nagelsweg 55 | 20097 Hamburg <br> <br> ORACLE Deutschland B.V. & Co. KG <br> Hauptverwaltung: Riesstr. 25, D-80992 München <br> Registergericht: Amtsgericht München, HRA 95603 <br> <br> Komplementärin: ORACLE Deutschland Verwaltung B.V. <br> Rijnzathe 6, 3454PV De Meern, Niederlande <br> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697 <br> Geschäftsführer: Jürgen Kunz, Marcel van de Molen, Alexander van der Ven <br> <br> <a class="moz-txt-link-rfc2396E" href="http://www.oracle.com/commitment"><http://www.oracle.com/commitment></a> <br> <br> <br> <br> Oracle is committed to developing practices and products that help <br> protect the environment <br> <br> <br> <br> </blockquote> <br> <br> --------------------------------------------------------------------- <br> To unsubscribe, e-mail: <a class="moz-txt-link-abbreviated" href="mailto:[email protected]">[email protected]</a> <br> For additional commands, e-mail: <a class="moz-txt-link-abbreviated" href="mailto:[email protected]">[email protected]</a> <br> <br> <br> </blockquote> <br> <br> <div class="moz-signature">-- <br> <meta http-equiv="CONTENT-TYPE" content="text/html; "> <p><a href="http://www.oracle.com/" target="_blank"><img src="cid:[email protected]" name="Grafik1" align="bottom" border="0" height="26" width="114"></a><br> <font color="#666666"><font face="Verdana, Arial, Helvetica, sans-serif"><font style="font-size: 8pt;" size="1">Bernd Eilers | Software Engineer<br> Phone: +49 40 23 646 967 <br> </font></font></font></p> <p><font style="font-size: 8pt;" size="1"><font color="#666666"><font face="Verdana, Arial, Helvetica, sans-serif">ORACLE Deutschland B.V. & Co. KG | Nagelsweg 55 | 20097 Hamburg</font></font></font></p> <p><font style="font-size: 8pt;" size="1"><font color="#666666"><font face="Verdana, Arial, Helvetica, sans-serif">ORACLE Deutschland B.V. & Co. KG<br> Hauptverwaltung: Riesstr. 25, D-80992 München<br> Registergericht: Amtsgericht München, HRA 95603<br> <br> Komplementärin: ORACLE Deutschland Verwaltung B.V.<br> Rijnzathe 6, 3454PV De Meern, Niederlande<br> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697<br> Geschäftsführer: Jürgen Kunz, Marcel van de Molen, Alexander van der Ven </font></font></font> </p> <table border="0" cellpadding="0" cellspacing="0" width="100%"> <tbody> <tr> <td valign="top" width="44"> <p align="left"><a href="http://www.oracle.com/commitment" target="_blank"><img src="cid:[email protected]" name="Grafik2" align="absbottom" border="0" height="28" width="44"></a></p> </td> <td width="100%"> <p align="left"><font color="#4b7d42"><font face="Verdana, Arial, Helvetica, sans-serif"><font style="font-size: 6pt;" size="1">Oracle is committed to developing practices and products that help protect the environment</font></font></font></p> </td> </tr> </tbody> </table> <p><br> <br> </p> </div> </body> </html> --------------070305030709000006000803 Content-Type: image/gif; name="Grafik1" Content-Transfer-Encoding: base64 Content-ID: <[email protected]> Content-Disposition: inline; filename="Grafik1" R0lGODlhcgAaAMQAAP8QEP/w8P/Q0P/g4P8wMP9gYP8gIP+goP9QUP+QkP+wsP9wcP+AgP9A QP/AwP8AAP///wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA ACH5BAAAAAAALAAAAAByABoAAAX/ICSOZGmeaKqubOu+cCzPdC0GCqPvvBKUDp5wp1gFdIPV IDfcOQa6A8yRaO4gweQoUHh4v2Bw4SdihM9egyBF8ALIpkEXDR44vA2XwkD/IgYGAAZbbX1n BGRmhl8AWiUJYicCAIsPDFh4LXOGjQ4NAAhrEIUPBQcOqKmoB5sEZV4JqqoHhQVxlGAOJQG4 AAunsqg/dw95KwdfBgzBqKIEBMaQDwCiKpOwEIq6KANuJg14igZwEHPULMTGKbxetjF8D1It xK7aKl8lCl8DAfALW1+2rUi3QtqgGN2mwYCXzYvAON624LoEQZ9DEfRcEFSBIE2DjyBDfnyI qRgMcA8a/z5AZQWXOxEdHxwUgfIgsX/zMqlAWclVCQFefLpAqdJBAFJn3owgtpJEQkslEWjU mYLnomokcDlS4s0eFzQNqvXDEyzmAwFPyaXYmGJBuxlzELWIacueiE0AFGyF4HbaNTquUEpF RxUFUC9YXQzARYAkEJSNipKQ9kDu0i9SKKNZ9iXswMIoYgI4oJYFskpgzkkmcdrviELqmPVt 1HpRAaaLqCE1lBgjvEpqRtgl8Tevygd7TcCuiKvPm8OVEkAIYLYPThSsRIpEQJr1x94QBIT0 9FH6QJC6AiRAoB1kEb7tRb4MvyB+AwSlbejfz7+///8ABijggAQWaOCBCCao4A+CDDbo4IMQ RijhhBQKGAIAOw== --------------070305030709000006000803 Content-Type: image/gif; name="Grafik2" Content-Transfer-Encoding: base64 Content-ID: <[email protected]> Content-Disposition: inline; filename="Grafik2" R0lGODlhLAAcAMQAAP///0x9RHmec9Pf0aa+oleFULHGrmKNW+nv6I+uivT39N7n3W6VZ7zP uZu2lsjXxYSmfwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA ACH5BAAAAAAALAAAAAAsABwAQAXhICCOZGmeaDoaReC+sKAMR7GopIA6boIvLgPAVSAgALxj SYdrOp/Q6NPgYqIEPkTBV2taTwgGbEw+PAZo3FfKbqMQQfeJ8DoY0Hg0wxhAKBgHCl5XLgMq DwFCAAIFPIYDgidrOUQGkSh0cQMJAQyXOXKhoqOkpaanI2aoIwMErq+uCY0BMlANKpxkYwQG BH82TpMicD0pAgQACToDAchqhAG3KQ4HIlQNnErG0IYqB9EAiEQvwEsozC43hy0CAeXMwsLD LS8MAvf497ruCZ+gOA0gfNt3AIK0J/JWKVzIsEQIADs= --------------070305030709000006000803-- --------------000206060805060408060300--