[emacs-w3m:13122] [emacs-w3m/emacs-w3m] Refactor codebase for utf-8 (#9)
Boruch Baum <[email protected]>
| Newsgroups | gmane.emacs.w3m |
|---|---|
| Message-ID | <emacs-w3m/emacs-w3m/pull/[email protected]> |
The code maintains inconsistent use of character encoding. Maybe
because parts of the code were written so long ago, before utf-8
became a defacto world-wide standard, other character encoding were
set for various files. The purpose of this commit is to try to make
the project consistently -utf-8 throughout.
This commit was originally applied in four months ago in the git
repository which I was using for development befor the project had its
own official git repository. During the intervening four months it was
publicly available for testing and was the version which I was using.
I received no complaints, and observed nothing suspicious; however, I
don't use Japanese, and I don't know whether anyone else bothered to
test.
In practice, much of the work should be easy to test just by using the
menu system in Japanese, and by using emacs-w3m for the various
shimbuns.
The character sets that had been in use included those which the w3
consortium say are to be especially avoided
https://www.w3.org/International/questions/qa-choosing-encodings#avoid.
Most that have explicit encoding are set to iso-2022-7bit, and file
w3m-bug.el is encoded for 'euc-japan'.
Since this was a huge and mind-numbing task, I automated it.
Step 1 was a few `sed` commands to change the first line of the *.el files.
Step 2 was to run `iconv` on the files.
#+BEGIN_SRC conf
for file in *.el ; \
do echo "$file" ; \
iconv -c -f iso-2022-jp -t utf-8 "$file" > "$file".new ; \
done
#+END_SRC
+ In general, the conversion operation succeeded, in that it did
transform Japanese text from unintelligble ASCII escape sequences
into Japanese characters (likewise unintelligble to me, but with
samples verified by google translate).
+ Some files complained when not using the `-c` flag to force
completion, but it doesn't seem important:
#+BEGIN_QUOTE
sb-dennou-new.el
iconv: illegal input sequence at position 47
sb-kyoko-np.el
iconv: illegal input sequence at position 1169
sb-nikkangendai.el
iconv: illegal input sequence at position 1463
sb-tech-on.el
iconv: illegal input sequence at position 2723
sb-wikimedia.el
iconv: illegal input sequence at position 4937
shimbun.el
iconv: illegal input sequence at position 61716
#+END_QUOTE
+ Step 3 was to eyeball the results, and edit obvious errors.
+ Latin diacritics were sometimes clobbered by the `iconv` operation.
For example, é ó ú Á ź in lists
of month names for European languages.
+ On a few rare occassions, '@' was clobbered in regexes and in
comments.
+ Some files, such as `w3m-bug.el` were fine without `iconv` and
`iconv ruined them.
You can view, comment on, or merge this pull request online at:
https://github.com/emacs-w3m/emacs-w3m/pull/9
-- Commit Summary --
* Refactor codebase for utf-8
-- File Changes --
M ChangeLog (6)
M bookmark-w3m.el (2)
M mew-w3m.el (2)
M mime-w3m.el (2)
M octet.el (2)
M shimbun/nnshimbun.el (2)
M shimbun/sb-2ch.el (8)
M shimbun/sb-airs.el (2)
M shimbun/sb-aljazeera.el (2)
M shimbun/sb-arch-bluegate.el (2)
M shimbun/sb-asahi-html.el (2)
M shimbun/sb-asahi-mytown.el (110)
M shimbun/sb-asahi.el (540)
M shimbun/sb-atmarkit.el (34)
M shimbun/sb-atom-hash.el (2)
M shimbun/sb-atom.el (2)
M shimbun/sb-bbc.el (2)
M shimbun/sb-bbdb-ml.el (2)
M shimbun/sb-cgi-board.el (2)
M shimbun/sb-chise.el (2)
M shimbun/sb-cnet-jp.el (10)
M shimbun/sb-cnet.el (2)
M shimbun/sb-cnn-jp.el (2)
M shimbun/sb-coldsync.el (2)
M shimbun/sb-debian-jp.el (2)
M shimbun/sb-debian.el (2)
M shimbun/sb-debugmagazin-de.el (2)
M shimbun/sb-dennou.el (6)
M shimbun/sb-digiko.el (2)
M shimbun/sb-elips.el (2)
M shimbun/sb-emacs-w3m.el (2)
M shimbun/sb-emacswiki.el (2)
M shimbun/sb-engadget-ja.el (4)
M shimbun/sb-excite.el (4)
M shimbun/sb-exconn.el (2)
M shimbun/sb-f1fan.el (4)
M shimbun/sb-fau.el (2)
M shimbun/sb-ffii.el (2)
M shimbun/sb-fml.el (2)
M shimbun/sb-geocrawler.el (2)
M shimbun/sb-glimpse.el (2)
M shimbun/sb-gnome.el (2)
M shimbun/sb-haiku-os.el (2)
M shimbun/sb-hash.el (2)
M shimbun/sb-heise.el (2)
M shimbun/sb-hns.el (2)
M shimbun/sb-ibm-dev.el (4)
M shimbun/sb-impress.el (40)
M shimbun/sb-infoshop.el (2)
M shimbun/sb-itmedia.el (44)
M shimbun/sb-japantimes.el (2)
M shimbun/sb-javaconf.el (2)
M shimbun/sb-jpilot.el (2)
M shimbun/sb-jpo.el (84)
M shimbun/sb-kantei.el (52)
M shimbun/sb-kde.el (2)
M shimbun/sb-kyoko-np.el (8)
M shimbun/sb-laut-de.el (2)
M shimbun/sb-linuxce-jp.el (2)
M shimbun/sb-lotusex.el (24)
M shimbun/sb-lump.el (2)
M shimbun/sb-m17n.el (2)
M shimbun/sb-macosx-jp.el (2)
M shimbun/sb-mailarc.el (2)
M shimbun/sb-mailman.el (20)
M shimbun/sb-mainichi.el (48)
M shimbun/sb-makanai.el (4)
M shimbun/sb-marc-aims.el (2)
M shimbun/sb-meadow.el (2)
M shimbun/sb-mew.el (2)
M shimbun/sb-mhonarc.el (2)
M shimbun/sb-mozilla-jp.el (2)
M shimbun/sb-msdn.el (2)
M shimbun/sb-msports-nifty.el (6)
M shimbun/sb-multi.el (2)
M shimbun/sb-n24-de.el (2)
M shimbun/sb-namazu.el (2)
M shimbun/sb-netbsd.el (2)
M shimbun/sb-nikkangendai.el (22)
M shimbun/sb-nikkansports.el (54)
M shimbun/sb-nikkei.el (292)
M shimbun/sb-nytimes.el (2)
M shimbun/sb-opentechpress-jp.el (4)
M shimbun/sb-orca.el (2)
M shimbun/sb-palmfan.el (46)
M shimbun/sb-pcweb-column.el (2)
M shimbun/sb-perlentaucher-de.el (2)
M shimbun/sb-pilot-link.el (2)
M shimbun/sb-pilot-mailsync.el (2)
M shimbun/sb-plucker.el (2)
M shimbun/sb-pocketgames.el (4)
M shimbun/sb-rakuten.el (8)
M shimbun/sb-redhat.el (2)
M shimbun/sb-rediff.el (2)
M shimbun/sb-rss-blogs.el (2)
M shimbun/sb-rss-hash.el (2)
M shimbun/sb-rss.el (2)
M shimbun/sb-ruby.el (2)
M shimbun/sb-sankei.el (62)
M shimbun/sb-savannah.el (2)
M shimbun/sb-scipy.el (2)
M shimbun/sb-security-memo.el (2)
M shimbun/sb-slashdot-jp.el (4)
M shimbun/sb-slashdot.el (2)
M shimbun/sb-sml.el (2)
M shimbun/sb-sourceforge-jp.el (2)
M shimbun/sb-spiegel.el (2)
M shimbun/sb-sponichi.el (28)
M shimbun/sb-squeak-dev.el (2)
M shimbun/sb-squeak-ja.el (2)
M shimbun/sb-sueddeutsche-de.el (2)
M shimbun/sb-tcup.el (12)
M shimbun/sb-tdiary-ml.el (2)
M shimbun/sb-tdiary.el (2)
M shimbun/sb-tech-on.el (50)
M shimbun/sb-texfaq.el (2)
M shimbun/sb-text.el (10)
M shimbun/sb-the-onion.el (2)
M shimbun/sb-the-register.el (2)
M shimbun/sb-tigris.el (2)
M shimbun/sb-toshiba.el (2)
M shimbun/sb-vinelinux.el (14)
M shimbun/sb-w3m-dev.el (2)
M shimbun/sb-welt-de.el (2)
M shimbun/sb-wiki.el (2)
M shimbun/sb-wikimedia.el (10)
M shimbun/sb-wincefan.el (20)
M shimbun/sb-wired-jp.el (6)
M shimbun/sb-x51.el (10)
M shimbun/sb-xemacs.el (2)
M shimbun/sb-yahoo-auctions.el (4)
M shimbun/sb-yahoo-sports.el (4)
M shimbun/sb-yahoo.el (164)
M shimbun/sb-yomiuri-html.el (2)
M shimbun/sb-yomiuri.el (196)
M shimbun/sb-yoshirin.el (20)
M shimbun/sb-zdnet-jp.el (8)
M shimbun/sb-zeit-de.el (2)
M shimbun/shimbun.el (128)
M w3m-antenna.el (2)
M w3m-bookmark.el (23)
M w3m-bug.el (18)
M w3m-ccl.el (2)
M w3m-cookie.el (2)
M w3m-dtree.el (15)
M w3m-ems.el (2)
M w3m-favicon.el (2)
M w3m-fb.el (2)
M w3m-form.el (2)
M w3m-hist.el (2)
M w3m-image.el (2)
M w3m-lnum.el (2)
M w3m-mail.el (2)
M w3m-namazu.el (2)
M w3m-perldoc.el (2)
M w3m-proc.el (40)
M w3m-rss.el (2)
M w3m-save.el (2)
M w3m-search.el (2)
M w3m-session.el (26)
M w3m-symbol.el (82)
M w3m-tabmenu.el (4)
M w3m-ucs.el (2)
M w3m-util.el (2)
M w3m-weather.el (320)
M w3m-xmas.el (2)
M w3m.el (213)
M w3mhack.el (2)
-- Patch Links --
https://github.com/emacs-w3m/emacs-w3m/pull/9.patch
https://github.com/emacs-w3m/emacs-w3m/pull/9.diff
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/emacs-w3m/emacs-w3m/pull/9