Re: Issues with japanese support

From: Masayuki Ataka <[email protected]>
Subject: Re: Issues with japanese support
Date: Wed, 16 Mar 2005 22:58:19 +0900 (JST)

> I found two problems for fill in japanese.  I will write another
> mail, later.
> 
Now, I'll explain the problems in fill code, and the patches.
I want to check in the changes before next AUCTeX release, if no objection.

Hereafter, I use following notation in the sample text.
  J  .. Japanese char (or char matches LaTeX-nospace-between-char-regexp)
  A  .. ASCII char (or char not matches LaTeX-nospace-between-char-regexp)
  |  .. fill column
  >< .. line break

[1]

Currently, AUCTeX can break lines between CJK chars:
  ...JJJJ>|<JJJ...
because we check if the char before fill column is breakable.

But if ASCII chars are mixed in the text and no spaces are 
between Japanese and ASCII
  ...JJJAA><AAA...
we skip back to the beginning of line (or the first space).

The following patch breaks line between Japanese and ASCII
  ...JJJ|AA><AAA...

*** latex.el.~5.347.~	2005-03-18 04:41:40.000000000 +0900
--- latex.el	2005-03-19 23:06:43.242091586 +0900
***************
*** 2618,2629 ****
    ;; COMPATIBILITY for Emacs < 22.1 and XEmacs
    (if (fboundp 'fill-move-to-break-point)
        (fill-move-to-break-point linebeg)
!     ;; Cancel `forward-char' which is called just before
!     ;; `LaTeX-fill-move-to-break-point' if the char before point matches
!     ;; `LaTeX-nospace-between-char-regexp'.
!     (if (and (featurep 'mule)
! 	     (TeX-looking-at-backward LaTeX-nospace-between-char-regexp 1))
! 	(backward-char 1)
        (skip-chars-backward "^ \n"))
      ;; Prevent infinite loops: If we cannot find a place to break
      ;; while searching backward, search forward again.
--- 2618,2631 ----
    ;; COMPATIBILITY for Emacs < 22.1 and XEmacs
    (if (fboundp 'fill-move-to-break-point)
        (fill-move-to-break-point linebeg)
!     (if (featurep 'mule)
!  	(if (TeX-looking-at-backward (concat LaTeX-nospace-between-char-regexp ".?") 2)
!  	    ;; Cancel `forward-char' which is called just before
!  	    ;; `LaTeX-fill-move-to-break-point' if the char before point matches
!  	    ;; `LaTeX-nospace-between-char-regexp'.
!  	    (backward-char 1)
!  	  (re-search-backward (concat " \\|\n\\|" LaTeX-nospace-between-char-regexp) linebeg t)
!  	  (forward-char 1))
        (skip-chars-backward "^ \n"))
      ;; Prevent infinite loops: If we cannot find a place to break
      ;; while searching backward, search forward again.
***************

[2]

Next patch prevents breaking line between Japanese and ASCII.

Japanese pTeX neglect the line break if the char at the end of
line is Japanese.
  ...JJJ|JJJ... => ...JJJJJJ...
User who put a space between Japanese and ASCII
  ...JJJ AAA JJJ...
worries that such a space would be neglected by line break.
  ...JJJ|AAA JJJ... => ...JJJAAA JJJ...

Japanese pTeX ragards newline as a space if the char at the end
of line is ASCII.
  ...AAA|AAA... => ...AAA AAA...
User who does not put a space between Japanese and ASCII
  ...AAAJJJAAA...
worries that such a space would be automatically inserted.
  ...AAA|JJJAAA... => ...AAA JJJAAA...

Some tricks are known for the both cases, but we need to know
which manner of space does user use in order to use the trick.

These problems, still remains in Emacs 22, are all solved if we
do not break line between Japanese and ASCII.


*** 2649,2654 ****
--- 2651,2670 ----
  	  (fill-find-break-point linebeg)
  	(when (fboundp 'kinsoku-process) ;XEmacs
  	  (kinsoku-process)))))
+   ;; Prevent line break between 2-byte char and 1-byte char.
+   (when (and enable-multibyte-characters
+ 	     (or (and (not (looking-at LaTeX-nospace-between-char-regexp))
+ 		      (TeX-looking-at-backward LaTeX-nospace-between-char-regexp))
+ 		 (and (not (TeX-looking-at-backward LaTeX-nospace-between-char-regexp))
+ 		      (looking-at LaTeX-nospace-between-char-regexp))))
+     (re-search-backward
+      (concat LaTeX-nospace-between-char-regexp LaTeX-nospace-between-char-regexp
+ 	     LaTeX-nospace-between-char-regexp
+ 	     "\\|"
+ 	     ".\\ca\\ +\\ca") linebeg t)
+     (if (looking-at "..\\c>")
+ 	(forward-char 1)
+       (forward-char 2)))
    ;; Cater for \verb|...| (and similar) contructs which should not be
    ;; broken. (FIXME: Make it work with shortvrb.sty (also loaded by
    ;; doc.sty) where |...| is allowed.  Arbitrary delimiters may be

---
email: [email protected]
Name:: Masayuki Ataka // (Japan)