Re: UTF-8 and ispell

2007/9/29, Rafael Laboissiere <[email protected]>:
> Yes, Perl understands "\xxx" escape sequences in strings where "xxx" is an
> octal number [1].  However, this does not help us here because when parsi=
ng
> the info-aspell file, DictionariesCommon.pm sees the string as ASCII, i.e.
> containing the "\" and [0-7] characters.

So Perl's \xxx sequences are basically the same as in S-Lang.
>
> [1] http://perldoc.perl.org/perlreref.html#ESCAPE-SEQUENCES
This is for regular expression, not normal strings, but I think the escape
sequences are mostly the same.

\x{263a} A wide hexadecimal value

\x{263a} in Perl seems to be the same as \x{263a} in s-lang, both give me
"=E2=98=BA" - a copyright sign. But I don't think this works the same in ut=
f-8 and
ascii mode - for that you need hexadecimal numbers (below 256 of course).

> At any rate, the strings in jed-ispell-dicts.sl are too long for aspell-bg
> and ispell_init.sl fails here with the error message:
>
> /var/cache/dictionaries-common/jed-ispell-dicts.sl:232: String too long f=
or buffer: found '??'
>
> Is this normal?

That sounds like a problem. I guess the string is longer than 256 character=
s.
From the S-Lang manual:
Although there is no imposed limit on the length of a string, string
  literals must be less than 256 characters in length.  It is possible
  to construct strings longer than this by string concatenation, e.g.,

             "This is the first part of a long string"
              + " and this is the second part"

Since DictionariesCommon generates S-Lang code, this limitation applies.

Broadly, there are three ways to fix this:
- catch the "String too long" error in a catch block. Tough luck for
Bulgarian speakers.
- Split the string up, as suggested in the manual. An example of how to do
this can be found in the autotext.sl mode.
- Instead of generating S-Lang code, generate some data file and provide a
S-Lang script to parse those data. One way to do this would be to
generate XML and parse that with the expat module. Another way would be
to store the data in a SQLite table, and in fact the next version of
autotext.sl may do that. Or maybe the readascii.sl library provided with
slsh can be used for this. Note that if the string isn't sourced by
S-Lang, you don't get the "\x{__}" substitution.

I think I'd go for the second option.