Tokenization of word-initial specials
Ciarán Ó Duibhín <[email protected]> Fri, 14 Jun 2013 12:28:52 +0100
| Newsgroups | gmane.comp.gnu.aspell.devel |
|---|---|
| Message-ID | <32221C0714884606B6E91731DD718A24@InneallChiarin> |
This is a multi-part message in MIME format.
--===============1594404242616156767==
Content-Type: multipart/alternative;
boundary="----=_NextPart_000_0009_01CE68FA.BAECB1F0"
This is a multi-part message in MIME format.
------=_NextPart_000_0009_01CE68FA.BAECB1F0
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
With reference to aspell source file modules/tokenizer/basic.cpp, =
procedure TokenizerBasic::advance(), the following block of code occurs =
at around the 18th line of the procedure:
if (is_begin(*cur) && is_word(cur[1]))
{
cur_pos +=3D cur->width;
++cur;
}
This code applies to the case where the relevant *.dat file declares a =
non-letter to be a valid word-initial symbol, and a token beginning with =
this symbol is found in text being checked.
As the code stands, the valid word-initial symbol is not stored with the =
extracted token (unlike a valid non-letter in the middle or at the end =
of a token). I would suggest the inclusion of a statement
word.append(*cur);
as an additional first statement within the braces.
The non-retention of the initial symbol in the token produces the =
situation where, given a dictionary which contains the form 'twas but =
not the form twas , the token 'twas in text is refused, and the =
suggested replacement is 'twas .
Inclusion of the suggested statement repairs this behaviour, and in =
doing so it makes aspell conformant to its stated behaviour in =
http://aspell.net/man-html/Words-With-Symbols-in-Them.html:
"The case where the symbol can appear at the beginning or end of =
the word is more difficult to deal with...
Aspell currently handles this case by first trying to spell =
check the word with the symbol and if that fails, try it without."
The effect of the proposed change on English should be an improvement, =
though not a significant one: English examples are few and unimportant =
('tis, 'twas, 'twill, 'twould). However in many languages word-initial =
(and word-final) apostrophes are common, and moreover ASCII hex 27 is =
not used as a quotation mark. An apostrophe is not a discardable =
punctuation mark, but part of the spelling of the word; removing the =
apostrophe produces a different word (or more usually, non-word). =
Nevertheless this is what aspell normally does; it includes in the =
dictionary the residue of the word without any marginal apostrophe, and =
(per the above quotation) checks the token less marginal apostrophe =
against the dictionary. This strategy may be serviceable, if ugly, for =
some languages, but the texts I wish to check contain so many words of =
this type that it would be necessary to admit legions of non-words into =
the dictionary and the whole operation breaks down.
The better way to proceed is to add the words to the dictionary complete =
with their marginal apostrophes, and to check the tokens complete with =
their marginal apostrophes against the dictionary. For this checking to =
work in the case of word-initial apostrophes, the suggested change to =
aspell is a necessary first step. At least two further steps will be =
needed also, before aspell will be able to reproduce the ability of the =
MS Word spell-checker to handle these situations satisfactorily.
I considered making a bug report on this, but I thought it needed more =
explanation than a bug report would normally contain. Also it would be =
necessary to be sure that there are no circumstances in which the =
present behaviour is preferable - if there are, any change should be =
made to depend on a configuration option.
This proposal concerns valid word-initial symbols in general, including =
in particular ASCII hex 27, and is independent of any consideration of =
the Unicode apostrophe.
My experiments have been conducted using the Hatier port of aspell for =
Windows at http://www.niversoft.com/downloads/aspell-0.60.5-msvc.tar.bz2 =
.
Ciar=E1n =D3 Duibh=EDn
------=_NextPart_000_0009_01CE68FA.BAECB1F0
Content-Type: text/html;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content=3D"text/html; charset=3Diso-8859-1" =
http-equiv=3DContent-Type>
<META name=3DGENERATOR content=3D"MSHTML 9.00.8112.16490">
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV><FONT size=3D2 face=3DArial>With reference to aspell source file=20
modules/tokenizer/basic.cpp, procedure TokenizerBasic::advance(), the =
following=20
block of code occurs at around the 18th line of the =
procedure:</FONT></DIV>
<DIV><FONT size=3D2 face=3DArial></FONT> </DIV>
<DIV><FONT size=3D2 face=3DArial> if (is_begin(*cur) =
&&=20
is_word(cur[1]))<BR> =
{<BR> =20
cur_pos +=3D cur->width;<BR> =20
++cur;<BR> }</FONT></DIV>
<DIV><FONT size=3D2 face=3DArial></FONT> </DIV>
<DIV><FONT size=3D2 face=3DArial>This code applies to the case where the =
relevant=20
*.dat file declares a non-letter to be a valid word-initial symbol, and =
a token=20
beginning with this symbol is found in text being checked.</FONT></DIV>
<DIV><FONT size=3D2 face=3DArial></FONT> </DIV>
<DIV><FONT size=3D2 face=3DArial>As the code stands, the valid =
word-initial symbol=20
is not stored with the extracted token (unlike a valid non-letter in the =
middle=20
or at the end of a token). I would suggest the inclusion of a=20
statement</FONT></DIV>
<DIV><FONT size=3D2 face=3DArial></FONT> </DIV>
<DIV><FONT size=3D2 face=3DArial> =20
word.append(*cur);</FONT></DIV>
<DIV><FONT size=3D2 face=3DArial></FONT> </DIV>
<DIV><FONT size=3D2 face=3DArial>as an additional first statement within =
the=20
braces.</FONT></DIV>
<DIV><FONT size=3D2 face=3DArial></FONT> </DIV><FONT size=3D2 =
face=3DArial>
<DIV><BR>The non-retention of the initial symbol in the token produces =
the=20
situation where, given a dictionary which contains the form =
<EM>'twas</EM> but=20
not the form <EM>twas</EM> , the token <EM>'twas</EM> in text is =
refused, and=20
the suggested replacement is <EM>'twas</EM> .</DIV>
<DIV> </DIV>
<DIV>Inclusion of the suggested statement repairs this behaviour, and in =
doing=20
so it makes aspell conformant to its stated behaviour in <A=20
href=3D"http://aspell.net/man-html/Words-With-Symbols-in-Them.html">http:=
//aspell.net/man-html/Words-With-Symbols-in-Them.html</A>:<BR>  =
; =20
<EM>"The case where the symbol can appear at the beginning or end of the =
word is=20
more difficult to deal=20
with...<BR> Aspell =
currently=20
handles this case by first trying to spell check the word with the =
symbol and if=20
that fails, try it without."</EM></DIV>
<DIV> </DIV>
<DIV><BR>The effect of the proposed change on English should be an =
improvement,=20
though not a significant one: English examples are few and unimportant =
('tis,=20
'twas, 'twill, 'twould). However in many languages word-initial =
(and=20
word-final) apostrophes are common, and moreover ASCII hex 27 is not =
used as a=20
quotation mark. An apostrophe is not a discardable punctuation =
mark, but=20
part of the spelling of the word; removing the apostrophe produces a =
different=20
word (or more usually, non-word). Nevertheless this is what aspell =
normally does; it includes in the dictionary the residue of the word=20
without any marginal apostrophe, and (per the above quotation) =
checks the=20
token less marginal apostrophe against the dictionary. This =
strategy may=20
be serviceable, if ugly, for some languages, but the texts I wish to =
check=20
contain so many words of this type that it would be necessary to admit =
legions=20
of non-words into the dictionary and the whole operation breaks =
down.</DIV>
<DIV> </DIV>
<DIV>The better way to proceed is to add the words to the dictionary =
complete=20
with their marginal apostrophes, and to check the tokens complete with =
their=20
marginal apostrophes against the dictionary. For this checking to =
work in=20
the case of word-initial apostrophes, the suggested change to aspell is =
a=20
necessary first step. At least two further steps will be needed =
also,=20
before aspell will be able to reproduce the ability of the MS Word =
spell-checker=20
to handle these situations satisfactorily.</DIV>
<DIV> </DIV>
<DIV><BR>I considered making a bug report on this, but I thought it =
needed more=20
explanation than a bug report would normally contain. Also it =
would be=20
necessary to be sure that there are no circumstances in which the =
present=20
behaviour is preferable =97 if there are, any change should be made to =
depend on a=20
configuration option.</DIV>
<DIV> </DIV>
<DIV><BR>This proposal concerns valid word-initial symbols in general, =
including=20
in particular ASCII hex 27, and is independent of any consideration of =
the=20
Unicode apostrophe.</DIV>
<DIV> </DIV>
<DIV><BR>My experiments have been conducted using the Hatier port of =
aspell for=20
Windows at <A=20
href=3D"http://www.niversoft.com/downloads/aspell-0.60.5-msvc.tar.bz2">ht=
tp://www.niversoft.com/downloads/aspell-0.60.5-msvc.tar.bz2</A>=20
.<BR></DIV>
<DIV> </DIV>
<DIV>Ciar=E1n =D3 Duibh=EDn</DIV></FONT></BODY></HTML>
------=_NextPart_000_0009_01CE68FA.BAECB1F0--
--===============1594404242616156767==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Aspell-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/aspell-devel
--===============1594404242616156767==--