Checking of word-marginal specials

This is a multi-part message in MIME format.

--===============5077015479605475357==
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_0013_01CE6DE1.314333C0"

This is a multi-part message in MIME format.

------=_NextPart_000_0013_01CE6DE1.314333C0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

This is the second part (change #2) of my consideration of apostrophes =
and hyphens in aspell.  The first part (change #1) was "Tokenization of =
word-initial specials" dated June 14 2013.

Currently, when *.dat marks apostrophe as valid initially, the =
dictionary form well validates the token 'well (in addition to the token =
well).  And, when *.dat marks apostrophe as valid finally, the =
dictionary form well also validates the token well' .  However, neither =
of the tokens 'well or well' should ever be validated by the form well, =
but approved only if those exact forms are present in the dictionary.

There are two cases: when the apostrophe is encountered in a token in a =
position, initial or final, where it IS NOT valid in *.dat (and note =
that this applies to en.dat), it is immediately dropped from the token, =
and only the token without the apostrophe is checked against the =
dictionary.  (Before change #1, even a valid initial apostrophe was =
dropped from the token, but not a valid final apostrophe.)  So if =
"trying the token without the special" is done with the intention of =
accepting a token of English which has contrived to include a =
neighbouring quotation mark, this is a non-existent situation whose =
removal will have no effect.

When the apostrophe is encountered in a token in a position, initial or =
final, where it IS valid in *.dat, the token should be accepted only if =
the dictionary contains the word including the apostrophe - the current =
practice of accepting the token, merely because the corresponding form =
without the apostrophe is in the dictionary, is to accept an invalid =
word, possibly resulting from a mistaken use of the apostrophe (ASCII =
hex 27) as a quotation mark.  (Remember that languages which accept =
valid word-marginal apostrophes in *.dat do not use ASCII hex 27 as a =
quotation mark.)

The code for "trying the token with and without any initial or final =
special" is found in procedure SensitiveCompare in =
modules/speller/default/language.cpp at around line 428.  The suggested =
change #2 is to remove the code which, when the token begins or ends =
with a valid special, and has failed to match the dictionary, compares =
the token minus the special to the dictionary.  (Note again that a token =
will never be found to begin or end with an INVALID special, as that =
special will have been dropped during tokenization.)  Specifically, I =
suggest removal of the four separate lines which use the special() =
function.  Having no previous experience of C++ programming I cannot say =
that everything has been done which ought to be done, but the concept =
has been tried and shown to work.  I do not at present see any reason to =
make it conditional, ie. I cannot see any situation where the present =
behaviour is preferable.

This suggestion will enable a language like Italian, for example, to =
have a new it.dat in which word-final apostrophe is allowed, and =
non-words like anch may be replaced in the dictionary by anch' .  Even =
for English, a new en.dat allowing marginal apostrophes and a new =
dictionary (with, for example, 'twas and 'twill in place of twas and =
twill, and adding 'tis and 'twould) could produce an improvement, but =
only with English texts in which an encoding distinction has been made =
between apostrophe and quotation mark.  The main beneficiaries of the =
suggestion will be among languages other than English.

As before, my experiments have been conducted using the Hatier port of =
aspell for Windows at =
http://www.niversoft.com/downloads/aspell-0.60.5-msvc.tar.bz2 .

Third and final part to follow.

Ciar=E1n =D3 Duibh=EDn


------=_NextPart_000_0013_01CE6DE1.314333C0
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content=3D"text/html; charset=3Diso-8859-1" =
http-equiv=3DContent-Type>
<META name=3DGENERATOR content=3D"MSHTML 9.00.8112.16490">
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV><FONT size=3D2 face=3DArial>This is the second part (change =
#2)&nbsp;of my=20
consideration of apostrophes and hyphens in aspell.&nbsp; The first part =
(change=20
#1)&nbsp;was "Tokenization of word-initial specials" dated June 14=20
2013.</FONT></DIV>
<DIV><FONT size=3D2 face=3DArial></FONT>&nbsp;</DIV>
<DIV><FONT size=3D2 face=3DArial>Currently, when *.dat marks apostrophe =
as valid=20
initially, the dictionary form <EM>well</EM> validates the token =
<EM>'well</EM>=20
(in addition to the token <EM>well</EM>).&nbsp; And, when *.dat marks =
apostrophe=20
as valid finally, the dictionary form <EM>well</EM> also validates the =
token=20
<EM>well'</EM> .&nbsp; However, neither of the tokens <EM>'well</EM> or=20
<EM>well'</EM> should ever be validated by the form <EM>well</EM>, but =
approved=20
only if those exact forms are present in the dictionary.</FONT></DIV>
<DIV><FONT size=3D2 face=3DArial></FONT>&nbsp;</DIV>
<DIV><FONT size=3D2 face=3DArial>There are two cases: when the =
apostrophe is=20
encountered in a token in a position, initial or final, where it IS NOT =
valid in=20
*.dat (and note that this applies to en.dat), it is immediately dropped =
from the=20
token, and only the token without the apostrophe is checked against the=20
dictionary.&nbsp; (Before change #1, even a valid initial apostrophe was =
dropped=20
from the token, but not a valid final apostrophe.)&nbsp; So if "trying =
the token=20
without the special" is done with the intention of accepting a token of =
English=20
which has&nbsp;contrived to include a&nbsp;neighbouring quotation mark, =
this is=20
a non-existent situation whose removal will have no effect.</FONT></DIV>
<DIV><FONT size=3D2 face=3DArial></FONT>&nbsp;</DIV>
<DIV><FONT size=3D2 face=3DArial>When the apostrophe is encountered in a =
token in a=20
position, initial or final, where it IS valid in *.dat, the token should =
be=20
accepted only if the dictionary contains the word including the =
apostrophe =97 the=20
current practice of accepting the token, merely because the =
corresponding form=20
without the apostrophe is in the dictionary, is to accept an invalid =
word,=20
possibly resulting from a mistaken use of the apostrophe (ASCII hex 27) =
as a=20
quotation mark.&nbsp; (Remember that languages which accept valid =
word-marginal=20
apostrophes in *.dat do not use ASCII hex 27 as a quotation =
mark.)</FONT></DIV>
<DIV><FONT size=3D2 face=3DArial></FONT>&nbsp;</DIV>
<DIV><FONT size=3D2 face=3DArial>The code for "trying the token with and =
without any=20
initial or final special" is found in procedure SensitiveCompare in=20
modules/speller/default/language.cpp at around line 428.&nbsp; The =
suggested=20
change #2 is to remove the code which, when the token begins or ends =
with a=20
valid special, and has failed to match the dictionary, compares the =
token minus=20
the special to the dictionary.&nbsp; (Note again that a token will never =
be=20
found to begin or end with an INVALID special, as that special will have =
been=20
dropped during tokenization.)&nbsp; Specifically, I suggest removal of =
the four=20
separate lines which use the special() function.&nbsp; Having&nbsp;no =
previous=20
experience of C++ programming I cannot say that everything has been done =
which=20
ought to be done, but the concept has been tried and shown to =
work.&nbsp; I do=20
not at present see any reason to make it conditional, ie. I cannot see =
any=20
situation where the present behaviour is preferable.</FONT></DIV>
<DIV><FONT size=3D2 face=3DArial></FONT>&nbsp;</DIV>
<DIV><FONT size=3D2 face=3DArial>This suggestion will enable a language =
like=20
Italian, for example, to have a new it.dat in which word-final =
apostrophe is=20
allowed, and non-words like anch may be replaced in the dictionary by =
anch'=20
.&nbsp; Even for English, a new en.dat allowing marginal apostrophes and =
a new=20
dictionary (with, for example, 'twas and 'twill in place of twas and =
twill, and=20
adding 'tis and 'twould) could produce an improvement, but only with =
English=20
texts in which an encoding distinction has been made between apostrophe =
and=20
quotation mark.&nbsp; The main beneficiaries of the suggestion will be =
among=20
languages other than English.</FONT></DIV>
<DIV><FONT size=3D2 face=3DArial></FONT>&nbsp;</DIV>
<DIV><FONT size=3D2 face=3DArial>As before, my experiments have been =
conducted using=20
the Hatier port of aspell for Windows at <A=20
href=3D"http://www.niversoft.com/downloads/aspell-0.60.5-msvc.tar.bz2">ht=
tp://www.niversoft.com/downloads/aspell-0.60.5-msvc.tar.bz2</A>=20
.<BR></FONT></DIV>
<DIV><FONT size=3D2 face=3DArial>Third and final part to =
follow.</FONT></DIV>
<DIV><FONT size=3D2 face=3DArial>&nbsp;</DIV>
<DIV>Ciar=E1n =D3 Duibh=EDn</DIV></FONT>
<P><FONT size=3D2 face=3DArial></FONT>&nbsp;</P></BODY></HTML>

------=_NextPart_000_0013_01CE6DE1.314333C0--



--===============5077015479605475357==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Aspell-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/aspell-devel

--===============5077015479605475357==--