Tokenization of words containing hyphens
Ciarán Ó Duibhín <[email protected]> Fri, 21 Jun 2013 12:31:48 +0100
| Newsgroups | gmane.comp.gnu.aspell.devel |
|---|---|
| Message-ID | <80C18C54747B4530BE6A7CE4E4ECBDAB@InneallChiarin> |
This is a multi-part message in MIME format.
--===============4645998932824395695==
Content-Type: multipart/alternative;
boundary="----=_NextPart_000_000A_01CE6E7B.4CB8B810"
This is a multi-part message in MIME format.
------=_NextPart_000_000A_01CE6E7B.4CB8B810
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
This is the third and last part (change #3) of my consideration of =
apostrophes and hyphens in aspell.
Languages may have words containing an internal hyphen, but with the =
components not being themselves words of the language (a possible =
English example is hotch-potch). In such languages it is well to allow =
a word-internal hyphen in *.dat and put such "compounds" in the =
dictionary. No new code is required for this. However, with the change =
in status of the hyphen, all hyphenated compounds not explicitly =
included in the dictionary will now be rejected, even if their =
components are all in the dictionary. To avoid this, new code is =
needed, for languages supporting internal hyphen, to examine a rejected =
word, and if it contains an internal hyphen, to check the components =
separately. If all the components are accepted, so is the compound. =
The hyphen itself will not be included in the separate components on =
either side of it.
There is something else we can do, when a hyphen is found in a token: we =
can check whether the component before AND INCLUDING the hyphen might be =
a known prefix; or whether the component after AND INCLUDING the hyphen =
might be a known suffix. Thus the dictionary could be allowed to =
include prefixes (including a final hyphen) and suffixes (including an =
initial hyphen), and we can modify *.dat to allow this. Code must be =
added to support matching of prefixes and suffixes, to be activated if =
*.dat allows initial/terminal hyphen, and when a rejected token contains =
an internal hyphen.
The extra code for processing a token containing an internal hyphen, =
after the token has been rejected as a whole, is positioned in =
modules/speller/default/speller_impl.cpp, in procedure =
SpellerImpl::check at around line 190. The new code is placed before =
the checking for two words run together without a space, though this may =
not be the best place for it. NOTE that I don't understand the purpose =
of parameters 3-6 to procedure check, or the corresponding parameters to =
procedure check2, and probably have not used them correctly. But the =
concept is shown to work.
Here is the additional code:
unsigned i=3D0;
while (*(word+i)!=3D 0) {
if ((i > 0) && (i < word_end-word-1) && (*(word+i)=3D=3D'-')) {
if (lang_->special('-').end) { /* test up to hyphen as prefix, =
test remainder recursively as word */
char t =3D *(word+i+1);
*(word+i+1) =3D (char) 0;
if (check2(word, try_uppercase, *ci, gi)) {
*(word+i+1) =3D t;
if (check(word+i+1, word_end, try_uppercase, =
run_together_limit, ci, gi))
return true;
}
else
*(word+i+1) =3D t;
}
if (lang_->special('-').middle) { /* test up to hyphen as word, =
test remainder recursively as word, then as suffix */
*(word+i) =3D (char) 0;
if (check2(word, try_uppercase, *ci, gi)) {
*(word+i) =3D '-';
if (check(word+i+1, word_end, try_uppercase, =
run_together_limit, ci, gi))
return true;
else {
if (lang_->special('-').begin) {
if (check(word+i, word_end, try_uppercase, =
run_together_limit, ci, gi))
return true;
}
}
}
else
*(word+i) =3D '-';
}
}
++i;
}
For this code to work as intended, change #2 is also necessary. =
Consider the token spell-check . We must test to see if the dictionary =
contains a prefix spell- or a suffix -check or plain words spell and =
check. We would expect to find no such prefix or suffix, but to find =
the two plain words. But unless change #2 is made, the token spell- =
will be accepted as matching the dictionary form spell and the process =
will be ended prematurely, albeit with the right result in this case.
As before, my experiments have been conducted using the Hatier port of =
aspell for Windows at =
http://www.niversoft.com/downloads/aspell-0.60.5-msvc.tar.bz2 . The =
changes suggested in these three messages have been made to this source =
and compiled using VC++ 2005. On the evidence so far, the changes =
appear to be working as intended, thereby solving the three problems I =
reported to aspell-user on 19 May 2013, and allowing aspell to treat the =
tokenization of apostrophes and hyphens in a similar way to the MS Word =
spell-checker. As far as I can see, no existing functionality is =
adversely affected by these changes.
Ciar=E1n =D3 Duibh=EDn
------=_NextPart_000_000A_01CE6E7B.4CB8B810
Content-Type: text/html;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content=3D"text/html; charset=3Diso-8859-1" =
http-equiv=3DContent-Type>
<META name=3DGENERATOR content=3D"MSHTML 9.00.8112.16490">
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV><FONT size=3D2 face=3DArial>This is the third and last part =
(change=20
#3) of my consideration of apostrophes and hyphens in =
aspell.</FONT></DIV>
<DIV><FONT size=3D2 face=3DArial></FONT> </DIV>
<DIV><FONT size=3D2 face=3DArial>Languages may have words containing an =
internal=20
hyphen, but with the components not being themselves words of the =
language (a=20
possible English example is <EM>hotch-potch</EM>). In such =
languages it is=20
well to allow a word-internal hyphen in *.dat and put such "compounds" =
in the=20
dictionary. No new code is required for this. However, with =
the=20
change in status of the hyphen, all hyphenated compounds not explicitly =
included=20
in the dictionary will now be rejected, even if their components are all =
in the=20
dictionary. To avoid this, new code is needed, for languages =
supporting=20
internal hyphen, to examine a rejected word, and if it contains an =
internal=20
hyphen, to check the components separately. If all the components =
are=20
accepted, so is the compound. The hyphen itself will not be =
included in=20
the separate components on either side of it.</FONT></DIV>
<DIV><FONT size=3D2 face=3DArial></FONT> </DIV>
<DIV><FONT size=3D2 face=3DArial>There is something else we can do, when =
a hyphen is=20
found in a token: we can check whether the component before AND =
INCLUDING the=20
hyphen might be a known prefix; or whether the component after AND =
INCLUDING the=20
hyphen might be a known suffix. Thus the dictionary could be =
allowed to=20
include prefixes (including a final hyphen) and suffixes (including an =
initial=20
hyphen), and we can modify *.dat to allow this. Code must be added =
to=20
support matching of prefixes and suffixes, to be activated if *.dat =
allows=20
initial/terminal hyphen, and when a rejected token contains an internal=20
hyphen.</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=3D2 face=3DArial>The extra code for processing a token =
containing an=20
internal hyphen, after the token has been rejected as a whole, is =
positioned in=20
modules/speller/default/speller_impl.cpp, in procedure =
SpellerImpl::check at=20
around line 190. The new code is placed before the checking for =
two words=20
run together without a space, though this may not be the best place for=20
it. NOTE that I don't understand the purpose of parameters 3=966 =
to=20
procedure check, or the corresponding parameters to procedure check2, =
and=20
probably have not used them correctly. But the concept is shown to =
work.</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=3D2 face=3DArial>Here is the additional =
code:</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=3D2 face=3DArial> unsigned=20
i=3D0;<BR> while (*(word+i)!=3D 0)=20
{<BR> if ((i > 0) && (i <=20
word_end-word-1) && (*(word+i)=3D=3D'-'))=20
{<BR> if =
(lang_->special('-').end)=20
{ /* test up to hyphen as prefix, test remainder recursively as =
word=20
*/<BR> char t =3D=20
*(word+i+1);<BR> =20
*(word+i+1) =3D (char) 0;<BR> =
if=20
(check2(word, try_uppercase, *ci, gi)) =
{<BR> =20
*(word+i+1) =3D t;<BR> =20
if (check(word+i+1, word_end, try_uppercase,=20
run_together_limit, ci, gi))<BR> =20
return true;<BR> =20
}<BR> else<BR> =20
*(word+i+1) =3D=20
t;<BR> =20
}<BR> if=20
(lang_->special('-').middle) { /* test up to hyphen as word, =
test=20
remainder recursively as word, then as suffix=20
*/<BR> *(word+i) =3D =
(char)=20
0;<BR> if (check2(word,=20
try_uppercase, *ci, gi)) {<BR> =
*(word+i) =3D =
'-';<BR> =20
if (check(word+i+1, word_end, try_uppercase, =
run_together_limit, ci,=20
gi))<BR> =
return=20
true;<BR> else=20
{<BR> if=20
(lang_->special('-').begin) =
{<BR> =20
if (check(word+i, word_end, try_uppercase,=20
run_together_limit, ci, =
gi))<BR> =20
return=20
true;<BR> =20
}<BR> =20
}<BR> =20
}<BR> =20
else<BR> =
*(word+i) =3D=20
'-';<BR> =20
}<BR> }<BR> =20
++i;<BR> }</FONT></DIV>
<DIV> </DIV>
<DIV><FONT size=3D2 face=3DArial>For this code to work as intended, =
change #2 is=20
also necessary. Consider the token <EM>spell-check</EM> . We =
must=20
test to see if the dictionary contains a prefix <EM>spell-</EM> or a =
suffix=20
<EM>-check</EM> or plain words <EM>spell</EM> and <EM>check</EM>. =
We would=20
expect to find no such prefix or suffix, but to find the two plain =
words. =20
But unless change #2 is made, the token <EM>spell-</EM> will be accepted =
as=20
matching the dictionary form <EM>spell</EM> and the process will be =
ended=20
prematurely, albeit with the right result in this case.</FONT></DIV>
<DIV><FONT size=3D2 face=3DArial></FONT> </DIV>
<DIV><FONT size=3D2 face=3DArial>As before, my experiments have been =
conducted using=20
the Hatier port of aspell for Windows at <A=20
href=3D"http://www.niversoft.com/downloads/aspell-0.60.5-msvc.tar.bz2">ht=
tp://www.niversoft.com/downloads/aspell-0.60.5-msvc.tar.bz2</A>=20
. The changes suggested in these three messages have been made to =
this=20
source and compiled using VC++ 2005. On the evidence so far, the =
changes=20
appear to be working as intended, thereby solving the three problems I =
reported=20
to aspell-user on 19 May 2013, and allowing aspell to treat the =
tokenization of=20
apostrophes and hyphens in a similar way to the MS Word =
spell-checker. As=20
far as I can see, no existing functionality is adversely affected by =
these=20
changes.<BR></FONT></DIV><FONT size=3D2 face=3DArial>
<DIV>Ciar=E1n =D3 Duibh=EDn</DIV>
<DIV> </DIV></FONT></BODY></HTML>
------=_NextPart_000_000A_01CE6E7B.4CB8B810--
--===============4645998932824395695==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Aspell-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/aspell-devel
--===============4645998932824395695==--