Re: Request for Comment: Callable CRM114 Classifiers (libcrm114)

Well, this is branching off into a UTF-8 discussion. Considering that
bit: the core elements of CRM114 currently process bytes:

1- tokenizing (which, as pointed out by Paolo, is done through regexes
exclusively - if we ditch all the [:graph:]+ code speedups in there)
2- feature hashing (which doesn't mind what it is fed: it's just a
byte series, so it can be a UTF-8 byte series; it simply does not
care)
3- filename extraction (which is just another tokenization, but I
mention it separately, just as the next one: ...
4- variable identifier extraction.

1+2 work on the content. (1) _should_ support UTF-8, but I never
tested if TRE accepts Unicode characters when using it byte-wide APIs;
if it doesn't, that means we might face a conversion from bytes to
integers (more correctly: expanding input bytes to wchar_t elements)
before feeding the content to any regex.

3+4 work on the script, but MAY work on the input data, depending on
your script. Here the question is: do we wish to support Unicode
filenames? And Unicode variable identifiers in the script? Which are
two different questions: the first (filenames) might be useful for
non-English developers working on Unicode OS's, but it means
mmap/fopen/et al must be fed wchar_t-based strings for filenames,
converted from the UTF-8 input/script bytes.

Supporting Unicode variable identifiers as well may or may not be a
simplification (my guess it /is/, actually) once you accept Unicode
filenames, as you then implicitly have moved your scripts into the
UTF-8 realm as well (as there can be literal filenames in there) which
means you'll have to do your share of UTF-8 <-> wchar_t[] conversions
in the code. It doesn't come free for 3+4.


Doesn't support CRM114 UTF-8 content right now already?
If we don't, the only thing I can think of that would keep CRM114 from
supporting classifying UTF-8 /content/ right now, as it exists today,
would be the TRE library not supporting Unicode regexes or content
when we use it's 'char'/byte-based APIs. If that is the case (is it?
If it is: do we have tests for this? I don't.) we'll hit the same
trouble with libcrm114, as Unicode support is a different story,
compared to 32/64/whatever-bittish or threading support.

--

Regarding the original RFC from Bill: please allow a few days for me
to consider an answer there. It deserves some /serious/ thought.

--

BTW: Latin for:

delete: deleo
disappear: evanesco; dilabor
eradicate: exstirpo
removal: amotio; migratio
remove: removeo (no, really! :-)); amoveo; tollo; segrego; emigro/commigro
retire: recedo; decedo; abeo
kill: interficio; occido; caedo; neco

(some good use of my English->Latin dictionary) which would lead to
two suggestions:

abi (= begone! [faul feature!])

or the immediately understandable (though on very slippery
latin-oriented ground):

ex

- which has the nice added property to sound exactly like 'axe' as in
'to axe [a feature]'; well known lingo for all software folk who ever
were on any deadline  ;-)


-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/