Re: SWI-Prolog 7.1.3

On 19/12/2013, at 2:37 AM, Jan Wielemaker wrote:

>> The first thing I note is that there are two arguments that
>> specify a set of characters, but there is no set-of-characters
>> data type.  (See pllib.htm section 1.5.)
> 
> That was noted.  ECliPSe is 8-bit only, so it doesn't matter too
> much.  Joachim and I decided that we would look into a more abstract
> character set notion later.  Trying to solve that right away would
> have complicated getting to an agreement on the primitives too much.

This seems back to front to me.
I know about the XP slogan YAGNI,
but in this case, since SWI Prolog has been dealing with
Unicode characters after a fashion for some time,
you *KNOW* that you *WILL* need it.
What makes the most sense is to DESIGN for the full case
but IMPLEMENT for the limited case (at first).

In the 21st century, a set of string handling operations
that isn't *designed* to help programmers cope with the
weirdness of Unicode is, well, I'm looking for polite
terms.  It's like designing a programming language in
which integers cannot be any bigger than 16 bits.

We already *KNOW* this isn't going to *WORK* long term.

I got a copy of "Unicode Demystified" last week, and have
been working my way through it.  It turns out that Unicode
3.2 was already worse than I realised.  Let me give you an
example, which I know is relevant to at least one person
who has posted several times to this mailing list.

The Indic scripts are alphabetic, but they take the
consonants as a basic framework and attach the vowels
to them.  Some vowels go on the right of the consonant,
some on the left, some on both sides and this is an
oversimplification.  Using adjacency to indicate fusion
and "." to indicate side by side, we might have
"LO.IG.K/" where the / cancels the implied "a" vowel.
(Problem here: "K" by itself indicates "KA"; you need
the virama to cancel the "A".)  So what do we do about
the vowels?  There are two possibilities:
 - encode the letters in their visual order (as
   displayed on the page)
 - encode the letters in the logical order (as
   pronounced).
I'd read all the stuff about Devanagari and so on, and
understood clearly that Unicode had unequivocally plumped
for logical order (L,O,G,I,K,/).  What I _hadn't_ noticed
that the "round trip compatibility with existing standards"
part had caused them to adopt _visual order_ for Thai and
Lao.  So Unicode uses *both* principles within a single
strongly related family of scripts.  Ouch.

Then there is the fun with vowel signs that are written on
both sides of a consonant.  Is "<X>" to be encoded as
(<>,X), as (X,<>), or as (<,X,>)?  More than one alternative
may be available within the same script!

All of this means that working "one character at a time",
the way that is so *obvious* and *easy* in most
European (Latin, Greek, Cyrillic, Armenian, Georgian)
scripts is fraught with extreme difficulty in other
scripts.  I'll get to what seems to me to be the obvious
conclusion shortly.

> Indeed.  We'll check out pllib.htm before deciding on a
> character set notion.

One of the things I have found extremely frustrating about
writing pllib.htm is that people seem to be treating it as
a specific concrete proposal to be taken or left, picking
what they like, and going on.  But

	PLLIB.HTM IS SUPPOSED TO BE
	THE OPENING OF A CONVERSATION.

I wanted it to engender a *discussion*.

Now here's "what seems to me to be the obvious conclusion".

	EUROPEAN PROLOG IMPLEMENTORS ARE THE WRONG
	PEOPLE TO DESIGN TEXT HANDLING OPERATIONS
	BY THEMSELVES.

I include myself as someone of European descent who used to
be a Prolog implementor.  I live in a country with a non-
European official language, but it uses a Latin-based script
and worked very nicely in an 8-bit world.

Over a thousand million people use an Indic script.
Over a thousand million people use a Han script.
Over 400 million people use the Arabic script (it's
probably a lot more, it's used in _lots_ of places).

Did you know that there is a mechanism in Unicode
for *describing* Chinese characters that are not
yet encoded?  It's recursive, and written prefix,
with U+2FF0...U+2FFB.  From the Wikipedia,
	the character “” can be described as “⿰書史”.
A little "tree" built this way should normally be
treated as a single combining character sequence;
such trees can have up to 6 leaves.  (I do not know
whether ideographic variant selectors can be mixed in
with this.  I don't really _want_ to know.)

An *adequate* design team for string operations in the
21st century will include at least one person
intimately familiar with CJK text processing,
at least one person intimately familiar with Indic
text processing, at least one person intimately familiar
with Arabic text processing, none of these people needing
deep understanding of Prolog, *and* at least one Prolog
implementor.

What I *can* be sure of is that we need *building blocks*,
not complete packaged predicates, and that we certainly
need to be able to deal with the patterns in UAX29 and
UAX31.
	
>>  - Problem: if I'm reading the documentation correctly,
>>    this doesn't do at all what I want.  I want a string
>>    splitter that takes *RUNS* of separator characters
>>    as separators.  Suppose the input is
>> 	"Lorem ipsum.  Dolor sit amet. "
>>    If the separators include spaces, and the number of
>>    Substrings is one more than the number of spaces,
>>    then this would return
>> 	["Lorem","ipsum","","Dolor","sit","amet",""]
>>                          ^^                      ^^
>>    with empty strings that are worse than useless to me.
> 
> You deal with runs of separator characters by making
> SepChars and PadChars the same.  Then you never and up
> with empty strings.  I guess the most typical case for
> this is if these sets represent white space.

In that case, the documentation needs rewriting.
The documentation is absolutely unambiguous:
	The number of SubStrings is one more than
	the number of separators found.
In this example, the number of separators is 6 so there
is no wiggle room whatsoever, the number of substrings
MUST be seven.  The documentation says NOTHING about
the number of substrings depending in any way whatsoever
on what the PadChars might be.

In any case, this doesn't address a problem that comes up
more often than you might expect: a "record" where the
separators are *different*.  For example, we might have
	<label>:<field>{,<field>}*
where commas in the <label> are just data and colons in
the <field>s are just data.

If you give me a building block that can split off ONE
field, I can do

	parse_record(String, Label, Fields) :-
	    split_one_field(String, Label, ":", Rest),
	    split_all_fields(Rest, ",", Fields).

	split_all_fields(String, Sep, [Field|Fields]) :-
	    split_one_field(String, Field, Sep, Rest),
	    !,
	    split_all_fields(Rest, Sep, Fields).
	split_all_fields(Rest, _, [Rest]).

If you give me a "packaged solution" that splits off all
the fields, I'm stuck.

>>    Note in particular that if you use this with Unix filenames
>>    it will go wrong.  The parts of "/foo//bar///ugh" are
>>    "foo", "bar", and "ugh" -- there are no empty parts there.
> 
> So,
> 
>   ?- split_string("/foo//bar///ugh", "/", "/", L).
>   L = ["foo", "bar", "ugh"].

Yes, but THAT IS NOT COMPATIBLE WITH THE DOCUMENTATION.

Here it is again:
	The SepChars argument provides the characters that
	act as separators and thus the length ofSubStrings
	is one more than the number of separators found.

That is not ambiguous.  SepChars is "/".  There are six
slashes in the string.  Therefore according to the
documentation there *MUST* be seven strings in the answer;
there *cannot* be 3.

In fact, I don't even need a building block for splitting
off one field.

The building blocks I need are
	(1) Find all of these characters (string argument).
	    (So that I can use ": " as a separator,
	    with ":" and " " in other combinations
	    being plain data.)
	(2) Find any of these characters (set argument).
	(3) Count leading or trailing characters in a set.
	(4) Given RSTART and RLENGTH, split a string _here_,
	    in O(1) time and space.
Given those, I can trivially *program* any string splitting
and trimming operations I want in Prolog.

>> Consider another example.  I have some data that need to be
>> treated the way M4 treats macro arguments.  That is, the
>> string is to be split at commas, and then LEADING white space
>> is to be removed but TRAILING white space must NOT be.
>> 
>>  - Problem: this is simply inexpressible.  There is *NO*
>>    built-in predicate in section 5.2.1 that can be used to
>>    remove leading characters in some set without also
>>    removing trailing characters in the same set.
> 
> That is indeed true.  No clue how useful this is.  I don't
> recall I ever needed this.  I guess the same holds for the
> ECLiPSe people.  More below.

The thing is, if you give me the *building blocks*, I can
do it when I need it and you don't get involved at all.
> 
>> Consider another example.  I have the contents of a file
>> as a string.  I want to split it into lines at \n characters
>> and to remove trailing white space from each line.   But it
>> is important NOT to remove leading white space.  Yes, I have
>> real data like this.
>> 
>>  - Problem: this is the reverse of the previous problem.
> 
> That sounds like a fair use case.  We could of course solve
> this by adding two padding sets to create a split_string/5
> or replace PadChars with a term that provides distict
> leading and trailing padding.

But this is a step in the wrong direction.  The right way
is *not* to provide even more complex packaged solutions
but to provide simple building blocks.  Packaged solutions
you find handy can be provided in _Prolog_ in an optional
module.
>>  - Problem: the documentation of split_string/4 does not make it
>>    clear whether separators are 'characters' as the user might
>>    perceive them (that is, a base character plus zero or _more_
>>    floating diacriticals) or 'codepoints'.  I have a melancholy
>>    suspicion that the implementation is not diacritical-aware.
> 
> Strings are sequences of code points.  There is a package utf8proc
> that will become string aware to realise several of these operations.
> I have not yet decided how these will be integrated.

The thing is, given efficient "find this string" and "split this
string once _here_" building blocks, it becomes a trivial task to
program the specific combination I want here in Prolog, and it
ceases to be a problem for you at all.
> 
>> Consider another family of examples.  I have a lot of data in
>> <file> = <record>*
>> <record> = <field>*
>> format.  Some of it is AWK-style data, where the separator needs
>> to be a run of white space and fields can't be empty, as
> 
> This works fine (disregarding Unicode white space):
> 
>    split_fields(Line, Fields) :-
> 	split_string(Line, " \t", " \t", Fields).

Again, this contradicts the documentation in
http://www.swi-prolog.org/pldoc/man?section=string-predicates

>> split_string/4 is *too specialised*.
> 
> I think that the examples above indicate that is not true.

No, they don't.  What they DO illustrate is that what you
think split_string/4 does is inconsistent with what the
documentation says it does.  For several of the cases I
mentioned you did not offer a solution.

> It can do the familar AWK processing.

Only by contradicting the documentation.

> It can do /etc/passwd processing.

Which was never in dispute.  Trouble is, I almost never
process /etc/passwd in Prolog, and these days the
getpwnam(3C) functions in UNIX usually get it from LDAP.

> You can map
> strings to numbers (with its problems, such as localization and various
> representations used in various languages). The only thing it can not do
> is deal with quoted material.

Not true.  You have already conceded that it cannot handle
 - multi-character separators
 - trimming on one side but not the other
and I have shown that it cannot handle
 - non-uniform separators.

Given *simpler* building blocks, it is easy to program all
of these.

> I very much have my doubt about this.  You are most likely
> 
> in the domain of some data exchange or programming language
> and they all differ in the details here: escape the quote
> as "" or \", yes or no newlines allowed, allowed escape
> sequences, etc. etc.

Actually, that's pretty much my point.
It is precisely *because* there is such variation
that read_string/[3,5] are of such little use.

What's needed is *either* a configurable tokeniser that
can handle lots of things with a little tweaking *OR*
good building blocks.

Now read_string/3 is a bad building block because it
requires you to know how many code-points, which in general
you can't know, except for 1.
And read_string/5 is a *packaged solution*, not a building
block.
>> But again, this is both *overspecific* -- if I am using
>> single-character separators and allowing empty fields the
>> odds are that I don't want _any_ trimming -- and
>> *underpowered* -- it doesn't allow different trimming
>> at each end, it doesn't allow "squishing" internal runs of
>> white space to single spaces, it doesn't do a whole lot of
>> things, and it requires every field to be split the same
>> way.
> 
> So, you can do all, except for strip _only_ leading or
> _only_ trailing padding, which could be added easily.

Where do you can "do all" from? 
 - squishing?  No.
 - fields with different separators?  No.
 - multi-character separators (like, oh, "\r\n")?  No.
 - ...
Given the right *building blocks*, these things are easy.
Given a packaged solution, even things that are *possible*
may be difficult to write or obscure to read.
>> So instead of
>> 	compile_uniset(zs+zl+zp+"\t\n\f", Layout),
>> 	read_string(Stream, Layout, Layout, Sep, Wanted)
>> do
>> 	compile_uniset(zs+zl+zp+"\t\n\f", Layout),
>> 	compile_uniset(not(Layout), Non_Layout),
>> 	uniset_skip(Stream, Layout),
>> 	uniset_read_string(Stream, Non_Layout, Wanted),
>> 
>> Now _this_ is a building block that is useful in a tokeniser...
> 
> Not so sure. You get a non-layout sequence. It is hard to act on
> delimiters.

How is it hard to act on delimiters?  The delimiter is still
THERE in the stream.  How hard is it to read one more character
if you want it?

> For example, using read_string/5 we can implement reading
> Name(Arg1, Arg2, ...) using:
> 
> my_read_term(In, Term) :-
> 	read_string(In, "(", " \t\n", _, NameS),

Right here this goes wrong:  "foo(foo(1,2),3)"
will give you NameS = ["foo","foo","1,2),3)"].
Also note that it will accept "f o o(1)",
giving NameS = ["f o o"].

> 	atom_string(Name, NameS),
> 	read_string(In, ",)", " \t\n", Del, Arg1),
> 	(   Del == 0')
> 	->  (   Arg1 == ""
> 	    ->	Args = []
> 	    ;	Args = [Arg1]
> 	    )
> 	;   read_args(Del, In, Arg1, Args)
> 	),
> 	compound_name_arguments(Term, Name, Args).
> 
> read_args(0'), _, Last, [Last]).
> read_args(0',, In, Arg, [Arg|T]) :-
> 	read_string(In, ",)", " \t\n", Del, Arg1),
> 	read_args(Del, In, Arg1, T).
> 
> Which looks pretty reasonable to me.

It's pretty easy to break.

>> For "complete" read "complicate".
> 
> Regular expressions are fairly well understood things that are
> part of pretty much every standard and language.

That doesn't mean they are not complicated.
(It's news to me that they are part of standard Fortran or
standard COBOL or any of several other programming language
standards I keep handy.)

I agree that they are a practical necessity these days.
(Thank you Vassily Bykov!  Without you, my Smalltalk would lack them.)
> 
> Having delimeters as sequences is probably useful to have.  That
> is not in your character set approach either.

That's because my character set approach is not meant to be a
complete packaged solution!

>> These days, it really *really* does not make sense to "get
>> characters at index."  If you pick up a CJK ideograph and
>> DON'T pick up the plane E variant selector with it, and
>> then drop it down somewhere else, it will look *wrong*.
> 
> That is in general surely true, but there is enough code and
> there are enough languages where position picking works nice.
> Given that we have the position picking in atoms, you want at
> least the same set on strings to simplify porting code that
> uses atoms as strings to using strings right away.  Next, you
> can consider adding new primitives that handle unicode strings
> at a more appropriate level.

That's a good way to end up with a big collection of things
most of which don't quite work. 

>> Step 1.
>>   Given a string, the operation of returning a substring
>>   should take O(1) time and O(1) space.
> 
> That is likely to happen.  I'm wondering whether or not we
> want an interface that can tell us that a string is a substring
> at a certain location of another string?

For a public interface, definitely not.
> 
>>   Oh yeah, there needs to be a
>> 
>> 	get_string/[1,2]
>> 
>>   predicate that reads one "character" from (the current|an) input
>>   stream and unifies the last argument with it, no matter how many
>>   code-points are involved.
> 
> Also makes sense, although the name seems a bit misleading to me.

OK, call it

	get_combining_character_sequence_as_string/[1,2]

> It does require advanced Unicode processing though, and that is not
> something I see happen anytime soon unless someone is willing to do
> it.

"Advanced"?  By Unicode standards, this is grade 2.
> 
>> Ordinary programmers don't have a hope in the hot place of
>> getting this right, *especially* if they are seduced into working
>> with one code point at a time by giving them only an antique
>> interface.

I am planning to add this to my Smalltalk soon.  That's one
reason why I got Unicode Demystified.
>> 
>> Unicode?
>> 
>> Unicode is *horrifyingly* complex and is only getting worse.
>> Too much thinking about it and you get Krantzberg syndrome.
>> (http://www.tor.com/stories/2008/07/down-on-the-farm)

> Thanks for the comments. Despite your opening, I don't see that much
> wrong with what there is now. I think I've showed that split_string/4
> and read_string/5 are sensible.

No.  You have showed that (a) the documentation is wrong
and (b) they still can't do many of the simple things I want
to do, whereas an approach based on *building blocks* could
do everything they can do *and* most of the things I want.

> 
> P.s.	Considering characters, I sense some motivation to use
> 	atoms rather than code points, so you can represent a
> 	character with its (diacritical | variant selector*).
> 	Right?

Unicode has to work at several levels.
Code points *exist*, so there had *better* be a way of
representing code points as numbers.

It doesn't really make a lot of sense to represent '⿰'
as an atom; it's not a "character" but only part of a character.
Arguably the *right* way to represent a combining character
sequence (especially in a multithreaded environment) is a string,
not an atom.

Amongst other things, case conversion doesn't in general map
code points to code points, but sequences to sequences, so
even for that, strings seem like the right idea.

-------------- next part --------------
HTML attachment scrubbed and removed
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 22px-Saw_sawndip.svg.png
Type: image/png
Size: 625 bytes
Desc: not available
URL: <https://lists.iai.uni-bonn.de/pipermail/swi-prolog/attachments/20131219/5e589aa4/attachment.png>
_______________________________________________
SWI-Prolog mailing list
[email protected]
https://lists.iai.uni-bonn.de/mailman/listinfo.cgi/swi-prolog