Re: SWI-Prolog 7.1.3

Hi Richard,

This is a bit long to answer in detail. I choose to ignore Unicode for
the time being, except being able to represent Unicode strings as
sequences of Unicode code points. It is way too complicated for me. I'm
afraid this simply has to wait for resources from people with an
interest to get these things resolved for their language. The
organizations that support SWI-Prolog now with money and/or programming
effort do not really have in interest in this AFAIK.  They mostly want
SWI-Prolog to be able to read, represent and write Unicode strings.

I picked up your section 10 of pllib.htm. That is a bit easier to digest.
I'm tempted to take string_leading_count/4 and string_trailing_count/4
on board. In my experience with sub_string/5 (yes, basically agree with
your observation about the argument order, but there is ISO sub_atom/5
that won't disappear and having two so similar predicates with different
argument order is IMHO worse than having both (consistently) wrong),
I don't see much need for versions with a reduced number of arguments.
Same thing happens with format/1,2,3.  Too often I decide to format to
user_error, so I change

	format('Hello world~n').
into	format(user_error, 'Hello world~n').

which doesn't do what I want (I guess it should give an error, which it
doesn't do right now because the format string doesn't make it look into
the argument list).

Just a format/2,3 would have been sufficient.

Can we call these string_leading_LENGTH?  That seems more consistent
with string_length?

	Cheers --- Jan

P.s.	Fixed docs for split_string/4.

On 12/19/2013 02:02 AM, Richard A. O'Keefe wrote:
>
> On 19/12/2013, at 2:37 AM, Jan Wielemaker wrote:
>
>>> The first thing I note is that there are two arguments that
>>> specify a set of characters, but there is no set-of-characters
>>> data type.  (See pllib.htm section 1.5.)
>>
>> That was noted.  ECliPSe is 8-bit only, so it doesn't matter too
>> much.  Joachim and I decided that we would look into a more abstract
>> character set notion later.  Trying to solve that right away would
>> have complicated getting to an agreement on the primitives too much.
>
> This seems back to front to me.
> I know about the XP slogan YAGNI,
> but in this case, since SWI Prolog has been dealing with
> Unicode characters after a fashion for some time,
> you *KNOW* that you *WILL* need it.
> What makes the most sense is to DESIGN for the full case
> but IMPLEMENT for the limited case (at first).
>
> In the 21st century, a set of string handling operations
> that isn't *designed* to help programmers cope with the
> weirdness of Unicode is, well, I'm looking for polite
> terms.  It's like designing a programming language in
> which integers cannot be any bigger than 16 bits.
>
> We already *KNOW* this isn't going to *WORK* long term.
>
> I got a copy of "Unicode Demystified" last week, and have
> been working my way through it.  It turns out that Unicode
> 3.2 was already worse than I realised.  Let me give you an
> example, which I know is relevant to at least one person
> who has posted several times to this mailing list.
>
> The Indic scripts are alphabetic, but they take the
> consonants as a basic framework and attach the vowels
> to them.  Some vowels go on the right of the consonant,
> some on the left, some on both sides and this is an
> oversimplification.  Using adjacency to indicate fusion
> and "." to indicate side by side, we might have
> "LO.IG.K/" where the / cancels the implied "a" vowel.
> (Problem here: "K" by itself indicates "KA"; you need
> the virama to cancel the "A".)  So what do we do about
> the vowels?  There are two possibilities:
>  - encode the letters in their visual order (as
>    displayed on the page)
>  - encode the letters in the logical order (as
>    pronounced).
> I'd read all the stuff about Devanagari and so on, and
> understood clearly that Unicode had unequivocally plumped
> for logical order (L,O,G,I,K,/).  What I _hadn't_ noticed
> that the "round trip compatibility with existing standards"
> part had caused them to adopt _visual order_ for Thai and
> Lao.  So Unicode uses *both* principles within a single
> strongly related family of scripts.  Ouch.
>
> Then there is the fun with vowel signs that are written on
> both sides of a consonant.  Is "<X>" to be encoded as
> (<>,X), as (X,<>), or as (<,X,>)?  More than one alternative
> may be available within the same script!
>
> All of this means that working "one character at a time",
> the way that is so *obvious* and *easy* in most
> European (Latin, Greek, Cyrillic, Armenian, Georgian)
> scripts is fraught with extreme difficulty in other
> scripts.  I'll get to what seems to me to be the obvious
> conclusion shortly.
>
>> Indeed.  We'll check out pllib.htm before deciding on a
>> character set notion.
>
> One of the things I have found extremely frustrating about
> writing pllib.htm is that people seem to be treating it as
> a specific concrete proposal to be taken or left, picking
> what they like, and going on.  But
>
> PLLIB.HTM IS SUPPOSED TO BE
> THE OPENING OF A CONVERSATION.
>
> I wanted it to engender a *discussion*.
>
> Now here's "what seems to me to be the obvious conclusion".
>
> EUROPEAN PROLOG IMPLEMENTORS ARE THE WRONG
> PEOPLE TO DESIGN TEXT HANDLING OPERATIONS
> BY THEMSELVES.
>
> I include myself as someone of European descent who used to
> be a Prolog implementor.  I live in a country with a non-
> European official language, but it uses a Latin-based script
> and worked very nicely in an 8-bit world.
>
> Over a thousand million people use an Indic script.
> Over a thousand million people use a Han script.
> Over 400 million people use the Arabic script (it's
> probably a lot more, it's used in _lots_ of places).
>
> Did you know that there is a mechanism in Unicode
> for *describing* Chinese characters that are not
> yet encoded?  It's recursive, and written prefix,
> with U+2FF0...U+2FFB.  From the Wikipedia,
> the character “Saw sawndip.svg” can be described as “⿰書史”.
> A little "tree" built this way should normally be
> treated as a single combining character sequence;
> such trees can have up to 6 leaves.  (I do not know
> whether ideographic variant selectors can be mixed in
> with this.  I don't really _want_ to know.)
>
> An *adequate* design team for string operations in the
> 21st century will include at least one person
> intimately familiar with CJK text processing,
> at least one person intimately familiar with Indic
> text processing, at least one person intimately familiar
> with Arabic text processing, none of these people needing
> deep understanding of Prolog, *and* at least one Prolog
> implementor.
>
> What I *can* be sure of is that we need *building blocks*,
> not complete packaged predicates, and that we certainly
> need to be able to deal with the patterns in UAX29 and
> UAX31.
>>>  - Problem: if I'm reading the documentation correctly,
>>>    this doesn't do at all what I want.  I want a string
>>>    splitter that takes *RUNS* of separator characters
>>>    as separators.  Suppose the input is
>>> "Lorem ipsum.  Dolor sit amet. "
>>>    If the separators include spaces, and the number of
>>>    Substrings is one more than the number of spaces,
>>>    then this would return
>>> ["Lorem","ipsum","","Dolor","sit","amet",""]
>>>                          ^^                      ^^
>>>    with empty strings that are worse than useless to me.
>>
>> You deal with runs of separator characters by making
>> SepChars and PadChars the same.  Then you never and up
>> with empty strings.  I guess the most typical case for
>> this is if these sets represent white space.
>
> In that case, the documentation needs rewriting.
> The documentation is absolutely unambiguous:
> The number of SubStrings is one more than
> the number of separators found.
> In this example, the number of separators is 6 so there
> is no wiggle room whatsoever, the number of substrings
> MUST be seven.  The documentation says NOTHING about
> the number of substrings depending in any way whatsoever
> on what the PadChars might be.
>
> In any case, this doesn't address a problem that comes up
> more often than you might expect: a "record" where the
> separators are *different*.  For example, we might have
> <label>:<field>{,<field>}*
> where commas in the <label> are just data and colons in
> the <field>s are just data.
>
> If you give me a building block that can split off ONE
> field, I can do
>
> parse_record(String, Label, Fields) :-
>     split_one_field(String, Label, ":", Rest),
>     split_all_fields(Rest, ",", Fields).
>
> split_all_fields(String, Sep, [Field|Fields]) :-
>     split_one_field(String, Field, Sep, Rest),
>     !,
>     split_all_fields(Rest, Sep, Fields).
> split_all_fields(Rest, _, [Rest]).
>
> If you give me a "packaged solution" that splits off all
> the fields, I'm stuck.
>
>>>    Note in particular that if you use this with Unix filenames
>>>    it will go wrong.  The parts of "/foo//bar///ugh" are
>>>    "foo", "bar", and "ugh" -- there are no empty parts there.
>>
>> So,
>>
>>   ?- split_string("/foo//bar///ugh", "/", "/", L).
>>   L = ["foo", "bar", "ugh"].
>
> Yes, but THAT IS NOT COMPATIBLE WITH THE DOCUMENTATION.
>
> Here it is again:
> The SepChars argument provides the characters that
> act as separators and thus the length ofSubStrings
> is one more than the number of separators found.
>
> That is not ambiguous.  SepChars is "/".  There are six
> slashes in the string.  Therefore according to the
> documentation there *MUST* be seven strings in the answer;
> there *cannot* be 3.
>
> In fact, I don't even need a building block for splitting
> off one field.
>
> The building blocks I need are
> (1) Find all of these characters (string argument).
>     (So that I can use ": " as a separator,
>     with ":" and " " in other combinations
>     being plain data.)
> (2) Find any of these characters (set argument).
> (3) Count leading or trailing characters in a set.
> (4) Given RSTART and RLENGTH, split a string _here_,
>     in O(1) time and space.
> Given those, I can trivially *program* any string splitting
> and trimming operations I want in Prolog.
>
>>> Consider another example.  I have some data that need to be
>>> treated the way M4 treats macro arguments.  That is, the
>>> string is to be split at commas, and then LEADING white space
>>> is to be removed but TRAILING white space must NOT be.
>>>
>>>  - Problem: this is simply inexpressible.  There is *NO*
>>>    built-in predicate in section 5.2.1 that can be used to
>>>    remove leading characters in some set without also
>>>    removing trailing characters in the same set.
>>
>> That is indeed true.  No clue how useful this is.  I don't
>> recall I ever needed this.  I guess the same holds for the
>> ECLiPSe people.  More below.
>
> The thing is, if you give me the *building blocks*, I can
> do it when I need it and you don't get involved at all.
>>
>>> Consider another example.  I have the contents of a file
>>> as a string.  I want to split it into lines at \n characters
>>> and to remove trailing white space from each line.   But it
>>> is important NOT to remove leading white space.  Yes, I have
>>> real data like this.
>>>
>>>  - Problem: this is the reverse of the previous problem.
>>
>> That sounds like a fair use case.  We could of course solve
>> this by adding two padding sets to create a split_string/5
>> or replace PadChars with a term that provides distict
>> leading and trailing padding.
>
> But this is a step in the wrong direction.  The right way
> is *not* to provide even more complex packaged solutions
> but to provide simple building blocks.  Packaged solutions
> you find handy can be provided in _Prolog_ in an optional
> module.
>>>  - Problem: the documentation of split_string/4 does not make it
>>>    clear whether separators are 'characters' as the user might
>>>    perceive them (that is, a base character plus zero or _more_
>>>    floating diacriticals) or 'codepoints'.  I have a melancholy
>>>    suspicion that the implementation is not diacritical-aware.
>>
>> Strings are sequences of code points.  There is a package utf8proc
>> that will become string aware to realise several of these operations.
>> I have not yet decided how these will be integrated.
>
> The thing is, given efficient "find this string" and "split this
> string once _here_" building blocks, it becomes a trivial task to
> program the specific combination I want here in Prolog, and it
> ceases to be a problem for you at all.
>>
>>> Consider another family of examples.  I have a lot of data in
>>> <file> = <record>*
>>> <record> = <field>*
>>> format.  Some of it is AWK-style data, where the separator needs
>>> to be a run of white space and fields can't be empty, as
>>
>> This works fine (disregarding Unicode white space):
>>
>>    split_fields(Line, Fields) :-
>> split_string(Line, " \t", " \t", Fields).
>
> Again, this contradicts the documentation in
> http://www.swi-prolog.org/pldoc/man?section=string-predicates
>
>>> split_string/4 is *too specialised*.
>>
>> I think that the examples above indicate that is not true.
>
> No, they don't.  What they DO illustrate is that what you
> think split_string/4 does is inconsistent with what the
> documentation says it does.  For several of the cases I
> mentioned you did not offer a solution.
>
>> It can do the familar AWK processing.
>
> Only by contradicting the documentation.
>
>> It can do /etc/passwd processing.
>
> Which was never in dispute.  Trouble is, I almost never
> process /etc/passwd in Prolog, and these days the
> getpwnam(3C) functions in UNIX usually get it from LDAP.
>
>> You can map
>> strings to numbers (with its problems, such as localization and various
>> representations used in various languages). The only thing it can not do
>> is deal with quoted material.
>
> Not true.  You have already conceded that it cannot handle
>  - multi-character separators
>  - trimming on one side but not the other
> and I have shown that it cannot handle
>  - non-uniform separators.
>
> Given *simpler* building blocks, it is easy to program all
> of these.
>
>> I very much have my doubt about this.  You are most likely
>> in the domain of some data exchange or programming language
>> and they all differ in the details here: escape the quote
>> as "" or \", yes or no newlines allowed, allowed escape
>> sequences, etc. etc.
>
> Actually, that's pretty much my point.
> It is precisely *because* there is such variation
> that read_string/[3,5] are of such little use.
>
> What's needed is *either* a configurable tokeniser that
> can handle lots of things with a little tweaking *OR*
> good building blocks.
>
> Now read_string/3 is a bad building block because it
> requires you to know how many code-points, which in general
> you can't know, except for 1.
> And read_string/5 is a *packaged solution*, not a building
> block.
>>> But again, this is both *overspecific* -- if I am using
>>> single-character separators and allowing empty fields the
>>> odds are that I don't want _any_ trimming -- and
>>> *underpowered* -- it doesn't allow different trimming
>>> at each end, it doesn't allow "squishing" internal runs of
>>> white space to single spaces, it doesn't do a whole lot of
>>> things, and it requires every field to be split the same
>>> way.
>>
>> So, you can do all, except for strip _only_ leading or
>> _only_ trailing padding, which could be added easily.
>
> Where do you can "do all" from?
>  - squishing?  No.
>  - fields with different separators?  No.
>  - multi-character separators (like, oh, "\r\n")?  No.
>  - ...
> Given the right *building blocks*, these things are easy.
> Given a packaged solution, even things that are *possible*
> may be difficult to write or obscure to read.
>>> So instead of
>>> compile_uniset(zs+zl+zp+"\t\n\f", Layout),
>>> read_string(Stream, Layout, Layout, Sep, Wanted)
>>> do
>>> compile_uniset(zs+zl+zp+"\t\n\f", Layout),
>>> compile_uniset(not(Layout), Non_Layout),
>>> uniset_skip(Stream, Layout),
>>> uniset_read_string(Stream, Non_Layout, Wanted),
>>>
>>> Now _this_ is a building block that is useful in a tokeniser...
>>
>> Not so sure. You get a non-layout sequence. It is hard to act on
>> delimiters.
>
> How is it hard to act on delimiters?  The delimiter is still
> THERE in the stream.  How hard is it to read one more character
> if you want it?
>
>> For example, using read_string/5 we can implement reading
>> Name(Arg1, Arg2, ...) using:
>>
>> my_read_term(In, Term) :-
>> read_string(In, "(", " \t\n", _, NameS),
>
> Right here this goes wrong:  "foo(foo(1,2),3)"
> will give you NameS = ["foo","foo","1,2),3)"].
> Also note that it will accept "f o o(1)",
> giving NameS = ["f o o"].
>
>> atom_string(Name, NameS),
>> read_string(In, ",)", " \t\n", Del, Arg1),
>> (   Del == 0')
>> ->  (   Arg1 == ""
>>    ->Args = []
>>    ;Args = [Arg1]
>>    )
>> ;   read_args(Del, In, Arg1, Args)
>> ),
>> compound_name_arguments(Term, Name, Args).
>>
>> read_args(0'), _, Last, [Last]).
>> read_args(0',, In, Arg, [Arg|T]) :-
>> read_string(In, ",)", " \t\n", Del, Arg1),
>> read_args(Del, In, Arg1, T).
>>
>> Which looks pretty reasonable to me.
>
> It's pretty easy to break.
>
>>> For "complete" read "complicate".
>>
>> Regular expressions are fairly well understood things that are
>> part of pretty much every standard and language.
>
> That doesn't mean they are not complicated.
> (It's news to me that they are part of standard Fortran or
> standard COBOL or any of several other programming language
> standards I keep handy.)
>
> I agree that they are a practical necessity these days.
> (Thank you Vassily Bykov!  Without you, my Smalltalk would lack them.)
>>
>> Having delimeters as sequences is probably useful to have.  That
>> is not in your character set approach either.
>
> That's because my character set approach is not meant to be a
> complete packaged solution!
>
>>> These days, it really *really* does not make sense to "get
>>> characters at index."  If you pick up a CJK ideograph and
>>> DON'T pick up the plane E variant selector with it, and
>>> then drop it down somewhere else, it will look *wrong*.
>>
>> That is in general surely true, but there is enough code and
>> there are enough languages where position picking works nice.
>> Given that we have the position picking in atoms, you want at
>> least the same set on strings to simplify porting code that
>> uses atoms as strings to using strings right away.  Next, you
>> can consider adding new primitives that handle unicode strings
>> at a more appropriate level.
>
> That's a good way to end up with a big collection of things
> most of which don't quite work.
>
>>> Step 1.
>>>   Given a string, the operation of returning a substring
>>>   should take O(1) time and O(1) space.
>>
>> That is likely to happen.  I'm wondering whether or not we
>> want an interface that can tell us that a string is a substring
>> at a certain location of another string?
>
> For a public interface, definitely not.
>>
>>>   Oh yeah, there needs to be a
>>>
>>> get_string/[1,2]
>>>
>>>   predicate that reads one "character" from (the current|an) input
>>>   stream and unifies the last argument with it, no matter how many
>>>   code-points are involved.
>>
>> Also makes sense, although the name seems a bit misleading to me.
>
> OK, call it
>
> get_combining_character_sequence_as_string/[1,2]
>
>> It does require advanced Unicode processing though, and that is not
>> something I see happen anytime soon unless someone is willing to do
>> it.
>
> "Advanced"?  By Unicode standards, this is grade 2.
>>
>>> Ordinary programmers don't have a hope in the hot place of
>>> getting this right, *especially* if they are seduced into working
>>> with one code point at a time by giving them only an antique
>>> interface.
>
> I am planning to add this to my Smalltalk soon.  That's one
> reason why I got Unicode Demystified.
>>>
>>> Unicode?
>>>
>>> Unicode is *horrifyingly* complex and is only getting worse.
>>> Too much thinking about it and you get Krantzberg syndrome.
>>> (http://www.tor.com/stories/2008/07/down-on-the-farm)
>
>> Thanks for the comments. Despite your opening, I don't see that much
>> wrong with what there is now. I think I've showed that split_string/4
>> and read_string/5 are sensible.
>
> No.  You have showed that (a) the documentation is wrong
> and (b) they still can't do many of the simple things I want
> to do, whereas an approach based on *building blocks* could
> do everything they can do *and* most of the things I want.
>
>>
>> P.s.Considering characters, I sense some motivation to use
>> atoms rather than code points, so you can represent a
>> character with its (diacritical | variant selector*).
>> Right?
>
> Unicode has to work at several levels.
> Code points *exist*, so there had *better* be a way of
> representing code points as numbers.
>
> It doesn't really make a lot of sense to represent '⿰'
> as an atom; it's not a "character" but only part of a character.
> Arguably the *right* way to represent a combining character
> sequence (especially in a multithreaded environment) is a string,
> not an atom.
>
> Amongst other things, case conversion doesn't in general map
> code points to code points, but sequences to sequences, so
> even for that, strings seem like the right idea.
>
_______________________________________________
SWI-Prolog mailing list
[email protected]
https://lists.iai.uni-bonn.de/mailman/listinfo.cgi/swi-prolog