7.1 string operations (was Re: Ann: SWI-Prolog 6.5.3)

On 11/22/2013 01:38 AM, Richard A. O'Keefe wrote:
> OK.  If you think strings are a good data type, shouldn't they have
> more operations?  And should the manual still be saying
> 	new code should consider using atoms

According to grep, that is no longer in the manual now.

> ?  Would it be a good idea to replace string_to_list/2 (which name

string_to_list/2 has been replaced with string_codes/2.

> suggests conversion that goes ONLY left to right) with
> 	string_codes(?String, ?Codes)
> 	string_chars(?String, ?Chars)

Just added that.

> like atom_codes/2 and atom_chars/2?  Shouldn't there be a
> number_string/2 to go with number_{codes,chars}/2?

Added number_string/2 as well.

> Would it be a good idea to
> document that/whether strings work in UTF-8, and whether "characters"
> in the documentation of string_length/2 and sub_string/5 means "bytes"
> or "Unicode code-points"?  Would it be a good idea to offer the

Strings are sequences of characters.  SWI-Prolog is Unicode aware,
so they are sequences of Unicode characters.   I think there is little
point repeating that over and over.  UTF-8 plays no role here.  In fact,
like atoms, the representation is either a byte-array of ISO-Latin-1
codes or an array of 32-bits characters (actually wchar_t, which means
UTF-16 on Windows, but that is an issue that should be resolved
globally).

>     span_left(String, Set, Before[, Length[, After]])
>     span_right(String, Set[[, Before], Length], After)
> predicates that Xerox Quintus Prolog did?  -- quite helpful for
> tokenising.

Found the docs.  I'm not really convinced.  Set is an atom,
code list or not(Set) as I understand it.  These simple
sets do not play very well with Unicode.  Even removing
white space is already getting hard this way.  You'd want
at least something that allows for character ranges and
Unicode classes similar to regex.

We probably want a split-like predicate, similar to atomic_list_concat/3.
The naming is a bit hard as there is no atom in this name.
Maybe atomic_list_string(?List, +Separator, ?String)?

Then there is long pending thing called regular expressions.  Strings
is the right moment to introduce them.  But, how?  Notable, which code
base and according to which of the many regex standards?  Is there a
code base that provides a function pointer to get the next character?

	Cheers --- Jan