New text re. URLs and character encodings

Hello URLers,

This is an updated proposal for addressing internationalization
issues in the URL process draft. This proposal maintains the
intention of my original proposal, while (hopefully) addressing
valuable comments made by Larry Masenter and others.


Regards,	Martin.


> INTERNET-DRAFT                                    Larry Masinter
> <draft-ietf-url-process-00>                          Dan Zigmond
> January 4, 1997                             Harald T. Alvestrand
> expires June 4, 1997
> 
>                Guidelines and Process for new URL Schemes


> 2.2.1 Clear mapping from other name spaces
> 
>    In many cases, new URL schemes are defined as ways to translate
>    other protocols and name spaces into the general framework of
>    URLs. The "ftp" URL scheme translates from the FTP protocol, while
>    the "mid" URL scheme translates from the Message-ID field of
>    messages.
> 
>    In either case, the description of the mapping must be complete,
>    must describe how character sets get encoded or not in URLs, must
>    describe exactly how all legal values of the base standard can be
>    represented using the URL scheme, and exactly which modifiers,
>    alternate forms and other artifacts from the base standards are
>    included or not included.

In my first proposal, I have explicitly included a requirement to
fully document the issue of character encoding. It turns out that
it is already here, with a "must", so that no further text is needed.
However, because the term "character set" is often confused with
"charset" (to which it is not equivalent), and because it is
individual characters, and not character sets, that get encoded,
I propose to change this paragraph as follows:

   In either case, the description of the mapping must be complete,
   must describe how characters get encoded or not in URLs (see
   Section X.Y), must
   describe exactly how all legal values of the base standard can be
   represented using the URL scheme, and exactly which modifiers,
   alternate forms and other artifacts from the base standards are
   included or not included.


>    In all cases, encoding rules must be made clear: What octets are
>    put into the URL, and if other octets need to be represented, what
>    convention is used to represent them?  Any departure from the %xx
>    convention needs special justification.

This paragraph also benefits from an update, to refer to characters
as well as to octets. Encoding rules, as of the current syntax
draft (not even including my proposed additions), are concerned
both with character->octet and with octet->%HH encoding. The
paragraph should be changed to:

   In all cases, encoding rules must be made clear: What characters
   or octets are put into the URL, and if other characters or octets
   need to be represented, what convention is used to represent them?
   Any departure from the UTF-8 convention for characters and the %xx
   convention for octets needs special justification.



The following section should be added:

   X.Y Character Encoding

[Note that I have avoided to use the term "internationalization",
because some don't like it to be used (although there is clear
evidence that it is widely accepted in the literature about the
subject).]

   When describing URL schmemes in which (some of) the elements of
   the URL are actually representations of sequences of characters,
   care should be taken not to introduce unnecessary variety in the
   ways in which characters are encoded into octets and then into
   URL characters. Unless there is some compelling reason for a
   particular scheme to do otherwise, translating character sequences
   into UTF-8 [RFC 2044] and then subsequently using the %HH encoding
   for unsafe octets is recommended.

[This paragraph is from Larry, I made two changes: There is no "new"
in the first line because the draft as a whole anyway applies only
to new URL schemes, and the "characters" in the last line is changed
to "octets" to be more precise.]

   Note: RFC 2044 specifies UTF-8 in terms of Unicode Version 1.1,
   corresponding to ISO 10646 without ammendments. It is widespread
   consensus that this should indeed be Unicode Version 2.0,
   corresponding to ISO 10646 including ammendment 5
   (and future additions thereto).

[This note should be kept to avoid ambiguities, up to the point where
an updated version of RFC 2044 will become available.]

   X.Y.1  Avoiding Variants

   If a protocol allows multiple ways to access the same resource,
   e.g. if multiple alternative character encodings or byte sequences
   can be used to access a particular resource, it is preferable for
   the URL scheme to reduce the variability, even at the cost of
   introducing a translation when mapping from URL to protocol.
   This will reduce the number of alternative URLs for the same
   resource, and encourage consistency. In particular, if a
   protocol supports multiple character encodings for strings,
   and there is enough context in the protocol to know these
   encodings, it is preferable to denote UTF-8 as the character
   encoding used in URLs to reduce ambiguity.

[This is more or less the case that Larry proposed, but less
specific. In particular, UTF-8 is also suggested for cases where
there are various encodings but none of them is UTF-8.]



   X.Y.2  Protocol-Specific Character Encodings

   If a protocol uses a single but protocol-specific character
   encoding which is different from UTF-8, the encoding used in
   URLs should nevertheless be UTF-8, and a mapping from URL to
   protocol encoding should be defined, because generic URL software is
   unlikely to be aware of and to be able to handle protocol-specific
   character encoding conventions and will therefore be unable to
   handle the correct characters in a user interface.

[This is designed to handle cases such as IMAP, where a single,
but highly protocol-specific encoding of characters has been
defined. The wording of the justification give here (which is
the main justification for all of the effort to define UTF-8
as the preferred URL encoding) is in part taken from a private
mail by Chris Newman, the author of the IMAP URL spec.]



   X.Y.3  Handling Unspecified Character Encodings

   There may be protocols and schemes that allow the use of various
   character encodings by implicit agreement between server and
   client, without providing a means to identify the encoding used.
   In this case, the URL should reflect the raw octet sequence used
   by the protocol. To aid in more consistent treatment of URLs
   for such protocols, UTF-8 should at least be included in the
   encodings that can be used by implicit agreement. Preferably,
   the protocol is upgraded so that UTF-8 is the preferred encoding
   or so that the encoding accepted by the server is made explicit,
   and in these cases, UTF-8 can be used in URLs.


[This is written to handle cases such as FTP,... ]



> 6. References

[RFC 2044]    F. Yergeau, "UTF-8, A Transformation Format of Unicode
                  and ISO 10646", Alis Technologies, October 1996.