[DBD::Pg 2/2] Commit UTF-8 design notes/discussion between DWC/GSM

Committed by David Christensen <[email protected]>

Subject: [DBD::Pg 2/2] Commit UTF-8 design notes/discussion between DWC/GSM

---
 TODO.utf8 |  161 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 161 insertions(+), 0 deletions(-)

diff --git a/TODO.utf8 b/TODO.utf8
new file mode 100644
index 0000000..5260bac
--- /dev/null
+++ b/TODO.utf8
@@ -0,0 +1,161 @@
+Summary of design changes from discussions with GSM and DWC re: utf-8 in DBD::Pg
+================================================================================
+
+Behavior of the pg_unicode/pg_utf8_strings connection attribute
+---------------------------------------------------------------
+We will utilize a connect attribute (enabled by default) to enable the
+use of an immediate SET client_encoding.  The current name of this is
+"pg_utf8_strings", but DWC prefers something non-encoding specific;
+examples wanted, but "pg_unicode" or "pg_internal" seem best.
+
+If the "pg_internal" attribute is explicitly provided in the DBI
+connect attributes it will be one of (0, 1), to enable/disable the
+pg_internal behavior explicitly.  If not provided, we check the
+initial "server_encoding" and "client_encoding" settings.
+
+The logic for setting "pg_internal" when unspecified is:
+
+ - If "server_encoding" is "SQL_ASCII" set pg_internal to 0.
+
+ - If "client_encoding" <> "server_encoding", or perhaps better yet if
+   the pg_setting("client_encoding") returns a different value than
+   the default version for that setting, then we assuming that the
+   client encoding choice is *explicit* and the user will be wanting
+   to get raw octets back from DBI, thus set pg_internal to 0.
+
+ - Otherwise set pg_internal to 1.
+
+Immediately after the connection initialization completes, we will
+check for the set pg_internal flag; if set, we issue a "SET
+client_encoding TO 'utf-8'" and commit.
+
+
+Proposal for an "encoding" DBD attribute interface
+--------------------------------------------------
+
+DWC suggested a DBD::db attribute handle, suggested to be called
+"encoding" which when set would effectively pass-thru to the
+underlying: "SET client_encoding = $blah" and *disable* the
+pg_internal flag.  Specifically, by setting the encoding attribute,
+you are effectively indicating that you want the data from PostgreSQL
+back
+
+If such a mechanism *was* instituted, we could utilize `pg_encoding =>
+'blah'` as the connection-level attribute and just tie the underlying
+implementation of the pg_internal mechanism to this, by having a
+keyword ('internal') as the special-case encoding, which could be
+enabled/disabled via $dbh->{pg_encoding} = 'internal';
+
+This would allow us to pass-through utf-8 *without* setting the SvUTF8
+flag by setting $dbh->{pg_encoding} = 'utf-8'.
+
+
+Behavior changes if pg_internal is set
+--------------------------------------
+
+There will be two distinct changes that need to take place,
+specifically input and output.
+
+When processing the result sets returned by the server, if pg_internal
+is set, we can either fiat that the "client_encoding" is set to UTF-8
+as it was originally when we switched it on connection, or verify that
+the libpq's result set charset/encoding is equal to UTF-8.  I believe
+this is available as an int, which could be cached when we do the
+original "SET client_encoding" and/or initial setup tests, which
+should prevent accidental corruption.
+
+ - if we decide to go this route and detect the charset change, we can
+   issue a notice/warning from DBD::Pg that the client_encoding has
+   changed and then turn off the pg_internal flag.
+
+ - if everything checks out, we use the usual dequote_* methods and
+   set the SvUTF8 flag on either text-based bytes, or set only on the
+   ASCII datums.
+
+ - a possible option to benchmark would be to directly use the
+   "utf8::upgrade" method from the perl internals (or some Sv-creation
+   method based on (char*)) to take advantage of any perl-specific
+   enhancements already in place.  This may be just as fast since perl
+   already needs to copy the (char*) contents into the SV, and may
+   already have fast-tracked code-paths for this type of operation,
+   since we know the data will be valid UTF8.
+
+When processing data coming *in* from the user i.e., (SV*) we consider
+the following:
+
+ - if pg_internal is 0, pass through the normal methods unabashed.
+
+ - if pg_internal is 1 and incoming SV's UTF8 flag is 1, we
+   do nothing; the underlying (char*) will already be in utf-8 data.
+
+ - if pg_internal is 1 and incoming SV's UTF8 flag is 0, we need
+   special consideration for hi-bit characters; since we've
+   effectively co-opted the expected client_encoding and forced UTF8,
+   we need to treat the raw data as octets.  We have a couple choices:
+
+     - treat as latin-1/perl raw.  This may be a good default choice,
+       but I'm not 100% convinced; in any case we would need to
+       convert from raw to utf-8 using utf8::upgrade.
+
+     - treat as original client_encoding.  This may be the least
+       changed expectation as far as the user is concerned, but
+       requires us to either:
+
+       a) switch client_encoding for query to the original
+          client_encoding, while somehow still retaining the utf-8
+          client encoding for result set retrieval, or,
+
+       b) actually use Encode to transcode from the original
+          client_encoding to UTF8.  I think GSM is particularly
+          against bringing Encode into the picture just due to
+          additional complexity issues.
+
+
+Implementation considerations/ideas
+-----------------------------------
+
+DWC feels strongly that we should avoid setting the SvUTF8 flag on any
+retrieved/created SV which does not require it; as such, an operation
+that can quickly check whether there are any hi-bit characters in a
+given (char*) would need to be weighed against the possible
+inconvenience of *always* setting the SvUTF8 flag on eligible strings,
+regardless of whether it is full ASCII.
+
+Considering that we already utilize strlen() which traverses the
+entire string, even a naïve replacement which simply tracks the number
+of hi-bit chars encountered while traversing for that length may be a
+low-enough overhead that it will not be an undue burden in detecting
+this situation.
+
+We can also take advantage of two algorithmic enhancements if we know two things:
+
+- firstly, if we somehow already have the length of the initial
+  string/structure, (via the libpq structures), we can unroll the
+  hi-bit detection into the largest-supported unit-sized words; i.e.,
+  on a 64-bit machine we could check 8 bytes at a time for presence of
+  high-bits, simply by checking against the (char*) against mask
+  0x88888888; if that result is non-zero then we have at least one
+  high-bit character in the batch.  (We can do compile-time checks to
+  determine the largest word size and have different versions of the
+  loop depending on said determintion.)  We would look at the (length
+  % word size) and utilize a different mask (presumably from a local
+  static LUT) on the remainder for the final detection.  This makes
+  the number of operations in the worst-case scenario O(n/wordsize +
+  1)
+
+- if we don't care about the *total* number of hi-bit chars (which we
+  may or may not), we can short-circuit the custom_strlen at the first
+  hi-bit and return (current_length) + system strlen(*currentp).
+  i.e., this would detect cases where there is a hi-bit present while
+  falling back to the (presumed) faster system-level strlen on the
+  rest of the buffer.
+
+We will look at updating/benchmarking various implementations of a
+combined strlen()/has_high_bit function to use in determining
+auto-upgrade to utf8 behavior.  This will allow us to skip an SV copy
+for the data that is potentially modified in-place.
+
+Which is better here depends on what is more expensive; this hi-bit
+check or the unequivocable copy of the SV data in case we have hi-bit
+data that needs to be utf8::upgrade'd.
+
-- 
1.7.0.5