Re: DBD::mysql path forward

What Night Light's post says to me is that there is high risk of causing data 
corruption if any changes are made under the DBD::mysql name where DBD::mysql 
has not been exhaustively tested to guarantee that its behavior is backwards 
compatible.

This makes a stronger case to me that the DBD::mysql Git master (that which 
includes the 4.042 changes and any other default breaking changes) should rename 
the Perl driver package name, I suggest DBD::mysql2 version 5.0, and that any 
changes not guaranteed backwards compatible for whatever reason go there.

If the Git legacy maintenance branch 4.041/3 can have careful security patches 
applied that don't require any changes to user code to prevent breakage, it gets 
them, and otherwise only DBD::mysql2 gets any changes.

By doing what I said, we can be guaranteed that users with no control over how 
DBD::mysql gets upgraded for them will introduce corruption simply for upgrading.

-- Darren Duncan

On 2017-09-19 5:46 AM, Night Light wrote:
> Dear Perl gurus,
>
> This is my first post. I'm using Perl with great joy, and I'd like to express my
> gratitude for all you are doing to keep Perl stable and fun to use.
>
> I'd like to ask to object to re-releasing this version and discuss on how to
> make 4.043 backwards compatible instead.
> This change will with 100% certainty corrupt all BLOB data written to the
> database when the developer did not read the release notes before applying the
> latest version of DBD::mysql (and changed its code consequently).
> Knowing that sysadmins have the habit of not always reading the release notes of
> each updated package the likelihood that this will happen will therefore high.
> I myself wasn't even shown the release notes as it was a dependency of an
> updated package that I applied.
> The exposure of this change is big as DBD::mysql affects multiple applications
> and many user bases.
> I believe deliberately introducing industry wide database corruption is
> something that will significantly harm peoples confidence in using Perl.
> I believe that not providing backwards compatibility is not in line with the
> Perl policy that has been carefully put together by the community to maintain
> the quality of Perl as it is today.
> http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DEPRECATION
>
> I therefore believe the only solution is an upgrade that is by default backwards
> compatible, and where it is the user who decides when to start UTF8 encode the
> input values of a SQL request instead.
> If it is too time consuming or too difficult it should be considered to park the
> UTF8-encoding "fix" and release a version with the security fix first.
>
> I have the following objections against this release:
>
> 1. the upgrade will corrupt more records than it fixes (it does more harm than good)
> 2. the reason given for not providing backward compatibility ("because it was
> hard to implement") is not plausible given the level of unwanted side effects.
>    This especially knowing that there is already a mechanism in place to signal
> if its wants UTF8 encoding or not (mysql_enable_utf8/mysql_enable_utf8mb4).
> 3. it costs more resources to coordinate/discuss a "way forward" or options than
> to implement a solution that addresses backwards compatibility
> 4. it is unreasonable to ask for changing existing source knowing that depending
> modules may not be actively maintained or proprietary
>    It can be argued that such module should always be maintained but it does not
> change the fact that a good running Perl program becomes unusable
> 5. it does not inform the user that after upgrading existing code will start
> write corrupt BLOB records
> 6. it does not inform the user about the fact that a code review of all existing
> code is necessary, and how it needs to be changed and tested
> 7. it does not give the user the option to decide how the BLOB's should be
> stored/encoded (opt in)
> 8. it does not provide backwards compatibility
>    By doing so it does not respect the Perl policy that has been carefully put
> together by the community to maintain the quality of Perl as it is today.
>    http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DEPRECATION
> 9. it blocks users from using DBD::mysql upgrades as long as they have not
> rewritten their existing code
> 10. not all users from DBD::mysql can be warned beforehand about the side
> effects as it is not known which private parties have code that use DBD::mysql
> 12. I believe development will go faster when support for backwards
> compatibility is addressed
> 13. having to write 1 extra line for each SQL query value is a monks job that
> will make the module less attractive to use
>
> About forking to DBD::mariadb?:
> The primary reason to create such a module is when the communication protocol of
> Mariadb has become incompatible with Mysql.
> To use this namespace to fix a bug in DBD::mysql does not meet that criteria and
> causes confusion for developers and unnecessary pollution of the DBD namespace.
>
> ---
>
> For people that do not know the impact of the change that is pending to be
> committed:
> (see Github issue that includes 3 reports of companies that suffered data loss
> https://github.com/perl5-dbi/DBD-mysql/issues/117 )
>
> Issue: some UTF8 characters are not properly displayed after retrieval
> Cause: SQL query values are not UTF8 encoded when sent to the database but they
> are all decoded once retrieved.
> Occurence: Only records with string data that can only be written with UTF8. It
> can be considered rare as people haven't reported this issue after 10 years of
> usage.
> Regional impact: Only affects countries which characters need UTF8 encoding and
> only affects string values.
> Steps to recover from it: Read string data unencoded and write it encoded.
>
> Changes of upgrade pending to be re-released:
> SQL query values are both UTF8 encoded when sent to the database as when its
> retrieved (including BLOB fields).
> BLOB fields will be excluded from encoding only if you specify its data type.
>
> Side effects from installing upgrade:
> - BLOB data will be written after UTF8 encoding and will therefore be corrupt
> - no possibility to detect if a BLOB field is corrupt or not. Only when known
> when the INSERT/UPDATE took place, and when the upgrade was installed
> - existing data will still display incorrect
>
> Occurence: every INSERT/UPDATE statement will start writing corrupted BLOB data
> Regional impact: worldwide
> Steps to recover from it corrupted BLOBs? You cannot. Your binary blobs are
> encoded as if they were UTF8 strings. Your binary data is unrecoverable (as in
> "gone forever").
> If you are a dentist you have to ask your customers to come back to make another
> x-ray as the made photo's are gone.
>
> What is asked from the developer to prevent this from happening?
> - do not miss reading the release notes before upgrading
> - review all source code (including written by other included modules) and
> specify the data type of each SQL parameter value
>   before: $dbh->do('INSERT INTO test (BLOB1,BLOB2,BLOB3,BLOB4)
> VALUES(?,?,?,?)',undef,$col1,$col2,$col3);
>   after:  $dbh->do('INSERT INTO test (BLOB1,BLOB2,BLOB3,BLOB4) VALUES(?,?,?,?)');
>           $sth->bind_param(1, $file, SQL_BLOB);
>           $sth->bind_param(2, $file, SQL_BLOB);
>           $sth->bind_param(3, $file, SQL_BLOB);
>           ...
>   One line more for each SQL statement. This will be a time consuming monks task
> during which the user will ask why this is necessary while it worked before.
> - upgrade scripts need to be written to UTF8 encode existing string data
> - retest all source code
>