Bug #74312 [Com]: Filter BOMs out of PHP output

Edit report at https://bugs.php.net/bug.php?id=74312&edit=1

 ID:                 74312
 Comment by:         furun at arcor dot de
 Reported by:        furun at arcor dot de
 Summary:            Filter BOMs out of PHP output
 Status:             Re-Opened
 Type:               Bug
 Package:            Unicode Engine related
 Operating System:   Win7
 PHP Version:        7.0.17
 Block user comment: N
 Private report:     N

 New Comment:

Referencing http://markmail.org/message/besjw22hxlpwlvdh "don't do magic"...

Argument: See BOM handling as header, and not as implicit magic:
Standard headers for text files like used in binarys, which define the encoding and so the handling precisely, should "maybe" be defined long long long time ago. Now we are left with text/script uncertainty, editors can not do better then guess the encoding, and users need to correct them manually, and there are many. Text files are so in general more "dirty" then binary files, and specially for scripts is this not useful, and need additional documentations and definitions beside of the file/data it self. The BOM could bring at least for UTF encoding a little certainty in text/script files. UTF8 become the favorite encoding, and would be a good argument for php to tolerate it. 
The BOM can then be seen as a header like for ZIP PNG or EXE files, just for a text format, even if this is (unfortunately) uncommon for text files. A header can tell a software that the data is actually a JPG and not a PNG, even if the file extension tells different. TXT or PHP not tell anything about encoding. (A good reason why php should not change anything, php can not know the intention of the script, like nikic points out in a], so this point is not only philosophical. There is a point where implicit magic become voodoo. And why i don't use zend.multibyte.) So headers can get exceptional handling, as by definition are not part of the files data, but only tell what data it is in the files.
The header tolerance would then not be implicit magic anymore, but a standard behavior for a header, the argument implicit magical is then out of the way for BOM.
(I would then put BOMs in all my PHP7s, like i did with C#s and TXTs...)


(...But this don't work for ?> cleaning, where it is indeed more controversial. This came here anyway late in the discussion, and i just jump in to it. The question is if implicit magic dose more harm, or save more harm. I found a lot of forum entry for "Headers already sent" errors because of empty chars after last ?>, wen i check the commonness for BOM problems. I have there a pragmatic point of view, if a implicit action saves way much more problems then it causes, and its intuitive right, i would do it. But i understand that the point of view of developers which are responsible for a code language which is used world-wide, is maybe different, and if they come to a conservative result.)


Previous Comments:
------------------------------------------------------------------------
[2017-04-02 01:10:37] spam2 at rhsoft dot net

> and i expect code incompatibility (not BOM) by working on PHP5 > PHP7. 
> so nothing like "5 min google search"

what did you not understand in "that ship sailed long ago"?
PHP7.0 is GA since 2015
PHP7.1 is GA since 2016

just because you have not finished "working on PHP5 > PHP7" don't mean you can go ahead and make backward incompatible changes at this point in time - to make it clear: our current deploayable code base of 250000 line sof code is PHP7 only (return types, scalar type hints, declare(strict_types=1);) and in a short will be PHP7.1 only by introduce void-return-types all over the place

and that's what "that ship sailed long ago" means

------------------------------------------------------------------------
[2017-04-01 23:43:26] furun at arcor dot de

(nikic)

I read the link you post here, tanks.
http://markmail.org/message/besjw22hxlpwlvdh
Needles to say that i am on the arguments from Sammy Kaye Powers.

(Repeating my self now...:
I searched a little in files, and found more BOMs in the files then i expected. It was a quick search. Like expected, in plugins mainly, and not the main codes. In JS and CSS files too, but browsers are now all handling it, JS BOMs don't cause problems anymore in browsers. Some projects have a "don't use BOM and ending ?>" advice. And i found in quick search several posts in forums BOM and ?> related, the "headers already sent by" error is maybe most common, but by fare not the only one. The BOM is nasty, because it creates errors all over the place, "wearing an invisibility cloak so-to-speak". In my case it was a broken captsha image generator, because of a 3-party plugin far away. It was a nerving bug search because 2 scripts had a BOM, and i searched a while on the wrong sides. Because i clean this bug source in my codes, i was not trained to search the bug there, and i expect code incompatibility (not BOM) by working on PHP5 > PHP7. So nothing like "5 min google search".!
  Experienced forum workers maybe know it better, but i think it is a source for recurring nasty issues... and i would fix it like discusses there, in the lexer.

Mono develop onces destroy one of my C# files, it was a file to handle UFT-8 encoding problems, and then gets problems it self. I was suddenly simply defect, [?]-Chars, and i never worked on it for long time. Since then, all my C# files have the UTF-8 BOM to tell Mono, "don't destroy it again, it is a UTF-8, leave it so". So against the advice, "do not use UFT-8 BOMs".

Link from Sammy Kaye Powers, and so i saw it too.
http://stackoverflow.com/search?q=php+bom
)

------------------------------------------------------------------------
[2017-04-01 21:32:40] spam2 at rhsoft dot net

> My opinion is, intentional BOM is rare, and the PHP5 > PHP7 
> upgrade is the right time to the change, in case it breaks 
> existing code, wen all developers are busy with maintenance 
> and compatibility anyway.

that ship sailed long ago 

> If the BOM and maybe even ?> handling is done behind an ini 
> setting, its even better, it gives developers the choice

no it don't - it only makes it unpredictable if, when and where your code breaks since you as developer normally have no control about ini-settings

------------------------------------------------------------------------
[2017-04-01 21:22:48] furun at arcor dot de

(nikic)

a) A technical argument could be, it could cause other problems if you just strip the BOM, and not process some UTF-8/encoding stuf. php must/should maybe then care some how what the BOM says about the encoding. My question would be then, if php still does not care about file encoding, would it break something/more then leaf BOMs in the files? I think if a BOM is there, php could presume correct and indented encoding. For UTF/encoding handling then is definitely the code responsible and not php, even UTF error corrections. (i use a script for this. Far back, wen some software switched from ASCII to UTF8, i have seen some confused files, and i fixed them by script, instead to manually correct all [?]-characters).
I miss there maybe something, but my suggestion would be to just delete/ignore the BOM, and keep the file as it is. (Keep the present behavior in default configuration.)
(I never used zend.multibyte, i like if php gives all out exactly as it is. The only exception is BOM and empty chars after ?>, where i see recurring problems.)

b) A other technical argument is backwards compatibility. My suggestion is, if a BOM output is intentional, it could be done explicit in code. And it would be a other bug source if a file not have, but should have a BOM for the output. Because the BOM is in the editor likely only visible in the options and not in the text editor it self, so not visible in first view (NotePad++ for example). Explicit BOM export would be more clean. And i would speculate, a intentional export of a BOM is way less probable, then a buggy behavior with unintentional BOMs. And a developer is focused on this specific behavior to export BOMs, other then unintentional BOMs.
(PHP could define constants for all important BOMs, for explicit output, if not done already.)
The discussion i searched here is the possible backwards compatibility break, and if problems are probable and recurring as unintended BOMs. 
My opinion is, intentional BOM is rare, and the PHP5 > PHP7 upgrade is the right time to the change, in case it breaks existing code, wen all developers are busy with maintenance and compatibility anyway. 
If the BOM and maybe even ?> handling is done behind an ini setting, its even better, it gives developers the choice. The question and discussion is then, should it be the default setting to handle or not handle the BOM by php. I would vote with my present opinion, to strip BOM by default in ini setting, in PHP-7, maybe not in PHP-5(?) if backwards compatibility is a big issue.


The big question is, is BOM and empty chars after ?> a recurring and potentially nasty issue?
In my opinion it is, which could be fixed pragmatically simply by changing some php roles. Correct my if i am wrong there.

------------------------------------------------------------------------
[2017-04-01 17:04:32] spam2 at rhsoft dot net

> to reeducate the entire planet

yes, because such "education" is a prerequisite anyways - or where you born with knowing programming languages and even if why that one piece was missing

> or to take the stone out of the way

there is no stone - frankly looking at 90% of PHP code out there (leading to prohibit as much as possible 3rd party code on our sevrers for a decade now) there are way too less stones checking if someone should prdocuece code running on sevrers for which he is most of the time not responsile at his own and innocent sysadmins and attacked 3rd party sites have to chew the outcome later

------------------------------------------------------------------------


The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at

    https://bugs.php.net/bug.php?id=74312


--
Edit this bug report at https://bugs.php.net/bug.php?id=74312&edit=1