Binarycloud And Encodings

Hi

We (Alex, Jcm and me) just talked on IRC and reached a decision 
regarding UTF-8 in binarycloud.

In r3/binarycloud:

  * PHP files: All files are in UTF-8 encoding without BOM. Comments may 
contain characters with a char code > 127 (they are multi byte in 
UTF-8), all other code must consist of characters fitting in 7 Bit 
ASCII. All strings in these files are translated. This means they do not 
contain localized strings but merely string IDs like 
"Creole.DatabaseConnectionFailed".
  * Template files: All Smarty templates are UTF-8 without BOM.
  * Configuration files: All base configuration files are UTF-8 and 
configuration values and keys must be 7 Bit ASCII. This is no problem 
since it only contains things like "switch this on" and "switch this 
off". If localization is ever required here, it must be done with 
translation.
  * NDF files: NDF must be UTF-8, keys must be 7 Bit ASCII, values must 
be 7 Bit ASCII, if localization is required, this must be done via XLIFF 
translation files and the yet to be defined NDF translation system.
  * Translation files: UTF-8, keys must be 7 Bit ASCII, English 
language. Everything else can be whatever is required to create a good 
translation.

In r3/vortex:

Same restrictions as in r3/binarycloud apply. After all this is a public 
application repository and all IDs being English allows for a maximum of 
useability with English being the lingua franca of the world.

In $WORKSPACE/app:

You are free to use whatever you want and helps you best. Follow the 
r3/binarycloud policy to get a well working system.


This way all data coming into the system by files (php, tpl, conf, ndf 
and translation) is UTF-8. We have to handle data coming from the user 
by GET or POST data now.

We unify this by converting all GET and POST data from the user to 
UTF-8. This makes some things a bit harder. For example, if you are a 
German developer and only handle German inputs, you can simply handle 
"Umlaute" like äüö since they fit into ISO-8859-1. With everything 
UTF-8, you suddenly have to care about multi byte input. However, you 
can be *sure* that it is UTF-8 and don't have to do conversion from 
funny browser inputs (Windows encoding, Mac Encoding etc.). However, 
there is the great advantage that your code will continue to work for 
all locales.

This is done at system startup - transparently for the user.


The next thing to care about is the output encoding. The browser can 
specify preferred encodings on the HTTP request. We use PEAR::I18Nv2 to 
resolve the best or falling back to a configureable standard one.

On system startup, we create an output handler. This enables us to use 
I18Nv2's automatic charset conversion feature. On Project shutdown, the 
output buffer is flushed and the conversion takes place.

Note that this will change Project and htdocs files yet again - for 
good, I hope. I will create a detailed ticket for this change later on, 
though.


As you can see above, the internal encoding always is UTF-8 as far as 
binarycloud can help developers here. However, when he gets data from a 
database, text files or wherever else, he has to make sure by himself 
that he converts the received data to UTF-8 by using iconv, for example, 
if necessary.


I hope that this encoding policy ensures that

1.) All input encodings can be handled unifiedly
2.) All output encoding requests can be answered to the best possible point.
3.) String handling is simplified internally

Please examine this policy and answer if you find unclear or errornous 
points.


I attached an overview file and committed the OmniGraffle file to 
subversion.

*m

-- 
GPG key: http://www.ggnore.net/~mholtgrewe/personal/files/public_key.asc

_______________________________________________
dev mailing list
dev-PnctHDZWAvB/Cz2I37pSEPZ4XP/[email protected]
http://lists.binarycloud.com/mailman/listinfo/dev