Re: Characters in an uploaded text file being corrupted

Thanks - will check that out, but only issue is that this will need 
end-users to try sort something like this out, and they won't necessarily be 
the most technical types as such, and, I also wouldn't have too much control 
over what they'd originally used to generate the content.

Am thinking that might be able to handle character set encoding processing a 
bit more simply using python, which is currently my other primary 
programming language, so might try out something like creating a bit of a 
cross platform utility app that they can try use initially to control 
something like this before uploading content, but, let's see...

Stay well

Jacob Kruger
Blind Biker
Skype: BlindZA
'...fate had broken his body, but not his spirit...'

----- Original Message ----- 
From: "Carl Roett" <[email protected]>
To: <[email protected]>
Sent: Sunday, May 27, 2012 6:47 PM
Subject: Re: [PHP-WIN] Characters in an uploaded text file being corrupted


> Do not use Microsoft Word to save files that need to be read-in by other
> software. Ever. It *always* finds a way to screw it up. Even if you cut 
> and
> paste it into the other program, I've heard cases where Word has put
> unprintable control characters into the pasted output, that showed-up as
> spaces in the form, then corrupted the output at runtime.
>
> I suggest you open the saved file in NetBeans. In the NetBeans
> settings dialogue, enable the options to "show control characters" and
> "process file as unicode"  (iirc). That will cause NetBeans to display all
> characters your font set contains, display a square box for the ones it
> doesn't, and display the various icons for control characters.
>
> If column alignment in your file is important (for example a set number of
> spaces between tokens and operators), you should be using a monospace 
> font.
> The best one I've ever used is available here:
> http://code.google.com/p/buddypress-media/downloads/detail?name=ttf-bitstream-vera-1.10.zip
> It's also really easy to read.
>
> For text streams in production site, take a look at BP-Media's sanitizer
> classes. Unicode to ASCII to HTML entity conversion is a tricky business,
> and these functions will ensure you get the right kind of output no matter
> what users throw at you. For example, people embedding unicode sequences 
> in
> an ASCII stream.
> http://code.google.com/p/buddypress-media/source/browse/bp_media/trunk/core/database/class.database.sanitizers.php
>
> ^C^
>
> ===================================================
>
>
>
> On Sun, May 27, 2012 at 8:57 AM, Jacob Kruger <[email protected]> 
> wrote:
>
>> Ok, specifically told that storage field to make use of UTF8_unicode_ci,
>> and tried re-uploading text file that specifically saved from word using
>> UTF8 encoding, and, the text content is still full of garbage 
>> characters..?
>>
>> Suppose might try something like copying/pasting from text file in
>> notepad, into something like a textarea field, but, don't think that 
>> would
>> always be the perfect process in this site.
>>
>> Will try a couple of other options with regard to DB encoding, and see if
>> come right, but, let's see...
>>
>> Jacob Kruger
>> Blind Biker
>> Skype: BlindZA
>> '...fate had broken his body, but not his spirit...'
>>
>> ----- Original Message ----- From: "Niel Archer" <
>> [email protected]>
>> To: <[email protected]>
>> Sent: Sunday, May 27, 2012 4:22 PM
>> Subject: Re: [PHP-WIN] Characters in an uploaded text file being 
>> corrupted
>>
>>
>>  Hi
>>>
>>>  Ok, and, FWIW, this test text file is a word document that have saved 
>>> as
>>>> a
>>>> text file, and when now resaved it using specifically unicode encoding,
>>>> it
>>>> seemd to eliminate this issue,
>>>>
>>>
>>> Make sure the MySQL Db, table, field are correctly set for the encoding.
>>> MySQL does not default to UTF-8 so you have to manually set it when
>>> creating the Db/table. IIRC, newer versions can also have the encoding
>>> set per field.
>>>
>>>  but would have thought there might be a
>>>> relatively simple way to handle something like encoding conversion in 
>>>> PHP
>>>> itself..?
>>>>
>>>
>>> PHP cannot read minds, unfortunately. There are ways to handle encoding
>>> conversions, but I don't think anyone would call them 'simple'. ;-)
>>> See the Multibyte String extension for one way.
>>>
>>> --
>>> Niel Archer
>>>
>>>
>>> --
>>> PHP Windows Mailing List (http://www.php.net/)
>>> To unsubscribe, visit: http://www.php.net/unsub.php
>>>
>>>
>>
>> --
>> PHP Windows Mailing List (http://www.php.net/)
>> To unsubscribe, visit: http://www.php.net/unsub.php
>>
>>
> 


-- 
PHP Windows Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php