Re: Gedcom files encoding

On 09/05/2012 11:25 PM, Ron Savage wrote:
> Hi Jose
>
> On 05/09/12 22:07, Jose Joao Dias de Almeida wrote:
>> Dear Gedcom-ers,
>> I just star with Gedcom.pm and things are beginning to work!
>>
>> But I have problems with files in Unicode.
>>
>> When files are in utf8 + BOM --> it returns error in first line (the BOM)
>>
>> If I remove the BOM and try again, apparently it does not pay attention
>> to "1 CHAR UTF-8"
>>
>> Is there any extra thing to say in this cases?
>> Um abraço
>> J.Joao
>>
>> 0 HEAD
>> 1 GEDC
>> 2 VERS 5.5
>> 2 FORM LINEAGE-LINKED
>>
>> 1 LANG Portuguese
>> 1 SOUR MYHERITAGE
>> ...
>
> I just checked the source code of Gedcom.pm V 1.16, and the only
> reference to utf8 is on line 390, where it is writing an XML file.
>
> So, you're right, of course.
>
> What do you think should happen?
>
> Perhaps if the code detects 1 CHAR UTF-8 the input file should be closed
> and re-opened in utf-8 mode, yes?

I think that would solve the problem.

Probably a simple
  if(/1 CHAR UTF-8/){ binmode(...) }
would also work.

One extra thing:
Some of the unicode files sometime include the initial byte order marker 
(BOM) in order to sign the unicode format used.

Bytes 	        Encoding Form
00 00 FE FF 	UTF-32, big-endian
FF FE 00 00 	UTF-32, little-endian
FE FF   	UTF-16, big-endian
FF FE   	UTF-16, little-endian
EF BB BF 	UTF-8

it would be nice if we could at least skip/ignore them (for exemple 
Myheritage tools are generating gedcom files with BOMs)

eg:

$ged =~ 
s/^(\x00\x00\xFE\xFF|\xFF\xFE\x00\x00|\xFF\xFE|\xFE\xFF|\xEF\xBB\xBF)//; 
  ## remove BOM !

Um abraço
J.Joao