Re: Gedcom files encoding

Ron,
   any solution you came up is fine!

   but:
 > I believe binmode does not work after the file has been read. That is,
 > it must be executed immediately after opening the file, before the 1st
 > read.

   Normally binmode words well even after some readind (but it may 
depend on buffering). For example:

use strict;
my $file=shift or die;

binmode(STDOUT,":utf8");
open(F,"<",$file) or die;

while(<F>){
   s/^(\xFF\xFE|\xFE\xFF|\xEF\xBB\xBF)//;     ## remove some BOMs by hand
   if( /CHAR\s+(\S+)/ ){binmode(F,":encoding($1)");
          print "--Afer binmode(:encoding($1)---\n" }
   print $_;
}

works fine in my machine (linux, perl5.12). Other alternative using BOM:

use File::BOM qw(:all);
my $file=shift or die;

    open_bom(F, $file)                    ## use BOM if present
or open    (F, $file)                    ## if no BOM find:
or die("cant open_bom nor open...\n");

binmode(STDOUT,":utf8");
while(<F>){
   if( /CHAR\s+(\S+)/ ){binmode(F,":encoding($1)");
          print "--Afer binmode(:encoding($1)---\n" }
   print $_;
}

also words fine for me.

In attach a ged file with BOM and CHAR and another with just CHAR.
I changed the real file introducing some utf8 code before the CHAR UTF-8 
directive (which is not possible) in order to show the behaviour of 
binmode and BOM.

Um abraço
J.Joao
 >
 > But see File::BOM. Gedcom needs to use that.
 >
 > The string 1 CHAR UTF-8 obviously must be read in before knowing the
 > file is intended to be utf8, so File::BOM won't solve that problem.
 >

On 09/07/2012 01:20 AM, Ron Savage wrote:
> Hi Jose
>
> On 07/09/12 02:06, Jose Joao Dias de Almeida wrote:
>>
>>
>> On 09/05/2012 11:25 PM, Ron Savage wrote:
>>> Hi Jose
>>>
>>> On 05/09/12 22:07, Jose Joao Dias de Almeida wrote:
>>>> Dear Gedcom-ers,
>>>> I just star with Gedcom.pm and things are beginning to work!
>>>>
>>>> But I have problems with files in Unicode.
>>>>
>>>> When files are in utf8 + BOM --> it returns error in first line (the
>>>> BOM)
>>>>
>>>> If I remove the BOM and try again, apparently it does not pay attention
>>>> to "1 CHAR UTF-8"
>>>>
>>>> Is there any extra thing to say in this cases?
>>>> Um abraço
>>>> J.Joao
>>>>
>>>> 0 HEAD
>>>> 1 GEDC
>>>> 2 VERS 5.5
>>>> 2 FORM LINEAGE-LINKED
>>>>
>>>> 1 LANG Portuguese
>>>> 1 SOUR MYHERITAGE
>>>> ...
>>>
>>> I just checked the source code of Gedcom.pm V 1.16, and the only
>>> reference to utf8 is on line 390, where it is writing an XML file.
>>>
>>> So, you're right, of course.
>>>
>>> What do you think should happen?
>>>
>>> Perhaps if the code detects 1 CHAR UTF-8 the input file should be closed
>>> and re-opened in utf-8 mode, yes?
>>
>> I think that would solve the problem.
>>
>> Probably a simple
>> if(/1 CHAR UTF-8/){ binmode(...) }
>> would also work.
>
> I believe binmode does not work after the file has been read. That is,
> it must be executed immediately after opening the file, before the 1st
> read.
>
> But see File::BOM. Gedcom needs to use that.
>
> The string 1 CHAR UTF-8 obviously must be read in before knowing the
> file is intended to be utf8, so File::BOM won't solve that problem.
>
>> One extra thing:
>> Some of the unicode files sometime include the initial byte order marker
>> (BOM) in order to sign the unicode format used.
>>
>> Bytes Encoding Form
>> 00 00 FE FF UTF-32, big-endian
>> FF FE 00 00 UTF-32, little-endian
>> FE FF UTF-16, big-endian
>> FF FE UTF-16, little-endian
>> EF BB BF UTF-8
>>
>> it would be nice if we could at least skip/ignore them (for exemple
>> Myheritage tools are generating gedcom files with BOMs)
>>
>> eg:
>>
>> $ged =~
>> s/^(\x00\x00\xFE\xFF|\xFF\xFE\x00\x00|\xFF\xFE|\xFE\xFF|\xEF\xBB\xBF)//;
>> ## remove BOM !
>>
>> Um abraço
>> J.Joao