Re: mdb-export encoding issue
Jakob Egger <[email protected]> Fri, 23 Sep 2011 08:53:51 +0200
| Newsgroups | gmane.comp.db.mdb-tools.devel |
|---|---|
| Message-ID | <[email protected]> |
--===============1496000692657289723== Content-Type: multipart/alternative; boundary="Apple-Mail=_372A8247-5A07-49D9-8D20-3DF191A00379" --Apple-Mail=_372A8247-5A07-49D9-8D20-3DF191A00379 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 In Jet4, all text is stored using the UCS2 encoding. However, Access = uses a special trick to reduce storage requirements: ASCII characters = are stored as single byte characters, and all others are stored as two = byte characters. A null byte is used to switch between the two encoding = methods. In your field, such a NULL byte will appear between the ASCII = text and the hebrew characters. Apparently, this NULL byte causes = mdb-export to believe it has reached the end of the string. This = shouldn't happen. Which version of mdb-export are you using? There are a lot of old = versions "in the wild". It is best if you compile the most current = version from github yourself so you can ensure you are using a recent = version. I'd be glad to try reading your file with the newest version, if you are = interested just send me a copy of the database to my private email = address. Best regards, Jakob On 23.09.2011, at 08:03, =D7=90=D7=A8=D7=99=D7=90=D7=9C =D7=A7=D7=9C=D7=92= =D7=A1=D7=91=D7=9C=D7=93 Ariel Klagsbald wrote: > =20 > =20 > 2011/9/21 Nirgal <[email protected]>: > > Jet4 always use unicode (UCS2) internally. > > > > Output should be utf-8, unless you set env var MDBICONV (there is no = underscore). > > > I tried again and again, and MDBICONV seems to have no effect (a = strange fact by itself). Any more ideas please? Maybe I'm wrong, and it = isn't an encoding problem. Can something else cause emd-export to ignore = half of the field? > =20 > =20 > [ak@ch ~/]$ setenv MDBICONV UTF-8 > [ak@ch ~/]$ mdb-export -QHd^ WebStructure.mdb FilePaths | grep = '^130419\^' > = 130419^9817^0113-20000101-010645-45_^Hebrew|HWomen|HinuchYeladimShlomBayit= |HinuchYeladim|R0113-5|R0113-2^01/01/00 00:00:00^84^20223203^0113^1^0^45 = =EF=BF=BD=EF=BF=BD=D7=96=EF=BF=BD =EF=BF=BD=D7=98=EF=BF=BD=EF=BF=BD=D7=99 = =EF=BF=BD=EF=BF=BD=D7=98=EF=BF=BD=EF=BF=BD, =EF=BF=BD=EF=BF=BD' = =EF=BF=BD=EF=BF=BD=D7=9A, =D7=9A=D7=99'=D7=91^0^0^0^0 > [ar@ch ~/]$ setenv MDBICONV iso-8859-1 > [ar@ch ~/]$ mdb-export -QHd^ WebStructure.mdb FilePaths | grep = '^130419\^' > = 130419^9817^0113-20000101-010645-45_^Hebrew|HWomen|HinuchYeladimShlomBayit= |HinuchYeladim|R0113-5|R0113-2^01/01/00 00:00:00^84^20223203^0113^1^0^45 = =EF=BF=BD=EF=BF=BD=D7=96=EF=BF=BD =EF=BF=BD=D7=98=EF=BF=BD=EF=BF=BD=D7=99 = =EF=BF=BD=EF=BF=BD=D7=98=EF=BF=BD=EF=BF=BD, =EF=BF=BD=EF=BF=BD' = =EF=BF=BD=EF=BF=BD=D7=9A, =D7=9A=D7=99'=D7=91^0^0^0^0 > [ar@ch ~/]$ setenv MDBICONV nothingatall > [ar@ch ~/]$ mdb-export -QHd^ WebStructure.mdb FilePaths | grep = '^130419\^' > = 130419^9817^0113-20000101-010645-45_^Hebrew|HWomen|HinuchYeladimShlomBayit= |HinuchYeladim|R0113-5|R0113-2^01/01/00 00:00:00^84^20223203^0113^1^0^45 = =EF=BF=BD=EF=BF=BD=D7=96=EF=BF=BD =EF=BF=BD=D7=98=EF=BF=BD=EF=BF=BD=D7=99 = =EF=BF=BD=EF=BF=BD=D7=98=EF=BF=BD=EF=BF=BD, =EF=BF=BD=EF=BF=BD' = =EF=BF=BD=EF=BF=BD=D7=9A, =D7=9A=D7=99'=D7=91^0^0^0^0 > [ar@ch ~/]$ > =20 > =20 > See? MDBICONV has no effect. The 10th field (it's hebrew) seems the = same (even if your terminal doesn't show hebrew, you can see there's no = difference), and the 3rd field is still truncated. Only the numbers = appear. > =20 > =20 > =20 > Any help please?!? > =20 > > > > On Wednesday 21 September 2011 09:55:12 =D7=90=D7=A8=D7=99=D7=90=D7=9C= =D7=A7=D7=9C=D7=92=D7=A1=D7=91=D7=9C=D7=93 Ariel Klagsbald wrote: > >> I hope this is the place to post such a problem. And I also hope my > >> diagnosys is correct (that it's really is an encoding problem. I'm = not > >> sure). > >> > >> Well, I have a large mdb file, in which one of the fields contains = strings like > >> > >> 0007-20101223-214033-=D7=A9=D7=9E=D7=95=D7=AA-=D7=91=D7=92=D7=93=D7=A8= _=D7=A9=D7=9D.mp3 > >> > >> or > >> > >> 0007-20110714-213442-=D7=99=D7=95=D7=9D_=D7=98=D7=95=D7=91_=D7=A9=D7=A0= =D7=99_=D7=A9=D7=9C_=D7=92=D7=9C=D7=95=D7=99=D7=95=D7=AA.mp3 > >> > >> That is, part english, part numbers and part Hebrew (yes, that's > >> hebrew, in case you can't see it in your browser). > >> > >> When I use mdb-export to extract data from this file, I get the > >> numbers correctly, but only them. The hebrew and english parts are > >> simply missing (even the '3' in the 'mp3' suffix). That is, when I > >> extract the latter example I get only > >> > >> 0007-20110714-213442 > >> > >> I'll add that other fields contain only hebrew (e.g. > >> =D7=99=D7=95=D7=9D =D7=98=D7=95=D7=91 =D7=A9=D7=A0=D7=99 =D7=A9=D7=9C= =D7=92=D7=9C=D7=95=D7=99=D7=95=D7=AA, =D7=99=D7=91' =D7=AA=D7=9E=D7=95=D7= =96, =D7=AA=D7=A9=D7=A2'=D7=90 > >> in the example ebove), and they seem to be extracted correctly. = That > >> is, I get some gibberish which I guess is the correct data, only my > >> terminal can't present it. > >> > >> I though it might be an encoding problem, so I've played a bit with > >> MDB_ICONV, MDB_JET_CHARSET, MDB_JET3_CHARSET and MDB_JET4_CHARSET = but > >> it showed no difference. > >> The file seems to be JET4 (so mdb-ver claims). I've no idea what > >> encoding does it use (I don't know how to find out. Any ideas?), = but I > >> guess it's utf-8 (only a guess). > >> > >> I'll be grateful for any help! > >> Ariel. > >> > >> = --------------------------------------------------------------------------= ---- > >> All the data continuously generated in your IT infrastructure = contains a > >> definitive record of customers, application performance, security > >> threats, fraudulent activity and more. Splunk takes this data and = makes > >> sense of it. Business sense. IT sense. Common sense. > >> http://p.sf.net/sfu/splunk-d2dcopy1 > >> _______________________________________________ > >> mdbtools-dev mailing list > >> [email protected] > >> https://lists.sourceforge.net/lists/listinfo/mdbtools-dev > >> > > > =20 > =20 > = --------------------------------------------------------------------------= ---- > All of the data generated in your IT infrastructure is seriously = valuable. > Why? It contains a definitive record of application performance, = security > threats, fraudulent activity, and more. Splunk takes this data and = makes > sense of it. IT sense. And common sense. > = http://p.sf.net/sfu/splunk-d2dcopy2_______________________________________= ________ > mdbtools-dev mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/mdbtools-dev --Apple-Mail=_372A8247-5A07-49D9-8D20-3DF191A00379 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; = "><div>In Jet4, all text is stored using the UCS2 encoding. However, = Access uses a special trick to reduce storage requirements: ASCII = characters are stored as single byte characters, and all others are = stored as two byte characters. A null byte is used to switch between the = two encoding methods. In your field, such a NULL byte will appear = between the ASCII text and the hebrew characters. Apparently, this NULL = byte causes mdb-export to believe it has reached the end of the string. = This shouldn't happen.</div><div><br></div><div>Which version of = mdb-export are you using? There are a lot of old versions "in the wild". = It is best if you compile the most current version from github yourself = so you can ensure you are using a recent = version.</div><div><br></div><div>I'd be glad to try reading your file = with the newest version, if you are interested just send me a copy of = the database to my private email address.</div><div><br></div><div>Best = regards,</div><div>Jakob</div><div><br></div><br><div><div>On = 23.09.2011, at 08:03, =D7=90=D7=A8=D7=99=D7=90=D7=9C =D7=A7=D7=9C=D7=92=D7= =A1=D7=91=D7=9C=D7=93 Ariel Klagsbald wrote:</div><br = class=3D"Apple-interchange-newline"><blockquote type=3D"cite"><div = dir=3D"rtl"><div style=3D"TEXT-ALIGN: right"> </div> <div dir=3D"ltr"> </div> <div dir=3D"ltr">2011/9/21 Nirgal <<a = href=3D"mailto:[email protected]">[email protected]</a= >>:</div> <div dir=3D"ltr">> Jet4 always use unicode (UCS2) internally.</div> <div dir=3D"ltr">></div> <div dir=3D"ltr">> Output should be utf-8, unless you set env var = MDBICONV (there is no underscore).</div> <div dir=3D"ltr">></div> <div dir=3D"ltr"> I tried again and again, and MDBICONV seems to = have no effect (a strange fact by itself). Any more ideas please? Maybe = I'm wrong, and it isn't an encoding problem. Can something else cause = emd-export to ignore half of the field?</div> <div dir=3D"ltr"> </div> <div dir=3D"ltr"> </div> <div dir=3D"ltr">[ak@ch ~/]$ setenv MDBICONV UTF-8<br>[ak@ch ~/]$ = mdb-export -QHd^ WebStructure.mdb FilePaths | grep = '^130419\^'<br>130419^9817^0113-20000101-010645-45_^Hebrew|HWomen|HinuchYe= ladimShlomBayit|HinuchYeladim|R0113-5|R0113-2^01/01/00 = 00:00:00^84^20223203^0113^1^0^45 =EF=BF=BD=EF=BF=BD=D7=96=EF=BF=BD = =EF=BF=BD=D7=98=EF=BF=BD=EF=BF=BD=D7=99 =EF=BF=BD=EF=BF=BD=D7=98=EF=BF=BD=EF= =BF=BD, =EF=BF=BD=EF=BF=BD' =EF=BF=BD=EF=BF=BD=D7=9A, = =D7=9A=D7=99'=D7=91^0^0^0^0</div> <div dir=3D"ltr">[ar@ch ~/]$ setenv MDBICONV iso-8859-1<br>[ar@ch ~/]$ = mdb-export -QHd^ WebStructure.mdb FilePaths | grep = '^130419\^'<br>130419^9817^0113-20000101-010645-45_^Hebrew|HWomen|HinuchYe= ladimShlomBayit|HinuchYeladim|R0113-5|R0113-2^01/01/00 = 00:00:00^84^20223203^0113^1^0^45 =EF=BF=BD=EF=BF=BD=D7=96=EF=BF=BD = =EF=BF=BD=D7=98=EF=BF=BD=EF=BF=BD=D7=99 =EF=BF=BD=EF=BF=BD=D7=98=EF=BF=BD=EF= =BF=BD, =EF=BF=BD=EF=BF=BD' =EF=BF=BD=EF=BF=BD=D7=9A, = =D7=9A=D7=99'=D7=91^0^0^0^0</div> <div dir=3D"ltr">[ar@ch ~/]$ setenv MDBICONV nothingatall<br>[ar@ch ~/]$ = mdb-export -QHd^ WebStructure.mdb FilePaths | grep = '^130419\^'<br>130419^9817^0113-20000101-010645-45_^Hebrew|HWomen|HinuchYe= ladimShlomBayit|HinuchYeladim|R0113-5|R0113-2^01/01/00 = 00:00:00^84^20223203^0113^1^0^45 =EF=BF=BD=EF=BF=BD=D7=96=EF=BF=BD = =EF=BF=BD=D7=98=EF=BF=BD=EF=BF=BD=D7=99 =EF=BF=BD=EF=BF=BD=D7=98=EF=BF=BD=EF= =BF=BD, =EF=BF=BD=EF=BF=BD' =EF=BF=BD=EF=BF=BD=D7=9A, = =D7=9A=D7=99'=D7=91^0^0^0^0</div> <div dir=3D"ltr">[ar@ch ~/]$<br></div> <div dir=3D"ltr"> </div> <div dir=3D"ltr"> </div> <div dir=3D"ltr"> See? MDBICONV has no effect. The 10th field (it's = hebrew) seems the same (even if your terminal doesn't show hebrew, you = can see there's no difference), and the 3rd field is still truncated. = Only the numbers appear.</div> <div dir=3D"ltr"> </div> <div dir=3D"ltr"> </div> <div dir=3D"ltr"> </div> <div dir=3D"ltr"> Any help please?!?</div> <div dir=3D"ltr"> </div> <div dir=3D"ltr">></div> <div dir=3D"ltr">> On Wednesday 21 September 2011 09:55:12 =D7=90=D7=A8= =D7=99=D7=90=D7=9C =D7=A7=D7=9C=D7=92=D7=A1=D7=91=D7=9C=D7=93 Ariel = Klagsbald wrote:</div> <div dir=3D"ltr">>> I hope this is the place to post such a = problem. And I also hope my</div> <div dir=3D"ltr">>> diagnosys is correct (that it's really is an = encoding problem. I'm not</div> <div dir=3D"ltr">>> sure).</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> Well, I have a large mdb file, in which one of = the fields contains strings like</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> = 0007-20101223-214033-=D7=A9=D7=9E=D7=95=D7=AA-=D7=91=D7=92=D7=93=D7=A8_=D7= =A9=D7=9D.mp3</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> or</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> = 0007-20110714-213442-=D7=99=D7=95=D7=9D_=D7=98=D7=95=D7=91_=D7=A9=D7=A0=D7= =99_=D7=A9=D7=9C_=D7=92=D7=9C=D7=95=D7=99=D7=95=D7=AA.mp3</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> That is, part english, part numbers and part = Hebrew (yes, that's</div> <div dir=3D"ltr">>> hebrew, in case you can't see it in your = browser).</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> When I use mdb-export to extract data from = this file, I get the</div> <div dir=3D"ltr">>> numbers correctly, but only them. The hebrew = and english parts are</div> <div dir=3D"ltr">>> simply missing (even the '3' in the 'mp3' = suffix). That is, when I</div> <div dir=3D"ltr">>> extract the latter example I get only</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> 0007-20110714-213442</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> I'll add that other fields contain only hebrew = (e.g.</div> <div dir=3D"ltr">>> =D7=99=D7=95=D7=9D =D7=98=D7=95=D7=91 = =D7=A9=D7=A0=D7=99 =D7=A9=D7=9C =D7=92=D7=9C=D7=95=D7=99=D7=95=D7=AA, = =D7=99=D7=91' =D7=AA=D7=9E=D7=95=D7=96, =D7=AA=D7=A9=D7=A2'=D7=90</div> <div dir=3D"ltr">>> in the example ebove), and they seem to be = extracted correctly. That</div> <div dir=3D"ltr">>> is, I get some gibberish which I guess is the = correct data, only my</div> <div dir=3D"ltr">>> terminal can't present it.</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> I though it might be an encoding problem, so = I've played a bit with</div> <div dir=3D"ltr">>> MDB_ICONV, MDB_JET_CHARSET, MDB_JET3_CHARSET = and MDB_JET4_CHARSET but</div> <div dir=3D"ltr">>> it showed no difference.</div> <div dir=3D"ltr">>> The file seems to be JET4 (so mdb-ver claims). = I've no idea what</div> <div dir=3D"ltr">>> encoding does it use (I don't know how to find = out. Any ideas?), but I</div> <div dir=3D"ltr">>> guess it's utf-8 (only a guess).</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> I'll be grateful for any help!</div> <div dir=3D"ltr">>> Ariel.</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> = --------------------------------------------------------------------------= ----</div> <div dir=3D"ltr">>> All the data continuously generated in your IT = infrastructure contains a</div> <div dir=3D"ltr">>> definitive record of customers, application = performance, security</div> <div dir=3D"ltr">>> threats, fraudulent activity and more. Splunk = takes this data and makes</div> <div dir=3D"ltr">>> sense of it. Business sense. IT sense. Common = sense.</div> <div dir=3D"ltr">>> <a = href=3D"http://p.sf.net/sfu/splunk-d2dcopy1">http://p.sf.net/sfu/splunk-d2= dcopy1</a></div> <div dir=3D"ltr">>> = _______________________________________________</div> <div dir=3D"ltr">>> mdbtools-dev mailing list</div> <div dir=3D"ltr">>> <a = href=3D"mailto:[email protected]">[email protected]= ceforge.net</a></div> <div dir=3D"ltr">>> <a = href=3D"https://lists.sourceforge.net/lists/listinfo/mdbtools-dev">https:/= /lists.sourceforge.net/lists/listinfo/mdbtools-dev</a></div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">></div> <div dir=3D"ltr"> </div> <div dir=3D"ltr"> </div></div> = --------------------------------------------------------------------------= ----<br>All of the data generated in your IT infrastructure is seriously = valuable.<br>Why? It contains a definitive record of application = performance, security<br>threats, fraudulent activity, and more. Splunk = takes this data and makes<br>sense of it. IT sense. And common = sense.<br><a = href=3D"http://p.sf.net/sfu/splunk-d2dcopy2_______________________________= ________________">http://p.sf.net/sfu/splunk-d2dcopy2_____________________= __________________________</a><br>mdbtools-dev mailing = list<br>[email protected]<br>https://lists.sourceforge.ne= t/lists/listinfo/mdbtools-dev<br></blockquote></div><br></body></html>= --Apple-Mail=_372A8247-5A07-49D9-8D20-3DF191A00379-- --===============1496000692657289723== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2dcopy2 --===============1496000692657289723== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ mdbtools-dev mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/mdbtools-dev --===============1496000692657289723==--