Re: mdb-export encoding issue
אריאל קלגסבלד Ariel Klagsbald <[email protected]> Fri, 23 Sep 2011 09:03:09 +0300
| Newsgroups | gmane.comp.db.mdb-tools.devel |
|---|---|
| Message-ID | <CAOejwRTt4o0mBFOa7-0LiSz=Q294Y6Dm5L9zOwid_xhKm==qkA@mail.gmail.com> |
--===============1298820313014975876== Content-Type: multipart/alternative; boundary=20cf303f6df45198dd04ad95900d --20cf303f6df45198dd04ad95900d Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable 2011/9/21 Nirgal <[email protected]>: > Jet4 always use unicode (UCS2) internally. > > Output should be utf-8, unless you set env var MDBICONV (there is no underscore). > I tried again and again, and MDBICONV seems to have no effect (a strange fact by itself). Any more ideas please? Maybe I'm wrong, and it isn't an encoding problem. Can something else cause emd-export to ignore half of the field? [ak@ch ~/]$ setenv MDBICONV UTF-8 [ak@ch ~/]$ mdb-export -QHd^ WebStructure.mdb FilePaths | grep '^130419\^' 130419^9817^0113-20000101-010645-45_^Hebrew|HWomen|HinuchYeladimShlomBayit|= HinuchYeladim|R0113-5|R0113-2^01/01/00 00:00:00^84^20223203^0113^1^0^45 =EF=BF=BD=EF=BF=BD=D7=96=EF=BF=BD =EF=BF= =BD=D7=98=EF=BF=BD=EF=BF=BD=D7=99 =EF=BF=BD=EF=BF=BD=D7=98=EF=BF=BD=EF=BF= =BD, =EF=BF=BD=EF=BF=BD' =EF=BF=BD=EF=BF=BD=D7=9A, =D7=9A=D7=99'=D7=91^0^0^= 0^0 [ar@ch ~/]$ setenv MDBICONV iso-8859-1 [ar@ch ~/]$ mdb-export -QHd^ WebStructure.mdb FilePaths | grep '^130419\^' 130419^9817^0113-20000101-010645-45_^Hebrew|HWomen|HinuchYeladimShlomBayit|= HinuchYeladim|R0113-5|R0113-2^01/01/00 00:00:00^84^20223203^0113^1^0^45 =EF=BF=BD=EF=BF=BD=D7=96=EF=BF=BD =EF=BF= =BD=D7=98=EF=BF=BD=EF=BF=BD=D7=99 =EF=BF=BD=EF=BF=BD=D7=98=EF=BF=BD=EF=BF= =BD, =EF=BF=BD=EF=BF=BD' =EF=BF=BD=EF=BF=BD=D7=9A, =D7=9A=D7=99'=D7=91^0^0^= 0^0 [ar@ch ~/]$ setenv MDBICONV nothingatall [ar@ch ~/]$ mdb-export -QHd^ WebStructure.mdb FilePaths | grep '^130419\^' 130419^9817^0113-20000101-010645-45_^Hebrew|HWomen|HinuchYeladimShlomBayit|= HinuchYeladim|R0113-5|R0113-2^01/01/00 00:00:00^84^20223203^0113^1^0^45 =EF=BF=BD=EF=BF=BD=D7=96=EF=BF=BD =EF=BF= =BD=D7=98=EF=BF=BD=EF=BF=BD=D7=99 =EF=BF=BD=EF=BF=BD=D7=98=EF=BF=BD=EF=BF= =BD, =EF=BF=BD=EF=BF=BD' =EF=BF=BD=EF=BF=BD=D7=9A, =D7=9A=D7=99'=D7=91^0^0^= 0^0 [ar@ch ~/]$ See? MDBICONV has no effect. The 10th field (it's hebrew) seems the same (even if your terminal doesn't show hebrew, you can see there's no difference), and the 3rd field is still truncated. Only the numbers appear. Any help please?!? > > On Wednesday 21 September 2011 09:55:12 =D7=90=D7=A8=D7=99=D7=90=D7=9C = =D7=A7=D7=9C=D7=92=D7=A1=D7=91=D7=9C=D7=93 Ariel Klagsbald wrote: >> I hope this is the place to post such a problem. And I also hope my >> diagnosys is correct (that it's really is an encoding problem. I'm not >> sure). >> >> Well, I have a large mdb file, in which one of the fields contains strings like >> >> 0007-20101223-214033-=D7=A9=D7=9E=D7=95=D7=AA-=D7=91=D7=92=D7=93=D7=A8_= =D7=A9=D7=9D.mp3 >> >> or >> >> 0007-20110714-213442-=D7=99=D7=95=D7=9D_=D7=98=D7=95=D7=91_=D7=A9=D7=A0= =D7=99_=D7=A9=D7=9C_=D7=92=D7=9C=D7=95=D7=99=D7=95=D7=AA.mp3 >> >> That is, part english, part numbers and part Hebrew (yes, that's >> hebrew, in case you can't see it in your browser). >> >> When I use mdb-export to extract data from this file, I get the >> numbers correctly, but only them. The hebrew and english parts are >> simply missing (even the '3' in the 'mp3' suffix). That is, when I >> extract the latter example I get only >> >> 0007-20110714-213442 >> >> I'll add that other fields contain only hebrew (e.g. >> =D7=99=D7=95=D7=9D =D7=98=D7=95=D7=91 =D7=A9=D7=A0=D7=99 =D7=A9=D7=9C = =D7=92=D7=9C=D7=95=D7=99=D7=95=D7=AA, =D7=99=D7=91' =D7=AA=D7=9E=D7=95=D7= =96, =D7=AA=D7=A9=D7=A2'=D7=90 >> in the example ebove), and they seem to be extracted correctly. That >> is, I get some gibberish which I guess is the correct data, only my >> terminal can't present it. >> >> I though it might be an encoding problem, so I've played a bit with >> MDB_ICONV, MDB_JET_CHARSET, MDB_JET3_CHARSET and MDB_JET4_CHARSET but >> it showed no difference. >> The file seems to be JET4 (so mdb-ver claims). I've no idea what >> encoding does it use (I don't know how to find out. Any ideas?), but I >> guess it's utf-8 (only a guess). >> >> I'll be grateful for any help! >> Ariel. >> >> ---------------------------------------------------------------------------= --- >> All the data continuously generated in your IT infrastructure contains a >> definitive record of customers, application performance, security >> threats, fraudulent activity and more. Splunk takes this data and makes >> sense of it. Business sense. IT sense. Common sense. >> http://p.sf.net/sfu/splunk-d2dcopy1 >> _______________________________________________ >> mdbtools-dev mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/mdbtools-dev >> > --20cf303f6df45198dd04ad95900d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"rtl"><div style=3D"TEXT-ALIGN: right">=C2=A0</div> <div dir=3D"ltr">=C2=A0</div> <div dir=3D"ltr">2011/9/21 Nirgal <<a href=3D"mailto:contact_mdbtools@ni= rgal.com">[email protected]</a>>:</div> <div dir=3D"ltr">> Jet4 always use unicode (UCS2) internally.</div> <div dir=3D"ltr">></div> <div dir=3D"ltr">> Output should be utf-8, unless you set env var MDBICO= NV (there is no underscore).</div> <div dir=3D"ltr">></div> <div dir=3D"ltr">=C2=A0 I tried again and again, and MDBICONV seems to have= no effect (a strange fact by itself). Any more ideas please? Maybe I'm= wrong, and it isn't an encoding problem. Can something else cause emd-= export to ignore half of the field?</div> <div dir=3D"ltr">=C2=A0</div> <div dir=3D"ltr">=C2=A0</div> <div dir=3D"ltr">[ak@ch ~/]$ setenv MDBICONV UTF-8<br>[ak@ch ~/]$ mdb-expor= t -QHd^ WebStructure.mdb FilePaths | grep '^130419\^'<br>130419^981= 7^0113-20000101-010645-45_^Hebrew|HWomen|HinuchYeladimShlomBayit|HinuchYela= dim|R0113-5|R0113-2^01/01/00 00:00:00^84^20223203^0113^1^0^45 =EF=BF=BD=EF= =BF=BD=D7=96=EF=BF=BD =EF=BF=BD=D7=98=EF=BF=BD=EF=BF=BD=D7=99 =EF=BF=BD=EF= =BF=BD=D7=98=EF=BF=BD=EF=BF=BD, =EF=BF=BD=EF=BF=BD' =EF=BF=BD=EF=BF=BD= =D7=9A, =D7=9A=D7=99'=D7=91^0^0^0^0</div> <div dir=3D"ltr">[ar@ch ~/]$ setenv MDBICONV iso-8859-1<br>[ar@ch ~/]$ mdb-= export -QHd^ WebStructure.mdb FilePaths | grep '^130419\^'<br>13041= 9^9817^0113-20000101-010645-45_^Hebrew|HWomen|HinuchYeladimShlomBayit|Hinuc= hYeladim|R0113-5|R0113-2^01/01/00 00:00:00^84^20223203^0113^1^0^45 =EF=BF= =BD=EF=BF=BD=D7=96=EF=BF=BD =EF=BF=BD=D7=98=EF=BF=BD=EF=BF=BD=D7=99 =EF=BF= =BD=EF=BF=BD=D7=98=EF=BF=BD=EF=BF=BD, =EF=BF=BD=EF=BF=BD' =EF=BF=BD=EF= =BF=BD=D7=9A, =D7=9A=D7=99'=D7=91^0^0^0^0</div> <div dir=3D"ltr">[ar@ch ~/]$ setenv MDBICONV nothingatall<br>[ar@ch ~/]$ md= b-export -QHd^ WebStructure.mdb FilePaths | grep '^130419\^'<br>130= 419^9817^0113-20000101-010645-45_^Hebrew|HWomen|HinuchYeladimShlomBayit|Hin= uchYeladim|R0113-5|R0113-2^01/01/00 00:00:00^84^20223203^0113^1^0^45 =EF=BF= =BD=EF=BF=BD=D7=96=EF=BF=BD =EF=BF=BD=D7=98=EF=BF=BD=EF=BF=BD=D7=99 =EF=BF= =BD=EF=BF=BD=D7=98=EF=BF=BD=EF=BF=BD, =EF=BF=BD=EF=BF=BD' =EF=BF=BD=EF= =BF=BD=D7=9A, =D7=9A=D7=99'=D7=91^0^0^0^0</div> <div dir=3D"ltr">[ar@ch ~/]$<br></div> <div dir=3D"ltr">=C2=A0</div> <div dir=3D"ltr">=C2=A0</div> <div dir=3D"ltr">=C2=A0See? MDBICONV has no effect. The 10th field (it'= s hebrew) seems the same (even if your terminal doesn't show hebrew, yo= u can see there's no difference), and the 3rd field is still truncated.= Only the numbers appear.</div> <div dir=3D"ltr">=C2=A0</div> <div dir=3D"ltr">=C2=A0</div> <div dir=3D"ltr">=C2=A0</div> <div dir=3D"ltr">=C2=A0 Any help please?!?</div> <div dir=3D"ltr">=C2=A0</div> <div dir=3D"ltr">></div> <div dir=3D"ltr">> On Wednesday 21 September 2011 09:55:12 =D7=90=D7=A8= =D7=99=D7=90=D7=9C =D7=A7=D7=9C=D7=92=D7=A1=D7=91=D7=9C=D7=93 Ariel Klagsba= ld wrote:</div> <div dir=3D"ltr">>> I hope this is the place to post such a problem. = And I also hope my</div> <div dir=3D"ltr">>> diagnosys is correct (that it's really is an = encoding problem. I'm not</div> <div dir=3D"ltr">>> sure).</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> Well, I have a large mdb file, in which one of th= e fields contains strings like</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> 0007-20101223-214033-=D7=A9=D7=9E=D7=95=D7=AA-=D7= =91=D7=92=D7=93=D7=A8_=D7=A9=D7=9D.mp3</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> or</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> 0007-20110714-213442-=D7=99=D7=95=D7=9D_=D7=98=D7= =95=D7=91_=D7=A9=D7=A0=D7=99_=D7=A9=D7=9C_=D7=92=D7=9C=D7=95=D7=99=D7=95=D7= =AA.mp3</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> That is, part english, part numbers and part Hebr= ew (yes, that's</div> <div dir=3D"ltr">>> hebrew, in case you can't see it in your brow= ser).</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> When I use mdb-export to extract data from this f= ile, I get the</div> <div dir=3D"ltr">>> numbers correctly, but only them. The hebrew and = english parts are</div> <div dir=3D"ltr">>> simply missing (even the '3' in the '= mp3' suffix). That is, when I</div> <div dir=3D"ltr">>> extract the latter example I get only</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> 0007-20110714-213442</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> I'll add that other fields contain only hebre= w (e.g.</div> <div dir=3D"ltr">>> =C2=A0=D7=99=D7=95=D7=9D =D7=98=D7=95=D7=91 =D7= =A9=D7=A0=D7=99 =D7=A9=D7=9C =D7=92=D7=9C=D7=95=D7=99=D7=95=D7=AA, =D7=99= =D7=91' =D7=AA=D7=9E=D7=95=D7=96, =D7=AA=D7=A9=D7=A2'=D7=90</div> <div dir=3D"ltr">>> in the example ebove), and they seem to be extrac= ted correctly. That</div> <div dir=3D"ltr">>> is, I get some gibberish which I guess is the cor= rect data, only my</div> <div dir=3D"ltr">>> terminal can't present it.</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> I though it might be an encoding problem, so I= 9;ve played a bit with</div> <div dir=3D"ltr">>> MDB_ICONV, MDB_JET_CHARSET, MDB_JET3_CHARSET and = MDB_JET4_CHARSET but</div> <div dir=3D"ltr">>> it showed no difference.</div> <div dir=3D"ltr">>> The file seems to be JET4 (so mdb-ver claims). I&= #39;ve no idea what</div> <div dir=3D"ltr">>> encoding does it use (I don't know how to fin= d out. Any ideas?), but I</div> <div dir=3D"ltr">>> guess it's utf-8 (only a guess).</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> =C2=A0 I'll be grateful for any help!</div> <div dir=3D"ltr">>> Ariel.</div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">>> -------------------------------------------------= -----------------------------</div> <div dir=3D"ltr">>> All the data continuously generated in your IT in= frastructure contains a</div> <div dir=3D"ltr">>> definitive record of customers, application perfo= rmance, security</div> <div dir=3D"ltr">>> threats, fraudulent activity and more. Splunk tak= es this data and makes</div> <div dir=3D"ltr">>> sense of it. Business sense. IT sense. Common sen= se.</div> <div dir=3D"ltr">>> <a href=3D"http://p.sf.net/sfu/splunk-d2dcopy1">h= ttp://p.sf.net/sfu/splunk-d2dcopy1</a></div> <div dir=3D"ltr">>> _______________________________________________</= div> <div dir=3D"ltr">>> mdbtools-dev mailing list</div> <div dir=3D"ltr">>> <a href=3D"mailto:[email protected].= net">[email protected]</a></div> <div dir=3D"ltr">>> <a href=3D"https://lists.sourceforge.net/lists/li= stinfo/mdbtools-dev">https://lists.sourceforge.net/lists/listinfo/mdbtools-= dev</a></div> <div dir=3D"ltr">>></div> <div dir=3D"ltr">></div> <div dir=3D"ltr">=C2=A0</div> <div dir=3D"ltr">=C2=A0</div></div> --20cf303f6df45198dd04ad95900d-- --===============1298820313014975876== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2dcopy2 --===============1298820313014975876== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ mdbtools-dev mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/mdbtools-dev --===============1298820313014975876==--