Re: (no subject)
Jakob Egger <[email protected]> Sat, 30 Jan 2016 11:27:39 +0100
| Newsgroups | gmane.comp.db.mdb-tools.devel |
|---|---|
| Message-ID | <[email protected]> |
--===============4206805617080937563== Content-Type: multipart/alternative; boundary="Apple-Mail=_ABEAF34D-47CE-4269-AF96-493D6393CD0F" --Apple-Mail=_ABEAF34D-47CE-4269-AF96-493D6393CD0F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii First of all: Page Usage Maps are only used for improving performance, = you don't need to use them if you just want to read data. Every table has two main page usage bitmaps: the bitmap with all the = used pages, and the bitmap with free pages. The used_pages bitmap tells you which pages in the database belong to = that table. You can find the same info by just looping through all pages = and checking the table id in the page header. Anyway, here are two = simple examples: If page index 6 belongs to a table, then the page usage = bitmap would be 01000000 in binary or the single byte 0x80. If pages 6 = and 9 belong to a table, the page usage bitmap would be two bytes: 0x80 = 0x02. The two different types of bitmaps are for small and for large = databases. If a bitmap for the full database fits on a single page (less = than 32736 pages), then type 0 bitmaps are used. Otherwise type 1 = bitmaps are used, meaning that the page usage bitmap is spread over = multiple pages (see hacking file for details) The free space usage map is only necessary if you want to support = writing: it tells Access on which pages it can find space to write a new = row. Then there are usage maps related to the indexes, and apparently there = are also usage maps related to columns at the end of the tdef pages... I = don't know how those work, I assume those are also some kinds of = optimisations. I hope this helps, but as I said before, I'd recommend ignoring the page = usage bitmaps at first. In my experience, they are also sometimes = corrupted, so sometimes you need to fall back to scanning the full = database anyway. Jakob > On 28 Jan 2016, at 23:34, Yannick Heinrich = <[email protected]> wrote: >=20 > This helps me to understand the process :)=20 >=20 > May be you can give me some more explanations about usage maps. >=20 > In the HACKING file of the mdv-tools directory, it describes the = different pages type in the file. 0x05 is for "Page Usage Bitmaps". >=20 > But how is the link done with the others pages (data and table = definition) ? Inside the descriptions of these pages, the document = refers to "usage bit mask".=20 >=20 > What is the difference between Page Usage Bitmaps and usage bit mask ? >=20 > There is also two types of Page Usage (0x00 and 0x01). Are those types = present after the page usage bitmaps header ? >=20 > Regards > Yannick >=20 > 2016-01-12 22:23 GMT+01:00 Jakob Egger <[email protected] = <mailto:[email protected]>>: > MSysObjects just tells you the name, type, etc. of the table. You can = ignore it if you just want to read the data.=20 >=20 > If you want to read the records, you need information from the tdef = page. The tdef pages tell you how what fields are in a table, what types = they have, and so on. You need that information to read the data pages. >=20 > What does a data page look like? Well, it has a header that tells you = what table it belongs to (table id =3D index of the tdef page). Then it = is filled with records, starting from the end. The header stores the = offsets of the records. >=20 > What do records look like? > Each record has all the fixed length fields in the front (use info = from tdef page to parse). All the variable length fields are at the end, = starting from the back. You could actually parse variable length fields = without the info from the tdef page (at least text columns). The NULL = map (which fields are NULL) is also stored at the end of the record. >=20 > How do you find data pages for a specific table? > 1) Look at the page usage bitmap for the table (complicated) > 2) Just loop through all pages in the file and look at the table id = (easy, fast enough for all but the largest databases) >=20 > Hope this helps a little. Reading MDB files is unfortunately not = trivial. >=20 > Jakob >=20 >=20 >> On 12 Jan 2016, at 13:16, Yannick Heinrich = <[email protected] <mailto:[email protected]>> wrote: >>=20 >> Hello again :) >>=20 >> I'm moving forward with my go library and I have arrived at the point = where I need to read the records from the=20 >> data pages. >>=20 >> =46rom what I understood by reading the source code of libmdb, all = the secrets are contained in the MSysObjects table. >>=20 >> Is there a good reference or description of the MSysObjects table and = object ? What is the basic approach ? >>=20 >>=20 >> Regards >> Yannick Heinrich >>=20 >>=20 >>=20 >> 2015-12-14 13:34 GMT+01:00 Yannick Heinrich = <[email protected] <mailto:[email protected]>>: >> Hello, >>=20 >> I'm currently trying to create a small Go tool that could deal with = mdb files. >>=20 >> I'm currently reading the HACKING file at = https://github.com/brianb/mdbtools/blob/master/HACKING = <https://github.com/brianb/mdbtools/blob/master/HACKING> to be able to = decode a Jet4 file for now. >>=20 >> I don't really understand the definition of tdef_pg in the data page = section. >>=20 >> Is it the index of the page within the whole file ? Is it an offset = relative to the beginning of the file ? >>=20 >> I fall on this page speaking about tdef_pg on sourceforge : = http://sourceforge.net/p/mdbtools/mailman/message/3267842/ = <http://sourceforge.net/p/mdbtools/mailman/message/3267842/> >>=20 >> I created a small program printing all the headers of the data pages = and I did not get the offset mentionned in this page. >> Moreover, on this page, a page_id is mentioned but no trace in the = HACKING file. >>=20 >> Could someone give me more information about the tdef_pg pointer and = more generally,=20 >> how is a data definition page found from the data page header ? >>=20 >> Yannick >>=20 >>=20 >>=20 >>=20 >> = --------------------------------------------------------------------------= ---- >>=20 >> _______________________________________________ >> mdbtools-dev mailing list >> [email protected] = <mailto:[email protected]> >> https://lists.sourceforge.net/lists/listinfo/mdbtools-dev = <https://lists.sourceforge.net/lists/listinfo/mdbtools-dev> >>=20 >>=20 >> = --------------------------------------------------------------------------= ---- >> Site24x7 APM Insight: Get Deep Visibility into Application = Performance >> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month >> Monitor end-to-end web transactions and take corrective actions now >> Troubleshoot faster and improve end-user experience. Signup Now! >> = http://pubads.g.doubleclick.net/gampad/clk?id=3D267308311&iu=3D/4140______= _________________________________________ = <http://pubads.g.doubleclick.net/gampad/clk?id=3D267308311&iu=3D/4140_____= __________________________________________> >> mdbtools-dev mailing list >> [email protected] = <mailto:[email protected]> >> https://lists.sourceforge.net/lists/listinfo/mdbtools-dev = <https://lists.sourceforge.net/lists/listinfo/mdbtools-dev> >=20 >=20 --Apple-Mail=_ABEAF34D-47CE-4269-AF96-493D6393CD0F Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div class=3D"">First of all: Page Usage Maps are only used = for improving performance, you don't need to use them if you just want = to read data.</div><div class=3D""><br class=3D""></div><div = class=3D"">Every table has two main page usage bitmaps: the bitmap with = all the used pages, and the bitmap with free pages.</div><div = class=3D""><br class=3D""></div><div class=3D"">The used_pages bitmap = tells you which pages in the database belong to that table. You can find = the same info by just looping through all pages and checking the table = id in the page header. Anyway, here are two simple examples: If page = index 6 belongs to a table, then the page usage bitmap would be 01000000 = in binary or the single byte 0x80. If pages 6 and 9 belong to a table, = the page usage bitmap would be two bytes: 0x80 0x02.</div><div = class=3D""><br class=3D""></div><div class=3D"">The two different types = of bitmaps are for small and for large databases. If a bitmap for the = full database fits on a single page (less than 32736 pages), then = type 0 bitmaps are used. Otherwise type 1 bitmaps are used, meaning that = the page usage bitmap is spread over multiple pages (see hacking file = for details)</div><div class=3D""><br class=3D""></div><div class=3D"">The= free space usage map is only necessary if you want to support writing: = it tells Access on which pages it can find space to write a new = row.</div><div class=3D""><br class=3D""></div><div class=3D"">Then = there are usage maps related to the indexes, and apparently there are = also usage maps related to columns at the end of the tdef pages... I = don't know how those work, I assume those are also some kinds of = optimisations.</div><div class=3D""><br class=3D""></div><div class=3D"">I= hope this helps, but as I said before, I'd recommend ignoring the page = usage bitmaps at first. In my experience, they are also sometimes = corrupted, so sometimes you need to fall back to scanning the full = database anyway.</div><div class=3D""><br class=3D""></div><div = class=3D"">Jakob</div><br class=3D""><div><blockquote type=3D"cite" = class=3D""><div class=3D"">On 28 Jan 2016, at 23:34, Yannick Heinrich = <<a href=3D"mailto:[email protected]" = class=3D"">[email protected]</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><div dir=3D"ltr" = class=3D"">This helps me to understand the process :) <div = class=3D""><br class=3D""></div><div class=3D"">May be you can give me = some more explanations about usage maps.</div><div class=3D""><br = class=3D""></div><div class=3D"">In the HACKING file of the mdv-tools = directory, it describes the different pages type in the file. 0x05 is = for "Page Usage Bitmaps".</div><div class=3D""><br class=3D""></div><div = class=3D"">But how is the link done with the others pages (data and = table definition) ? Inside the descriptions of these pages, the document = refers to "usage bit mask". </div><div class=3D""><br = class=3D""></div><div class=3D"">What is the difference between Page = Usage Bitmaps and usage bit mask ?</div><div class=3D""><br = class=3D""></div><div class=3D"">There is also two types of Page Usage = (0x00 and 0x01). Are those types present after the page usage bitmaps = header ?</div><div class=3D""><br class=3D""></div><div = class=3D"">Regards</div><div class=3D"">Yannick</div><div = class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">2016-01-12= 22:23 GMT+01:00 Jakob Egger <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:[email protected]" target=3D"_blank" = class=3D"">[email protected]</a>></span>:<br class=3D""><blockquote = class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc = solid;padding-left:1ex"><div style=3D"word-wrap:break-word" = class=3D"">MSysObjects just tells you the name, type, etc. of the table. = You can ignore it if you just want to read the data. <div = class=3D""><div class=3D""><br class=3D""></div><div class=3D"">If you = want to read the records, you need information from the tdef page. The = tdef pages tell you how what fields are in a table, what types they = have, and so on. You need that information to read the data = pages.</div><div class=3D""><br class=3D""></div><div class=3D"">What = does a data page look like? Well, it has a header that tells you what = table it belongs to (table id =3D index of the tdef page). Then it is = filled with records, starting from the end. The header stores the = offsets of the records.</div><div class=3D""><br class=3D""></div><div = class=3D"">What do records look like?</div><div class=3D"">Each record = has all the fixed length fields in the front (use info from tdef page to = parse). All the variable length fields are at the end, starting from the = back. You could actually parse variable length fields without the info = from the tdef page (at least text columns). The NULL map (which fields = are NULL) is also stored at the end of the record.</div><div = class=3D""><br class=3D""></div><div class=3D"">How do you find data = pages for a specific table?</div><div class=3D"">1) Look at the page = usage bitmap for the table (complicated)</div><div class=3D"">2) Just = loop through all pages in the file and look at the table id (easy, fast = enough for all but the largest databases)</div><div class=3D""><br = class=3D""></div><div class=3D"">Hope this helps a little. Reading MDB = files is unfortunately not trivial.</div><div class=3D""><br = class=3D""></div><div class=3D"">Jakob</div><div class=3D""><br = class=3D""></div><div class=3D""><br class=3D""><div = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div = class=3D""><div class=3D"">On 12 Jan 2016, at 13:16, Yannick Heinrich = <<a href=3D"mailto:[email protected]" target=3D"_blank" = class=3D"">[email protected]</a>> wrote:</div><br = class=3D""></div></div><div class=3D""><div class=3D""><div = class=3D""><div dir=3D"ltr" class=3D"">Hello again :)<div class=3D""><br = class=3D""></div><div class=3D"">I'm moving forward with my go library = and I have arrived at the point where I need to read the records from = the </div><div class=3D"">data pages.</div><div class=3D""><br = class=3D""></div><div class=3D"">=46rom what I understood by reading the = source code of libmdb, all the secrets are contained in the MSysObjects = table.</div><div class=3D""><br class=3D""></div><div class=3D"">Is = there a good reference or description of the MSysObjects table and = object ? What is the basic approach ?</div><div class=3D""><br = class=3D""></div><div class=3D""><br class=3D""></div><div = class=3D"">Regards</div><div class=3D"">Yannick Heinrich</div><div = class=3D""><br class=3D""></div><div class=3D""><br = class=3D""></div></div><div class=3D"gmail_extra"><br class=3D""><div = class=3D"gmail_quote">2015-12-14 13:34 GMT+01:00 Yannick Heinrich <span = dir=3D"ltr" class=3D""><<a href=3D"mailto:[email protected]" = target=3D"_blank" class=3D"">[email protected]</a>></span>:<br= class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr" = class=3D""><span style=3D"font-size:12.8px" class=3D"">Hello,</span><div = style=3D"font-size:12.8px" class=3D""><br class=3D""></div><div = style=3D"font-size:12.8px" class=3D"">I'm currently trying to create a = small Go tool that could deal with mdb files.</div><div = style=3D"font-size:12.8px" class=3D""><br class=3D""></div><div = style=3D"font-size:12.8px" class=3D"">I'm currently reading the HACKING = file at <a = href=3D"https://github.com/brianb/mdbtools/blob/master/HACKING" = target=3D"_blank" = class=3D"">https://github.com/brianb/mdbtools/blob/master/HACKING</a> = ;to be able to decode a Jet4 file for now.</div><div = style=3D"font-size:12.8px" class=3D""><br class=3D""></div><div = style=3D"font-size:12.8px" class=3D"">I don't really understand the = definition of tdef_pg in the data page section.</div><div = style=3D"font-size:12.8px" class=3D""><br class=3D""></div><div = style=3D"font-size:12.8px" class=3D"">Is it the index of the page within = the whole file ? Is it an offset relative to the beginning of the file = ?</div><div style=3D"font-size:12.8px" class=3D""><br = class=3D""></div><div style=3D"font-size:12.8px" class=3D"">I fall on = this page speaking about tdef_pg on sourceforge : <a = href=3D"http://sourceforge.net/p/mdbtools/mailman/message/3267842/" = target=3D"_blank" = class=3D"">http://sourceforge.net/p/mdbtools/mailman/message/3267842/</a><= /div><div style=3D"font-size:12.8px" class=3D""><br class=3D""></div><div = style=3D"font-size:12.8px" class=3D"">I created a small program printing = all the headers of the data pages and I did not get the offset = mentionned in this page.</div><div style=3D"font-size:12.8px" = class=3D"">Moreover, on this page, a page_id is mentioned but no trace = in the HACKING file.</div><div style=3D"font-size:12.8px" class=3D""><br = class=3D""></div><div style=3D"font-size:12.8px" class=3D"">Could = someone give me more information about the <span = style=3D"font-size:12.8px" class=3D"">tdef_pg pointer and more = generally, </span></div><div style=3D"font-size:12.8px" = class=3D""><span style=3D"font-size:12.8px" class=3D"">how is a data = definition page found from the data page header ?</span></div><span = class=3D""><font color=3D"#888888" class=3D""><div = style=3D"font-size:12.8px" class=3D""><span style=3D"font-size:12.8px" = class=3D""><br class=3D""></span></div><div style=3D"font-size:12.8px" = class=3D"">Yannick</div><div style=3D"font-size:12.8px" class=3D""><br = class=3D""></div><div style=3D"font-size:12.8px" class=3D""><span = style=3D"font-size:12.8px" class=3D""><br class=3D""></span></div><div = style=3D"font-size:12.8px" class=3D""><span style=3D"font-size:12.8px" = class=3D""><br class=3D""></span></div></font></span></div> <br = class=3D"">---------------------------------------------------------------= ---------------<br class=3D""> <br class=3D"">_______________________________________________<br = class=3D""> mdbtools-dev mailing list<br class=3D""> <a href=3D"mailto:[email protected]" target=3D"_blank" = class=3D"">[email protected]</a><br class=3D""> <a href=3D"https://lists.sourceforge.net/lists/listinfo/mdbtools-dev" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">https://lists.sourceforge.net/lists/listinfo/mdbtools-dev</a><b= r class=3D""> <br class=3D""></blockquote></div><br class=3D""></div></div></div> = --------------------------------------------------------------------------= ----<br class=3D"">Site24x7 APM Insight: Get Deep Visibility into = Application Performance<br class=3D"">APM + Mobile APM + RUM: Monitor 3 = App instances at just $35/Month<br class=3D"">Monitor end-to-end web = transactions and take corrective actions now<br class=3D"">Troubleshoot = faster and improve end-user experience. Signup Now!<br class=3D""><a = href=3D"http://pubads.g.doubleclick.net/gampad/clk?id=3D267308311&iu=3D= /4140_______________________________________________" target=3D"_blank" = class=3D"">http://pubads.g.doubleclick.net/gampad/clk?id=3D267308311&i= u=3D/4140_______________________________________________</a><span = class=3D""><br class=3D"">mdbtools-dev mailing list<br class=3D""><a = href=3D"mailto:[email protected]" target=3D"_blank" = class=3D"">[email protected]</a><br class=3D""><a = href=3D"https://lists.sourceforge.net/lists/listinfo/mdbtools-dev" = target=3D"_blank" = class=3D"">https://lists.sourceforge.net/lists/listinfo/mdbtools-dev</a><b= r class=3D""></span></div></blockquote></div><br = class=3D""></div></div></div></blockquote></div><br = class=3D""></div></div> </div></blockquote></div><br class=3D""></body></html>= --Apple-Mail=_ABEAF34D-47CE-4269-AF96-493D6393CD0F-- --===============4206805617080937563== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140 --===============4206805617080937563== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ mdbtools-dev mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/mdbtools-dev --===============4206805617080937563==--