Proposed changes to HACKING doc

I have been collecting changes/corrections to the HACKING document.  
Some of these tidbits are apparently already known (since the code uses 
them), but they haven't found their way into the doc.  Some appear to be 
new discoveries I have made.  And some appear to be just plain incorrect 
info in the HACKING doc.  Perhaps the most interesting item is #8.

However, I am very new to MDBTools.  I would appreciate it if some of 
you people who have been digging into these files for years could 
review/critique these observations.  While it makes sense to keep the 
doc as up-to-date as possible, we don't want to get it wrong.

Please confirm/deny any of these 9 assertions that you can.

Thanks.

dw

==============
1) From file.c we learn that:

     /* get the db password located at 0x42 bytes into the file */

So apparently on the Database Definition Page, the 14 bytes starting at 
0x42 are the (encrypted) database password.

==============
2) Also from file.c:

     mdb->f->db_key = mdb_get_int32(mdb->pg_buf, 0x3e);

Apparently the 4 bytes at 0x3e on the Database Definition Page are the 
database key.

==============
3) Per my experimentation, Access 2007 apparently uses Jet v5.  So now 
the known values for the byte at Offset 0x14 of the Database Definition 
Page are:

Jet 3: 0
Jet 4: 1
Jet 5: 2

Has anyone tried Access 2010?

==============
4) From table.c we see that:

         pcol->is_long_auto = col[fmt->col_flags_offset]&  0x04 ? 1 : 0;
	pcol->is_uuid_auto = col[fmt->col_flags_offset]&  0x40 ? 1 : 0;

My own observations suggest bit 2 indicates whether the column can be null.

Thus, the known bits of the column bitmask are:

0x01: variable length column
0x02: can be null
0x04: is auto long
0x40: is auto guid

So, this line from HACKING is insufficient:

| ???? | 1 byte  | bitmask     | low order bit indicates variable columns |

==============
5) Examining the raw data for columns in a v4 database, it looks like 
the first two bytes after the (second) col_num are an LCID.  That would 
explain why there are commonly 0x409 (meaning English).

==============
6) Looking at one of the Windows headers (dbdaoint.h 
<http://www.koders.com/c/fidE758EB80731C033AA2D2015B9A0EAF6F1C2C1ECE.aspx?s=matrix>), 
we see the DataTypeEnum enum has a remarkable correlation with the 
values for column data types.  That suggests that the other values in 
the enum may also apply:

dbdaoint.h:

	dbBoolean	= 1,
	dbByte	= 2,
	dbInteger	= 3,
	dbLong	= 4,
	dbCurrency	= 5,
	dbSingle	= 6,
	dbDouble	= 7,
	dbDate	= 8,
	dbBinary	= 9,
	dbText	= 10,
	dbLongBinary	= 11,
	dbMemo	= 12,
	dbGUID	= 15,
	dbBigInt    = 16,
	dbVarBinary    = 17,
	dbChar    = 18,
	dbNumeric    = 19,
	dbDecimal    = 20,
	dbFloat    = 21,
	dbTime    = 22,
	dbTimeStamp    = 23

==============
7) I believe I have more info about the data at the end of a tabledef:

| ???? | n bytes | ???         |                                          |
| 0xFF | 2 bytes | ???         | End of the tableDef  ?                   |

It appears the struct for this data actually looks like this:

#pragma pack(1)
typedef struct {
     short col_num;
     int arg1;
     int arg2;
} TDef_Unknown;

The first two bytes are either a column number, or 0xffff to indicate 
the end of the list.  Of the > 80 tables I have looked at, only columns 
of type OLE and Memo have entries in this table.  In fact, only OLE and 
Memo fields with a length of 0xffff.  Contra-wise, while not all OLE and 
Memo fields have entries in this list, all the OLE and Memo fields with 
a length of 0xffff have entries in this list.

I haven't yet determined the meaning of arg1 and arg2.  However, I do 
believe my guess about them being ints is correct.

I can tell you that of the 192 TDef_Unknowns that I have found:

- In all 192 cases, arg2 = arg1 + 1
- arg1 and arg2 are all unique values.  ie None of the other 
TDef_Unknown entries anywhere in the db use the same values.
- I don't believe arg1 and arg2 are page numbers (too big), or point to 
primary keys in some database table (can't find any correlation).

Here's some entries to show what I mean:

col_num arg1 arg2 coltype colsize

2 00005c02 00005c03 12 65535
3 00005c04 00005c05 12 65535
4 00005c06 00005c07 12 65535
5 00005c08 00005c09 12 65535
6 00005c0a 00005c0b 12 65535
7 00005c0c 00005c0d 12 65535
8 00005c0e 00005c0f 12 65535
9 00005c10 00005c11 12 65535
10 00005c12 00005c13 12 65535
11 00005c14 00005c15 12 65535
12 00005c16 00005c17 12 65535
14 00005c18 00005c19 12 65535
15 00005c1a 00005c1b 12 65535
16 00005c1c 00005c1d 12 65535
17 00005c1e 00005c1f 12 65535
18 00005c20 00005c21 12 65535
19 00005c22 00005c23 12 65535
10 00006a02 00006a03 12 65535
11 00008402 00008403 12 65535
12 00008404 00008405 12 65535
13 00008406 00008407 12 65535
14 00008408 00008409 12 65535

These records are all from a single table that contains many Memo 
fields.  Notice that while the numbers always increase, they don't 
always do so sequentially.

So, I don't believe arg1 and arg2 are any type of datatype or flag 
fields (doesn't make sense with incrementing fields).  Nor do I think 
they have anything to do with the # of records in a table (I have large 
numbers in TDEF_Unknown for tables with 0 records).

So far, I'm stumped.  But I feel certain enough about the fact they 
these are records, the record lengths, the list terminator and the 
contents of the first bytes to suggest putting them in the doc.

==============
8) The definitions for index information seem to be wrong.

The problem starts with this area of HACKING:

| ???? | 4 bytes | num_idx     | Number of indexes in table               |
| ???? | 4 bytes | num_real_idx| Number of indexes in table (repeat)      |

Which should be:

| ???? | 4 bytes | num_idx     | Number of indexes in table               |
| ???? | 4 bytes | num_idx_col | Number of index column entries           |

Also this:

| ???? | 4 bytes | index_num2  | Number of the index (repeat)             |

should be this

| ???? | 4 bytes | index_cols_index  | Index into index cols list         |

Lastly

| ???? | 1 byte | primary_key | 0x01 if this index is primary |

Should be

| ???? | 1 byte | index_flags | see MDB_IDX_* flags |

Since it can contain any of these flags (from mdbtools.h):

	MDB_IDX_UNIQUE = 0x01,
	MDB_IDX_IGNORENULLS = 0x02,
	MDB_IDX_REQUIRED = 0x08

So, when walking the list of index definitions, you use num_idx to tell 
you how many indices there are.  Then for each index, you use 
index_Cols_index to index into the column definitions table to find the 
columns for that index.  The reason num_idx and num_idx_col might be 
different is not because some of the indices are "real" (whatever that 
was supposed to mean), but simply because more than one index uses the 
same column list.  I've seen this happen if there is one index that 
includes nulls, and one that doesn't.

I have a re-written mdb_read_indices that (I believe) correctly parses 
this data if anyone is interested.

==============
9) The Memo Field Definition is incorrect.  This:

| ???? | 3 bytes | memo_len    | Total length of the memo                 |
| ???? | 1 bytes | bitmask     | See values                               |

should be:

| ???? | 4 bytes | memo_len    | Total length of the memo w/flags    |

Where the top 2 bits are the flags described.  The bitmask *can't* be an 
entire byte long.  OLE fields can hold up to 1gig.  That requires at 
least 30 bits, leaving only 2 bits for flags.  In fact, this quote 
<http://office.microsoft.com/en-us/access-help/access-2007-specifications-HA010030739.aspx> 
from makes me wonder if sometimes 0xC0000000 is ignored?

     Number of characters in a Memo field: 65,535 when entering data through the user interface; 2 gigabytes of character storage when entering data programmatically

Cuz 2gigs is going to take 31 bits

------------------------------------------------------------------------------
Free Software Download: Index, Search & Analyze Logs and other IT data in 
Real-Time with Splunk. Collect, index and harness all the fast moving IT data 
generated by your applications, servers and devices whether physical, virtual
or in the cloud. Deliver compliance at lower cost and gain new business 
insights. http://p.sf.net/sfu/splunk-dev2dev

_______________________________________________
mdbtools-dev mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/mdbtools-dev