Re: Patches + thoughts

Nirgal Vourgère <[email protected]> Sat, 19 Feb 2011 14:08:35 +0000

Newsgroups	gmane.comp.db.mdb-tools.devel
Message-ID	<[email protected]>

On Friday 18 February 2011 19:38:19 Brian Bruns wrote:
> I'd like to keep quote/escape (particularly quote, I can't think of a
> case where escape is more than one char) as a string since it allows a
> multicharacter delimiter, not as uncommon a convention as you might
> think.

Right.
Attached is a revised version of export patch.

That does not really anwser my question: "Should we escape the escape character ?"
My patch does.
If you or anyone don't like it, there is a #define DONT_ESCAPE_ESCAPE

Attached is an update for Binary type (should not be treated as Memo).

> I agree that the escaping needs some work as it is often not as simple
> as adding a simple leading character, but that gets a little
> complicated and we'd have to bake in some escaping styles rather than
> the generic switch we now have.

I agree.

But I bet the INSERT version of mdb-export doesn't support charcters < 32 right now, nor non-utf8 characters above 127 (OLE).

I guess the best solution would be to move the INSERT code into the schema exportation tools (backend.c).
The insert would then occur before foreign keys setup. We could also set up the sequences (autonum/serial) automatically then. And have a full mdb->sql conversion tool with both schema and data.

I don't really have the time for that. Would that be a good idea?

> On unicode2ascii, I have no clue how string compression would work for
> BIG5, but otherwise it basically punts on the overrun question, it's
> up to the calling function to properly allocate the size which is
> generally MDB_BIND_SIZE, which is 4 times the page size, so that
> should accommodate any code set.

I hit some surprised with props, because I was careless of the unciode conversion expanding size problem at first. I mean column name sizes.
Anyways. I was thinking about Memo fields. I though the limit was 16k, so that mdbtools might fail after utf-8 conversion.
But Memo field limit actually is 64k chars (192k bytes in utf-8), so we are in trouble anyways.

Maybe all LVal should be handled the same way, with no automatic binding. We now have 3 versions of lval reading in data.c...

Here's another hint: The table/column props internally have:
- Name
- type (like col_type)
- Buffer (pointer + size)
Right now, we are converting everything to a proper string, meaning we loose the type. We might want a kind of MdbVariant (type+buffer) that could be played with. Like numeric/date operations.

I have the feeling a MdbVariant might be usefull for binding too. We could probably wrap a LVal reader in it.

Ok, I stop daydreaming now... :)

------------------------------------------------------------------------------
The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb

_______________________________________________
mdbtools-dev mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/mdbtools-dev

export (text/x-patch, 2.5 KB)

Index: mdbtools-0.6pre1/src/util/mdb-export.c
===================================================================
--- mdbtools-0.6pre1.orig/src/util/mdb-export.c
+++ mdbtools-0.6pre1/src/util/mdb-export.c
@@ -26,40 +26,44 @@
 #undef MDB_BIND_SIZE
 #define MDB_BIND_SIZE 200000
 
-#define is_text_type(x) (x==MDB_TEXT || x==MDB_OLE || x==MDB_MEMO || x==MDB_DATETIME || x==MDB_BINARY)
+#define is_quote_type(x) (x==MDB_TEXT || x==MDB_OLE || x==MDB_MEMO || x==MDB_DATETIME || x==MDB_BINARY)
+#define is_binary_type(x) (x==MDB_OLE || x==MDB_BINARY)
 
 static char *escapes(char *s);
 
-void
-print_col(gchar *col_val, int quote_text, int col_type, int bin_length, char *quote_char, char *escape_char)
+//#define DONT_ESCAPE_ESCAPE
+static void
+print_col(gchar *col_val, int quote_text, int col_type, int bin_len, char *quote_char, char *escape_char)
 {
-	gchar *s;
-	unsigned char c;
+	size_t quote_len = strlen(quote_char); /* multibyte */
 
-	if (quote_text && is_text_type(col_type)) {
+	size_t orig_escape_len = escape_char ? strlen(escape_char) : 0;
+
+	/* double the quote char if no escape char passed */
+	if (!escape_char)
+		escape_char = quote_char;
+
+	if (quote_text && is_quote_type(col_type)) {
 		fputs(quote_char,stdout);
-		if (col_type == MDB_OLE || col_type == MDB_BINARY)	{
-			while (bin_length--) {
-				c = (unsigned char)*col_val++;
-				if (strlen(quote_char)==1 && c==quote_char[0] || c==escape_char[0]) {
-					if (escape_char)
-						fputs(escape_char,stdout);
-					else /* double the quote char if no escape char passed */
-						fputs(quote_char,stdout);
-				}
-				putc(c, stdout);
-			}
+		while (1) {
+			if (is_binary_type(col_type)) {
+				if (!bin_len--)
+					break;
+			} else /* use \0 sentry */
+				if (!*col_val)
+					break;
+
+			if (quote_len && !strncmp(col_val, quote_char, quote_len)) {
+				fprintf(stdout, "%s%s", escape_char, quote_char);
+				col_val += quote_len;
+#ifndef DONT_ESCAPE_ESCAPE
+			} else if (orig_escape_len && !strncmp(col_val, escape_char, orig_escape_len)) {
+				fprintf(stdout, "%s%s", escape_char, escape_char);
+				col_val += orig_escape_len;
+#endif
+			} else
+				putc(*col_val++, stdout);
 		}
-		else
-			for (s=col_val;(c=*s);s++) {
-				if (strlen(quote_char)==1 && c==quote_char[0] || c==escape_char[0]) {
-					if (escape_char)
-						fputs(escape_char,stdout);
-					else /* double the quote char if no escape char passed */
-						fputs(quote_char,stdout);
-				}
-				putc(c, stdout);
-			}
 		fputs(quote_char,stdout);
 	} else
 		fputs(col_val,stdout);

binaries (text/x-patch, 707 B)

Index: mdbtools-0.6pre1/src/libmdb/data.c
===================================================================
--- mdbtools-0.6pre1.orig/src/libmdb/data.c
+++ mdbtools-0.6pre1/src/libmdb/data.c
@@ -914,6 +914,13 @@
 			td = mdb_get_double(buf, start);
 			text = g_strdup_printf("%.16e", td);
 		break;
+		case MDB_BINARY:
+			if (size<0) {
+				text = g_strdup("");
+			} else {
+				text = g_malloc(size);
+				memcpy((char*)buf+start, text, size);
+			}
 		case MDB_TEXT:
 			if (size<0) {
 				text = g_strdup("");
@@ -927,7 +934,6 @@
 			text = mdb_date_to_string(mdb, start);
 		break;
 		case MDB_MEMO:
-		case MDB_BINARY:
 			text = mdb_memo_to_string(mdb, start, size);
 		break;
 		case MDB_MONEY: