Re: Re: content encoding etc. (updated)

Hatuka*nezumi - IKEDA Soji <[email protected]> Sun, 23 May 2004 18:42:14 +0900
Newsgroups gmane.mail.sqwebmail
Message-ID <[email protected]>
On Sat, 22 May 2004 10:20:48 -0400
Sam Varshavchik <[email protected]> wrote:

> Hatuka*nezumi - IKEDA Soji writes:
> 
> > Updated patch for 4.0.4. I splitted it to 4 subpatches.
> > 
> > # Changes on flowed format are removed.  I wish more suggestions.
> 
> 1) Small problem in encodebase64().  When generating multiple encoded words, 
> they need to be separated by spaces.  As it is, encodebase64() will generate 
> "=?...?==?...?=" with no intervening space.

Uh, sorry.


> 2) The logic in rfc2047_encode_callback that tries to avoid breaking up 
> multibyte characters is hard to follow.  I think there's a better approach.
> 
> First, convert the entire string to unicode.  Then, take a small chunk of 
> unicodes and convert it back to the original charset.  If the result is 
> larger than RFC2047_ENCODE_FOLDLENGTH, try again with a slightly smaller 
> chunk.  Encode it, then repeat until the entire unicode string is done.
> 
> If you know that, on average, the charset uses two octets per character then 
> you can make a pretty good guess as to how many unicodes to try, each time.

With 'small by small' approach, wrapped lines will be a bit ragged.
Because multibyte string may contain both multibyte and singlebyte 
characters.  However, it is compliant to MIME.


Anyway, additional fixes are attached (diff to 20040522 patch).

Thanks.


  --- nezumi
sqwebmail-4.0.4-20040522_fix0523.patch (application/octet-stream, 4 KB)
--- rfc822/rfc2047.c.orig	2004-05-22 12:33:15.000000000 +0900
+++ rfc822/rfc2047.c	2004-05-23 18:11:44.000000000 +0900
@@ -553,6 +553,13 @@
 
 		if ((rc=(*func)("?=", 2, arg)))
 			return rc;
+		if (*ptr)
+			/*
+			 * Encoded-words must be sepalated by
+			 * linear-white-space.
+			 */
+			if ((rc=(*func)(" ", 1, arg)))
+				return rc;
 	}
 	return 0;
 }
@@ -601,97 +608,90 @@
 	uptr = ustr;
 	while (*uptr)
 	{
-	unicode_char *uwptr;
 	unicode_char save_uc;
-	char *wstr;
-	size_t i;
+	char *wstr=NULL;
+	size_t i, end, j;
 
-		i = offset + 2 + strlen(charset) + 3;
+		if ((i = offset + 2 + strlen(charset) + 3) >
+		    RFC2047_ENCODE_FOLDLENGTH - 2)
+			/* Keep room for at least one character. */
+			i = RFC2047_ENCODE_FOLDLENGTH - 2;
 		offset = 0;
 
 		/*
 		 * Figure out where to break encoded-word.
-		 * Note that multibyte character boundaries must not be broken,
-		 * and at the end of string in encoded-word, context must be 
-		 * US-ASCII.
+		 * Take a small chunk of Unicode string and convert it back to
+		 * the original charset.  If the result exseeds line length,
+		 * try again with a shorter chunk.  
 		 */
-		uwptr = uptr;
-		while (*uwptr)
+		end = 0;
+		while (uptr[end] && end < (RFC2047_ENCODE_FOLDLENGTH - i) / 2)
+			end++; 
+			/*
+			 * FIXME: Unicode character with `combining'
+			 * property etc. should not be treated as
+			 * separate character.
+			 */
+		j = end;
+		while (j)
 		{
-		size_t elen;
-		unicode_char *prev_uwptr, *last_error=NULL;
-
-			prev_uwptr=uwptr;
-			uwptr++;	/* FIXME: Unicode character with
-					 * `combining' property etc. should
-					 * not be treated as separate
-					 * character.
-					 */
-			save_uc = *uwptr;
-			*uwptr = (unicode_char)0;
+			save_uc = uptr[j];
+			uptr[j] = (unicode_char)0;
 			wstr = (uiptr->u2c)(uiptr, uptr, &dummy);
-			*uwptr = save_uc;
+			uptr[j] = save_uc;
 
 			if (!wstr)
+			/* Possiblly a part of one character extracted to 
+			 * multiple Unicode characters (e.g. base unicode 
+			 * character of one combined character).  Try on 
+			 * shorter chunk.
+			 */
+			{
+				j--;	/* FIXME */
+				continue;
+			}
+
+			if (i + ((strlen(wstr) + 3-1) / 3) * 4 + 2 >
+			    RFC2047_ENCODE_FOLDLENGTH - 1)
 			/*
-			 * Possiblly a part of one character extracted to 
-			 * multiple unicode characters (e.g. base unicode
-			 * character of one combined character).
-			 * Seek the end of complete character.
+			 * Encoded string exceeded line length.
+			 * Try on shorter chunk.
 			 */
 			{
-				last_error = uwptr;
-				while (*uwptr)
-				{
-					uwptr++; /* FIXME: see above */
-					save_uc = *uwptr;
-					*uwptr = (unicode_char)0;
-					wstr=(uiptr->u2c)(uiptr, uptr, &dummy);
-					*uwptr = save_uc;
-					if (wstr)
-						break;
-				}
-				if (!wstr) /* Real error. Replace it. */
+			size_t	k=j;
+
+				j--;	/* FIXME */
+				if (j == 0)
+				/* Only one character exeeds line length.
+				 * Anyway, encode it. */
 				{
-					uwptr = prev_uwptr;
-					*uwptr = (unicode_char)0x003F;
-					continue;
+					j = k;
+					break;
 				}
-				prev_uwptr = last_error;
+				free(wstr);
+				continue;
 			}
-			elen = i + ((strlen(wstr) + 3-1) / 3) * 4 + 2;
-			free(wstr);
 
-			if (elen > RFC2047_ENCODE_FOLDLENGTH - 1 - 4)
-			{
-				/*
-				 * Exceeded line length.  If encoded string
-				 * have more than one character, rewind to
-				 * previous position and encode it.
-				 */
-				if (prev_uwptr != uptr &&
-				    prev_uwptr != last_error)
-					uwptr = prev_uwptr;
-				/*
-				 * Otherwise, only one character exceeds line 
-				 * length.  Anyway, encode it.
-				 */
-				break;
-			}
+			break;
 		}
 
-		save_uc = *uwptr;
-		*uwptr = (unicode_char)0;
-		wstr = (uiptr->u2c)(uiptr, uptr, NULL);
-		*uwptr = save_uc;
-		rc = encodebase64(wstr, charset, func, arg, 0, 0);
-		if (wstr) free(wstr);
+		if (!wstr)
+		{
+			end = 1;
+			rc = encodebase64("?", charset, func, arg, 0, 0);
+		}
+		else
+		{
+			end = j;
+			rc = encodebase64(wstr, charset, func, arg, 0, 0);
+			free(wstr);
+		}
 		if (rc)
 		{
 			free(ustr);
 			return rc;
 		}
-		uptr = uwptr;
+		uptr += end;
 
 		if (*uptr)
 			/*