Re: filenames on MS Windows

On Sun, 25 Oct 2020, Allin Cottrell wrote:

> I tried googling this but didn't find an answer -- sorry if I should have 
> just tried harder! My question is: can gnuplot on Windows handle a unicode 
> filename argument passed in UTF-16? As in
>
> path/to/wgnuplot.exe <UTF-16 input filename>

OK, that question was under-researched, but now I've done my 
homework. Sorry, this is a bit long but I hope I can arouse some 
interest in the topic.

Why bother with UTF-16 filename arguments? Nowadays a fair number of 
Windows users construct paths (directory names or filenames) which 
are "out of codepage" -- that is, unicode names which cannot be 
represented in the (retro) "system codepage", which is typically 
just an 8-bit encoding. Since Windows has supported unicode since NT 
came out, it's a reasonable expectation that any filename one can 
construct on the platform should be accessible via any program of 
interest. But a program that restricts itself to the "ANSI" form of 
filenames simply cannot access files with out-of-codepage paths. 
(Sane modern OSes don't have this problem because they use UTF-8 
throughout.)

So what about gnuplot? I may be wrong but it seems to me that 
gnuplot on Windows is stuck with "ANSI" filenames at present. Even 
with UNICODE and _UNICODE defined when compiling the program, the 
command-line arguments are retrieved in winmain.c using either _argv 
or __argv (depending on the compiler), and these get the ANSI-form 
arguments (as opposed to __wargv which gets the arguments in UTF-16 
form).

It would be easy to swap out __argv for __wargv but by itself this 
would be very disruptive. The subsequent code in winmain.c, and then 
the code in plot.c (gnu_main) to which the args array is passed, all 
assumes the elements of argv are plain "char *", not "wide char" 
arrays. Handling UTF-16, which is chock-full of NUL bytes, would 
require lots of messy "ifdefs".

I have a proposal for fixing this. I realise it may not be 
acceptable as it stands but maybe someone else might want to take it 
up. I'm attaching patches for src/win/winmain.c and src/misc.c for 
reference but here I'll try to explain the strategy.

1) In winmain.c, grab the command-line arguments as UTF-16 but 
immediately convert them to UTF-8, so they can handled by the 
regular string.h APIs, both here and in plot.c (gnu_main).

2) When we actually go to open a command-line file argument 
(loadpath_fopen, in misc.c, called from gnu_main), we first try 
opening the file using the filename as-is, but it that fails (and 
the filename validates as UTF-8) we convert it to UTF-16 and try 
again.

Since UTF-8 is a superset of ASCII, ASCII filename arguments should 
pass through transparently. Within-codepage non-ASCII filenames 
should get converted back to UTF-16 and opened OK. And the bonus is 
that out-of-codepage arguments should also be converted and opened 
OK.

I've tested this on Windows 10, with the system codepage set to 
Windows 1252 ("Western Europe"), and have successfully opened files 
with names in Russian and Greek. (I think this should also work if 
the user has the system codepage set to UTF-8 (65001), which is a 
"beta" option on Windows.)

My implementation uses GLib APIs (nice and simple) to convert from 
UTF-16 to UTF-8 and back again (if needed). GLib is required anyway 
if one is building the Cairo-based terminals. I suppose one could 
use native Windows APIs to the same purpose but I suspect it would 
be a lot more bother.

In my test setup this whole deal is triggered by the CFLAGS define

-DWIDE_ARGS

which is respected only when building for Windows -- and admittedly 
has only been tested when cross-compiling for Windows from Linux 
using Mingw-w64. In my mingw Makefile, I have:

WIDE_ARGS = 1

...

ifdef WIDE_ARGS
   CFLAGS += -DWIDE_ARGS
   CFLAGS += $(shell pkg-config --cflags glib-2.0)
endif

--
Allin Cottrell
Department of Economics
Wake Forest University

_______________________________________________
gnuplot-beta mailing list
[email protected]
Membership management via: https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

--- misc.c.orig	2020-10-31 09:56:21.939348916 -0400
+++ misc.c	2020-10-31 16:43:54.466247260 -0400
@@ -48,6 +48,8 @@
 # include <fcntl.h>
 # if defined(__WATCOMC__) || defined(_MSC_VER)
 #  include <io.h>        /* for setmode() */
+# elif defined(WIDE_ARGS)
+#  include <glib.h>
 # endif
 #endif
 
@@ -581,6 +583,31 @@
     while (lf_pop());
 }
 
+static FILE *
+alt_fopen(const char *filename, const char *mode)
+{
+    FILE *fp = fopen(filename, mode);
+
+#ifdef WIDE_ARGS
+    if (fp == NULL && g_utf8_validate(filename, -1, NULL)) {
+	gunichar2 *wfname;
+
+	wfname = g_utf8_to_utf16(filename, -1, NULL, NULL, NULL);
+	if (wfname != NULL) {
+	    size_t len = strlen(mode) + 1;
+	    wchar_t *wmode = calloc(len, sizeof *wmode);
+	    
+	    mbstowcs(wmode, mode, len);
+	    fp = _wfopen(wfname, wmode);
+	    free(wmode);
+	    g_free(wfname);
+	}
+    }
+#endif
+
+    return fp;
+}
+
 FILE *
 loadpath_fopen(const char *filename, const char *mode)
 {
@@ -602,7 +629,7 @@
 	    return (FILE *) 0;
     } else
 #endif /* PIPES */
-    if ((fp = fopen(filename, mode)) == (FILE *) NULL) {
+    if ((fp = alt_fopen(filename, mode)) == (FILE *) NULL) {
 	/* try 'loadpath' variable */
 	char *fullname = NULL, *path;
 
@@ -611,7 +638,7 @@
 	    fullname = gp_realloc(fullname, strlen(path) + 1 + strlen(filename) + 1, "loadpath_fopen");
 	    strcpy(fullname, path);
 	    PATH_CONCAT(fullname, filename);
-	    if ((fp = fopen(fullname, mode)) != NULL) {
+	    if ((fp = alt_fopen(fullname, mode)) != NULL) {
 		/* free(fullname); */
 		loadpath_fontname = fullname;
 		fullname = NULL;

--- winmain.c.orig	2020-10-31 10:00:07.180096397 -0400
+++ winmain.c	2020-10-31 16:45:23.760210002 -0400
@@ -86,6 +86,9 @@
 # include "caca.trm"
 # undef TERM_PUBLIC_PROTO
 #endif
+#ifdef WIDE_ARGS
+# include <glib.h>
+#endif
 
 
 /* workaround for old header files */
@@ -424,6 +427,42 @@
     }
 }
 
+#ifdef WIDE_ARGS
+
+static int alt_winargs (int *pargc, char ***pargv)
+{
+    int argc_w = 0;
+    LPWSTR *argv_w;
+    int err = 0;
+
+    /* get args as UTF-16 */
+    argv_w = CommandLineToArgvW(GetCommandLineW(), &argc_w);
+
+    if (argv_w == NULL) {
+	err = 1;
+    } else {
+	/* convert args to UTF-8, so they can be passed
+	   and read as char * by gnu_main()
+	*/
+	char **argv_u8 = calloc(argc_w, sizeof *argv_u8);
+	int i;
+
+	for (i=0; i<argc_w && !err; i++) {
+	    argv_u8[i] = g_utf16_to_utf8(argv_w[i], -1, NULL, NULL, NULL);
+	    if (argv_u8[i] == NULL) {
+		err = 1;
+	    }
+	}
+	*pargc = argc_w;
+	*pargv = argv_u8;
+	/* we're done with this */
+	LocalFree(argv_w);
+    }
+
+    return err;
+}
+
+#endif
 
 #ifndef WGP_CONSOLE
 int WINAPI
@@ -441,10 +480,14 @@
 #endif
 
 #ifndef WGP_CONSOLE
-# if defined( __MINGW32__) && !defined(_W64)
+# if defined(__MINGW32__) && !defined(_W64)
 #  define argc _argc
 #  define argv _argv
-# else /* MSVC, WATCOM, MINGW-W64 */
+# elif defined(WIDE_ARGS)
+    int argc = 0;
+    char **argv = NULL;
+    alt_winargs(&argc, &argv);
+# else /* MSVC, WATCOM */
 #  define argc __argc
 #  define argv __argv
 # endif