RE: [INTERNALS-WIN] Question about using UTF-8 encoding for server variables used by PHP

Hi Pierre,

I think that there is a difference of philosophy that is at the root of the issue here.  PHP considers decoding of the URL to be a task for the application.  IIS considers decoding of the URL to be a server function.

The main reason IIS owns decoding is that the IIS core server takes some actions on the decoded URL before it even calls into application code.  The results of these actions sometimes have significant security ramifications.  Historically, there have been a number of exploited vulnerabilities that were a result of an application decoding the URL in a way that is not identical to the way that IIS decodes it.  Any change to the way that we do this would be perceived as a security risk (and based on my experience, it would eventually be exploited in some way.)

I am not familiar with the internals of Apache, but I would suspect that it could have similar problems.  In particular, Apache may need to process a .htaccess file using a path derived from the URL.  For example, I have seen applications that filter URLs using the FilesMatch directive.  If an application decodes the URL in a way that does not result in an exact match to the URL that Apache used, it seems there is opportunity there for a problem to occur.

Finally, getting back to Unicode APIs, the point I was trying to make about SCRIPT_FILENAME is that unless PHP uses Unicode APIs to open the script file, it will not be possible to support multiple code pages for content.  The problem is that the non-Unicode APIs do not support UTF-8 and require the path to be expressed in the default system code page.  This is not possible if the URL was encoded in some other code page, even though IIS itself can handle different code pages.

Thanks,
-Wade

-----Original Message-----
From: Pierre Joye [mailto:[email protected]] 
Sent: Friday, January 22, 2010 6:07 PM
To: Wade Hilmo
Cc: Johannes Schlüter; Ruslan Yakushev; Stanislav Malyshev; [email protected]
Subject: Re: [INTERNALS-WIN] Question about using UTF-8 encoding for server variables used by PHP

hi Wade,

2010/1/22 Wade Hilmo <[email protected]>:

> I have some concerns about putting the raw URL into the REQUEST_URI variable.

I think there is a confusion about the problem Ruslan is trying to
solve, manage url like: http://mysite.com/2009/10/русские-символ,
which are encoded as described in the RFC. None of the files actually
use UTF-8, index.php is not suddenly named индекс.php, even for
Wordpress :).

http://mysite.com/2009/10/русские-символ is actually processed by the
rewrite module and will end (example) as:

http://mysite.com/index.php?year=2009&month=10&title=русские-символ

Or whatever else is defined as rewrite rule.

> If applications are using data from that value to open files, there is a potential for security vulnerabilities.  The problem can occur because IIS looks up configuration information using the canonicalized version of the URL for the request.  This process of canonicalization includes decoding the URL, but also some other things.  For example, the kernel mode HTTP parser underneath IIS does the work of deciding the code page of the incoming URL and then hands IIS a Unicode version of the URL for processing.  If an application starts with the raw URL and does its own decoding, it could potentially use a different code page form what the HTTP parser used.  If the application then opens a file, it could be using a path that does not match the IIS configuration for the request.  Since some of the configuration can affect security, problems can be introduced.  The most common security bugs that arise from this problem are bypassing authentication or authorization rules, or serving script source code back to the client because a clever URL managed to trick the application into opening a script file meant for another script handler.
>
> In general, we would be happy to do the right thing for PHP so that support for non-ASCII URLs and filenames works well.  It looks like we could potentially get there by ensuring that SCRIPT_FILENAME is correct.

Exactly, see php-src/sapi/cgi/cgi_main.c, line 1000+, there is a
simple explanation about how php will process the (f)cgi inputs and
the implementation follows this text.

>  Unfortunately, I understand that not all of the file I/O in PHP uses Unicode APIs.

Nothing in PHP uses the Unicode APIs (except the IO_REPARSE
implementation I added in 5.3.1, in TSRM) , and that won't be possible
to change any time soon (5.2.x or 5.3.x).

>  This is not a problem if the URL is sent using a code page that matches the server's system because our default is to populate all of the server variables using the system's default code page.  We've introduced an option in the FastCGI 1.5 handler that we just released to encode any or all of the server variables using UTF-8 encoding.  This is effective at preserving the URL for the application, but the non-Unicode file APIs in Windows do not work with UTF-8.  That means that PHP would have to convert the paths to Unicode and use only Unicode APIs to access the file system.

Why introduce such thing now at this stage? URL are encoded and can be
easily decode in PHP applications, I do that since more than a decade
without issues. It was always the responsability of the application to
deal with that. However as long as this option is disabled by default,
it won't break anything.

> I would be interested in hearing your thoughts on these issues.

Cheers,
-- 
Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org