Bug#1081266: apache2: Reverse proxy via mod_rewrite broken after upgrade to 2.4.62-1~deb12u1

Package: apache2
Version: 2.4.62-1~deb12u1
Severity: important
X-Debbugs-Cc: [email protected], [email protected]

Dear Maintainer,

After upgrading apache2 packages, we noticed that our SEO rewriting rules in apache2 no longer worked and Tomcat tried to access non-existing file paths with URL encoded questionmarks.

I have first noticed that is issue affects Debian 12, but I can confirm that it also affects Debian 11, so this happens in oldstable, apache2 2.4.62-1~deb11u1, too.

To show the issue, you'll want to enable the following mods:
a2enmod lbmethod_byrequests proxy proxy_ajp proxy_balancer slotmem_shm rewrite

I have set up a balancer worker in mods-available/proxy_balancer.conf:
<Proxy balancer://tomcat>
        BalancerMember ajp://localhost:8009 secret=youllneverknow
</Proxy>

I have narrowed the issue down to using a proxy RewriteRule inside a Directory block. So to reproduce, set up /etc/apache2/sites-available/000-default.conf like this:

<VirtualHost *:80>
        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        <Directory "/var/www/html">
                DirectoryIndex index.html
                RewriteEngine On
                RewriteRule ^/?(.*?)$ balancer://tomcat/demo/index.jsp?rewrite=$1 [P,L,env=AJP_REDIRECT_REAL_URL:$1,QSA]
        </Directory>
</VirtualHost>

To illustrate the issue, I have set up a simple /demo/ application in Tomcat 10, but the problem is caused by the Apache2 webserver, so this part is not relevant here.

Before the upgrade, i.e. with apache <= 2.4.61-1~deb12u1, a request to http://127.0.0.1/foo/bar/?someparam will result in the following request being proxied to tomcat, as is expected:
        GET /demo/index.jsp?rewrite=foo/bar/&someparam

After the upgrade to 2.4.62-1~deb12u1, the same requests gets mangled:
        GET /demo/index.jsp%3Frewrite=foo/bar/&someparam?rewrite=foo/bar/&someparam

You can see that the complete parameter string is added twice now, with the leading ? being escaped the first time around, which in turn causes the path to be completely messed up, so Tomcat won't be able to find the file and returns a 404 status.

When turning on debug logging in apache2, one can see that the request path is still fine during mod_rewrite processing, it only gets broken during mod_proxy processing. The issue does not occur, when the RewriteRule is placed outside of the Directory block. Unfortunately, this is not a viable workaround for us, we really need to be able to use this inside <Directory> and we need the full flexibility of mod_rewrite too, so we cannot implement the same thing using ProxyPass, either. For now, the only resolution is to downgrade the apache2 packages:

apt -y --allow-downgrades install apache2=2.4.61-1~deb12u1 apache2-data=2.4.61-1~deb12u1 apache2-bin=2.4.61-1~deb12u1 apache2-utils=2.4.61-1~deb12u1

After the downgrade, the RewriteRule with the proxy directive is back to working as expected. As 2.4.62-1~deb12u1 contains security fixes, it feels like having to pin the previous apache2 version is not a good solution, but upgrading it is not possible until this is fixed.

If I had to guess, this may be caused by the following change:
mod_proxy: Fix canonicalisation and FCGI env (PATH_INFO, SCRIPT_NAME) for
     "balancer:" URLs set via SetHandler, also allowing for "unix:" sockets
     with BalancerMember(s).  PR 69168.  [Yann Ylavic]


-- Package-specific info:

-- System Information:
Debian Release: 12.7
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable'), (500, 'oldstable')
Architecture: amd64 (x86_64)

Kernel: Linux 6.5.11-8-pve (SMP w/24 CPU threads; PREEMPT)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE
Locale: LANG=C, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages apache2 depends on:
ii  apache2-bin                2.4.62-1~deb12u1
ii  apache2-data               2.4.62-1~deb12u1
ii  apache2-utils              2.4.62-1~deb12u1
ii  init-system-helpers        1.65.2
ii  media-types                10.0.0
ii  perl                       5.36.0-7+deb12u1
ii  procps                     2:4.0.2-3
ii  sysvinit-utils [lsb-base]  3.06-4

Versions of packages apache2 recommends:
ii  ssl-cert  1.1.2

Versions of packages apache2 suggests:
pn  apache2-doc                                      <none>
pn  apache2-suexec-pristine | apache2-suexec-custom  <none>
pn  www-browser                                      <none>

Versions of packages apache2-bin depends on:
ii  libapr1                  1.7.2-3
ii  libaprutil1              1.6.3-1
ii  libaprutil1-dbd-sqlite3  1.6.3-1
ii  libaprutil1-ldap         1.6.3-1
ii  libbrotli1               1.0.9-2+b6
ii  libc6                    2.36-9+deb12u8
ii  libcrypt1                1:4.4.33-2
ii  libcurl4                 7.88.1-10+deb12u7
ii  libjansson4              2.14-2
ii  libldap-2.5-0            2.5.13+dfsg-5
ii  liblua5.3-0              5.3.6-2
ii  libnghttp2-14            1.52.0-1+deb12u1
ii  libpcre2-8-0             10.42-1
ii  libssl3                  3.0.14-1~deb12u1
ii  libxml2                  2.9.14+dfsg-1.3~deb12u1
ii  perl                     5.36.0-7+deb12u1
ii  zlib1g                   1:1.2.13.dfsg-1

Versions of packages apache2-bin suggests:
pn  apache2-doc                                      <none>
pn  apache2-suexec-pristine | apache2-suexec-custom  <none>
pn  www-browser                                      <none>

Versions of packages apache2 is related to:
ii  apache2      2.4.62-1~deb12u1
ii  apache2-bin  2.4.62-1~deb12u1

-- Configuration Files:
/etc/apache2/apache2.conf changed [not included]
/etc/apache2/conf-available/charset.conf changed [not included]
/etc/apache2/conf-available/localized-error-pages.conf changed [not included]
/etc/apache2/conf-available/other-vhosts-access-log.conf changed [not included]
/etc/apache2/conf-available/security.conf changed [not included]
/etc/apache2/conf-available/serve-cgi-bin.conf changed [not included]
/etc/apache2/mods-available/actions.conf changed [not included]
/etc/apache2/mods-available/alias.conf changed [not included]
/etc/apache2/mods-available/autoindex.conf changed [not included]
/etc/apache2/mods-available/cache_disk.conf changed [not included]
/etc/apache2/mods-available/cgid.conf changed [not included]
/etc/apache2/mods-available/dav_fs.conf changed [not included]
/etc/apache2/mods-available/deflate.conf changed [not included]
/etc/apache2/mods-available/dir.conf changed [not included]
/etc/apache2/mods-available/http2.conf changed [not included]
/etc/apache2/mods-available/info.conf changed [not included]
/etc/apache2/mods-available/ldap.conf changed [not included]
/etc/apache2/mods-available/mime.conf changed [not included]
/etc/apache2/mods-available/mime_magic.conf changed [not included]
/etc/apache2/mods-available/mpm_event.conf changed [not included]
/etc/apache2/mods-available/mpm_prefork.conf changed [not included]
/etc/apache2/mods-available/mpm_worker.conf changed [not included]
/etc/apache2/mods-available/negotiation.conf changed [not included]
/etc/apache2/mods-available/proxy.conf changed [not included]
/etc/apache2/mods-available/proxy_balancer.conf changed [not included]
/etc/apache2/mods-available/proxy_ftp.conf changed [not included]
/etc/apache2/mods-available/proxy_html.conf changed [not included]
/etc/apache2/mods-available/reqtimeout.conf changed [not included]
/etc/apache2/mods-available/setenvif.conf changed [not included]
/etc/apache2/mods-available/ssl.conf changed [not included]
/etc/apache2/mods-available/status.conf changed [not included]
/etc/apache2/mods-available/userdir.conf changed [not included]
/etc/apache2/ports.conf changed [not included]
/etc/apache2/sites-available/000-default.conf changed [not included]
/etc/apache2/sites-available/default-ssl.conf changed [not included]

-- no debconf information