[pfx] Re: cleanup(8): recipient_canonical can emit an over-long socketmap query with no operator guard; PERM then retried as 451

This is a multi-part message in MIME format.
--===============5024529205493796353==
Content-Type: multipart/alternative;
 boundary="------------6II2HDSY3NrtjmOBKDZss05g"
Content-Language: en-US

This is a multi-part message in MIME format.
--------------6II2HDSY3NrtjmOBKDZss05g
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi Wietse,

Thanks for the pipemap:{pcre, socketmap} suggestion. I tested both 
variants with postmap -q against real Postfix (3.9.11 and 3.11.5), which 
also covers the UTF-8 concern.

Two facts confirmed:

First, pcre: patterns count bytes, not characters. A 299-character / 
300-byte key (298 × 'a' followed by a two-byte é) trips if !/(.{300})/, 
confirming .{300} matches on byte length.

Second, Postfix validates the rewrite result as UTF-8 before returning 
it. This is what separates the two variants:

The negative pattern
if !/(.{300})/
/(.+)/ $1
ENDIF

never rewrites the key. It returns the whole, unmodified string when it 
is under the limit, and no result for anything at or above 300 bytes, at 
which point pipemap short-circuits and the socketmap is never queried. 
Because the key is never truncated, there is no encoding hazard 
regardless of where a multibyte character sits. Side note: I changed 
(.*) to (.+) as empty string result is not allowed in maps.

The positive pattern /^(.{1,300})/ $1 truncates at a byte boundary. When 
that boundary falls inside a multibyte character, Postfix rejects the 
malformed result with:
warning: ... non-UTF-8 value "...": malformed UTF-8 or invalid codepoint
fatal: table pcre:...: query error
A query error is a temporary failure, so the lookup defers rather than 
returning a clean result.

To quantify it, I placed a four-byte character (😅, F0 9F 98 85) across 
the 300-byte boundary at each alignment: three of the four offsets 
produce a query error, and only the alignment where the character ends 
exactly at byte 300 survives.

The failure is therefore data-dependent and intermittent.

So the negative pattern is the right fit for shielding an uncooperative 
socketmap from over-long input, without touching Postfix internals and 
without any UTF-8 truncation risk. Thanks again for the pointer.

As a longer-term direction, it would be valuable to be able to return an 
action — a 5.x.x REJECT — based on a header address before cleanup 
commits the message, rather than only accept-or-defer. That is not 
specific to recipient_canonical_maps; it is the general gap that today a 
malformed header address gives an operator no clean way to refuse the 
message at that stage.

One point worth flagging for /anyone/ running a split scheme like mine: 
envelope_recipient and header_recipient share the same 
recipient_canonical_maps, so this length guard applies to both 
indiscriminately. That makes it important to also enforce a 
recipient-length limit at the incoming relay (the RCPT check), and to 
keep the two limits in sync with a deliberate window gap — the incoming 
limit set safely below the guard's threshold. Without that gap, an 
over-long but otherwise legitimate envelope recipient would trip the 
guard and SRS would simply be skipped for it, instead of the guard only 
ever catching the oversized header-derived key it is meant for.

Regards,
Dmytro Alieksieiev
DevOps Engineer

On 15/07/2026 23:36, Wietse Venema via Postfix-users wrote:
> Dmytro Alieksieiev via Postfix-users:
>> Hi Wietse,
>>
>> Thanks - and glad we agree on the PERM/451 point; that alone would take
>> most of the pain out of this.
> That is easier said than done.
>
> Postfix table lookups currenly have three possible results:
>
> - DICT_ERR_NONE: the requested item was found, or it does not exist.
>
> - DICT_ERR_RETRY: typically, timeout
>
> - DICT_ERR_CONFIG: configuration error
>
> The difference between DICT_ERR_RETRY and DICT_ERR_CONFIG is in the
> error message; both result in the same program behavior, because
> configuration errors can be corrected.
>
> The Postfix mapping is:
>
> OK, NOTFOUND -> DICT_ERR_NONE
>
> TEMP -> DICT_ERR_RETRY
>
> PERM -> DICT_ERR_CONFIG
>
> Changing the maning of DICT_ERR_CONFIG into a "reject" would be a
> compatibility break, and adding a new "reject" status would require
> lots of changes everywhere Postfix looks up information.
>
> Until now there has een no use case for dictionaries to
> return an error that says "reject".
>
>> On header_checks: I don't think it can cover this one. header_checks
>> match a header as a single logical blob, not per-address, so I can't
>> express "reject/shorten the one over-long address token" ? and
>> rewriting/truncating text inside a folded, quote-swallowed header is
>> fragile (the very thing that makes it dangerous, the unterminated quote,
>> is what makes it hard to match a stable pattern against).
> Inded, this is an available option, now.
>
> Here's another option:
>
>      xxx_maps = pipemap:{prce:/path/to/file,socketmap:{your-stuff}}
>
> In the first example, /path/to/file contains a negative pattern to
> select with < 300 bytes. Longer strings will produce a "not found"
> result without using the socketmap:
>
> if !/(.{300})/
> /(.*)/ $1
> ENDIF
>
> In the second example, /path/to/file uses a positive pattern to
> to expose the socketmap only to the first 300 bytes of a string.
>
> /^(.{1,300}) $1
>
> No idea how this handles UTF-8 non-ASCII.
>
> This may be a reasonable option for making Postfix work with
> an uncooperative socketmap.
>
> 	Wietse
--------------6II2HDSY3NrtjmOBKDZss05g
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <p>Hi Wietse,<br>
      <br>
      Thanks for the pipemap:{pcre, socketmap} suggestion. I tested both
      variants with postmap -q against real Postfix (3.9.11 and 3.11.5),
      which also covers the UTF-8 concern.<br>
      <br>
      Two facts confirmed:<br>
      <br>
      First, pcre: patterns count bytes, not characters. A 299-character
      / 300-byte key (298 × 'a' followed by a two-byte é) trips if
      !/(.{300})/, confirming .{300} matches on byte length.<br>
      <br>
      Second, Postfix validates the rewrite result as UTF-8 before
      returning it. This is what separates the two variants:<br>
      <br>
      The negative pattern<br>
      if !/(.{300})/<br>
      /(.+)/ $1<br>
      ENDIF</p>
    <p>never rewrites the key. It returns the whole, unmodified string
      when it is under the limit, and no result for anything at or above
      300 bytes, at which point pipemap short-circuits and the socketmap
      is never queried. Because the key is never truncated, there is no
      encoding hazard regardless of where a multibyte character sits.
      Side note: I changed (.*) to (.+) as empty string result is not
      allowed in maps.<br>
      <br>
      The positive pattern /^(.{1,300})/ $1 truncates at a byte
      boundary. When that boundary falls inside a multibyte character,
      Postfix rejects the malformed result with:<br>
      warning: ... non-UTF-8 value "...": malformed UTF-8 or invalid
      codepoint<br>
      fatal: table pcre:...: query error<br>
      A query error is a temporary failure, so the lookup defers rather
      than returning a clean result.</p>
    <p>To quantify it, I placed a four-byte character (😅, F0 9F 98 85)
      across the 300-byte boundary at each alignment: three of the four
      offsets produce a query error, and only the alignment where the
      character ends exactly at byte 300 survives.</p>
    <p>The failure is therefore data-dependent and intermittent.<br>
      <br>
      So the negative pattern is the right fit for shielding an
      uncooperative socketmap from over-long input, without touching
      Postfix internals and without any UTF-8 truncation risk. Thanks
      again for the pointer.</p>
    <p>As a longer-term direction, it would be valuable to be able to
      return an action — a 5.x.x REJECT — based on a header address
      before cleanup commits the message, rather than only
      accept-or-defer. That is not specific to recipient_canonical_maps;
      it is the general gap that today a malformed header address gives
      an operator no clean way to refuse the message at that stage.<br>
      <br>
      One point worth flagging for <i>anyone</i> running a split scheme
      like mine: envelope_recipient and header_recipient share the same
      recipient_canonical_maps, so this length guard applies to both
      indiscriminately. That makes it important to also enforce a
      recipient-length limit at the incoming relay (the RCPT check), and
      to keep the two limits in sync with a deliberate window gap — the
      incoming limit set safely below the guard's threshold. Without
      that gap, an over-long but otherwise legitimate envelope recipient
      would trip the guard and SRS would simply be skipped for it,
      instead of the guard only ever catching the oversized
      header-derived key it is meant for.<br>
      <br>
    </p>
    <pre class="moz-signature" cols="72">Regards,
Dmytro Alieksieiev
DevOps Engineer</pre>
    <div class="moz-cite-prefix">On 15/07/2026 23:36, Wietse Venema via
      Postfix-users wrote:<br>
    </div>
    <blockquote type="cite"
      cite="mid:[email protected]">
      <pre wrap="" class="moz-quote-pre">Dmytro Alieksieiev via Postfix-users:
</pre>
      <blockquote type="cite">
        <pre wrap="" class="moz-quote-pre">Hi Wietse,

Thanks - and glad we agree on the PERM/451 point; that alone would take 
most of the pain out of this.
</pre>
      </blockquote>
      <pre wrap="" class="moz-quote-pre">
That is easier said than done.

Postfix table lookups currenly have three possible results:

- DICT_ERR_NONE: the requested item was found, or it does not exist.

- DICT_ERR_RETRY: typically, timeout

- DICT_ERR_CONFIG: configuration error

The difference between DICT_ERR_RETRY and DICT_ERR_CONFIG is in the
error message; both result in the same program behavior, because
configuration errors can be corrected.

The Postfix mapping is:

OK, NOTFOUND -&gt; DICT_ERR_NONE

TEMP -&gt; DICT_ERR_RETRY

PERM -&gt; DICT_ERR_CONFIG

Changing the maning of DICT_ERR_CONFIG into a "reject" would be a
compatibility break, and adding a new "reject" status would require
lots of changes everywhere Postfix looks up information.

Until now there has een no use case for dictionaries to
return an error that says "reject".

</pre>
      <blockquote type="cite">
        <pre wrap="" class="moz-quote-pre">On header_checks: I don't think it can cover this one. header_checks 
match a header as a single logical blob, not per-address, so I can't 
express "reject/shorten the one over-long address token" ? and 
rewriting/truncating text inside a folded, quote-swallowed header is 
fragile (the very thing that makes it dangerous, the unterminated quote, 
is what makes it hard to match a stable pattern against).
</pre>
      </blockquote>
      <pre wrap="" class="moz-quote-pre">
Inded, this is an available option, now.

Here's another option:

    xxx_maps = pipemap:{prce:/path/to/file,socketmap:{your-stuff}}

In the first example, /path/to/file contains a negative pattern to
select with &lt; 300 bytes. Longer strings will produce a "not found"
result without using the socketmap:

if !/(.{300})/
/(.*)/ $1
ENDIF

In the second example, /path/to/file uses a positive pattern to
to expose the socketmap only to the first 300 bytes of a string.

/^(.{1,300}) $1

No idea how this handles UTF-8 non-ASCII.

This may be a reasonable option for making Postfix work with
an uncooperative socketmap.

	Wietse</pre>
    </blockquote>
  </body>
</html>

--------------6II2HDSY3NrtjmOBKDZss05g--

--===============5024529205493796353==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Postfix-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]

--===============5024529205493796353==--