[pfx] Re: cleanup(8): recipient_canonical can emit an over-long socketmap query with no operator guard; PERM then retried as 451

Dmytro Alieksieiev via Postfix-users:
> Hi Wietse, Viktor,
> 
> I ran into a funky interaction between|recipient_canonical|header 
> rewriting and a socketmap map (PostSRSd, doing reverse-SRS), where a 
> single malformed/header/address can deterministically cancel an 
> otherwise-deliverable message. I think I've narrowed the root cause down 
> to something that postfix itself has no way to guard against, and I'd 
> love your read on it and maybe propose a way to solve this issue.
> 
> 
>     Setup
> 
> An incoming relay accepts mail for forwarding and hands it to an 
> outgoing relay. The outgoing relay does reverse-SRS via a socketmap and 
> also canonicalizes header recipients (so the|To:|in a DSN matches the 
> real MAIL TO, per RFC 3464):
> 
> |recipient_canonical_maps = socketmap:inet:127.0.0.1:10003:reverse 
> recipient_canonical_classes = envelope_recipient, header_recipient 
> remote_header_rewrite_domain = domain.invalid # remote clients -> 
> rewrite context = remote |
> 
> The socketmap server is PostSRSd 2.x, but ? importantly ? it is*not*the 
> culprit; any socketmap with a bounded request buffer behaves the same 
> (see below).

Would a header_checks pattern work around with this, like replacing
long text with a shorter text?

I agree that a PERM error should not result in a 451 status.

	Wietse
 
>     Trigger
> 
> A forwarded spam message (envelope recipient perfectly valid and 
> deliverable) carries a*cosmetically mangled|To:|or|Cc:|header*. The 
> interesting part is not the two-bare-|@|addr-spec ? that turns out to be 
> harmless ? but an*unbalanced double-quote combined with a long folded 
> continuation*:
> 
> |Cc: [[email protected]]@host.example.org" 
> <[email protected]@evil.example.net> 
> AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA 
> AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA 
> ... (~2 KB of folded junk, no closing quote) ... 
> AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA] |
> 
> The|"|after|host.example.org|is never closed. So|tok822|treats 
> everything that follows ? including the entire folded blob ? as*one 
> quoted-string localpart*, and in the remote rewrite context 
> appends|@domain.invalid|, producing a single ~2 KB address:
> 
> |"<~2000 chars of junk>"@domain.invalid |
> 
> 
>     What happens
> 
> |cleanup(8)|issues that ~2 KB string as the canonical-map (socketmap) 
> query. It exceeds the socketmap server's request buffer 
> (PostSRSd's|PAYLOAD_SIZE|is 512 bytes), so the server returns a 
> permanent error:
> 
> |warning: socketmap:inet:127.0.0.1:10003:reverse socketmap server 
> permanent error: Too big. warning: reverse lookup error for "" 
> <[email protected]@evil.example.net> ...[AAAA...]"@domain.invalid" 
> warning: ABCDEF: recipient_canonical_maps map lookup problem for ... -- 
> message not accepted, try again later ABCDEF: removed (canceled) |
> 
> and the upstream sender gets:
> 
> |451 4.3.0 Error: queue file write error (in reply to end of DATA command) |
> 
> Because it's a 451, the upstream relay retries the message for its whole 
> queue lifetime ? so one attacker-controlled cosmetic header turns into a 
> message that can never drain and slowly fills a relay with deferred 
> retries. (A different socketmap/version returns|PERM Invalid 
> query.|instead of|Too big.|for the same input; same class, same outcome.)
> 
> 
>     Root cause, and why I can't defend against it inside postfix
> 
> The address/content/is fine ? the same recipient handed to the map as a 
> normal-length key returns "not found" and delivers cleanly. What breaks 
> is purely that*cleanup builds a canonical-map query that is too large to 
> be represented*, and then treats the resulting|PERM|as fatal. Two 
> separate things line up:
> 
>  1. An*unbalanced quote*lets a folded header expand into one enormous
>     address token. Postfix is arguably being lenient/correct here, but
>     the result is a multi-KB "address".
>  2. *A failed lookup during/optional header/canonicalization is fatal to
>     the whole message.*The junk address erroring is fine ?
>     it/should/fail. What's disproportionate is the blast
>     radius:|header_recipient|rewriting is a cosmetic, best-effort nicety
>     (align a DSN|To:|for RFC 3464), yet its failure cancels a message
>     whose envelope recipient is perfectly valid and deliverable. A
>     cosmetic step failing shouldn't take down the delivery.
>  3. *A/permanent/map error (|PERM|) is reported to the client as
>     a/transient/|451|.*This is what turns a one-off into a queue-wedge:
>     the socketmap said/permanent/, but the upstream sees|451 queue file
>     write error|, so it retries the identical, deterministically-failing
>     message for the entire queue lifetime. Had the same condition
>     surfaced as a|5xx|, the message would simply bounce and the queue
>     would drain ? annoying, but not self-perpetuating.
> 
> And here's the part that motivated writing to you:*postfix gives an 
> operator no lever to prevent its own doomed query.*
> 
>   * There's no introspection point between "cleanup extracted this
>     header address" and "cleanup sends it to the map" ? I can't see or
>     cap the key that's about to be sent.
>   * Canonical/socketmap lookups have*no key-length limit*; cleanup will
>     happily emit a key it must know the socketmap can't accept.
>   * |header_checks|match a*whole logical header*, not individual
>     addresses, so I can't write a rule like "reject if any single
>     address in To/Cc exceeds N bytes or X symbols." I can only
>     pattern-match the header text as a blob, which is fragile against
>     folding.
>   * The failure surfaces as a generic|451 ... queue file write error|?
>     indistinguishable from disk-full, a database outage, etc. ? so I
>     can't even intercept/this specific/condition and handle it differently.
> 
> Net: the only place I/can/currently block this is a milter/content 
> filter in front of the relay (which is what I'll do), but that feels 
> like papering over a postfix-side sharp edge ? and it's a leaky patch, 
> because a milter parses headers/very/differently from postfix's|tok822|. 
> The filter's address parser doesn't reproduce the 
> unclosed-quote-swallows-the-fold behavior, so what the milter sees as 
> the recipient set and what|cleanup|will actually hand the socketmap are 
> two different things. I can approximate (flag unbalanced quotes / 
> over-long tokens on the raw header), but I can't reliably pre-empt 
> postfix's own tokenization from outside it.
> 
> 
>     Questions / possible directions
> 
>  1. Should a lookup failure during*optional header*canonicalization
>     (|header_recipient|) be fatal at all? Logging and leaving the header
>     untouched (continue, don't cancel) would avoid the whole class of
>     problem for cosmetic rewrites.
>  2. Could the canonical/socketmap client*cap the key length*it will send
>     ? e.g. if the rewritten address exceeds a representable/
>     configurable maximum, treat it as "no match" (skip the rewrite)
>     instead of emitting a query the server must reject? Postfix knows
>     the key before it sends it.
>  3. Failing that, is there appetite for a*per-address length
>     guard*operators could set (reject or skip addresses whose serialized
>     form exceeds N), given|header_checks|can't express per-address
>     limits today?
>  4. Would distinguishing a socketmap|PERM|on/header/canonicalization
>     from a genuine transport/tempfail ? with its own status rather than
>     the generic|queue file write error|? be reasonable, so operators can
>     act on it?
>  5. Most impactful, and independent of the above: should a
>     socketmap*|PERM|map to a permanent|5xx|*rather than a
>     transient|451|? The server explicitly said/permanent/, yet the
>     client is told to retry forever. Surfacing it as|5xx|(bounce)
>     instead of|451|(retry) would, on its own, prevent the deterministic
>     queue-wedge ? even if everything else stayed as-is.
> 
> 
>     Minimal reproduction
> 
> Stock postfix + any socketmap that bounds its request (e.g. PostSRSd, 
> 512-byte payload):
> 
> |recipient_canonical_maps = socketmap:inet:127.0.0.1:10003:reverse 
> recipient_canonical_classes = envelope_recipient, header_recipient 
> remote_header_rewrite_domain = domain.invalid 
> local_header_rewrite_clients = <something that excludes the injecting 
> client, so context = remote> |
> 
> Inject (clean envelope, junk only in the header) a message 
> whose|To:|or|Cc:|opens a double-quote that is never closed and is 
> followed by ~2 KB of folded content. cleanup issues an oversized 
> socketmap query, gets|PERM|, and cancels the message with|451 4.3.0 
> Error: queue file write error|. Balancing/closing the quote (so the 
> token stays a normal-length address) makes it deliver normally ? 
> confirming size, not content, is the fault.
> 
> Thanks for postfix, and for reading this far.
> 
> -- 
> Regards,
> Dmytro Alieksieiev
> DevOps Engineer

> _______________________________________________
> Postfix-users mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
_______________________________________________
Postfix-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]