Faulty bounces are "confirming" spam

A spam or so a day has been getting past qconfirm recently. It happens when
someone sends spam with an address that has a valid domain but an invalid user
name, and the mail exchanger for that domain bounces the confirmation message
to the address in the From header rather than the envelope address. This
"confirms" the message, and the spam ends up in my inbox.

Would it make sense to ignore confirmations that have an empty envelope sender?
Would it break anything?

Chris