thinking about processing of CC: recipients from whitelisted sender

Here is a scenario that I tend to see quite a bit:

1) I routinely converse with Charles.  Charles sends me an email, with a
CC: to Tom.  Charles is on my whitelist so his message is accepted.  Tom
I barely know or did not previously have an email address for.

2) Tom does a "reply all" joining the conversation.  When I receive his
message it is challenged, even though both Tom and I know Charles who
initiated the conversation.  I might also know Tom but don't have him in
the whitelist.

Has anyone thought about "social whitelisting"?  For example, in the
above scenario, if I had auto-whitelisted Tom when I got Charles' email
that contained Tom in the CC:, the conversation would continue normally
without interruption by TMDA.

Rationale:
1) Charles does not correspond with spammers.
2) This trades convenience of my correspondents vs. the possibility I
may have to blacklist an address now and then.
3) If Tom knows Charles well enough to be on a CC: email along with me,
then he probably deserves to be whitelisted.
4) In order for a spammer to exploit this method, he would have to know
one of my whitelist senders and forge a message from one of them to add
a sender to my whitelist (I use SPF so this is increasingly hard to
accomplish)
5) Mass virus emails from users' contact list will be dealt with before
reaching TMDA.

I don't think this can currently be done with TMDA, but I believe it
would be a pretty easy thing to add.

It seems simple enough... Am I missing something?  Are there drawbacks
that I'm missing?