net6501 - large clock drift under high load

I've been troubleshooting a problem with large clock drift on my net6501
for a while. It is running Debian 7, with kernel 3.2.58.

Of course, I've got a NTP server running on it.  And I generally have no
problems with time sync during normal system usage (low load)

But what I've discovered is that during periods of high system load (like
when I'm running rsnapshot on the box to do home backups, the clock starts
drifting quickly.  The CPU load on the net6501 around that time is very
high, around 2.5.  The clock drifts enough that ntp thinks that my time
sources are failing sanity checks.

So then, NTP quits synchronizing the time.  After a day or two of this,
with other high load times, the system time gets so far from the time of
the outside time servers, that NTP just quits running.

Has anybody else noticed this?  Is there anything I can do to address the
clock drift during high load times?  I never encountered this problem on
my net4801.

Thanks,
Bryan