ExtUtils::MakeMaker question, before I post to a larger audience

I'm probably just doing something really dumb, but I cannot figure out 
what...

I am using the cpan tool to install modules on a FreeBSD system that I 
do not have root permission on.  I have a provided a PREFIX for the 
makepl_arg, to install everything in my local path.  Almost everything 
works fine.

The modules (e.g. foo.pm) and the man1 pages are correctly installed 
relative to the PREFIX path.  However, man3 pages seem to ignore PREFIX. 
  In fact, even if I try using INSTALLMAN3DIR, and whether or not I 
specific INSTALLDIRS=perl, nothing seems to cause man3 man pages to be 
re-routed.  Instead, they try to be installed in the root distribution 
and the install fails:

> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> ERROR: Can't create '/usr/local/lib/perl5/5.8.8/man/man3'
> Do not have write permissions on '/usr/local/lib/perl5/5.8.8/man/man3'
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>  at /homes/pkasten/perl/lib/perl5/site_perl/5.8.8/Module/Build/Base.pm
> line 3569

In other words, with the PREFIX variable set (to ~/perl), man1 man pages 
are correctly placed in ~/perl/man/man1, but man3 man pages seem to want 
to go to /usr/local/lib/perl5/5.8.8/man/man3.  And, this does not change 
even if I also specify INSTALLMAN3DIR=~/perl/man/man3, with or without 
INSTALLDIRS=perl.

Here's the line from my ~/.cpan/CPAN/MyConfig.pm:
   'makepl_arg' => q[PREFIX=~/perl],

perl is 5.8.8 and ExtUtils::MakeMaker 6.62 is installed on my system. 
And the directory ~/perl/man/man3/ exists.

Am I doing something wrong?  Should I send this question somewhere else?

Thanks!
Phil