(racoon 727) MD5 works, SHA1 does not

[email protected] Fri, 10 Sep 2004 23:10:55 +0000
Newsgroups gmane.network.ipv6.kame.racoon
Message-ID <091020042310.16370.4142347F0003B45700003FF22161243646020E060E080E03D2029A9D0E@att.net>
All,

We have been trying to interoperate with two platforms, one, a PowerPC
and the other a Pentium with the BSD IPSec and Racoon. Both the
platforms are of our company itself.

Interop between PowerPC-to-PowerPC and Pentium-to-Pentium is fine.

When we hook up a PowerPC-to-Pentium, we get the error:

"auth fail in IPv4 ESP input", ONLY when running SHA1. 
There is no problem when running MD5!

This error string was traced to to esp_input.c and then, 
using more debugging, to this code:

if (bcmp(sum0, sum, siz) != 0) {
  ipseclog((LOG_WARNING, "auth fail in IPv4 ESP input: %s %s\n",
      ipsec4_logpacketstr(ip, spi), ipsec_logsastr(sav)));
  ipsecstat.in_espauthfail++;
  goto bad;
 }

So, the comparison is failing.

This is quite likely an endian issue in the SHA1 code since the MD5 is not
complaining.

I may not be asking this question in the right forum but if somebody knows
where in the SHA1 code I should look, please let me know.

I would appreciate any help!


Thanks,

    .arun