(racoon 745) Re: Endian-ness in sha1.c?

All,

Well, I was able to find and fix the problem.

There was a bug in sha1.c. Here is what works for me. If it makes
sense to you all, then please use it.

The bug was that if you have RACOON/IPSEC (BSD) and if the same code
is running on 2 platforms doing IPSec, one of the platforms being little-
endian (Pentium in my case) and the other being big-endian (PowerPC
in my case), then SHA1 will not work and you get the following error:
"auth fail in IPv4 ESP input" which is in the file esp_input.c and this
error is as a result of failure in bcmp().

The fix is to be made to sha1.c, in the routine sha1_step().

The original (problematic) code is:
#if BYTE_ORDER == LITTLE_ENDIAN
 struct sha1_ctxt tctxt;
 bcopy(&ctxt->m.b8[0], &tctxt.m.b8[0], 64);
 ctxt->m.b8[0] = tctxt.m.b8[3]; ctxt->m.b8[1] = tctxt.m.b8[2];
 ctxt->m.b8[2] = tctxt.m.b8[1]; ctxt->m.b8[3] = tctxt.m.b8[0];
........
........ and so on

The fixed lines of code  are:

 struct sha1_ctxt tctxt;    /* take this out of the #idfef */
#if BYTE_ORDER == BIG_ENDIAN   /* instead of LITTLE_ENDIAN */
 bcopy(&ctxt->m.b8[0], &tctxt.m.b8[0], 64);
#else
 bcopy(&ctxt->m.b8[0], &tctxt.m.b8[0], 64);
 ctxt->m.b8[0] = tctxt.m.b8[3]; ctxt->m.b8[1] = tctxt.m.b8[2];
 ctxt->m.b8[2] = tctxt.m.b8[1]; ctxt->m.b8[3] = tctxt.m.b8[0];
 ctxt->m.b8[4] = tctxt.m.b8[7]; ctxt->m.b8[5] = tctxt.m.b8[6];
...........
...........
and so on


-------------- Original message from [email protected]: -------------- 

Hello all and Jun-ichiro itojun Itoh,

Still trying to resolve an inter-op problem for SHA1 when using RACOON
between a little-endian (Pentium) machine and a big-endian (PowerPC)
machine. It works for MD5.

Looking at the SHA1 code (sha1.c), I see this line in sha1_step(),
#if BYTE_ORDER == LITTLE_ENDIAN and in that routine at least,
nothing for BIG_ENDIAN, so I am assuming that the default case always
works for BIG_ENDIAN and fixes need to be done only for LITTLE_E.

Is this assumption right? Does anybody have any suggestions about
the sha1 code and it's ability to deal with both the endian-s?

I would appreciate any help! Thanks.

    .arun