User authentication in IPsec

Hi all, 

I am trying to figure out whether IPsec support user authentication or not.
I think I understand that IKE supports certificate based authentication and
XAUTH extends the authentication to some legacy methods such as RADIUS.
However, it seems to me that these authentications all happen during the SA
establishment. After the SA is established and IPsec tunnel is up, everyone
who has access to the machine is able to use that tunnel. Is that right? I
came to this conclusion because I think the IPsec module has no way to
figure out who sends the packet when it tries to process an IP packet.
However, in section 4.4.2 of RFC2401 it says "user id" can be used as
selector to select SA. I am confused here. Can anyone clarify this? 

Thanks and best regards, 
Xiangfei