Sigma0 and L4 Core

I know that sigma0 is the memory manager and pager.
I would like to know if the server sigma0 could be merged into the core 
kernel?
Or would doing this cause L4 to lose a lot of it's speed and stability?
The reason I ask this is because I read that L4's IPC was designed to 
stay in the CPU cache, and I think making the kernel bigger would make 
this no longer possible.