Process Notification (pnotify) performance comparisons

In my tests, I have found no mesaurable difference in performance between
stock 2.6.14-rc2, pnotify with the subscriber list protected by the 
rwsem lock like it used to be, and pnotify with read protections 
implemented with RCU.

Problem:  Job allocates memory and uses rw semaphores quite frequently.
This seems to be illegal from within rcu_read_lock() / rcu_read_unlock().
Running with CONFIG_DEBUG_SPINLOCK_SLEEP confirms this.  I'm not sure some of 
the things we want to do in Job are possible (or at least not easily possible) 
if this kernel module subscriber can never sleep.  Even the keyring 
infrastructure needs to allocate memory sometimes.

My opinion is: Let's not use the RCU version of pnotify and stick with the
rwsem version.
 - Since you aren't supposed to sleep, many (most) users can't make use of
   pnotify.
 - The performance data shows no difference between the rwsem version
   with two subscribers (keyring and job) vs a stock kernel with the 
   non-modified keyring support enabled.
 - Using RCU for protection adds comlexity

Here are the details on the test runs.  The RCU version "illegally" can 
sleep but I didn't hit any problems in my tests.

IMPORTANT: The version of AIM7 used here is not currently tracking
the community version - ours is closer to the original AIM7.

General Test info:
- For the two tests with pnotify (old rwsem protected subscriber list and
  new wsem/rcu protected subscriber list), the tests were always fired off
  with the shell process being in a job container.  This means the tests 
  don't only compare RCU performance, but also some aspects of Linux job 
  performance.  All child processes will have Job and keyring as subscribers.
  Of course, the stock kernel doesn't have pnotify or job and uses the
  non-modified keyring.

- forkexit is simple and just forks and exits the number of times supplied on
  the command line.  It reports the number of times fork returned -1.

- The 2.6.14-rc2 kernel was used for each test.  The only variations in 
  patches and configuration related to which version of keyring, pnotify, 
  and Job were used (if any).

- jobtest is a mini job test suite.  One of the tests forks/exits enough 
  processes for PIDs to wrap.  A job_create is done for each forked
  process, which means job is always a subscribed kernenl module to the
  forked processes.  I snipped out the exessive output in the results.

- keyring support is enabled in all kernels.  For both pnotify kernel versions,
  keyrings have been changed to support pnotify for task struct entries and
  fork/exit calls.

- All kernels include the kdb patch.  Otherwise, all kerenels used the
  sn2_defconfig as a base.

- The test system was a 2-processor 1400 Mhz SGI Altix 350 (ia64)


pnotify & linux kernel job with RCU, pnotify-aware keyring
------------------------------------------------------------------------------
forkexit test runs (supplied number is the number of forks fired).

minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92468 times

real    0m0.916s
user    0m0.036s
sys     0m0.408s
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92469 times

real    0m0.914s
user    0m0.024s
sys     0m0.412s
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92468 times

real    0m0.909s
user    0m0.060s
sys     0m0.332s
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92467 times

real    0m0.911s
user    0m0.052s
sys     0m0.380s


minime1:~ # time jobtest
[snip]
Great.  All tests completed, no errors

real    0m11.339s
user    0m0.524s
sys     0m4.536s
minime1:~ # time jobtest
[snip]
Great.  All tests completed, no errors

real    0m11.385s
user    0m0.500s
sys     0m4.592s


Linux version 2.6.14-rc2 (erikj@attica) (gcc version 3.3.3 (SuSE Linux)) #1 SMP PREEMPT Wed Sep 28 12:11:05 CDT 2005
HOST = minime1
CPUS = 2
DIRS = 1
DISKS= 0
FS   = xfs
SCSI = non-xscsi
ID   = RCU pnotify+job
Run 1 of 1
Benchmark	Version	Machine	Run Date
AIM Multiuser Benchmark - Suite VII	"1.1"	AIM7 Run	Sep 28 16:00:45 2005

Tasks	Jobs/Min	JTI	Real	CPU	Jobs/sec/task
1	1962.2		100	3.0	0.5	32.7040
2	4041.7		99	2.9	0.8	33.6806
3	5734.0		98	3.0	1.2	31.8555
4	7232.1		99	3.2	1.5	30.1336
5	8591.7		96	3.4	1.8	28.6389
10	13655.6		93	4.3	3.4	22.7593
20	19812.8		92	5.9	6.7	16.5106
50	26626.4		90	10.9	16.5	8.8755
100	30021.7		87	19.4	32.8	5.0036
150	31080.9		86	28.1	49.1	3.4534
200	31727.0		84	36.7	65.2	2.6439
500	32047.0		84	90.8	163.1	1.0682
1000	31954.7		83	182.1	325.8	0.5326
2000	32633.4		83	356.7	653.9	0.2719



pnotify & linux kernel job WITHOUT RCU - all rwsem, pnotify-aware keyring
------------------------------------------------------------------------------
forkexit test runs (supplied number is the number of forks fired).

minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92468 times

real    0m0.913s
user    0m0.028s
sys     0m0.428s
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92470 times

real    0m1.032s
user    0m0.044s
sys     0m0.416s
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92469 times

real    0m1.048s
user    0m0.048s
sys     0m0.368s
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92469 times

real    0m0.948s
user    0m0.028s
sys     0m0.444s


minime1:/tmp # time jobtest
[snip]
Great.  All tests completed, no errors

real    0m11.620s
user    0m0.612s
sys     0m4.540s
minime1:/tmp # time jobtest
[snip]
Great.  All tests completed, no errors

real    0m11.268s
user    0m0.548s
sys     0m4.488s


Linux version 2.6.14-rc2 (erikj@attica) (gcc version 3.3.3 (SuSE Linux)) #2 SMP PREEMPT Wed Sep 28 15:01:33 CDT 2005
HOST = minime1
CPUS = 2
DIRS = 1
DISKS= 0
FS   = xfs
SCSI = non-xscsi
ID   = non-RCU pnotify+job
Run 1 of 1
Benchmark	Version	Machine	Run Date
AIM Multiuser Benchmark - Suite VII	"1.1"	AIM7 Run	Sep 28 17:17:56 2005

Tasks	Jobs/Min	JTI	Real	CPU	Jobs/sec/task
1	1964.9		100	3.0	0.5	32.7481
2	4020.7		99	2.9	0.8	33.5060
3	5630.4		98	3.1	1.2	31.2802
4	7218.6		97	3.2	1.5	30.0775
5	8637.6		97	3.4	1.8	28.7919
10	13857.1		94	4.2	3.5	23.0952
20	19429.1		93	6.0	6.7	16.1910
50	26126.8		88	11.1	16.5	8.7089
100	29950.6		86	19.4	32.7	4.9918
150	31052.1		86	28.1	49.1	3.4502
200	31817.2		85	36.6	65.4	2.6514
500	32101.1		84	90.7	163.1	1.0700
1000	31908.8		83	182.4	325.8	0.5318
2000	32718.3		82	355.8	653.7	0.2727



linux kernel without pnotify and without job, non-modified keyring enabled
------------------------------------------------------------------------------
forkexit test runs (supplied number is the number of forks fired).

minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92467 times

real    0m0.910s
user    0m0.024s
sys     0m0.356s
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92467 times

real    0m0.912s
user    0m0.032s
sys     0m0.472s
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92467 times

real    0m0.915s
user    0m0.044s
sys     0m0.400s
minime1:/tmp # time ./forkexit 100000
Fork returned an error: 92469 times

real    0m0.923s
user    0m0.036s
sys     0m0.388s

(no jobtest tests since kernel doesn't have pnotify or job)


---------------------------------------------------------------------------
Linux version 2.6.14-rc2 (erikj@attica) (gcc version 3.3.3 (SuSE Linux)) #2 SMP PREEMPT Wed Sep 28 14:53:02 CDT 2005
HOST = minime1
CPUS = 2
DIRS = 1
DISKS= 0
FS   = xfs
SCSI = non-xscsi
ID   = NO pnotify+job
Run 1 of 1
Benchmark	Version	Machine	Run Date
AIM Multiuser Benchmark - Suite VII	"1.1"	AIM7 Run	Sep 28 18:36:09 2005

Tasks	Jobs/Min	JTI	Real	CPU	Jobs/sec/task
1	1964.2		100	3.0	0.5	32.7371
2	4024.9		99	2.9	0.8	33.5408
3	5539.3		99	3.2	1.2	30.7741
4	7123.6		96	3.3	1.5	29.6818
5	8617.1		95	3.4	1.8	28.7237
10	13713.5		96	4.2	3.4	22.8558
20	19732.2		92	5.9	6.7	16.4435
50	26411.3		88	11.0	16.4	8.8038
100	30001.5		86	19.4	32.7	5.0003
150	31184.1		86	28.0	49.0	3.4649
200	31714.9		84	36.7	65.4	2.6429
500	32160.7		84	90.5	163.0	1.0720
1000	31876.8		83	182.6	325.7	0.5313
2000	32708.1		83	355.9	653.0	0.2726