Re: Poor thread performance on Linux vs. Solaris
Bill Soudan <[email protected]>
| Newsgroups | gmane.comp.lib.phil |
|---|---|
| Organization | SunGard Trading Systems, Inc. |
| Message-ID | <[email protected]> |
On Sunday 07 September 2003 11:47 pm, you wrote: > On Thu, 4 Sep 2003, Bill Soudan wrote: > > have you tried oprofile? It should be able to profile your app too. I have already, it was one of the tools I used to track the system time to the futex spinlock. I've also used Intel's proprietary vtune tool, which points to the same culprit. > How does 'vmstat 1' output look before, during and after such a freeze? > Does it show a sharp increase in context-switching? Is there a sharp > increase in runqueue length? [nr of tasks running] To my untrained eye, context switching appears very high and volatile for the duration of the test... the only time it drops is during the mysterious freezes, during which system time spikes up to 80-100%. I've included the vmstat output for the duration of the test. Notice the particularly bad freeze near the end of the test, 99% system time for 2 seconds. Runqueue length is usually 0 for the duration of the test. These results are with arjan's 2.6.0-test4 kernel + Boris Hu's futex_q_lock patch. For comparison, I've also attached the vmstat output from a run on a 2way 450mhz Ultra-60 running Solaris 5.8. > the system overhead alone is not a problem, Linux uses a kernel > mechanism for which other OSs often use userspace spinlocks. Interesting you mention this, I'd suspected this but I wasn't certain. So concievably we could be doing just as much lock thrashing on Solaris, we just wouldn't notice it since it'd be lumped in with the user time. I'd really like to find a tool that can measure lock contention. > But obviously the 'freezing' of your app and the low throughput is a > bad thing. Currently, the throughput is only about 4x of a Sun Ultra-60 box (dual 450mHz UltraSparc II). I don't believe we're I/O bound at all, so with a rough guess on our 4way 2.668ghz Xeon I'd expect at least an 8-10x performance increase. If I could translate the system time into user time, gain 30% more throughput, then I'll be at 6x. Whether I can get this by tweaking/fixing the kernel or fixing our app is the question. At the very least, the system time spikes look like a kernel problem to me. > one thing looks interesting - your top output shows ~10% idle time, is > that correct? Could you try the latest -mm kernel from: > > kernel.org/pub/linux/kernel/people/akpm/patches > > this also has some scheduler enhancements. (that might as well be a > degradation for your app though.) Yes, the idle time is correct. I'll give the mm kernel a shot later today or first thing tomorrow. Next on my list is to spend some more time with oprofile now that I have Boris' patch applied. Bill STATEMENT OF CONFIDENTIALITY The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain confidential or privileged information. If you are not the intended recipient, please notify SunGard Trading Systems immediately at (201) 499-5900 and destroy all copies of this message and any attachments.
vmstat.2.6.0-0.test4.1.32futex_q_lock
(text/plain, 6.8 KB)
/data/build/bsoudan/taap_UMA27_devel/bin> vmstat 1 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 0 0 0 3571524 7908 57916 0 0 34 9 336 723 15 10 75 0 0 0 0 3571524 7908 57916 0 0 0 0 1006 18 0 0 100 0 0 0 0 3571524 7908 57916 0 0 0 0 1007 16 0 0 100 0 0 0 0 3571540 7908 57916 0 0 0 0 1007 18 0 0 100 0 0 0 0 3571556 7916 57908 0 0 0 36 1031 42 0 1 99 0 0 0 0 3571556 7916 57908 0 0 0 0 1007 14 0 0 100 0 0 0 0 3571556 7916 57908 0 0 0 0 1008 18 0 0 100 0 0 1 0 3571396 7916 58044 0 0 0 0 2662 10165 43 16 41 0 0 1 0 3571380 7916 58044 0 0 0 0 3309 13028 68 21 11 0 0 0 0 3571348 7924 58036 0 0 0 228 3292 13393 64 20 16 0 0 0 0 3571348 7924 58104 0 0 0 0 3265 13002 69 20 11 0 0 0 0 3571348 7924 58172 0 0 0 0 3352 14240 67 22 11 0 0 0 0 3571348 7924 58172 0 0 0 0 3354 13652 69 20 11 0 0 0 0 3571268 7924 58172 0 0 0 0 3390 17100 67 21 11 0 0 2 0 3571268 7932 58164 0 0 0 68 3352 17269 65 21 14 0 0 1 0 3571268 7932 58164 0 0 0 0 3430 27358 66 24 11 0 0 1 0 3571268 7932 58164 0 0 0 40 3416 18307 65 22 13 0 0 0 0 3571268 7932 58164 0 0 0 0 3425 16091 67 21 12 0 0 0 0 3571268 7932 58164 0 0 0 0 3416 16119 64 24 12 1 0 0 0 3571204 7940 58156 0 0 0 68 3401 15631 64 26 10 0 0 1 0 3571204 7940 58224 0 0 0 0 3442 15894 66 25 9 0 0 0 0 3571204 7940 58224 0 0 0 0 3420 16503 67 20 13 0 0 1 0 3571204 7940 58224 0 0 0 0 3394 23001 63 25 12 0 0 1 0 3571204 7940 58224 0 0 0 0 3413 25372 62 24 13 1 0 0 0 3571204 7948 58216 0 0 0 68 3421 21533 65 24 11 0 0 1 0 3571204 7948 58284 0 0 0 0 3429 16453 65 22 13 0 0 0 0 3570628 7948 58284 0 0 0 0 3422 16711 67 22 11 0 0 1 0 3566100 7948 58284 0 0 0 0 3539 17518 63 25 12 0 0 1 0 3561492 7948 58284 0 0 0 0 3514 16667 64 25 11 1 0 0 0 3555684 7956 58276 0 0 0 68 3374 17230 64 25 12 0 0 0 0 3552996 7956 58276 0 0 0 0 3203 20928 65 20 15 0 0 0 0 3549668 7956 58276 0 0 0 0 2532 10802 69 16 14 0 0 0 0 3546148 7956 58276 0 0 0 0 1366 4829 66 21 13 0 0 0 0 3544292 7956 58276 0 0 0 0 1322 4920 61 26 14 1 0 0 0 3543908 7964 58268 0 0 0 64 1314 6799 41 48 11 0 0 0 0 3542884 7964 58268 0 0 0 0 1340 5767 43 45 12 0 0 0 0 3541604 7964 58268 0 0 0 0 1376 6473 72 15 13 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 0 0 0 3540772 7964 58268 0 0 0 0 1376 6319 64 23 13 0 0 0 0 3540260 7964 58268 0 0 0 0 1122 3750 21 73 6 1 0 0 0 3540260 7972 58260 0 0 0 56 1115 2674 9 89 2 0 0 0 0 3540260 7972 58260 0 0 0 0 1350 9071 35 56 10 0 0 0 0 3540260 7972 58260 0 0 0 16 1191 5283 20 76 4 0 0 0 0 3540260 7972 58260 0 0 0 0 1032 747 3 96 0 0 0 0 0 3540260 7972 58260 0 0 0 0 1139 3428 14 81 5 1 0 0 0 3539812 7980 58320 0 0 0 40 1312 6989 37 52 12 0 0 0 0 3538916 7980 58320 0 0 0 0 1360 11262 54 30 16 0 0 0 0 3538852 7980 58320 0 0 0 0 1147 3152 15 82 3 0 0 0 0 3537956 7980 58320 0 0 0 0 1338 7848 42 45 13 0 0 0 0 3537828 7980 58320 0 0 0 0 1354 9777 50 34 15 1 0 0 0 3537636 7988 58312 0 0 0 56 1296 7845 43 46 11 0 0 0 0 3537316 7988 58312 0 0 0 0 1261 7192 31 61 8 0 0 0 0 3537124 7988 58312 0 0 0 0 1266 6786 47 41 12 0 0 0 0 3536996 7988 58312 0 0 0 0 1350 6681 53 35 12 0 0 0 0 3536932 7988 58312 0 0 0 0 1331 7851 52 35 13 0 0 0 0 3536804 7996 58372 0 0 0 60 1318 7346 40 50 11 0 0 0 0 3536612 7996 58372 0 0 0 0 1338 7608 57 32 11 0 0 0 0 3536228 7996 58440 0 0 0 0 1207 3261 29 66 6 0 0 0 0 3536228 7996 58440 0 0 0 0 1098 2481 11 87 2 0 0 0 0 3536420 7996 58440 0 0 0 0 1339 9067 41 45 14 0 0 0 0 3536676 8004 58432 0 0 0 56 1255 8130 45 42 14 0 0 0 0 3536548 8004 58432 0 0 0 0 1379 6564 37 53 10 0 0 0 0 3536548 8004 58432 0 0 0 0 1306 5329 43 48 9 0 0 0 0 3536548 8004 58432 0 0 0 0 1322 6817 66 20 14 0 0 0 0 3536548 8004 58432 0 0 0 0 1106 1137 10 88 2 0 0 0 0 3537252 8012 58424 0 0 0 60 1388 7836 73 12 15 0 0 0 0 3537188 8012 58424 0 0 0 0 1347 7703 72 12 16 0 0 0 0 3537188 8012 58424 0 0 0 0 1368 7177 69 16 15 0 0 0 0 3537188 8012 58424 0 0 0 0 1371 6058 58 26 16 0 0 0 0 3537188 8012 58424 0 0 0 0 1362 6481 65 17 18 0 0 0 0 3536228 8020 58416 0 0 0 64 1194 3321 28 67 5 0 0 0 0 3535588 8020 58416 0 0 0 0 1381 7399 63 19 18 0 0 0 0 3535332 8020 58416 0 0 0 0 1342 6313 55 28 18 0 0 0 0 3535268 8020 58416 0 0 0 0 1010 36 1 99 0 0 0 0 0 3535268 8020 58416 0 0 0 0 1008 19 1 99 0 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 0 0 0 3535268 8028 58408 0 0 0 48 1396 7179 55 27 18 0 0 0 0 3536484 8028 58408 0 0 0 0 1377 6094 60 12 28 0 0 0 0 3537444 8028 58476 0 0 0 20 1364 324 27 2 71 0 0 0 0 3537444 8028 58476 0 0 0 0 1353 123 4 2 94 0 0 0 0 3537764 8028 58476 0 0 0 0 1359 108 1 2 98 0 0 0 0 3537764 8044 58460 0 0 0 80 1368 142 1 2 98
vmstat.solaris5.8
(text/plain, 10 KB)
procs memory page disk faults cpu r b w swap free re mf pi po fr de sr m0 m1 m2 s0 in sy cs us sy id 0 0 0 6702816 230264 0 0 0 0 0 0 0 0 0 0 1 478 3400 1050 0 6 94 0 0 0 6702816 230264 0 0 0 0 0 0 0 0 0 0 0 441 422 361 0 0 100 0 0 0 6702816 230264 2 106 0 0 0 0 0 0 0 0 0 449 594 377 0 0 99 0 0 0 6702816 230264 0 0 0 0 0 0 0 0 0 0 0 581 406 628 0 0 100 3 0 0 6702376 228584 0 216 0 0 0 0 0 0 0 0 0 3897 7449 5107 49 12 39 3 0 0 6702376 228536 0 13 0 0 0 0 0 1 1 1 18 3698 7337 2222 73 27 0 3 0 0 6702376 228400 0 7 0 0 0 0 0 0 0 0 0 4721 4860 2673 84 15 0 4 0 0 6702376 228392 0 0 0 0 0 0 0 0 0 0 0 4347 4785 2502 86 13 0 4 0 0 6702376 228392 0 1 0 0 0 0 0 0 0 0 0 4606 5762 3581 87 13 0 3 0 0 6702376 228392 0 0 0 0 0 0 0 0 0 0 0 5317 8362 5968 80 19 0 6 0 0 6702304 228376 0 12 0 0 0 0 0 0 0 0 0 5668 14403 8708 74 25 1 6 0 0 6702304 228216 0 46 0 0 0 0 0 0 0 0 0 4695 13084 8693 88 11 1 5 0 0 6700792 226728 0 347 0 0 0 0 0 0 0 0 0 4223 6723 4240 85 13 1 4 0 0 6698560 224504 0 359 0 0 0 0 0 0 0 0 0 4185 6376 3852 84 13 2 3 0 0 6696400 222280 0 365 0 0 0 0 0 0 0 0 0 4823 8043 5558 77 21 2 3 0 0 6693592 219576 0 501 0 0 0 0 0 0 0 0 4 4775 11036 5653 77 20 2 3 0 0 6689848 215920 0 482 0 0 0 0 0 0 0 0 0 4011 4977 2615 86 13 1 5 0 0 6687472 213544 0 252 0 0 0 0 0 0 0 0 0 3481 4607 1800 83 14 2 3 0 0 6685744 211760 0 254 0 0 0 0 0 0 0 0 0 3390 4483 1763 83 13 3 procs memory page disk faults cpu r b w swap free re mf pi po fr de sr m0 m1 m2 s0 in sy cs us sy id 3 0 0 6684736 210752 0 5 0 0 0 0 0 0 0 0 0 2882 3455 1187 92 8 0 3 0 0 6684736 210744 0 0 0 0 0 0 0 0 0 0 2 2842 5586 1386 91 9 0 3 0 0 6684736 210720 1 16 0 0 0 0 0 0 0 0 0 2725 3482 1059 91 9 0 3 0 0 6684736 210632 0 1 0 0 0 0 0 0 0 0 0 2780 3522 983 95 5 0 3 0 0 6684736 210632 0 0 0 0 0 0 0 0 0 0 0 2755 3510 1070 89 11 0 3 0 0 6684736 210632 0 0 0 0 0 0 0 0 0 0 0 2995 3323 1185 91 9 0 3 0 0 6684736 210632 0 0 0 0 0 0 0 0 0 0 2 2847 4517 1245 85 15 0 3 0 0 6684736 210632 0 0 0 0 0 0 0 0 0 0 0 2853 4576 1330 91 9 0 3 0 0 6684736 210632 0 0 0 0 0 0 0 0 0 0 0 2760 3438 971 93 7 0 3 0 0 6684736 210632 0 0 0 0 0 0 0 0 0 0 0 2856 5459 1110 87 13 0 3 0 0 6684736 210632 0 0 0 0 0 0 0 0 0 0 0 3121 3657 1541 90 10 0 3 0 0 6684736 210632 0 0 0 0 0 0 0 1 1 1 3 2764 3855 1076 85 14 0 3 0 0 6684736 210632 0 0 0 0 0 0 0 0 0 0 0 2920 5627 1564 87 13 0 3 0 0 6684736 210632 0 0 0 0 0 0 0 0 0 0 0 2808 3381 1080 93 7 0 3 0 0 6684736 210632 0 0 0 0 0 0 0 0 0 0 0 2840 3400 1090 92 8 0 3 0 0 6684736 210632 0 0 0 0 0 0 0 0 0 0 0 2760 3510 979 94 6 0 3 0 0 6684736 210632 0 0 0 0 0 0 0 1 1 1 20 2668 2918 1027 71 29 0 3 0 0 6684736 210632 0 0 0 0 0 0 0 0 0 0 0 3163 6525 2046 83 17 0 3 0 0 6684736 210632 0 0 0 0 0 0 0 0 0 0 0 2915 3537 1341 93 7 0 procs memory page disk faults cpu r b w swap free re mf pi po fr de sr m0 m1 m2 s0 in sy cs us sy id 3 0 0 6684736 210632 0 0 0 0 0 0 0 0 0 0 0 2824 7378 1143 89 11 0 3 0 0 6684736 210632 0 0 0 0 0 0 0 0 0 0 0 2837 3351 1023 91 9 0 3 0 0 6684736 210632 0 0 0 0 0 0 0 1 1 1 5 2940 3451 1353 85 15 0 3 0 0 6684736 210632 0 0 0 0 0 0 0 0 0 0 0 2805 5956 1357 87 13 0 3 0 0 6684736 210632 0 0 0 0 0 0 0 0 0 0 0 2884 3327 1027 93 7 0 3 0 0 6684736 210632 0 0 0 0 0 0 0 0 0 0 0 2890 3660 1245 91 9 0 2 0 0 6684880 210776 0 0 0 0 0 0 0 0 0 0 0 2780 5400 1005 89 10 0 3 0 0 6684880 210776 0 0 0 0 0 0 0 0 0 0 4 2972 3542 1473 85 12 2 4 0 0 6684880 210776 0 0 0 0 0 0 0 0 0 0 0 2894 5346 1434 85 13 2 3 0 0 6684880 210776 16 0 128 0 0 0 0 1 0 1 0 2831 3389 1040 92 7 1 2 0 0 6684880 210888 0 0 0 0 0 0 0 0 0 0 0 2797 3361 933 94 6 0 3 0 0 6684880 210888 2 106 0 0 0 0 0 0 0 0 0 2899 3679 1056 87 13 0 3 0 0 6684880 210888 17 255 56 0 0 0 0 0 0 0 7 2926 3602 1297 89 11 0 3 1 0 6683720 209816 25 255 80 0 0 0 0 0 0 0 9 2618 6652 1193 87 13 0 3 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2722 3500 943 89 10 0 3 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2732 3424 907 90 10 0 3 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2711 3553 973 89 10 0 3 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 1 2950 3529 1174 88 12 0 3 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2989 5437 1305 86 14 0 procs memory page disk faults cpu r b w swap free re mf pi po fr de sr m0 m1 m2 s0 in sy cs us sy id 3 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2834 5519 923 93 7 0 3 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 3004 3637 1062 90 10 0 3 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 3013 3632 1177 88 11 0 4 0 0 6684848 210880 1 52 0 0 0 0 0 8 8 8 1 3185 3821 1298 89 11 0 3 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 3025 5283 1361 92 8 0 2 0 0 6684848 210880 4 214 0 0 0 0 0 0 0 0 0 2909 3656 1072 89 11 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2729 3387 946 90 10 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2748 3289 887 92 8 0 2 0 0 6684848 210880 0 1 0 0 0 0 0 17 17 17 44 2786 2492 856 66 33 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2891 5141 1377 85 14 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2859 3408 1104 87 13 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2741 3346 916 91 9 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2717 3381 977 91 9 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2899 3359 1169 91 8 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 3004 6185 1596 91 9 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2817 4003 1112 89 11 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2780 3355 950 93 7 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 5 2732 3281 952 91 9 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2849 3434 1066 93 7 0 procs memory page disk faults cpu r b w swap free re mf pi po fr de sr m0 m1 m2 s0 in sy cs us sy id 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 1 3124 6069 1786 88 11 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2781 3432 1016 89 11 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2789 3483 1024 89 10 1 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 2 2720 3347 932 93 7 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2764 3317 944 91 9 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 3141 6137 1888 86 14 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2750 3412 936 94 6 0 3 0 0 6682944 209056 48 511 64 8 8 0 0 0 0 0 9 2498 4785 826 89 10 0 2 0 0 6684848 210880 3 30 16 0 0 0 0 0 1 1 3 2579 3300 846 84 14 2 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 3024 3645 1082 90 10 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 3281 6243 1784 87 12 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2997 5717 978 89 11 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 3163 3893 1057 87 12 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 3 3074 3833 996 88 12 0 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 3127 3856 1024 87 11 2 2 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 3458 6582 1718 83 16 0 1 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 3037 3694 1121 90 9 1 1 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 3029 3882 1103 89 11 0 0 0 0 6684848 210880 0 0 0 0 0 0 0 3 3 3 6 2644 2987 806 79 16 4 procs memory page disk faults cpu r b w swap free re mf pi po fr de sr m0 m1 m2 s0 in sy cs us sy id 0 0 0 6684848 210880 0 0 0 0 0 0 0 14 14 14 38 2873 2862 813 84 15 1 0 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2962 4926 1597 87 11 2 0 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2717 3109 877 94 6 0 0 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2697 5082 868 89 10 0 0 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 5 2705 3006 849 86 13 1 0 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2670 3062 842 90 9 1 0 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2754 4862 1186 87 10 3 0 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2950 3281 1118 86 12 2 0 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 3010 5001 970 86 11 3 0 0 0 6684848 210880 45 411 40 8 8 0 0 0 0 0 9 2805 5455 1004 82 16 2 1 0 0 6684376 210432 7 126 24 0 0 0 0 0 0 0 3 2678 3765 719 87 12 0 0 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 3218 5373 1592 84 11 5 1 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 3068 3573 1018 93 6 1 0 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2286 2305 430 46 7 48 0 0 0 6684848 210880 1 53 0 0 0 0 0 0 0 0 0 2280 2393 387 48 5 47 0 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2293 2288 401 47 3 50 1 0 0 6684848 210880 0 0 0 0 0 0 0 0 0 0 0 2353 4079 971 48 4 49 0 0 0 6683280 209376 57 548 64 0 0 0 0 0 0 0 8 2349 4559 610 54 11 35 0 0 0 6684496 210576 0 0 0 0 0 0 0 0 0 0 0 1375 1321 458 19 4 76 procs memory page disk faults cpu r b w swap free re mf pi po fr de sr m0 m1 m2 s0 in sy cs us sy id 0 0 0 6684496 210576 0 0 0 0 0 0 0 0 0 0 1 471 556 423 0 2 98 0 0 0 6684496 210576 0 0 0 0 0 0 0 0 0 0 0 454 442 397 0 0 100