Re: OMSA 8.3.0 not starting or segfaulting

Hi Chandra,

thanks for the information. Looks like nrpe might be the culprit:

# ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x00000000 0          root       600        1         
0x00000000 32769      root       600        1         
0x00000000 1113456642 root       600        1         
0x00000000 1113686019 root       600        1         
0x00000000 1113948164 root       600        1         
0x00000000 1113915397 root       600        1         
0x00000000 1113784326 root       600        1         
0x00000000 1113817095 root       600        1         
0x00000000 1113849864 root       600        1         
0x00000000 1113882633 root       600        1         
0x00000000 1113980938 root       600        1         
0x00000000 1114013707 root       600        1         
0x00000000 1114046476 root       600        1         
0x00000000 1114079245 root       600        1         
0x00000000 1114112014 root       600        1         
0x00000000 1114144783 root       600        1         
0x00000000 1114177552 root       600        1         
0x00000000 1114210321 root       600        1         
0x00000000 1117356050 root       600        1         
0x00000000 1117519891 root       600        1         
0x00000000 1117552660 root       600        1         
0x00000000 1116635157 root       600        1         
0x00000000 1116504086 root       600        1         
0x00000000 1116536855 root       600        1         
0x00000000 1116569624 root       600        1         
0x00000000 1116602393 root       600        1         
0x00000000 1117585434 root       600        1         
0x00000000 1117618203 root       600        1         
0xd95b59b9 16613404   root       666        2         
0x00000000 599719965  nrpe       600        1         
0x000003d4 16678942   root       644        1         
0x00000000 599752735  nrpe       600        1         
0x00000000 316276768  nrpe       600        1         
0x00000000 316309537  nrpe       600        1         
0x00000000 1608777762 nrpe       600        1         
0x00000000 174096419  nrpe       600        1         
0x00000000 1608810532 nrpe       600        1         
0x00000000 174161957  nrpe       600        1         
0x00000000 1608843302 nrpe       600        1         
0x00000000 1608876071 nrpe       600        1         
0x00000000 1339129896 nrpe       600        1         
0x00000000 1339162665 nrpe       600        1         
0x00000000 1550483498 nrpe       600        1         
0x00000000 1550516267 nrpe       600        1         
0x00000000 1339293740 nrpe       600        1         
0x00000000 1339326509 nrpe       600        1         
0x00000000 1339359278 nrpe       600        1         
0x00000000 1339392047 nrpe       600        1         
0x00000000 1937965104 nrpe       600        1         
0x00000000 1937997873 nrpe       600        1         
0x00000000 1783726130 nrpe       600        1         
0x00000000 1783758899 nrpe       600        1         
0x00000000 1977810996 nrpe       600        1         
0x00000000 1977843765 nrpe       600        1         
0x00000000 188186678  nrpe       600        1         
0x00000000 188219447  nrpe       600        1         
0x00000000 1131085880 nrpe       600        1         
0x00000000 1131118649 nrpe       600        1         
0x00000000 193953850  nrpe       600        1         
0x00000000 193986619  nrpe       600        1         
0x00000000 194052156  nrpe       600        1         
0x00000000 194084925  nrpe       600        1         
0x00000000 194150462  nrpe       600        1         
0x00000000 194183231  nrpe       600        1         
0x00000000 1331134528 nrpe       600        1         
0x00000000 1331167297 nrpe       600        1         
0x00000000 1504444482 nrpe       600        1         
0x00000000 1504477251 nrpe       600        1         
0x00000000 1687322692 nrpe       600        1         
0x00000000 1687355461 nrpe       600        1         
0x00000000 1687388230 nrpe       600        1         
0x00000000 1687420999 nrpe       600        1         
0x00000000 1723924552 nrpe       600        1         
0x00000000 1723957321 nrpe       600        1         
0x00000000 1759510602 nrpe       600        1         
0x00000000 1759543371 nrpe       600        1         
0x00000000 1828454476 nrpe       600        1         
0x00000000 1828487245 nrpe       600        1         
0x00000000 1889828942 nrpe       600        1         
0x00000000 1889861711 nrpe       600        1         
0x00000000 1949040720 nrpe       600        1         
0x00000000 1949073489 nrpe       600        1         
0x00000000 1949106258 nrpe       600        1         
0x00000000 1949139027 nrpe       600        1         
0x00000000 1968406612 nrpe       600        1         
0x00000000 1968439381 nrpe       600        1         
0x00000000 1317863510 nrpe       600        1         
0x00000000 1317896279 nrpe       600        1         
0x00000000 1317929048 nrpe       600        1         
0x00000000 1317961817 nrpe       600        1         
0x00000000 1306853466 nrpe       600        1         
0x00000000 1306886235 nrpe       600        1         
0x00000000 1412595804 nrpe       600        1         
0x00000000 1412628573 nrpe       600        1         
0x00000000 1346568286 nrpe       600        1         
0x00000000 1346601055 nrpe       600        1         
0x00000000 1346633824 nrpe       600        1         
0x00000000 1346666593 nrpe       600        1         
0x00000000 481525858  nrpe       600        1         
0x00000000 481558627  nrpe       600        1         
0x00000000 481591396  nrpe       600        1         
0x00000000 481624165  nrpe       600        1         
0x00000000 585138278  nrpe       600        1         
0x00000000 585171047  nrpe       600        1         
0x00000000 585203816  nrpe       600        1         
0x00000000 585236585  nrpe       600        1         
0x00000000 678527082  nrpe       600        1         
0x00000000 678559851  nrpe       600        1         
0x00000000 681640044  nrpe       600        1         
0x00000000 681672813  nrpe       600        1         
0x00000000 683901038  nrpe       600        1         
0x00000000 683933807  nrpe       600        1         
0x00000000 684097648  nrpe       600        1         
0x00000000 684130417  nrpe       600        1         
0x00000000 808976498  nrpe       600        1         
0x00000000 809009267  nrpe       600        1         
0x00000000 819200116  nrpe       600        1         
0x00000000 819232885  nrpe       600        1         
0x00000000 801800310  nrpe       600        1         
0x00000000 801833079  nrpe       600        1         
0x00000000 817791096  nrpe       600        1         
0x00000000 817823865  nrpe       600        1         
0x00000000 860520570  nrpe       600        1         
0x00000000 860553339  nrpe       600        1         
0x00000000 860586108  nrpe       600        1         
0x00000000 860618877  nrpe       600        1         
0x00000000 1074528382 nrpe       600        1         
0x00000000 1074561151 nrpe       600        1         

The server is in production, so a reboot is at the moment not possible.
I will try to find a different, where I can debug the problem better.

Regards,
Stefan

----- Original Message -----
> From: "Chandrasekhar R" <[email protected]>
> To: "stefan dietrich" <[email protected]>
> Cc: [email protected]
> Sent: Friday, April 29, 2016 10:38:42 AM
> Subject: RE: OMSA 8.3.0 not starting or segfaulting

> Hi Stefan,
> 
> Thanks for providing this. Based on this strace output, I think the number
> semaphores has reached the maximum count on your server. If possible, Reboot
> the server and see issue resolves.
> 
> Also, send us the output of "ipcs -s" command.
> 
> Regards
> Chandra
> 
> -----Original Message-----
> From: Dietrich, Stefan [mailto:[email protected]]
> Sent: Friday, April 29, 2016 1:45 PM
> To: R, Chandrasekhar
> Cc: linux-poweredge-Lists
> Subject: Re: OMSA 8.3.0 not starting or segfaulting
> 
> Hi Chandra,
> 
> this is the output:
> 
> # strace -f ./dchcfg command=getsystype
> execve("./dchcfg", ["./dchcfg", "command=getsystype"], [/* 36 vars */]) = 0
> brk(0) = 0x1dd2000
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
> 0x7f60e63c4000
> access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
> open("/etc/ld.so.cache", O_RDONLY) = 3
> fstat(3, {st_mode=S_IFREG|0644, st_size=110432, ...}) = 0 mmap(NULL, 110432,
> PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f60e63a9000
> close(3) = 0
> open("/opt/dell/srvadmin/lib64/libdchcfl.so.8", O_RDONLY) = 3 read(3,
> "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \230\0\0\0\0\0\0"..., 832) =
> 832 fstat(3, {st_mode=S_IFREG|0755, st_size=613566, ...}) = 0 mmap(NULL,
> 2271824, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f60e617e000
> mprotect(0x7f60e61a8000, 2097152, PROT_NONE) = 0 mmap(0x7f60e63a8000, 4096,
> PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2a000) =
> 0x7f60e63a8000
> close(3) = 0
> open("/lib64/libdl.so.2", O_RDONLY) = 3
> read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\r\0@9\0\0\0"...,
> 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=22536, ...}) = 0
> mmap(0x3940000000, 2109696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
> 0) = 0x3940000000 mprotect(0x3940002000, 2097152, PROT_NONE) = 0
> mmap(0x3940202000, 8192, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x3940202000
> close(3) = 0
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
> 0x7f60e617d000 open("/lib64/libpthread.so.0", O_RDONLY) = 3 read(3,
> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000^\300?9\0\0\0"..., 832) = 832
> fstat(3, {st_mode=S_IFREG|0755, st_size=145936, ...}) = 0 mmap(0x393fc00000,
> 2212848, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x393fc00000
> mprotect(0x393fc17000, 2097152, PROT_NONE) = 0 mmap(0x393fe17000, 8192,
> PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) =
> 0x393fe17000 mmap(0x393fe19000, 13296, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x393fe19000
> close(3) = 0
> open("/lib64/libm.so.6", O_RDONLY) = 3
> read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p>\200?9\0\0\0"..., 832)
> = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=599392, ...}) = 0
> mmap(0x393f800000, 2633912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
> 0) = 0x393f800000 mprotect(0x393f883000, 2093056, PROT_NONE) = 0
> mmap(0x393fa82000, 8192, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x82000) = 0x393fa82000
> close(3) = 0
> open("/lib64/libc.so.6", O_RDONLY) = 3
> read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\356A?9\0\0\0"..., 832)
> = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=1926520, ...}) = 0
> mmap(0x393f400000, 3750152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
> 0) = 0x393f400000 mprotect(0x393f58a000, 2097152, PROT_NONE) = 0
> mmap(0x393f78a000, 20480, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18a000) = 0x393f78a000
> mmap(0x393f78f000, 18696, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x393f78f000
> close(3) = 0
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
> 0x7f60e617c000 mmap(NULL, 4096, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f60e617b000 mmap(NULL, 4096,
> PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f60e617a000
> arch_prctl(ARCH_SET_FS, 0x7f60e617b700) = 0 mprotect(0x393f78a000, 16384,
> PROT_READ) = 0 mprotect(0x393fa82000, 4096, PROT_READ) = 0
> mprotect(0x393fe17000, 4096, PROT_READ) = 0 mprotect(0x3940202000, 4096,
> PROT_READ) = 0 mprotect(0x393f21f000, 4096, PROT_READ) = 0
> munmap(0x7f60e63a9000, 110432) = 0
> set_tid_address(0x7f60e617b9d0) = 966829
> set_robust_list(0x7f60e617b9e0, 24) = 0
> futex(0x7ffd9570474c, FUTEX_WAKE_PRIVATE, 1) = 0 futex(0x7ffd9570474c,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, 7f60e617b700) = -1
> EAGAIN (Resource temporarily unavailable) rt_sigaction(SIGRTMIN, {0x393fc05cb0,
> [], SA_RESTORER|SA_SIGINFO, 0x393fc0f7e0}, NULL, 8) = 0 rt_sigaction(SIGRT_1,
> {0x393fc05d40, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x393fc0f7e0}, NULL, 8) =
> 0 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
> getrlimit(RLIMIT_STACK, {rlim_cur=10240*1024, rlim_max=RLIM64_INFINITY}) = 0
> prctl(PR_SET_UNALIGN, NOPRINT) = -1 EINVAL (Invalid argument)
> rt_sigaction(SIGINT, {0x407f30, [INT], SA_RESTORER|SA_RESTART, 0x393f4326a0},
> {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGTERM, {0x407f30, [TERM],
> SA_RESTORER|SA_RESTART, 0x393f4326a0}, {SIG_DFL, [], 0}, 8) = 0
> brk(0) = 0x1dd2000
> brk(0x1df3000) = 0x1df3000
> semget(IPC_PRIVATE, 1, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on
> device)
> --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} ---
> +++ killed by SIGSEGV +++
> Segmentation fault
> 
> Regards,
> Stefan
> 
>> Thanks Stefan for these details. Your server is Dell Branded. Can you
>> please run the following command and send me the response?
>>
>> cd /opt/dell/srvadmin/sbin
>> strace -f ./dchcfg command=getsystype
>>
>> Regards
>> Chandra
>>
>> -----Original Message-----
>> From: Dietrich, Stefan [mailto:[email protected]]
>> Sent: Friday, April 29, 2016 12:52 PM
>> To: R, Chandrasekhar
>> Cc: linux-poweredge-Lists
>> Subject: Re: OMSA 8.3.0 not starting or segfaulting
>>
>> Hi Chandra,
>>
>> they should be Dell brand (?). This is from a R430 with a segfault:
>>
>> # smbios-sys-info-lite
>> Libsmbios: 2.2.27
>> System ID: 0x0639
>> Service Tag:
>> Express Service Code:
>> Aset Tag:
>> Product Name: PowerEdge R430
>> BIOS Version: 1.2.6
>> Vendor: Dell Inc.
>> Is Dell: 1
>> OEM String 1: Dell System
>> OEM String 2: 5[0000]
>> OEM String 3: 14[1]
>> OEM String 4: 17[537D0FC5205BB363]
>> OEM String 5: 17[537BA7D240AC7A6C]
>> OEM String 6: 18[0]
>> OEM String 7: 19[1]
>> OEM String 8: 19[1]
>>
>> I have removed the service and asset tag information.
>>
>> Regards,
>> Stefan
>>
>>> Hi Stefan,
>>>
>>> Are you seeing this segfault issue on Re-branded servers or Dell brand as well?
>>> Please get us the following command response on failing system:
>>>
>>> smbios-sys-info-lite
>>>
>>> Regards
>>> Chandra
>>>