Re: ra split dies with a floating point divide error when processing large file
Carter Bullard <[email protected]> Tue, 16 Oct 2018 10:23:18 +0200
| Newsgroups | gmane.network.argus |
|---|---|
| Message-ID | <[email protected]> |
--Apple-Mail-123DD999-2C18-4ADC-AB5A-06F53E19591A Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Yes, I use it all the time for this purpose. How are you splitting, size or= date ?? Can you send the complete commandline ?? Carter =09 Carter Bullard =E2=80=A2 CTO 150 E 57th Street Suite 12D New York, New York 10022-2795 Phone +1.212.588.9133 =E2=80=A2 Mobile +1.917.497.9494 > On Oct 16, 2018, at 6:07 AM, Russell Fulton <[email protected]> wrot= e: >=20 > The log rollover process broke and I was left with some large flow files s= o I tried to break them up with rasplit but if died about a third of the way= through the file... >=20 > rful011@secmgrprd02:/usr/local/tools/notify-framework$ sudo ls -lh /data/a= rgus/data/dmzo.big > -rw-r--r-- 1 argus argus 516G Oct 15 08:59 /data/argus/data/dmzo.big >=20 > [1]+ Floating point exception(core dumped) rasplit -r /data/argus/data/dm= zo.big -M time 1h -w "%Y/%m/%d/dmzo.%H.%M.%S" >=20 >=20 > Oct 15 18:07:36 secmgrprd02 kernel: [4584251.907460] traps: rasplit[5534] t= rap divide error ip:556d5ebf7bc4 sp:7ffc39708be0 error:0 in rasplit[556d5eb8= 9000+b3000] >=20 > I tried re running the job with a -t parameter to start processing at the b= eginning of the hour in which it crashed with the same result, this did outp= ut some data. >=20 > Tried again this time starting -t the hour after the point where is crashe= d and it still dies and does not produce an output. >=20 > I conclude it is something to do with the process of reading the file that= has the problem not the splitting. >=20 > Has anyone used rasplit on really big files? This one is weeks data and i= s nearly 500GB. >=20 > Russell --Apple-Mail-123DD999-2C18-4ADC-AB5A-06F53E19591A Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><head><meta http-equiv=3D"content-type" content=3D"text/html; charset=3D= utf-8"></head><body dir=3D"auto">Yes, I use it all the time for this purpose= . How are you splitting, size or date ??<div>Can you send the complete= commandline ??</div><div>Carter<br><br><div id=3D"AppleMailSignature" dir=3D= "ltr"><table id=3D"sig" width=3D"360" cellspacing=3D"0" cellpadding=3D"0" bo= rder-spacing=3D"0" class=3D"" style=3D"font-family: UICTFontTextStyleBody; -= webkit-text-size-adjust: auto; font-size: 13pt; width: 360px; margin: 0px; p= adding: 0px;"><tbody class=3D""><tr class=3D""><td width=3D"142" class=3D"" s= tyle=3D"width: 142px; margin: 0px; padding: 0px;"><a href=3D"http://qosient.= com/" title=3D"QoSient" class=3D"" style=3D"text-decoration: none; border: n= one; color: rgb(149, 79, 114);"><img src=3D"data:image/png;base64,iVBORw0KGg= oAAAANSUhEUgAAAR8AAABdCAYAAACcsGKeAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYW= R5ccllPAAACy9JREFUeNrsndt140YShmt85n2RgbERCI5gMBEMNgLBEZiOQHAEtCOgHIFGEZCKgN= oIyI2AzMArnGkcy5jqC9DVF4D/dw4eRElEo6vr7+oLqokAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= AAAAAAAAAAAAAAAMBN8CGTctSaz1/frivMlJTi7apGn53VBcBixKdW193bVTKNWsfQ2F/eroO6QD= ixad+uT8pWheFvX9U12OUs2Ebe2/4RZgFzaN6u3dt1ebv+EryelJMUqGIRSmUnH5scBWzSjb5zD9= OAqb1n34hOwoLDXRflNCWqXczhJWyynWkTiA/wasiXCKLDXTtEQpM7iWPgjqGB+IAY8zmukc5e9Y= yt+r/S8J2NapBPjqLW/80G5rBSWex1Ujbi7FMpu2wdxKuD+ICQbCPO0dSOc0h7REHGiOdiEJ12xn= yRbpgN8QHJwvZQ8zHDqszJEgVVMNN3HA228mVsE4gPiB627ynOJHBB9gnTFubSOneoOhpEaKr41O= p/hgu2y48yZadeGcL2S6IGU1qiMDTib3XE1c02YMeAyHN9beiSKiI1Cc+R0i957yBAk+rmCH8CM6= ZZ9ilufqH8J3hbgwA1N9x4ONvV8Ckwo/OKLj4hJypjCdCtTkLXxK9sAeA64km2ELA1RDy50hqGh7= e2DN9RvLkesP7Oa5/qxkty4o7iTrLmyhNhDgwsUHxOmuFLuWDH++vGhl97wnwPyEh8fnCMHDiR+Z= mWk9OlL+tVM5QEACTAJj79kOoX5vOv6loKvfD8qlHzW+798foJyFZ8NpoG+usCn/WR+CRkDzdiay= 5KxQZAkC3cXM9uwc9T0+3O/XSEDYZAzne853w+Gn7Xb8Yrmc9/W3AF9pHPKyM29+rzGFRMNBkjV/= WBifIq1agON+5Y3GsgueYPrxdSTiumHM59hNOOPuvnef6z8IbWMtFbPyT5d6D79SL+RTXuykEgXl= Q9hxDDCyN8/T0/Z+Zc9cg2j8L3KEd20c19XVX9PCubSDl5Z/l53EHcqzqpLOV8UXV1negPpcPf/T= jSg94uf07o+CZ1cNyQaw2vJhQRhl4l+eetPpL8PpyOZBJ+xRwe7gPYxSddr8Qk/fi7daKzn1nObk= I59xQ+w+ik9lVpvmQtcBW+ERK2TthwJ5JbkTO9m9etXHyk7CKRKdPmV51Qu6lyFp+PBvHhQqe18M= I4dH9UzO+eczk7B4NfmSFVbemt96psvquM/b37PU9PzO8eVB0saf+Wq+A+OQj4wdEm/ff1+8PulD= 2k51t2lojXtZylip5/tgxbXyfUY2Vpxzomtacu89BcYl5BcuWnIXN+o536m8IiXhsy5ySSyhywFQ= zbc458TJk2hxzVlUX4N6RPmjf39SJd5LPTlLOzlLNSz6Jrg3UAnwn2esWe1p2KohAcVrYBnLg2hM= NS79PZ5j5SiZCU+OiE5zKzI201zn0UEp8d+SflKzR2vZBM7vRk4lPTuuAcrvA0iHQK2Y1gg5/iUN= zermZh4vOkqTefhYVSI2hbz7bXCEe4G5LfnxdNfCQcM3eOngJbahxXehOmLnOk1HtpJbmvAA1DgD= Jz8dkEjBh1k/a1h39dArQfTnzLpYrP2vCN7vYUb/d3ReGzEE4RoeFZ6wzFhxMH6QwMNfklZjPVqe= TUwkXw+yE+mYgPN8/zFLi8vg1+igh15L5HSXo/kq/47CjOfCV3n9bDv0I485YRYYjPwsXnxIhAjG= FpR3ETgrXkvg/kJFQWH/EpKF6mzdKjAwoxIexaxgris1zxaSntSuCJ4udhLlUv6nIctu9ku4/4bC= juQgk3r1LM8K8mYnvZ5C4+F+HJqhw5zXzGPaV9Mzy1+LlEQz5J+n3E5xhZmOfaImbHPhbILhfx0e= XzedX0fmuCe56zw/+Me9I/Ipf7kSnnl8j3/6yug2H4s6e4qUq4N9NDJ7zjvv9TZu38v7mWTyc+15= WLzxzhIU0InyKj41eHcoXmoARIl6K2oHhHZpNG6F4C3/PKtJvcckOdc3XCHxzVsuduReJTzTTSHe= OAKXKpPDNimmofVh8J/aSJlocdtzGoHedkpK+SeWaIj/Cwq16R+Hya2UtWkXtWU9ThIqgxG/hnQ7= tpI5ThXxl3bGCC+Oga91p2OTeOz2wbrqXMIHfNbFh8VQLE1ckDnB6M+WhoSFy60Ybks8ql6Jk4EZ= kjPq8Jn+N1FI2WGdTtcErIjqm3KnJ99e30T7j48sSHlOG4XMdLF5975rOvaAqiTv/AiGETWXz+R8= hNvchhl84h64WHt4Vm/uEZTUGUJSxBg4zF56zpOX5Z8PNy55CdEfmI8+wwZAUQHyPcMTktLXPlS3= f66pRNgufRzymjwMpStpSk2KQ6Xnn8Ee69bPE5aKKfJZ5xzp080E+QPnqIT6oGXmgiuFzI4RwpRFo= LFx9d9NP3ut2CnrMhfnn9t4mOMiXxe0hqx2jjljhkYhsgKD69Ubk5kQdaxuTzcKoE56xTT6t4Yb4= 7RQ/7hXmWnE6tTLEf7FXT6YAFi0+P7v2dPeW98bAg/UFvc46hOWTSwBuHcqUWfIpcRm6v1j1cfPn= iM5z3xDl3rgJkerP695nOcGWiwNirfy1T37ltpuOGPOcI931mRLqEm68DXY5fqeTcksJzNJTV17FS= nWnWP9eJ0uYTcoHLldQ6/m9HfpkMLxQp6ZUHMfP51EL1UedQryanzqGXKQ1llEokvie55FlT2JJf= GtUYZWzJL01oR/I5nDcQn3WIT2Fx7jpxuG86ObQKZNAY0R/n1FMbwCWwIxaaqGdKWg1f8dEdbRND= eCvH+6xBfE6pnNwkQCmOVi7IfgSwdOPbRhSgRuiZpA81tEWEc6LNjsKc2xU6Ot1M6HzXID5J87rb= BOgUKQpqyZzYPFSj0z2/tAC1mudqZ3wX10lIHaS3E+qIOiFn0Qmh9OpkObrXLYlPmVqAbNni9hRm= OdomOhLH47o0vEug4acpmpubHTDEmeymObY5DV1KfEyd41ZAdAvizzdbq/gUFPfYptkNRhcJbT2d= slGO53KgnUQDcx3nXwzOV88wcmv4Tp+0pLYTJ7YTxLq0DHXnRoBS4mPqHN6fM1bMsPfW8L1rFR8i= fqW1cLRDRwHnGyvLMIxzzK0qVKMq6f3VqN/tyP3guqFC6sjia3v2QXgbjXPXyhFswrqNZKOTKks3= skmrPjs62GCu8EuKjy0yG0R3R/oXpoe26HJumatzLVV8OtKfWFsy992MRkZdaEd0GQ6FuKROzAw5= BJ17Sc9VtOR+HPLUa+cZcUqLT2jbDGWcMrxfqvjY5nltVxfLGVvPgrpeT5TXOzy1sPj6OrOpIW0E= y3oSskMI8Xk/bJe0zXHmMy9VfGzTDNmIz3iMfBIWnA3lvXW+nThc5IYCsZ6v8YgMTsQnactRfCQ6= xsE2PkP7JYvPnEjyRMyixofIDlnS3xuxPhF/yuSY/m3lM307S+xAy8vLWyjnvnv37AXzjP17Yy8Z= PGNjsc97e3wl+VQe5Uh0rxQuXUj5zjalRlAOqgyS7a9m7hGy/VWB6rP/3nv6Pr3yWV0vgdqIKNtI= cx0AAMD2uKYl6xJVBAAIOSwzjcVj7d0BANwoHZkn/DqIEAAgZBRkWy3qVx18Xp9oIGIAAJNA2Jbp= h6W8Ka8GDC9CdqhiAOLzYUFlbYk/hnfMsJT4Qt8vKw5LyMMKWv+7z5RX8nUAQMYiJLFzekp2PQAA= +EcU4/qGe+zEUgCAG2FIteH6CkeLKgMAhIiINkqMuOFZhyoCID0fbuQ5a/o2v1PS9FNKAQAAAAAA= AAAAAAAAAAAAAAAAAAAASM3/BRgAaVGndQa4eeAAAAAASUVORK5CYII=3D" alt=3D"QoSient" c= lass=3D"" style=3D"border: none; height: 48px; line-height: 48px; font-size:= 48px; font-family: 'Gill Sans', sans-serif; color: rgb(0, 0, 0);"></a></td>= <td width=3D"20" class=3D"" style=3D"width: 16px; min-width: 16px; max-width= : 16px; margin: 0px; padding: 0px;"> </td><td class=3D"" style=3D"margi= n: 0px; padding: 0px; border-collapse: collapse;"><table id=3D"sig2" cellspa= cing=3D"0" cellpadding=3D"0" border-spacing=3D"0" class=3D""><tbody class=3D= ""><tr class=3D""><td class=3D"" style=3D"white-space: nowrap; font-family: '= Gill Sans Light', sans-serif;"><a href=3D"mailto:[email protected]" class=3D= "" style=3D"text-decoration: none; border: none; color: rgb(149, 79, 114);">= <span class=3D"" style=3D"line-height: 16px; font-size: 16px; color: rgb(0, 0= , 176);">Carter Bullard</span> </a><span class=3D"" style=3D"line-heigh= t: 14px; font-size: 12px;">=E2=80=A2</span> <span class=3D"" style=3D"l= ine-height: 14px; font-size: 12px;">CTO</span></td></tr><tr class=3D""><td c= lass=3D"" style=3D"white-space: nowrap; font-family: 'Gill Sans Light', sans= -serif;"><span class=3D"" style=3D"line-height: 14px; font-size: 12px;">150 E= 57th Street Suite 12D</span></td></tr><tr class=3D""><td class=3D"" st= yle=3D"white-space: nowrap; font-family: 'Gill Sans Light', sans-serif;"><sp= an class=3D"" style=3D"line-height: 14px; font-size: 12px;">New York, New Yo= rk 10022-2795</span></td></tr><tr class=3D""><td class=3D"" style=3D"white-s= pace: nowrap; font-family: 'Gill Sans Light', sans-serif;"><span class=3D"" s= tyle=3D"line-height: 12px; font-size: 12px;">Phone +1.212.588.9133 =E2=80= =A2 Mobile +1.917.497.9494<br></span></td></tr></tbody></table></td></tr></t= body></table></div><div dir=3D"ltr"><br>On Oct 16, 2018, at 6:07 AM, Russell= Fulton <<a href=3D"mailto:[email protected]">[email protected].= nz</a>> wrote:<br><br></div><blockquote type=3D"cite"><div dir=3D"ltr"><s= pan>The log rollover process broke and I was left with some large flow files= so I tried to break them up with rasplit but if died about a third of the w= ay through the file...</span><br><span></span><br><span>rful011@secmgrprd02:= /usr/local/tools/notify-framework$ sudo ls -lh /data/argus/data/dmzo.big</sp= an><br><span>-rw-r--r-- 1 argus argus 516G Oct 15 08:59 /data/argus/data/dmz= o.big</span><br><span></span><br><span>[1]+ Floating point exception(c= ore dumped) rasplit -r /data/argus/data/dmzo.big -M time 1h -w "%Y/%m/%d/dmz= o.%H.%M.%S"</span><br><span></span><br><span></span><br><span>Oct 15 18:07:3= 6 secmgrprd02 kernel: [4584251.907460] traps: rasplit[5534] trap divide erro= r ip:556d5ebf7bc4 sp:7ffc39708be0 error:0 in rasplit[556d5eb89000+b3000]</sp= an><br><span></span><br><span>I tried re running the job with a -t parameter= to start processing at the beginning of the hour in which it crashed with t= he same result, this did output some data.</span><br><span></span><br><span>= Tried again this time starting -t the hour after the point where is crashed a= nd it still dies and does not produce an output.</span><br><span></span><br>= <span>I conclude it is something to do with the process of reading the file t= hat has the problem not the splitting.</span><br><span></span><br><span>Has a= nyone used rasplit on really big files? This one is weeks data a= nd is nearly 500GB.</span><br><span></span><br><span>Russell</span><br></div= ></blockquote></div></body></html>= --Apple-Mail-123DD999-2C18-4ADC-AB5A-06F53E19591A--