Re: i7 working for rtai-5.0-test1.tar.bz2
"Hasan Jamal" <[email protected]>
| Newsgroups | gmane.linux.real-time.rtai |
|---|---|
| Organization | Connect Tech Inc |
| Message-ID | <[email protected]> |
I did some debugging on user/latency and I found that it is hung on __ipipe_migrate_head The call chain is rt_make_hard_real_time -> steal_from_linux -> __ipipe_migrate_head So, it's some scheduling issue, how to move past this? -----Original Message----- From: Hasan Jamal [mailto:[email protected]] Sent: March-09-16 9:26 AM To: [email protected] Subject: FW: [Rtai] i7 working for rtai-5.0-test1.tar.bz2 Send to the list as suggested by Paolo, the nice guy. -----Original Message----- From: Paolo Mantegazza [mailto:[email protected]] Sent: March-09-16 9:14 AM To: Hasan Jamal Subject: Re: [Rtai] i7 working for rtai-5.0-test1.tar.bz2 The On 03/09/2016 02:58 PM, Hasan Jamal wrote: > After overnight test, I'd say kernel and RTAI modules are working > properly for 3.14.39. But, It looks RTAI modules have subtle bugs > during module load/unload, may need some flushing of cache? My next > job is to get user/latency work, I'd like to load modules manually for > it, what modules does it need? Hal+sched+sem+mbx+msg, while kernel space needs hal+sched+fifos. It is an easy information you can gather from the files "runinfo" in "testsuite" directories. Your problem is likely one of cold caches, to be warmed up. In my view also ovl_max maight be a bit high, but if it is generated just in the first records then that will confirm my assumption. The latency.c code I sent you has just a few lines more to check what happens after having warmed the cache up. You should try it, porting the same checks into the kernel test. BTW: keep this thread on the RTAI list. I sent you the code privately not to cut other RTAI users off, but just not to bother them with an attachment. I would like other people can learn something through the list discussions. Paolo. > > Following is the output of overnight test with intense load > > RTH| lat min| ovl min| lat avg| lat max| ovl max| overruns > RTD| 263| 112| 315| 3104| 47942| 0 > RTD| 262| 112| 314| 1277| 47942| 0 > RTD| 260| 112| 313| 1450| 47942| 0 > RTD| 262| 112| 313| 1130| 47942| 0 > RTD| 262| 112| 314| 3474| 47942| 0 > RTD| 261| 112| 314| 1126| 47942| 0 > RTD| 264| 112| 315| 2890| 47942| 0 > RTD| 262| 112| 314| 1455| 47942| 0 > RTD| 215| 112| 314| 1285| 47942| 0 > RTD| 263| 112| 314| 1310| 47942| 0 > RTD| 262| 112| 315| 3857| 47942| 0 > RTD| 247| 112| 313| 1058| 47942| 0 > RTD| 220| 112| 315| 2209| 47942| 0 > RTD| 263| 112| 315| 1829| 47942| 0 > RTD| 179| 112| 314| 1682| 47942| 0 > RTD| 262| 112| 313| 1788| 47942| 0 > RTD| 262| 112| 314| 3287| 47942| 0 > RTD| 158| 112| 314| 1328| 47942| 0 > RTD| 262| 112| 314| 1417| 47942| 0 > RTD| 262| 112| 315| 1286| 47942| 0 > ^CRTD| 262| 112| 315| 1286| 47942| > 0 > > -----Original Message----- > From: Hasan Jamal [mailto:[email protected]] > Sent: March-08-16 11:14 PM > To: 'Paolo Mantegazza' > Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2 > > Looks like 3.14.39 is working. What I found is that I always get 0 > overruns after reboot, but if start/stop /kern/latency then overruns > number increase randomly. So, I manually loaded all needed modules for > kern/latency and if I start/stop latency test then overrun number > never increases under default load. I'm running now with intense load > and will post result tomorrow morning. > > Thanks, > Hasan > -----Original Message----- > From: Hasan Jamal [mailto:[email protected]] > Sent: March-08-16 6:58 PM > To: 'Paolo Mantegazza' > Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2 > > I spent days getting Linux-3.16/RTAI4.1 built but went nowhere. > So, I decided the latest but it is also not working properly. > I'm building now for 3.14.39. > > I've to get it working by tomorrow, promised to the customer. > > I'm using the same system where old Linux-2.6.24 with RTAI works, now > the customer needs upgrade together other stuff like applications. > > The minimum requirement is Linux-3.11 from customer. > > I appreciate your help, any idea to make progress? > > Thanks, > Hasan > -----Original Message----- > From: Paolo Mantegazza [mailto:[email protected]] > Sent: March-08-16 6:42 PM > To: Hasan Jamal > Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2 > > Take into account that you have to wait a few seconds before seeing > something on the screen. > If user space does not work then there is something strange. > > Paolo > ________________________________________ > From: Hasan Jamal [[email protected]] > Sent: Wednesday, March 9, 2016 12:21 AM > To: Paolo Mantegazza > Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2 > > This does not update table, looks like hung. I was using kern/latency. > If kern/latency somehow works then why not user/latency? > > -----Original Message----- > From: Paolo Mantegazza [mailto:[email protected]] > Sent: March-08-16 6:13 PM > To: Hasan Jamal > Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2 > > User > ________________________________________ > From: Hasan Jamal [[email protected]] > Sent: Wednesday, March 9, 2016 12:08 AM > To: Paolo Mantegazza > Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2 > > OK, got it; is this user or kern? I just want to make sure even though > you mentioned 'user' previously. > > -----Original Message----- > From: Paolo Mantegazza [mailto:[email protected]] > Sent: March-08-16 6:02 PM > To: Hasan Jamal > Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2 > > Now you should. Paolo > ________________________________________ > From: Hasan Jamal [[email protected]] > Sent: Tuesday, March 8, 2016 11:37 PM > To: Paolo Mantegazza > Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2 > > I didn't get the attachment. > > -----Original Message----- > From: Paolo Mantegazza [mailto:[email protected]] > Sent: March-08-16 5:35 PM > To: Hasan Jamal > Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2 > > Please overwrite the attached latency.c in the RTAI subdirectory: > "testsuite/user/latency". > Then rerun make install from there, go and execute latency again. > Report if anything changed to the RTAI list. > Thanks, Paolo. > > ________________________________________ > From: Hasan Jamal [[email protected]] > Sent: Tuesday, March 8, 2016 10:50 PM > To: Paolo Mantegazza > Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2 > > Running your script of 'make' and will post result an hour later. > > -----Original Message----- > From: Hasan Jamal [mailto:[email protected]] > Sent: March-08-16 4:42 PM > To: 'Paolo Mantegazza'; '[email protected]' > Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2 > > OK, got it. It won't make any difference than what I'm doing. > Following is the output after running for 50mins > > RTD| 270| 0| 321| 1696| 46086| 26 > RTD| 270| 0| 320| 1852| 46086| 26 > RTH| lat min| ovl min| lat avg| lat max| ovl max| overruns > RTD| 270| 0| 321| 2051| 46086| 26 > RTD| 265| 0| 323| 2965| 46086| 26 > > Following is 'pas aux' output, ' heap_array' is using 100% of both CPU. > > root 2549 0.0 0.0 4448 776 pts/14 S+ 14:46 0:00 /bin/sh > ./run > root 2550 0.0 0.0 4448 1640 pts/14 S+ 14:46 0:00 /bin/sh > /usr/realtime/bin/rtai-load > root 2582 0.0 0.0 4448 1328 pts/14 S+ 14:46 0:00 /bin/sh > /usr/realtime/bin/rtai-load > root 2623 0.0 0.0 0 0 ? S 14:46 0:00 > [RTAI_KTHRD_SRVR] > root 2659 0.0 0.0 4336 624 pts/14 S+ 14:46 0:00 ./display > root 2661 100 5.2 209000 205752 pts/6 R 14:47 47:59 > ./heap_array > root 2662 99.5 5.2 209000 205688 pts/6 R 14:47 47:45 > ./heap_array > root 2679 0.0 0.0 22648 2620 pts/6 R+ 15:35 0:00 ps aux > > Following is the output of last few lines of 'dmesg' > > [ 164.259240] I-pipe: head domain RTAI registered. > [ 164.259244] RTAI[hal]: mounted. ISOL_CPUS_MASK: 0, LINUX CPU > ISOLATED > MAP: 0). > [ 164.259248] SYSINFO - # CPUs: 2, TIMER NAME: 'lapic', TIMER IRQ: > 16641, TIMER FREQ: 68593002, CLOCK NAME: 'tsc', CLOCK FREQ: 2194981000, CPU FREQ: > 2194981000, LINUX TIMER IRQ: 16641. > [ 177.017478] RTAI[malloc]: global heap size = 2097152 bytes, <BSD>. > [ 177.017521] , kstacks pool size = 524288 bytes. > [ 177.017524] RTAI[sched]: hard timer type/freq = lapic/68593002(Hz); > timing: oneshot; linear timed lists. > [ 177.017526] RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase freq > = > 2194981000 hz. > [ 177.017528] RTAI[sched]: timer setup = 124 ns, resched latency = 0 ns. > [ 191.216621] > [ 191.216621] > [ 191.216621] CPU USE SUMMARY > [ 191.216627] # 0 -> 140730 > [ 191.216628] # 1 -> 0 > [ 191.216629] END OF CPU USE SUMMARY > [ 191.216629] > [ 191.245947] SCHED releases registered named TASK THRSRV [ > 191.246059] > RTAI[malloc]: unloaded. > [ 191.342819] RTAI[sched]: unloaded (forced hard/soft/hard transitions: > traps 0, syscalls 0). > [ 1593.438447] RTAI[malloc]: global heap size = 2097152 bytes, <BSD>. > [ 1593.438491] , kstacks pool size = 524288 bytes. > [ 1593.438494] RTAI[sched]: hard timer type/freq = lapic/68593002(Hz); > timing: oneshot; linear timed lists. > [ 1593.438496] RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase freq > = > 2194981000 hz. > [ 1593.438498] RTAI[sched]: timer setup = 124 ns, resched latency = 0 ns. > [ 1604.719986] > [ 1604.719986] > [ 1604.719986] CPU USE SUMMARY > [ 1604.719992] # 0 -> 112287 > [ 1604.719993] # 1 -> 0 > [ 1604.719994] END OF CPU USE SUMMARY > [ 1604.719994] > [ 1604.734869] SCHED releases registered named TASK THRSRV [ > 1604.734973] > RTAI[malloc]: unloaded. > [ 1604.831806] RTAI[sched]: unloaded (forced hard/soft/hard transitions: > traps 0, syscalls 0). > [ 1606.783388] RTAI[malloc]: global heap size = 2097152 bytes, <BSD>. > [ 1606.783432] , kstacks pool size = 524288 bytes. > [ 1606.783436] RTAI[sched]: hard timer type/freq = lapic/68593002(Hz); > timing: oneshot; linear timed lists. > [ 1606.783438] RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase freq > = > 2194981000 hz. > [ 1606.783439] RTAI[sched]: timer setup = 124 ns, resched latency = 0 ns. > [ 1614.699614] > [ 1614.699614] > [ 1614.699614] CPU USE SUMMARY > [ 1614.699619] # 0 -> 78685 > [ 1614.699620] # 1 -> 0 > [ 1614.699622] END OF CPU USE SUMMARY > [ 1614.699622] > [ 1614.724600] SCHED releases registered named TASK THRSRV [ > 1614.724704] > RTAI[malloc]: unloaded. > [ 1614.822239] RTAI[sched]: unloaded (forced hard/soft/hard transitions: > traps 0, syscalls 0). > [ 1615.948754] RTAI[malloc]: global heap size = 2097152 bytes, <BSD>. > [ 1615.948798] , kstacks pool size = 524288 bytes. > [ 1615.948801] RTAI[sched]: hard timer type/freq = lapic/68593002(Hz); > timing: oneshot; linear timed lists. > [ 1615.948803] RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase freq > = > 2194981000 hz. > [ 1615.948805] RTAI[sched]: timer setup = 124 ns, resched latency = 0 ns. > [ 1620.153990] > [ 1620.153990] > [ 1620.153990] CPU USE SUMMARY > [ 1620.153995] # 0 -> 41611 > [ 1620.153997] # 1 -> 0 > [ 1620.153998] END OF CPU USE SUMMARY > [ 1620.153998] > [ 1620.180231] SCHED releases registered named TASK THRSRV [ > 1620.180351] > RTAI[malloc]: unloaded. > [ 1620.279942] RTAI[sched]: unloaded (forced hard/soft/hard transitions: > traps 0, syscalls 0). > [ 1621.160235] RTAI[malloc]: global heap size = 2097152 bytes, <BSD>. > [ 1621.160280] , kstacks pool size = 524288 bytes. > [ 1621.160283] RTAI[sched]: hard timer type/freq = lapic/68593002(Hz); > timing: oneshot; linear timed lists. > [ 1621.160285] RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase freq > = > 2194981000 hz. > [ 1621.160287] RTAI[sched]: timer setup = 125 ns, resched latency = 0 ns. > [ 2563.057731] perf interrupt took too long (2511 > 2500), lowering > kernel.perf_event_max_sample_rate to 50000 > > Let me know whether any other info that would help. > > Thanks, > Hasan > -----Original Message----- > From: Paolo Mantegazza [mailto:[email protected]] > Sent: March-08-16 4:29 PM > To: Hasan Jamal; [email protected] > Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2 > > I mean entering the Linux directory and: > while "true"; do make clean; make -j<all_my_cpus>; done Paolo > ________________________________________ > From: Hasan Jamal [[email protected]] > Sent: Tuesday, March 8, 2016 9:37 PM > To: Paolo Mantegazza; [email protected] > Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2 > > I made my own version of application, that writes to 125 MB of RAM > continuously and ran a few instances of the same application, 'ps aux' > showed full CPU usage. > I'll follow your suggestion for loading OS using ping, not sure what > you meant for "keept making the kernel and cleaning it"? > > Thanks, > Hasan > -----Original Message----- > From: Paolo Mantegazza [mailto:[email protected]] > Sent: March-08-16 3:27 PM > To: Hasan Jamal; [email protected] > Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2 > > How long did the test last? There was intense Linux load? > Take into account that you should test for a hour or twq, at least. To > create a somewhat intense Linux load in run "ping -f somewher" and > keept making the kernel and cleaning it. > Then report the final rows of the latency test. > Paolo > ________________________________________ > From: Hasan Jamal [[email protected]] > Sent: Tuesday, March 8, 2016 8:56 PM > To: Paolo Mantegazza; [email protected] > Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2 > >> BTW, have you disabled CPU frequency scaling (CPU throttling) >> BIOS/Linux > config? > No, I did not but built with disabled CPU frequency scaling does same. > > Following is the result after overnight test with no load but default > services running. > > RTH| lat min| ovl min| lat avg| lat max| ovl max| overruns > RTD| 1304| 0| 1815| 3118| 43940| 7 > RTD| 974| 0| 1823| 3900| 43940| 7 > > If I stop and start 'run' then overruns changes but the number does > not change over time, it seems that is somehow locked. > > Following is the output with intense load (disabled CPU frequency > scaling), again overrun number don't change from start. > > RTH| lat min| ovl min| lat avg| lat max| ovl max| overruns > RTD| 267| 0| 321| 1195| 7753| 10 > RTD| 267| 0| 322| 1166| 7753| 10 > > May I accept this result as good? Or some bugs are in RTAI? > > Thanks, > Hasan > > -----Original Message----- > From: Paolo Mantegazza [mailto:[email protected]] > Sent: March-07-16 6:06 PM > To: Hasan Jamal; [email protected] > Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2 > > It means that there have been 7 cycles that missed the deadline. > Somewhat strange, since latency has aperiod of 100000 ns, which is not > overtaken by a max overall latency of 32043 ns. > Let it run longer, under intense Linux load, and see what happens. > BTW, have you disabled CPU frequency scaling (CPU throttling) > BIOS/Linux config? Paolo. ________________________________________ From: Rtai [[email protected]] on behalf of Hasan Jamal [[email protected]] Sent: Monday, March 7, 2016 11:47 PM To: [email protected] Subject: [Rtai] i7 working for rtai-5.0-test1.tar.bz2 I've got latest RTAI5.0 working Linux-3.18.22. I'm wondering about overruns in latency test, are they normal? RTD| 1339| 0| 1837| 7611| 32043| 7 RTD| 979| 0| 1818| 5484| 32043| 7 RTH| lat min| ovl min| lat avg| lat max| ovl max| overruns RTD| 1414| 0| 1839| 2742| 32043| 7 RTD| 979| 0| 1819| 27438| 32043| 7 RTD| 1780| 0| 1824| 10873| 32043| 7 RTD| 974| 0| 1807| 11571| 32043| 7 RTD| 1570| 0| 1842| 10585| 32043| 7 RTD| 979| 0| 1835| 22838| 32043| 7 RTD| 1730| 0| 1814| 2773| 32043| 7 RTD| 984| 0| 1830| 16463| 32043| 7 RTD| 1720| 0| 1835| 11337| 32043| 7 RTD| 984| 0| 1822| 21159| 32043| 7 _______________________________________________ Rtai mailing list [email protected] https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai