Re: i7 working for rtai-5.0-test1.tar.bz2
Paolo Mantegazza <[email protected]>
| Newsgroups | gmane.linux.real-time.rtai |
|---|---|
| Message-ID | <VI1PR06MB14710A6B611B7F258A07610590B30@VI1PR06MB1471.eurprd06.prod.outlook.com> |
Let me see if I understood. You did something like:
.
-
sched_submit_work(p);
printk("before __schedule\n");
if (likely(__schedule())) {
printk("after __schedule\n");
return 0;
}
.
.
and never see the output of the second printk?
Paolo
________________________________________
From: Hasan Jamal [[email protected]]
Sent: Wednesday, March 9, 2016 7:17 PM
To: Paolo Mantegazza; [email protected]
Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
I've tried almost all of 3.16 and 3.18 from your repo in the last few days.
So far, 3.14.39 is promising potential.
I did printk on __ipipe_migrate_head and found it is hung on 'if
(likely(__schedule()))'
Using printk in __schedule will fill kernel buffer, how to narrow down more?
I'd like to see the bottom of the issue and have a solution.
Thanks for the help.
-----Original Message-----
From: Paolo Mantegazza [mailto:[email protected]]
Sent: March-09-16 12:08 PM
To: Hasan Jamal; [email protected]
Subject: Re: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
On 03/09/2016 05:33 PM, Hasan Jamal wrote:
> I did some debugging on user/latency and I found that it is hung on
> __ipipe_migrate_head The call chain is rt_make_hard_real_time ->
> steal_from_linux -> __ipipe_migrate_head
>
> So, it's some scheduling issue, how to move past this?
Is a thing I would like to know. If I recall it well, it has been signalled
another time.
I'm not able to reproduce it and following the code in the patch I find
nothing wrong. Other users close to me has been asked a few helping checks
but none failed in using user space. So I'm helpless now.
If It happened to me I would have followed the path in the kernel with a few
printk till the real locking point.
That it is likely that seeing it I would have discovered were my logic was
wrong.
The only suggestion is to try with the other two version available in
VULCANO CVS, 3.16 and 3.18.
Paolo
> -----Original Message-----
> From: Hasan Jamal [mailto:[email protected]]
> Sent: March-09-16 9:26 AM
> To: [email protected]
> Subject: FW: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>
> Send to the list as suggested by Paolo, the nice guy.
>
> -----Original Message-----
> From: Paolo Mantegazza [mailto:[email protected]]
> Sent: March-09-16 9:14 AM
> To: Hasan Jamal
> Subject: Re: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>
> The
> On 03/09/2016 02:58 PM, Hasan Jamal wrote:
>> After overnight test, I'd say kernel and RTAI modules are working
>> properly for 3.14.39. But, It looks RTAI modules have subtle bugs
>> during module load/unload, may need some flushing of cache? My next
>> job is to get user/latency work, I'd like to load modules manually
>> for it, what modules does it need?
> Hal+sched+sem+mbx+msg, while kernel space needs hal+sched+fifos. It is
> an easy information you can gather from the files "runinfo" in "testsuite"
> directories.
> Your problem is likely one of cold caches, to be warmed up. In my view
> also ovl_max maight be a bit high, but if it is generated just in the
> first records then that will confirm my assumption.
> The latency.c code I sent you has just a few lines more to check what
> happens after having warmed the cache up. You should try it, porting
> the same checks into the kernel test.
> BTW: keep this thread on the RTAI list. I sent you the code privately
> not to cut other RTAI users off, but just not to bother them with an
> attachment. I would like other people can learn something through the list
discussions.
> Paolo.
>> Following is the output of overnight test with intense load
>>
>> RTH| lat min| ovl min| lat avg| lat max| ovl max|
> overruns
>> RTD| 263| 112| 315| 3104| 47942|
> 0
>> RTD| 262| 112| 314| 1277| 47942|
> 0
>> RTD| 260| 112| 313| 1450| 47942|
> 0
>> RTD| 262| 112| 313| 1130| 47942|
> 0
>> RTD| 262| 112| 314| 3474| 47942|
> 0
>> RTD| 261| 112| 314| 1126| 47942|
> 0
>> RTD| 264| 112| 315| 2890| 47942|
> 0
>> RTD| 262| 112| 314| 1455| 47942|
> 0
>> RTD| 215| 112| 314| 1285| 47942|
> 0
>> RTD| 263| 112| 314| 1310| 47942|
> 0
>> RTD| 262| 112| 315| 3857| 47942|
> 0
>> RTD| 247| 112| 313| 1058| 47942|
> 0
>> RTD| 220| 112| 315| 2209| 47942|
> 0
>> RTD| 263| 112| 315| 1829| 47942|
> 0
>> RTD| 179| 112| 314| 1682| 47942|
> 0
>> RTD| 262| 112| 313| 1788| 47942|
> 0
>> RTD| 262| 112| 314| 3287| 47942|
> 0
>> RTD| 158| 112| 314| 1328| 47942|
> 0
>> RTD| 262| 112| 314| 1417| 47942|
> 0
>> RTD| 262| 112| 315| 1286| 47942|
> 0
>> ^CRTD| 262| 112| 315| 1286| 47942|
>> 0
>>
>> -----Original Message-----
>> From: Hasan Jamal [mailto:[email protected]]
>> Sent: March-08-16 11:14 PM
>> To: 'Paolo Mantegazza'
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> Looks like 3.14.39 is working. What I found is that I always get 0
>> overruns after reboot, but if start/stop /kern/latency then overruns
>> number increase randomly. So, I manually loaded all needed modules
>> for kern/latency and if I start/stop latency test then overrun number
>> never increases under default load. I'm running now with intense load
>> and will post result tomorrow morning.
>>
>> Thanks,
>> Hasan
>> -----Original Message-----
>> From: Hasan Jamal [mailto:[email protected]]
>> Sent: March-08-16 6:58 PM
>> To: 'Paolo Mantegazza'
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> I spent days getting Linux-3.16/RTAI4.1 built but went nowhere.
>> So, I decided the latest but it is also not working properly.
>> I'm building now for 3.14.39.
>>
>> I've to get it working by tomorrow, promised to the customer.
>>
>> I'm using the same system where old Linux-2.6.24 with RTAI works, now
>> the customer needs upgrade together other stuff like applications.
>>
>> The minimum requirement is Linux-3.11 from customer.
>>
>> I appreciate your help, any idea to make progress?
>>
>> Thanks,
>> Hasan
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: March-08-16 6:42 PM
>> To: Hasan Jamal
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> Take into account that you have to wait a few seconds before seeing
>> something on the screen.
>> If user space does not work then there is something strange.
>>
>> Paolo
>> ________________________________________
>> From: Hasan Jamal [[email protected]]
>> Sent: Wednesday, March 9, 2016 12:21 AM
>> To: Paolo Mantegazza
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> This does not update table, looks like hung. I was using kern/latency.
>> If kern/latency somehow works then why not user/latency?
>>
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: March-08-16 6:13 PM
>> To: Hasan Jamal
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> User
>> ________________________________________
>> From: Hasan Jamal [[email protected]]
>> Sent: Wednesday, March 9, 2016 12:08 AM
>> To: Paolo Mantegazza
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> OK, got it; is this user or kern? I just want to make sure even
>> though you mentioned 'user' previously.
>>
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: March-08-16 6:02 PM
>> To: Hasan Jamal
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> Now you should. Paolo
>> ________________________________________
>> From: Hasan Jamal [[email protected]]
>> Sent: Tuesday, March 8, 2016 11:37 PM
>> To: Paolo Mantegazza
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> I didn't get the attachment.
>>
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: March-08-16 5:35 PM
>> To: Hasan Jamal
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> Please overwrite the attached latency.c in the RTAI subdirectory:
>> "testsuite/user/latency".
>> Then rerun make install from there, go and execute latency again.
>> Report if anything changed to the RTAI list.
>> Thanks, Paolo.
>>
>> ________________________________________
>> From: Hasan Jamal [[email protected]]
>> Sent: Tuesday, March 8, 2016 10:50 PM
>> To: Paolo Mantegazza
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> Running your script of 'make' and will post result an hour later.
>>
>> -----Original Message-----
>> From: Hasan Jamal [mailto:[email protected]]
>> Sent: March-08-16 4:42 PM
>> To: 'Paolo Mantegazza'; '[email protected]'
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> OK, got it. It won't make any difference than what I'm doing.
>> Following is the output after running for 50mins
>>
>> RTD| 270| 0| 321| 1696| 46086|
> 26
>> RTD| 270| 0| 320| 1852| 46086|
> 26
>> RTH| lat min| ovl min| lat avg| lat max| ovl max|
> overruns
>> RTD| 270| 0| 321| 2051| 46086|
> 26
>> RTD| 265| 0| 323| 2965| 46086|
> 26
>> Following is 'pas aux' output, ' heap_array' is using 100% of both CPU.
>>
>> root 2549 0.0 0.0 4448 776 pts/14 S+ 14:46 0:00 /bin/sh
>> ./run
>> root 2550 0.0 0.0 4448 1640 pts/14 S+ 14:46 0:00 /bin/sh
>> /usr/realtime/bin/rtai-load
>> root 2582 0.0 0.0 4448 1328 pts/14 S+ 14:46 0:00 /bin/sh
>> /usr/realtime/bin/rtai-load
>> root 2623 0.0 0.0 0 0 ? S 14:46 0:00
>> [RTAI_KTHRD_SRVR]
>> root 2659 0.0 0.0 4336 624 pts/14 S+ 14:46 0:00
./display
>> root 2661 100 5.2 209000 205752 pts/6 R 14:47 47:59
>> ./heap_array
>> root 2662 99.5 5.2 209000 205688 pts/6 R 14:47 47:45
>> ./heap_array
>> root 2679 0.0 0.0 22648 2620 pts/6 R+ 15:35 0:00 ps aux
>>
>> Following is the output of last few lines of 'dmesg'
>>
>> [ 164.259240] I-pipe: head domain RTAI registered.
>> [ 164.259244] RTAI[hal]: mounted. ISOL_CPUS_MASK: 0, LINUX CPU
>> ISOLATED
>> MAP: 0).
>> [ 164.259248] SYSINFO - # CPUs: 2, TIMER NAME: 'lapic', TIMER IRQ:
>> 16641, TIMER FREQ: 68593002, CLOCK NAME: 'tsc', CLOCK FREQ:
>> 2194981000,
> CPU FREQ:
>> 2194981000, LINUX TIMER IRQ: 16641.
>> [ 177.017478] RTAI[malloc]: global heap size = 2097152 bytes, <BSD>.
>> [ 177.017521] , kstacks pool size = 524288 bytes.
>> [ 177.017524] RTAI[sched]: hard timer type/freq =
>> lapic/68593002(Hz);
>> timing: oneshot; linear timed lists.
>> [ 177.017526] RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase
>> freq =
>> 2194981000 hz.
>> [ 177.017528] RTAI[sched]: timer setup = 124 ns, resched latency = 0 ns.
>> [ 191.216621]
>> [ 191.216621]
>> [ 191.216621] CPU USE SUMMARY
>> [ 191.216627] # 0 -> 140730
>> [ 191.216628] # 1 -> 0
>> [ 191.216629] END OF CPU USE SUMMARY [ 191.216629] [ 191.245947]
>> SCHED releases registered named TASK THRSRV [ 191.246059]
>> RTAI[malloc]: unloaded.
>> [ 191.342819] RTAI[sched]: unloaded (forced hard/soft/hard transitions:
>> traps 0, syscalls 0).
>> [ 1593.438447] RTAI[malloc]: global heap size = 2097152 bytes, <BSD>.
>> [ 1593.438491] , kstacks pool size = 524288 bytes.
>> [ 1593.438494] RTAI[sched]: hard timer type/freq =
>> lapic/68593002(Hz);
>> timing: oneshot; linear timed lists.
>> [ 1593.438496] RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase
>> freq =
>> 2194981000 hz.
>> [ 1593.438498] RTAI[sched]: timer setup = 124 ns, resched latency = 0 ns.
>> [ 1604.719986]
>> [ 1604.719986]
>> [ 1604.719986] CPU USE SUMMARY
>> [ 1604.719992] # 0 -> 112287
>> [ 1604.719993] # 1 -> 0
>> [ 1604.719994] END OF CPU USE SUMMARY [ 1604.719994] [ 1604.734869]
>> SCHED releases registered named TASK THRSRV [ 1604.734973]
>> RTAI[malloc]: unloaded.
>> [ 1604.831806] RTAI[sched]: unloaded (forced hard/soft/hard transitions:
>> traps 0, syscalls 0).
>> [ 1606.783388] RTAI[malloc]: global heap size = 2097152 bytes, <BSD>.
>> [ 1606.783432] , kstacks pool size = 524288 bytes.
>> [ 1606.783436] RTAI[sched]: hard timer type/freq =
>> lapic/68593002(Hz);
>> timing: oneshot; linear timed lists.
>> [ 1606.783438] RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase
>> freq =
>> 2194981000 hz.
>> [ 1606.783439] RTAI[sched]: timer setup = 124 ns, resched latency = 0 ns.
>> [ 1614.699614]
>> [ 1614.699614]
>> [ 1614.699614] CPU USE SUMMARY
>> [ 1614.699619] # 0 -> 78685
>> [ 1614.699620] # 1 -> 0
>> [ 1614.699622] END OF CPU USE SUMMARY [ 1614.699622] [ 1614.724600]
>> SCHED releases registered named TASK THRSRV [ 1614.724704]
>> RTAI[malloc]: unloaded.
>> [ 1614.822239] RTAI[sched]: unloaded (forced hard/soft/hard transitions:
>> traps 0, syscalls 0).
>> [ 1615.948754] RTAI[malloc]: global heap size = 2097152 bytes, <BSD>.
>> [ 1615.948798] , kstacks pool size = 524288 bytes.
>> [ 1615.948801] RTAI[sched]: hard timer type/freq =
>> lapic/68593002(Hz);
>> timing: oneshot; linear timed lists.
>> [ 1615.948803] RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase
>> freq =
>> 2194981000 hz.
>> [ 1615.948805] RTAI[sched]: timer setup = 124 ns, resched latency = 0 ns.
>> [ 1620.153990]
>> [ 1620.153990]
>> [ 1620.153990] CPU USE SUMMARY
>> [ 1620.153995] # 0 -> 41611
>> [ 1620.153997] # 1 -> 0
>> [ 1620.153998] END OF CPU USE SUMMARY [ 1620.153998] [ 1620.180231]
>> SCHED releases registered named TASK THRSRV [ 1620.180351]
>> RTAI[malloc]: unloaded.
>> [ 1620.279942] RTAI[sched]: unloaded (forced hard/soft/hard transitions:
>> traps 0, syscalls 0).
>> [ 1621.160235] RTAI[malloc]: global heap size = 2097152 bytes, <BSD>.
>> [ 1621.160280] , kstacks pool size = 524288 bytes.
>> [ 1621.160283] RTAI[sched]: hard timer type/freq =
>> lapic/68593002(Hz);
>> timing: oneshot; linear timed lists.
>> [ 1621.160285] RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase
>> freq =
>> 2194981000 hz.
>> [ 1621.160287] RTAI[sched]: timer setup = 125 ns, resched latency = 0 ns.
>> [ 2563.057731] perf interrupt took too long (2511 > 2500), lowering
>> kernel.perf_event_max_sample_rate to 50000
>>
>> Let me know whether any other info that would help.
>>
>> Thanks,
>> Hasan
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: March-08-16 4:29 PM
>> To: Hasan Jamal; [email protected]
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> I mean entering the Linux directory and:
>> while "true"; do make clean; make -j<all_my_cpus>; done Paolo
>> ________________________________________
>> From: Hasan Jamal [[email protected]]
>> Sent: Tuesday, March 8, 2016 9:37 PM
>> To: Paolo Mantegazza; [email protected]
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> I made my own version of application, that writes to 125 MB of RAM
>> continuously and ran a few instances of the same application, 'ps aux'
>> showed full CPU usage.
>> I'll follow your suggestion for loading OS using ping, not sure what
>> you meant for "keept making the kernel and cleaning it"?
>>
>> Thanks,
>> Hasan
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: March-08-16 3:27 PM
>> To: Hasan Jamal; [email protected]
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> How long did the test last? There was intense Linux load?
>> Take into account that you should test for a hour or twq, at least.
>> To create a somewhat intense Linux load in run "ping -f somewher" and
>> keept making the kernel and cleaning it.
>> Then report the final rows of the latency test.
>> Paolo
>> ________________________________________
>> From: Hasan Jamal [[email protected]]
>> Sent: Tuesday, March 8, 2016 8:56 PM
>> To: Paolo Mantegazza; [email protected]
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>>> BTW, have you disabled CPU frequency scaling (CPU throttling)
>>> BIOS/Linux
>> config?
>> No, I did not but built with disabled CPU frequency scaling does same.
>>
>> Following is the result after overnight test with no load but default
>> services running.
>>
>> RTH| lat min| ovl min| lat avg| lat max| ovl max|
> overruns
>> RTD| 1304| 0| 1815| 3118| 43940|
> 7
>> RTD| 974| 0| 1823| 3900| 43940|
> 7
>> If I stop and start 'run' then overruns changes but the number does
>> not change over time, it seems that is somehow locked.
>>
>> Following is the output with intense load (disabled CPU frequency
>> scaling), again overrun number don't change from start.
>>
>> RTH| lat min| ovl min| lat avg| lat max| ovl max|
> overruns
>> RTD| 267| 0| 321| 1195| 7753|
> 10
>> RTD| 267| 0| 322| 1166| 7753|
> 10
>> May I accept this result as good? Or some bugs are in RTAI?
>>
>> Thanks,
>> Hasan
>>
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: March-07-16 6:06 PM
>> To: Hasan Jamal; [email protected]
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> It means that there have been 7 cycles that missed the deadline.
>> Somewhat strange, since latency has aperiod of 100000 ns, which is
>> not overtaken by a max overall latency of 32043 ns.
>> Let it run longer, under intense Linux load, and see what happens.
>> BTW, have you disabled CPU frequency scaling (CPU throttling)
>> BIOS/Linux config?
> Paolo. ________________________________________ From: Rtai
> [[email protected]] on behalf of Hasan Jamal
> [[email protected]]
> Sent: Monday, March 7, 2016 11:47 PM To:
> [email protected] Subject: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
> I've got latest RTAI5.0 working Linux-3.18.22. I'm wondering about
> overruns in latency test, are they normal? RTD| 1339| 0| 1837| 7611|
> 32043| 7
> RTD| 979| 0| 1818| 5484| 32043| 7 RTH| lat min| ovl min| lat avg| lat
> max| ovl max| overruns RTD| 1414| 0| 1839| 2742| 32043| 7 RTD| 979| 0|
> 1819| 27438| 32043| 7 RTD| 1780| 0| 1824| 10873| 32043| 7 RTD| 974| 0|
> 1807| 11571| 32043| 7 RTD| 1570| 0| 1842| 10585| 32043| 7 RTD| 979| 0|
> 1835| 22838| 32043| 7 RTD| 1730| 0| 1814| 2773| 32043| 7 RTD| 984| 0|
> 1830| 16463| 32043| 7 RTD| 1720| 0| 1835| 11337| 32043| 7 RTD| 984| 0|
> 1822| 21159| 32043| 7
>
>
>
> _______________________________________________
> Rtai mailing list
> [email protected]
> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
_______________________________________________
Rtai mailing list
[email protected]
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai