Re: i7 working for rtai-5.0-test1.tar.bz2

On 03/09/2016 05:33 PM, Hasan Jamal wrote:
> I did some debugging on user/latency and I found that it is hung on
> __ipipe_migrate_head
> The call chain is
> rt_make_hard_real_time -> steal_from_linux -> __ipipe_migrate_head
>
> So, it's some scheduling issue, how to move past this?
Is a thing I would like to know. If I recall it well, it has been 
signalled another time.
I'm not able to reproduce it and following the code in the patch I find 
nothing wrong. Other users close to me has been asked a few helping 
checks but none failed in using user space. So I'm helpless now.
If It happened to me I would have followed the path in the kernel with a 
few printk till the real locking point.
That it is likely that seeing it I would have discovered were my logic 
was wrong.
The only suggestion is to try with the other two version available in 
VULCANO CVS, 3.16 and 3.18.
Paolo
> -----Original Message-----
> From: Hasan Jamal [mailto:[email protected]]
> Sent: March-09-16 9:26 AM
> To: [email protected]
> Subject: FW: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>
> Send to the list as suggested by Paolo, the nice guy.
>
> -----Original Message-----
> From: Paolo Mantegazza [mailto:[email protected]]
> Sent: March-09-16 9:14 AM
> To: Hasan Jamal
> Subject: Re: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>
> The
> On 03/09/2016 02:58 PM, Hasan Jamal wrote:
>> After overnight test, I'd say kernel and RTAI modules are working
>> properly for 3.14.39. But, It looks RTAI modules have subtle bugs
>> during module load/unload, may need some flushing of cache? My next
>> job is to get user/latency work, I'd like to load modules manually for
>> it, what modules does it need?
> Hal+sched+sem+mbx+msg, while kernel space needs hal+sched+fifos. It is
> an easy information you can gather from the files "runinfo" in "testsuite"
> directories.
> Your problem is likely one of cold caches, to be warmed up. In my view also
> ovl_max maight be a bit high, but if it is generated just in the first
> records then that will confirm my assumption.
> The latency.c code I sent you has just a few lines more to check what
> happens after having warmed the cache up. You should try it, porting the
> same checks into the kernel test.
> BTW: keep this thread on the RTAI list. I sent you the code privately not to
> cut other RTAI users off, but just not to bother them with an attachment. I
> would like other people can learn something through the list discussions.
> Paolo.
>> Following is the output of overnight test with intense load
>>
>> RTH|    lat min|    ovl min|    lat avg|    lat max|    ovl max|
> overruns
>> RTD|        263|        112|        315|       3104|      47942|
> 0
>> RTD|        262|        112|        314|       1277|      47942|
> 0
>> RTD|        260|        112|        313|       1450|      47942|
> 0
>> RTD|        262|        112|        313|       1130|      47942|
> 0
>> RTD|        262|        112|        314|       3474|      47942|
> 0
>> RTD|        261|        112|        314|       1126|      47942|
> 0
>> RTD|        264|        112|        315|       2890|      47942|
> 0
>> RTD|        262|        112|        314|       1455|      47942|
> 0
>> RTD|        215|        112|        314|       1285|      47942|
> 0
>> RTD|        263|        112|        314|       1310|      47942|
> 0
>> RTD|        262|        112|        315|       3857|      47942|
> 0
>> RTD|        247|        112|        313|       1058|      47942|
> 0
>> RTD|        220|        112|        315|       2209|      47942|
> 0
>> RTD|        263|        112|        315|       1829|      47942|
> 0
>> RTD|        179|        112|        314|       1682|      47942|
> 0
>> RTD|        262|        112|        313|       1788|      47942|
> 0
>> RTD|        262|        112|        314|       3287|      47942|
> 0
>> RTD|        158|        112|        314|       1328|      47942|
> 0
>> RTD|        262|        112|        314|       1417|      47942|
> 0
>> RTD|        262|        112|        315|       1286|      47942|
> 0
>> ^CRTD|        262|        112|        315|       1286|      47942|
>> 0
>>
>> -----Original Message-----
>> From: Hasan Jamal [mailto:[email protected]]
>> Sent: March-08-16 11:14 PM
>> To: 'Paolo Mantegazza'
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> Looks like 3.14.39 is working. What I found is that I always get 0
>> overruns after reboot, but if start/stop /kern/latency then overruns
>> number increase randomly. So, I manually loaded all needed modules for
>> kern/latency and if I start/stop latency test then overrun number
>> never increases under default load. I'm running now with intense load
>> and will post result tomorrow morning.
>>
>> Thanks,
>> Hasan
>> -----Original Message-----
>> From: Hasan Jamal [mailto:[email protected]]
>> Sent: March-08-16 6:58 PM
>> To: 'Paolo Mantegazza'
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> I spent days getting Linux-3.16/RTAI4.1 built but went nowhere.
>> So, I decided the latest but it is also not working properly.
>> I'm building now for 3.14.39.
>>
>> I've to get it working by tomorrow, promised to the customer.
>>
>> I'm using the same system where old Linux-2.6.24 with RTAI works, now
>> the customer needs upgrade together other stuff like applications.
>>
>> The minimum requirement is Linux-3.11 from customer.
>>
>> I appreciate your help, any idea to make progress?
>>
>> Thanks,
>> Hasan
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: March-08-16 6:42 PM
>> To: Hasan Jamal
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> Take into account that you have to wait a few seconds before seeing
>> something on the screen.
>> If user space does not work then there is something strange.
>>
>> Paolo
>> ________________________________________
>> From: Hasan Jamal [[email protected]]
>> Sent: Wednesday, March 9, 2016 12:21 AM
>> To: Paolo Mantegazza
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> This does not update table, looks like hung. I was using kern/latency.
>> If kern/latency somehow works then why not user/latency?
>>
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: March-08-16 6:13 PM
>> To: Hasan Jamal
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> User
>> ________________________________________
>> From: Hasan Jamal [[email protected]]
>> Sent: Wednesday, March 9, 2016 12:08 AM
>> To: Paolo Mantegazza
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> OK, got it; is this user or kern? I just want to make sure even though
>> you mentioned 'user' previously.
>>
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: March-08-16 6:02 PM
>> To: Hasan Jamal
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> Now you should. Paolo
>> ________________________________________
>> From: Hasan Jamal [[email protected]]
>> Sent: Tuesday, March 8, 2016 11:37 PM
>> To: Paolo Mantegazza
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> I didn't get the attachment.
>>
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: March-08-16 5:35 PM
>> To: Hasan Jamal
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> Please overwrite the attached latency.c in the RTAI subdirectory:
>> "testsuite/user/latency".
>> Then rerun make install from there, go and execute latency again.
>> Report if anything changed to the RTAI list.
>> Thanks, Paolo.
>>
>> ________________________________________
>> From: Hasan Jamal [[email protected]]
>> Sent: Tuesday, March 8, 2016 10:50 PM
>> To: Paolo Mantegazza
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> Running your script of 'make' and will post result an hour later.
>>
>> -----Original Message-----
>> From: Hasan Jamal [mailto:[email protected]]
>> Sent: March-08-16 4:42 PM
>> To: 'Paolo Mantegazza'; '[email protected]'
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> OK, got it. It won't make any difference than what I'm doing.
>> Following is the output after running for 50mins
>>
>> RTD|        270|          0|        321|       1696|      46086|
> 26
>> RTD|        270|          0|        320|       1852|      46086|
> 26
>> RTH|    lat min|    ovl min|    lat avg|    lat max|    ovl max|
> overruns
>> RTD|        270|          0|        321|       2051|      46086|
> 26
>> RTD|        265|          0|        323|       2965|      46086|
> 26
>> Following is 'pas aux' output, ' heap_array' is using 100% of both CPU.
>>
>> root      2549  0.0  0.0   4448   776 pts/14   S+   14:46   0:00 /bin/sh
>> ./run
>> root      2550  0.0  0.0   4448  1640 pts/14   S+   14:46   0:00 /bin/sh
>> /usr/realtime/bin/rtai-load
>> root      2582  0.0  0.0   4448  1328 pts/14   S+   14:46   0:00 /bin/sh
>> /usr/realtime/bin/rtai-load
>> root      2623  0.0  0.0      0     0 ?        S    14:46   0:00
>> [RTAI_KTHRD_SRVR]
>> root      2659  0.0  0.0   4336   624 pts/14   S+   14:46   0:00 ./display
>> root      2661  100  5.2 209000 205752 pts/6   R    14:47  47:59
>> ./heap_array
>> root      2662 99.5  5.2 209000 205688 pts/6   R    14:47  47:45
>> ./heap_array
>> root      2679  0.0  0.0  22648  2620 pts/6    R+   15:35   0:00 ps aux
>>
>> Following is the output of last few lines of 'dmesg'
>>
>> [  164.259240] I-pipe: head domain RTAI registered.
>> [  164.259244] RTAI[hal]: mounted. ISOL_CPUS_MASK: 0, LINUX CPU
>> ISOLATED
>> MAP: 0).
>> [  164.259248] SYSINFO - # CPUs: 2, TIMER NAME: 'lapic', TIMER IRQ:
>> 16641, TIMER FREQ: 68593002, CLOCK NAME: 'tsc', CLOCK FREQ: 2194981000,
> CPU FREQ:
>> 2194981000, LINUX TIMER IRQ: 16641.
>> [  177.017478] RTAI[malloc]: global heap size = 2097152 bytes, <BSD>.
>> [  177.017521] , kstacks pool size = 524288 bytes.
>> [  177.017524] RTAI[sched]: hard timer type/freq = lapic/68593002(Hz);
>> timing: oneshot; linear timed lists.
>> [  177.017526] RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase freq
>> =
>> 2194981000 hz.
>> [  177.017528] RTAI[sched]: timer setup = 124 ns, resched latency = 0 ns.
>> [  191.216621]
>> [  191.216621]
>> [  191.216621] CPU USE SUMMARY
>> [  191.216627] # 0 -> 140730
>> [  191.216628] # 1 -> 0
>> [  191.216629] END OF CPU USE SUMMARY
>> [  191.216629]
>> [  191.245947] SCHED releases registered named TASK THRSRV [
>> 191.246059]
>> RTAI[malloc]: unloaded.
>> [  191.342819] RTAI[sched]: unloaded (forced hard/soft/hard transitions:
>> traps 0, syscalls 0).
>> [ 1593.438447] RTAI[malloc]: global heap size = 2097152 bytes, <BSD>.
>> [ 1593.438491] , kstacks pool size = 524288 bytes.
>> [ 1593.438494] RTAI[sched]: hard timer type/freq = lapic/68593002(Hz);
>> timing: oneshot; linear timed lists.
>> [ 1593.438496] RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase freq
>> =
>> 2194981000 hz.
>> [ 1593.438498] RTAI[sched]: timer setup = 124 ns, resched latency = 0 ns.
>> [ 1604.719986]
>> [ 1604.719986]
>> [ 1604.719986] CPU USE SUMMARY
>> [ 1604.719992] # 0 -> 112287
>> [ 1604.719993] # 1 -> 0
>> [ 1604.719994] END OF CPU USE SUMMARY
>> [ 1604.719994]
>> [ 1604.734869] SCHED releases registered named TASK THRSRV [
>> 1604.734973]
>> RTAI[malloc]: unloaded.
>> [ 1604.831806] RTAI[sched]: unloaded (forced hard/soft/hard transitions:
>> traps 0, syscalls 0).
>> [ 1606.783388] RTAI[malloc]: global heap size = 2097152 bytes, <BSD>.
>> [ 1606.783432] , kstacks pool size = 524288 bytes.
>> [ 1606.783436] RTAI[sched]: hard timer type/freq = lapic/68593002(Hz);
>> timing: oneshot; linear timed lists.
>> [ 1606.783438] RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase freq
>> =
>> 2194981000 hz.
>> [ 1606.783439] RTAI[sched]: timer setup = 124 ns, resched latency = 0 ns.
>> [ 1614.699614]
>> [ 1614.699614]
>> [ 1614.699614] CPU USE SUMMARY
>> [ 1614.699619] # 0 -> 78685
>> [ 1614.699620] # 1 -> 0
>> [ 1614.699622] END OF CPU USE SUMMARY
>> [ 1614.699622]
>> [ 1614.724600] SCHED releases registered named TASK THRSRV [
>> 1614.724704]
>> RTAI[malloc]: unloaded.
>> [ 1614.822239] RTAI[sched]: unloaded (forced hard/soft/hard transitions:
>> traps 0, syscalls 0).
>> [ 1615.948754] RTAI[malloc]: global heap size = 2097152 bytes, <BSD>.
>> [ 1615.948798] , kstacks pool size = 524288 bytes.
>> [ 1615.948801] RTAI[sched]: hard timer type/freq = lapic/68593002(Hz);
>> timing: oneshot; linear timed lists.
>> [ 1615.948803] RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase freq
>> =
>> 2194981000 hz.
>> [ 1615.948805] RTAI[sched]: timer setup = 124 ns, resched latency = 0 ns.
>> [ 1620.153990]
>> [ 1620.153990]
>> [ 1620.153990] CPU USE SUMMARY
>> [ 1620.153995] # 0 -> 41611
>> [ 1620.153997] # 1 -> 0
>> [ 1620.153998] END OF CPU USE SUMMARY
>> [ 1620.153998]
>> [ 1620.180231] SCHED releases registered named TASK THRSRV [
>> 1620.180351]
>> RTAI[malloc]: unloaded.
>> [ 1620.279942] RTAI[sched]: unloaded (forced hard/soft/hard transitions:
>> traps 0, syscalls 0).
>> [ 1621.160235] RTAI[malloc]: global heap size = 2097152 bytes, <BSD>.
>> [ 1621.160280] , kstacks pool size = 524288 bytes.
>> [ 1621.160283] RTAI[sched]: hard timer type/freq = lapic/68593002(Hz);
>> timing: oneshot; linear timed lists.
>> [ 1621.160285] RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase freq
>> =
>> 2194981000 hz.
>> [ 1621.160287] RTAI[sched]: timer setup = 125 ns, resched latency = 0 ns.
>> [ 2563.057731] perf interrupt took too long (2511 > 2500), lowering
>> kernel.perf_event_max_sample_rate to 50000
>>
>> Let me know whether any other info that would help.
>>
>> Thanks,
>> Hasan
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: March-08-16 4:29 PM
>> To: Hasan Jamal; [email protected]
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> I mean entering the Linux directory and:
>> while "true"; do make clean; make -j<all_my_cpus>; done Paolo
>> ________________________________________
>> From: Hasan Jamal [[email protected]]
>> Sent: Tuesday, March 8, 2016 9:37 PM
>> To: Paolo Mantegazza; [email protected]
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> I made my own version of application, that writes  to 125 MB of RAM
>> continuously and ran a few instances of the same application, 'ps aux'
>> showed full CPU usage.
>> I'll follow your suggestion for loading OS using ping, not sure what
>> you meant for "keept making the kernel and cleaning it"?
>>
>> Thanks,
>> Hasan
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: March-08-16 3:27 PM
>> To: Hasan Jamal; [email protected]
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> How long did the test last? There was intense Linux load?
>> Take into account that you should test for a hour or twq, at least. To
>> create a somewhat intense Linux load in run "ping -f somewher" and
>> keept making the kernel and cleaning it.
>> Then report the final rows of the latency test.
>> Paolo
>> ________________________________________
>> From: Hasan Jamal [[email protected]]
>> Sent: Tuesday, March 8, 2016 8:56 PM
>> To: Paolo Mantegazza; [email protected]
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>>> BTW, have you disabled CPU frequency scaling (CPU throttling)
>>> BIOS/Linux
>> config?
>> No, I did not but built with disabled CPU frequency scaling does same.
>>
>> Following is the result after overnight test with no load but default
>> services running.
>>
>> RTH|    lat min|    ovl min|    lat avg|    lat max|    ovl max|
> overruns
>> RTD|       1304|          0|       1815|       3118|      43940|
> 7
>> RTD|        974|          0|       1823|       3900|      43940|
> 7
>> If I stop and start 'run' then overruns changes but the number does
>> not change over time, it seems that is somehow locked.
>>
>> Following is the output with intense load (disabled CPU frequency
>> scaling), again overrun number don't change from start.
>>
>> RTH|    lat min|    ovl min|    lat avg|    lat max|    ovl max|
> overruns
>> RTD|        267|          0|        321|       1195|       7753|
> 10
>> RTD|        267|          0|        322|       1166|       7753|
> 10
>> May I accept this result as good? Or some bugs are in RTAI?
>>
>> Thanks,
>> Hasan
>>
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: March-07-16 6:06 PM
>> To: Hasan Jamal; [email protected]
>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> It means that there have been 7 cycles that missed the deadline.
>> Somewhat strange, since latency has aperiod of 100000 ns, which is not
>> overtaken by a max overall latency of 32043 ns.
>> Let it run longer, under intense Linux load, and see what happens.
>> BTW, have you disabled CPU frequency scaling (CPU throttling)
>> BIOS/Linux config?
> Paolo. ________________________________________ From: Rtai
> [[email protected]] on behalf of Hasan Jamal [[email protected]]
> Sent: Monday, March 7, 2016 11:47 PM To:
> [email protected] Subject: [Rtai] i7 working for rtai-5.0-test1.tar.bz2 I've got
> latest RTAI5.0 working Linux-3.18.22. I'm wondering about overruns in
> latency test, are they normal? RTD| 1339| 0| 1837| 7611| 32043| 7
> RTD| 979| 0| 1818| 5484| 32043| 7 RTH| lat min| ovl min| lat avg| lat
> max| ovl max| overruns RTD| 1414| 0| 1839| 2742| 32043| 7 RTD| 979| 0|
> 1819| 27438| 32043| 7 RTD| 1780| 0| 1824| 10873| 32043| 7 RTD| 974| 0|
> 1807| 11571| 32043| 7 RTD| 1570| 0| 1842| 10585| 32043| 7 RTD| 979| 0|
> 1835| 22838| 32043| 7 RTD| 1730| 0| 1814| 2773| 32043| 7 RTD| 984| 0|
> 1830| 16463| 32043| 7 RTD| 1720| 0| 1835| 11337| 32043| 7 RTD| 984| 0|
> 1822| 21159| 32043| 7
>
>
>
> _______________________________________________
> Rtai mailing list
> [email protected]
> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai

_______________________________________________
Rtai mailing list
[email protected]
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai