Re: i7 working for rtai-5.0-test1.tar.bz2

Just for your info. To be sure the patch from VULCANO CVS was OK  I've 
reinstalled 3.14.39 on a couple of  PCs and see that the user latency 
test work as expected.
Even If I do not understand yet what is blocking a reschedule 
(__schedule) in stealing a task to linux, I'm interested in 
understanding what I do not now, eventually solving the problem.
Paolo.

On 03/09/2016 07:17 PM, Hasan Jamal wrote:
> I've tried almost all of 3.16 and 3.18 from your repo in the last few days.
> So far, 3.14.39 is promising potential.
>
> I did printk on __ipipe_migrate_head and found it is hung on 'if
> (likely(__schedule()))'
>
> Using printk in __schedule will fill kernel buffer, how to narrow down more?
> I'd like to see the bottom of the issue and have a solution.
>
> Thanks for the help.
>
> -----Original Message-----
> From: Paolo Mantegazza [mailto:[email protected]]
> Sent: March-09-16 12:08 PM
> To: Hasan Jamal; [email protected]
> Subject: Re: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>
> On 03/09/2016 05:33 PM, Hasan Jamal wrote:
>> I did some debugging on user/latency and I found that it is hung on
>> __ipipe_migrate_head The call chain is rt_make_hard_real_time ->
>> steal_from_linux -> __ipipe_migrate_head
>>
>> So, it's some scheduling issue, how to move past this?
> Is a thing I would like to know. If I recall it well, it has been signalled
> another time.
> I'm not able to reproduce it and following the code in the patch I find
> nothing wrong. Other users close to me has been asked a few helping checks
> but none failed in using user space. So I'm helpless now.
> If It happened to me I would have followed the path in the kernel with a few
> printk till the real locking point.
> That it is likely that seeing it I would have discovered were my logic was
> wrong.
> The only suggestion is to try with the other two version available in
> VULCANO CVS, 3.16 and 3.18.
> Paolo
>> -----Original Message-----
>> From: Hasan Jamal [mailto:[email protected]]
>> Sent: March-09-16 9:26 AM
>> To: [email protected]
>> Subject: FW: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> Send to the list as suggested by Paolo, the nice guy.
>>
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: March-09-16 9:14 AM
>> To: Hasan Jamal
>> Subject: Re: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>
>> The
>> On 03/09/2016 02:58 PM, Hasan Jamal wrote:
>>> After overnight test, I'd say kernel and RTAI modules are working
>>> properly for 3.14.39. But, It looks RTAI modules have subtle bugs
>>> during module load/unload, may need some flushing of cache? My next
>>> job is to get user/latency work, I'd like to load modules manually
>>> for it, what modules does it need?
>> Hal+sched+sem+mbx+msg, while kernel space needs hal+sched+fifos. It is
>> an easy information you can gather from the files "runinfo" in "testsuite"
>> directories.
>> Your problem is likely one of cold caches, to be warmed up. In my view
>> also ovl_max maight be a bit high, but if it is generated just in the
>> first records then that will confirm my assumption.
>> The latency.c code I sent you has just a few lines more to check what
>> happens after having warmed the cache up. You should try it, porting
>> the same checks into the kernel test.
>> BTW: keep this thread on the RTAI list. I sent you the code privately
>> not to cut other RTAI users off, but just not to bother them with an
>> attachment. I would like other people can learn something through the list
> discussions.
>> Paolo.
>>> Following is the output of overnight test with intense load
>>>
>>> RTH|    lat min|    ovl min|    lat avg|    lat max|    ovl max|
>> overruns
>>> RTD|        263|        112|        315|       3104|      47942|
>> 0
>>> RTD|        262|        112|        314|       1277|      47942|
>> 0
>>> RTD|        260|        112|        313|       1450|      47942|
>> 0
>>> RTD|        262|        112|        313|       1130|      47942|
>> 0
>>> RTD|        262|        112|        314|       3474|      47942|
>> 0
>>> RTD|        261|        112|        314|       1126|      47942|
>> 0
>>> RTD|        264|        112|        315|       2890|      47942|
>> 0
>>> RTD|        262|        112|        314|       1455|      47942|
>> 0
>>> RTD|        215|        112|        314|       1285|      47942|
>> 0
>>> RTD|        263|        112|        314|       1310|      47942|
>> 0
>>> RTD|        262|        112|        315|       3857|      47942|
>> 0
>>> RTD|        247|        112|        313|       1058|      47942|
>> 0
>>> RTD|        220|        112|        315|       2209|      47942|
>> 0
>>> RTD|        263|        112|        315|       1829|      47942|
>> 0
>>> RTD|        179|        112|        314|       1682|      47942|
>> 0
>>> RTD|        262|        112|        313|       1788|      47942|
>> 0
>>> RTD|        262|        112|        314|       3287|      47942|
>> 0
>>> RTD|        158|        112|        314|       1328|      47942|
>> 0
>>> RTD|        262|        112|        314|       1417|      47942|
>> 0
>>> RTD|        262|        112|        315|       1286|      47942|
>> 0
>>> ^CRTD|        262|        112|        315|       1286|      47942|
>>> 0
>>>
>>> -----Original Message-----
>>> From: Hasan Jamal [mailto:[email protected]]
>>> Sent: March-08-16 11:14 PM
>>> To: 'Paolo Mantegazza'
>>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>>
>>> Looks like 3.14.39 is working. What I found is that I always get 0
>>> overruns after reboot, but if start/stop /kern/latency then overruns
>>> number increase randomly. So, I manually loaded all needed modules
>>> for kern/latency and if I start/stop latency test then overrun number
>>> never increases under default load. I'm running now with intense load
>>> and will post result tomorrow morning.
>>>
>>> Thanks,
>>> Hasan
>>> -----Original Message-----
>>> From: Hasan Jamal [mailto:[email protected]]
>>> Sent: March-08-16 6:58 PM
>>> To: 'Paolo Mantegazza'
>>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>>
>>> I spent days getting Linux-3.16/RTAI4.1 built but went nowhere.
>>> So, I decided the latest but it is also not working properly.
>>> I'm building now for 3.14.39.
>>>
>>> I've to get it working by tomorrow, promised to the customer.
>>>
>>> I'm using the same system where old Linux-2.6.24 with RTAI works, now
>>> the customer needs upgrade together other stuff like applications.
>>>
>>> The minimum requirement is Linux-3.11 from customer.
>>>
>>> I appreciate your help, any idea to make progress?
>>>
>>> Thanks,
>>> Hasan
>>> -----Original Message-----
>>> From: Paolo Mantegazza [mailto:[email protected]]
>>> Sent: March-08-16 6:42 PM
>>> To: Hasan Jamal
>>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>>
>>> Take into account that you have to wait a few seconds before seeing
>>> something on the screen.
>>> If user space does not work then there is something strange.
>>>
>>> Paolo
>>> ________________________________________
>>> From: Hasan Jamal [[email protected]]
>>> Sent: Wednesday, March 9, 2016 12:21 AM
>>> To: Paolo Mantegazza
>>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>>
>>> This does not update table, looks like hung. I was using kern/latency.
>>> If kern/latency somehow works then why not user/latency?
>>>
>>> -----Original Message-----
>>> From: Paolo Mantegazza [mailto:[email protected]]
>>> Sent: March-08-16 6:13 PM
>>> To: Hasan Jamal
>>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>>
>>> User
>>> ________________________________________
>>> From: Hasan Jamal [[email protected]]
>>> Sent: Wednesday, March 9, 2016 12:08 AM
>>> To: Paolo Mantegazza
>>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>>
>>> OK, got it; is this user or kern? I just want to make sure even
>>> though you mentioned 'user' previously.
>>>
>>> -----Original Message-----
>>> From: Paolo Mantegazza [mailto:[email protected]]
>>> Sent: March-08-16 6:02 PM
>>> To: Hasan Jamal
>>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>>
>>> Now you should. Paolo
>>> ________________________________________
>>> From: Hasan Jamal [[email protected]]
>>> Sent: Tuesday, March 8, 2016 11:37 PM
>>> To: Paolo Mantegazza
>>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>>
>>> I didn't get the attachment.
>>>
>>> -----Original Message-----
>>> From: Paolo Mantegazza [mailto:[email protected]]
>>> Sent: March-08-16 5:35 PM
>>> To: Hasan Jamal
>>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>>
>>> Please overwrite the attached latency.c in the RTAI subdirectory:
>>> "testsuite/user/latency".
>>> Then rerun make install from there, go and execute latency again.
>>> Report if anything changed to the RTAI list.
>>> Thanks, Paolo.
>>>
>>> ________________________________________
>>> From: Hasan Jamal [[email protected]]
>>> Sent: Tuesday, March 8, 2016 10:50 PM
>>> To: Paolo Mantegazza
>>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>>
>>> Running your script of 'make' and will post result an hour later.
>>>
>>> -----Original Message-----
>>> From: Hasan Jamal [mailto:[email protected]]
>>> Sent: March-08-16 4:42 PM
>>> To: 'Paolo Mantegazza'; '[email protected]'
>>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>>
>>> OK, got it. It won't make any difference than what I'm doing.
>>> Following is the output after running for 50mins
>>>
>>> RTD|        270|          0|        321|       1696|      46086|
>> 26
>>> RTD|        270|          0|        320|       1852|      46086|
>> 26
>>> RTH|    lat min|    ovl min|    lat avg|    lat max|    ovl max|
>> overruns
>>> RTD|        270|          0|        321|       2051|      46086|
>> 26
>>> RTD|        265|          0|        323|       2965|      46086|
>> 26
>>> Following is 'pas aux' output, ' heap_array' is using 100% of both CPU.
>>>
>>> root      2549  0.0  0.0   4448   776 pts/14   S+   14:46   0:00 /bin/sh
>>> ./run
>>> root      2550  0.0  0.0   4448  1640 pts/14   S+   14:46   0:00 /bin/sh
>>> /usr/realtime/bin/rtai-load
>>> root      2582  0.0  0.0   4448  1328 pts/14   S+   14:46   0:00 /bin/sh
>>> /usr/realtime/bin/rtai-load
>>> root      2623  0.0  0.0      0     0 ?        S    14:46   0:00
>>> [RTAI_KTHRD_SRVR]
>>> root      2659  0.0  0.0   4336   624 pts/14   S+   14:46   0:00
> ./display
>>> root      2661  100  5.2 209000 205752 pts/6   R    14:47  47:59
>>> ./heap_array
>>> root      2662 99.5  5.2 209000 205688 pts/6   R    14:47  47:45
>>> ./heap_array
>>> root      2679  0.0  0.0  22648  2620 pts/6    R+   15:35   0:00 ps aux
>>>
>>> Following is the output of last few lines of 'dmesg'
>>>
>>> [  164.259240] I-pipe: head domain RTAI registered.
>>> [  164.259244] RTAI[hal]: mounted. ISOL_CPUS_MASK: 0, LINUX CPU
>>> ISOLATED
>>> MAP: 0).
>>> [  164.259248] SYSINFO - # CPUs: 2, TIMER NAME: 'lapic', TIMER IRQ:
>>> 16641, TIMER FREQ: 68593002, CLOCK NAME: 'tsc', CLOCK FREQ:
>>> 2194981000,
>> CPU FREQ:
>>> 2194981000, LINUX TIMER IRQ: 16641.
>>> [  177.017478] RTAI[malloc]: global heap size = 2097152 bytes, <BSD>.
>>> [  177.017521] , kstacks pool size = 524288 bytes.
>>> [  177.017524] RTAI[sched]: hard timer type/freq =
>>> lapic/68593002(Hz);
>>> timing: oneshot; linear timed lists.
>>> [  177.017526] RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase
>>> freq =
>>> 2194981000 hz.
>>> [  177.017528] RTAI[sched]: timer setup = 124 ns, resched latency = 0 ns.
>>> [  191.216621]
>>> [  191.216621]
>>> [  191.216621] CPU USE SUMMARY
>>> [  191.216627] # 0 -> 140730
>>> [  191.216628] # 1 -> 0
>>> [  191.216629] END OF CPU USE SUMMARY [  191.216629] [  191.245947]
>>> SCHED releases registered named TASK THRSRV [ 191.246059]
>>> RTAI[malloc]: unloaded.
>>> [  191.342819] RTAI[sched]: unloaded (forced hard/soft/hard transitions:
>>> traps 0, syscalls 0).
>>> [ 1593.438447] RTAI[malloc]: global heap size = 2097152 bytes, <BSD>.
>>> [ 1593.438491] , kstacks pool size = 524288 bytes.
>>> [ 1593.438494] RTAI[sched]: hard timer type/freq =
>>> lapic/68593002(Hz);
>>> timing: oneshot; linear timed lists.
>>> [ 1593.438496] RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase
>>> freq =
>>> 2194981000 hz.
>>> [ 1593.438498] RTAI[sched]: timer setup = 124 ns, resched latency = 0 ns.
>>> [ 1604.719986]
>>> [ 1604.719986]
>>> [ 1604.719986] CPU USE SUMMARY
>>> [ 1604.719992] # 0 -> 112287
>>> [ 1604.719993] # 1 -> 0
>>> [ 1604.719994] END OF CPU USE SUMMARY [ 1604.719994] [ 1604.734869]
>>> SCHED releases registered named TASK THRSRV [ 1604.734973]
>>> RTAI[malloc]: unloaded.
>>> [ 1604.831806] RTAI[sched]: unloaded (forced hard/soft/hard transitions:
>>> traps 0, syscalls 0).
>>> [ 1606.783388] RTAI[malloc]: global heap size = 2097152 bytes, <BSD>.
>>> [ 1606.783432] , kstacks pool size = 524288 bytes.
>>> [ 1606.783436] RTAI[sched]: hard timer type/freq =
>>> lapic/68593002(Hz);
>>> timing: oneshot; linear timed lists.
>>> [ 1606.783438] RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase
>>> freq =
>>> 2194981000 hz.
>>> [ 1606.783439] RTAI[sched]: timer setup = 124 ns, resched latency = 0 ns.
>>> [ 1614.699614]
>>> [ 1614.699614]
>>> [ 1614.699614] CPU USE SUMMARY
>>> [ 1614.699619] # 0 -> 78685
>>> [ 1614.699620] # 1 -> 0
>>> [ 1614.699622] END OF CPU USE SUMMARY [ 1614.699622] [ 1614.724600]
>>> SCHED releases registered named TASK THRSRV [ 1614.724704]
>>> RTAI[malloc]: unloaded.
>>> [ 1614.822239] RTAI[sched]: unloaded (forced hard/soft/hard transitions:
>>> traps 0, syscalls 0).
>>> [ 1615.948754] RTAI[malloc]: global heap size = 2097152 bytes, <BSD>.
>>> [ 1615.948798] , kstacks pool size = 524288 bytes.
>>> [ 1615.948801] RTAI[sched]: hard timer type/freq =
>>> lapic/68593002(Hz);
>>> timing: oneshot; linear timed lists.
>>> [ 1615.948803] RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase
>>> freq =
>>> 2194981000 hz.
>>> [ 1615.948805] RTAI[sched]: timer setup = 124 ns, resched latency = 0 ns.
>>> [ 1620.153990]
>>> [ 1620.153990]
>>> [ 1620.153990] CPU USE SUMMARY
>>> [ 1620.153995] # 0 -> 41611
>>> [ 1620.153997] # 1 -> 0
>>> [ 1620.153998] END OF CPU USE SUMMARY [ 1620.153998] [ 1620.180231]
>>> SCHED releases registered named TASK THRSRV [ 1620.180351]
>>> RTAI[malloc]: unloaded.
>>> [ 1620.279942] RTAI[sched]: unloaded (forced hard/soft/hard transitions:
>>> traps 0, syscalls 0).
>>> [ 1621.160235] RTAI[malloc]: global heap size = 2097152 bytes, <BSD>.
>>> [ 1621.160280] , kstacks pool size = 524288 bytes.
>>> [ 1621.160283] RTAI[sched]: hard timer type/freq =
>>> lapic/68593002(Hz);
>>> timing: oneshot; linear timed lists.
>>> [ 1621.160285] RTAI[sched]: Linux timer freq = 250 (Hz), TimeBase
>>> freq =
>>> 2194981000 hz.
>>> [ 1621.160287] RTAI[sched]: timer setup = 125 ns, resched latency = 0 ns.
>>> [ 2563.057731] perf interrupt took too long (2511 > 2500), lowering
>>> kernel.perf_event_max_sample_rate to 50000
>>>
>>> Let me know whether any other info that would help.
>>>
>>> Thanks,
>>> Hasan
>>> -----Original Message-----
>>> From: Paolo Mantegazza [mailto:[email protected]]
>>> Sent: March-08-16 4:29 PM
>>> To: Hasan Jamal; [email protected]
>>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>>
>>> I mean entering the Linux directory and:
>>> while "true"; do make clean; make -j<all_my_cpus>; done Paolo
>>> ________________________________________
>>> From: Hasan Jamal [[email protected]]
>>> Sent: Tuesday, March 8, 2016 9:37 PM
>>> To: Paolo Mantegazza; [email protected]
>>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>>
>>> I made my own version of application, that writes  to 125 MB of RAM
>>> continuously and ran a few instances of the same application, 'ps aux'
>>> showed full CPU usage.
>>> I'll follow your suggestion for loading OS using ping, not sure what
>>> you meant for "keept making the kernel and cleaning it"?
>>>
>>> Thanks,
>>> Hasan
>>> -----Original Message-----
>>> From: Paolo Mantegazza [mailto:[email protected]]
>>> Sent: March-08-16 3:27 PM
>>> To: Hasan Jamal; [email protected]
>>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>>
>>> How long did the test last? There was intense Linux load?
>>> Take into account that you should test for a hour or twq, at least.
>>> To create a somewhat intense Linux load in run "ping -f somewher" and
>>> keept making the kernel and cleaning it.
>>> Then report the final rows of the latency test.
>>> Paolo
>>> ________________________________________
>>> From: Hasan Jamal [[email protected]]
>>> Sent: Tuesday, March 8, 2016 8:56 PM
>>> To: Paolo Mantegazza; [email protected]
>>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>>
>>>> BTW, have you disabled CPU frequency scaling (CPU throttling)
>>>> BIOS/Linux
>>> config?
>>> No, I did not but built with disabled CPU frequency scaling does same.
>>>
>>> Following is the result after overnight test with no load but default
>>> services running.
>>>
>>> RTH|    lat min|    ovl min|    lat avg|    lat max|    ovl max|
>> overruns
>>> RTD|       1304|          0|       1815|       3118|      43940|
>> 7
>>> RTD|        974|          0|       1823|       3900|      43940|
>> 7
>>> If I stop and start 'run' then overruns changes but the number does
>>> not change over time, it seems that is somehow locked.
>>>
>>> Following is the output with intense load (disabled CPU frequency
>>> scaling), again overrun number don't change from start.
>>>
>>> RTH|    lat min|    ovl min|    lat avg|    lat max|    ovl max|
>> overruns
>>> RTD|        267|          0|        321|       1195|       7753|
>> 10
>>> RTD|        267|          0|        322|       1166|       7753|
>> 10
>>> May I accept this result as good? Or some bugs are in RTAI?
>>>
>>> Thanks,
>>> Hasan
>>>
>>> -----Original Message-----
>>> From: Paolo Mantegazza [mailto:[email protected]]
>>> Sent: March-07-16 6:06 PM
>>> To: Hasan Jamal; [email protected]
>>> Subject: RE: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>>>
>>> It means that there have been 7 cycles that missed the deadline.
>>> Somewhat strange, since latency has aperiod of 100000 ns, which is
>>> not overtaken by a max overall latency of 32043 ns.
>>> Let it run longer, under intense Linux load, and see what happens.
>>> BTW, have you disabled CPU frequency scaling (CPU throttling)
>>> BIOS/Linux config?
>> Paolo. ________________________________________ From: Rtai
>> [[email protected]] on behalf of Hasan Jamal
>> [[email protected]]
>> Sent: Monday, March 7, 2016 11:47 PM To:
>> [email protected] Subject: [Rtai] i7 working for rtai-5.0-test1.tar.bz2
>> I've got latest RTAI5.0 working Linux-3.18.22. I'm wondering about
>> overruns in latency test, are they normal? RTD| 1339| 0| 1837| 7611|
>> 32043| 7
>> RTD| 979| 0| 1818| 5484| 32043| 7 RTH| lat min| ovl min| lat avg| lat
>> max| ovl max| overruns RTD| 1414| 0| 1839| 2742| 32043| 7 RTD| 979| 0|
>> 1819| 27438| 32043| 7 RTD| 1780| 0| 1824| 10873| 32043| 7 RTD| 974| 0|
>> 1807| 11571| 32043| 7 RTD| 1570| 0| 1842| 10585| 32043| 7 RTD| 979| 0|
>> 1835| 22838| 32043| 7 RTD| 1730| 0| 1814| 2773| 32043| 7 RTD| 984| 0|
>> 1830| 16463| 32043| 7 RTD| 1720| 0| 1835| 11337| 32043| 7 RTD| 984| 0|
>> 1822| 21159| 32043| 7
>>
>>
>>
>> _______________________________________________
>> Rtai mailing list
>> [email protected]
>> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
>
>

_______________________________________________
Rtai mailing list
[email protected]
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai