Re: Problems with a new Ryzen-based PC

The fact that it works with a "Core i5-7500" is good new for me. The user memory access from the kernel for >= 6xxx INTEL CPUs has bean the main delay factor for a 5.1 RTAI release, which could have been done quite a bit of time ago. The related troubles being related to something I could have not dreamed without the help of a few RTAI users having >= 6xxx Intel CPUs. 
Your further check make us confident those troubles are fixed.

Paolo.

P.S: if your Ryzen system is going to gather just dust and just if it will be so, I dare asking you to gift it to us, to see if we can do something to fix it, without any obligation on both sides :-).

________________________________________
From: Duane AW. Jeffery [[email protected]]
Sent: Wednesday, January 17, 2018 10:23 PM
To: Paolo Mantegazza; '[email protected]'
Subject: RE: [Rtai] Problems with a new Ryzen-based PC

I received my new Core i5-7500 and Gigabyte H270M-D3H board - things are much happier here.  RTAI-5.1-test2 works and loads my modules with no issues and my code seems to be working.

We're having a stability issue with one piece of it but I believe it is unrelated to RTAI and it simply needs more processing horsepower to keep up (we're used to running on 6-core FX 6300, and this i5 has only 4 cores - Our program has 5 threads, all of which consume the majority of a core).  I'm in the middle of ordering a 6-core i5-8400 and a Z370-HD3P board to test.

Still not sure why the much cheaper Ryzen doesn't even run RTAI/comedi stably, but for now, I'm not too concerned about it since I have some workable hardware.

-Duane

-----Original Message-----
From: Paolo Mantegazza [mailto:[email protected]]
Sent: Wednesday, January 17, 2018 4:05 PM
To: Paolo Mantegazza <[email protected]>; Duane AW. Jeffery <[email protected]>; '[email protected]' <[email protected]>
Subject: RE: [Rtai] Problems with a new Ryzen-based PC

I've checked the COMEDI  ni_pccmio interrupt handler of my colleague. He has two of them, related to a DA card and to an AD one.
The latter is shared with two USB ports but he can use it with RTAII without any glitch, provided nothing is executed on those ports.
It is programmed for asynchronous AD input at 20 KHz and he sees no troubles.
So, without having my hands on the hardware, I'm sorry but at the moment  I've no further clue about your troubles.
Paolo
________________________________________
From: Rtai [[email protected]] on behalf of Paolo Mantegazza [[email protected]]
Sent: Friday, January 12, 2018 4:06 PM
To: Duane AW. Jeffery; '[email protected]'
Subject: Re: [Rtai] Problems with a new Ryzen-based PC

As a further remark I should say that I'm not sure that an interrupt handled by RTAI should appear in the Linux irqs list.
Was it so with the version it worked with?
Paolo

On 01/12/2018 03:10 PM, Duane AW. Jeffery wrote:
> Definitely reads 11-fasteoi on the the IRQ line.
>
> Should I be finding a "xx-fasteoi" designation?  This is all a bit over my head.
>
> Could this be because of some BIOS setting on my motherboard?
>
> -Duane
>
> -----Original Message-----
> From: Paolo Mantegazza [mailto:[email protected]]
> Sent: Friday, January 12, 2018 4:23 AM
> To: Duane AW. Jeffery <[email protected]>
> Subject: Re: [Rtai] Problems with a new Ryzen-based PC
>
> I was not able to read it on my home PC. OK now I can see it in my office.
> Nonetheless the fastio qualification is somewhat blurred. Is it "11-fastio" or "ll-fastio"
> In any case I never saw any "xx-fastio", did you?.  It is possible that the problem is related to it.
> Paolo
>
> On 01/11/2018 11:31 PM, Duane AW. Jeffery wrote:
>> I had attached a screen grab with cat /proc/interrupts reply
>>
>> -Duane
>>
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: Thursday, January 11, 2018 5:16 PM
>> To: Duane AW. Jeffery <[email protected]>
>> Subject: RE: [Rtai] Problems with a new Ryzen-based PC
>>
>> WHat does " cat /proc/interrupts/ report?
>> Paolo
>>
>> ________________________________________
>> From: Duane AW. Jeffery [[email protected]]
>> Sent: Thursday, January 11, 2018 9:56 PM
>> To: Paolo Mantegazza
>> Subject: RE: [Rtai] Problems with a new Ryzen-based PC
>>
>> In the thinking that it could be comedi related, I pulled a fresh set of comedi sources from github and tried recompiling them - no change.
>>
>> comedi_board_info produces the expected output about my board, and comedi_soft_calibrate produces sensible output for my board as well.  comedi_calibrate fails during some setup routine - this is as normal for our systems in past.
>>
>> Attached is a screen grab of my lsmod and /proc/interrupts.  It
>> appears that comedi has installed itself onto IRQ 35.  (this is with
>> my A/D card present and my kernel module not yet insmod'd)
>>
>> System definitely crashes on attempting to insmod my kernel module.  Same crash dump as a couple weeks ago if it happens in text mode.  If in X, then it just locks the system solid.
>>
>> On insmod'ing the RTAI modules, dmesg has a message that says Linux Timer IRQ: 4353.
>>
>> Any ideas what else I can try?
>>
>> -Duane
>>
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: Wednesday, December 27, 2017 12:08 PM
>> To: Duane AW. Jeffery <[email protected]>
>> Subject: RE: [Rtai] Problems with a new Ryzen-based PC
>>
>> It seem that it could be comedi related.
>> In that view take into account that RTAI is now based on the content of COMEDI gits, even if it is likely that the trouble you witness is not related to the COMEDI support you are using.
>>
>> You should try first to see what comedi support helper tool say, e.g. using comedi_board_info and comedi_calibrate, which should work independently from RTAI. I've gathered some dust in comedi usage, so you should first try looking at comedi docs for further help.
>>
>> At the moment the only suggestion I have is a possible conflict in the interrupt number, i.e. comedi uses an interrupt number already in use.
>>
>> Paolo
>> ________________________________________
>> From: Duane AW. Jeffery [[email protected]]
>> Sent: Wednesday, December 27, 2017 5:37 PM
>> To: Paolo Mantegazza; [email protected]
>> Subject: RE: [Rtai] Problems with a new Ryzen-based PC
>>
>> I'm happy to try any suggestions you have or go looking wherever might be useful.  I'm not sure even how to find whose handler might be causing the trouble - the system dies with the photographed kernel panic from December 21st on insmod of my module.
>>
>> I inherited this module from another developer I no longer have access to, so I'm not sure what interrupts it might be connected to.  It uses Comedi to retrieve A/D data from a National Instruments A/D card.  We have a separate stub version for testing that leaves out the comedi calls and it gets loaded when the system detects the A/D card isn't present on bootup.  This version does not crash the system on insmod.  So I'm guessing its related to the interrupts from the A/D card.
>>
>> -Duane
>>
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: Wednesday, December 27, 2017 11:30 AM
>> To: Duane AW. Jeffery <[email protected]>; [email protected]
>> Subject: RE: [Rtai] Problems with a new Ryzen-based PC
>>
>> Glad that kernel the posix support is fixed. The trouble was due to a
>> rewrite of the user space posix timer interface, whereas I forgot to
>> keep the compatibility with its kernel space counterpart. Thanks for
>> signaling it-
>>
>> As for the interrupt handler I need to know something more. Can you help  me understanding whose handler is causing the trouble?
>>
>> Paolo
>>
>> ________________________________________
>> From: Duane AW. Jeffery [[email protected]]
>> Sent: Wednesday, December 27, 2017 3:50 PM
>> To: Paolo Mantegazza; [email protected]
>> Subject: RE: [Rtai] Problems with a new Ryzen-based PC
>>
>> Contents of this machine's /usr/realtime/calibration/latencies file:
>>
>> 1221 1345 99999
>>
>> These latencies appear to be in-line with what I see when running the /usr/realtime/testsuite/kern/latency test, and latencies in the ~1.5 us range are within parameters for my application.
>>
>> Paolo, the fix you sent me seems to eliminate the compiler whining on #include <rtai_posix.h>.
>>
>> Not sure my app actually needs to include that file, since it compiled fine without it, but I put it back the way it originally was after installing your new tarball and the complaints stopped, so I think you squished that bug.
>>
>> Machine still kernel panics on inserting my module with the same interrupt handler complaint though.
>>
>> -Duane
>>
>> -----Original Message-----
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: Friday, December 22, 2017 12:44 PM
>> To: Duane AW. Jeffery <[email protected]>; [email protected]
>> Subject: RE: [Rtai] Problems with a new Ryzen-based PC
>>
>> RTAI calibrate kernel and user space latencies  the very first time it is run. The related results are written in a file store in the calibration directory of the RTAI install directory and are reused at any subsequent execution of an RTAI application.
>>    So you simply have to look for the existence of such a file and check if the latencies test show an average latency close to zero when run within a quiet environment.
>> Latency is a single line file containing the kernel and space latency and the period used for the calibration, in nanosecs.
>>
>> As for the rest of your problems  I'll have a look to the posix compilation problems ASAP, and see what I can do for the interrupt handler afterward.
>> Can you provide me a simple kernel test of yours for the posix timer case?
>>
>> What count more for me, at the moment, is to have it confirmed that the INTEL fixes are OK for recent AMDs also.
>>
>> Paolo
>>
>>
>> ________________________________________
>> From: Duane AW. Jeffery [[email protected]]
>> Sent: Friday, December 22, 2017 4:12 PM
>> To: Paolo Mantegazza; [email protected]
>> Subject: RE: [Rtai] Problems with a new Ryzen-based PC
>>
>> Yes, working in kernel space.  Our module RTAD uses rtai_math.
>>
>> I am able to run the kernel and user space latency tests without issue, but not sure about the latency calibration file - I've never needed to work with it before.  Where can I find it and how will I know its been created correctly?
>>
>> -Duane
>>
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: Friday, December 22, 2017 4:04 AM
>> To: Duane AW. Jeffery <[email protected]>; [email protected]
>> Subject: Re: [Rtai] Problems with a new Ryzen-based PC
>>
>> The problem is of another kind, i.e. interrupt related. What I would like to know first is if you can run the kernel/user space latency tests and if the latency calibration file is created correctly.
>> As for the timer the fix should not be a problem, but a small, stand alone, test case showing the problem will be of help.
>> Mine works, but from the use of rtai_math I can infer that you are working in kernel space. Is that correct?
>>
>> Paolo.
>>
>> On 12/21/2017 11:17 PM, Duane AW. Jeffery wrote:
>> Well, I tried the 5.1-test1, and it seems to fix that problem on rtai_sched insmod.
>>
>> It created a new problem where my RTAI kernel module portion of my software #include's <rtai_posix.h>, which made it fail to compile on rt_ptimer_create not matching its prototype in rtai_tasklets.h.
>>
>> I was able to compile my code without #include'ing <rtai_posix.h> to clear that error.
>>
>> Now I'm having another issue that appeared before when I insmod my kernel module, which uses comedi calls.  It seems to still be rtai_sched that blows up.
>>
>> Crash attached.
>>
>> -Duane
>>
>> From: Paolo Mantegazza [mailto:[email protected]]
>> Sent: Thursday, December 21, 2017 9:26 AM
>> To: Duane AW. Jeffery
>> <[email protected]><mailto:[email protected]>;
>> [email protected]<mailto:[email protected]>
>> Subject: Re: [Rtai] Problems with a new Ryzen-based PC
>>
>> On 12/21/2017 03:18 PM, Duane AW. Jeffery wrote:
>> I am indeed not.  I'm running probably the RTAI 5.0-test2 package - Its whatever RTAI was at when 4.1.18 was added to HAL patch support.  Generally speaking I get the system working and then don't update anything after that until forced to.
>>
>> Should I try RTAI 5.1-test1?  Is there a fix related to rt_lxrt_invoke in 5.1-test1?
>> I expect so. Here is an explanation I gave to somebody else:
>>
>>
>>
>>
>> What I pointed out to the list is the problem of the argument passing from user to kernel space, where what has worked unchanged for almost 2 decades, not only on x86 archs, failed to run on recent Ix-6xxx INTEL. Here it is how it is explained in the configuration help:
>>
>> config RTAI_USE_STACK_ARGS
>>
>>           bool "Keep using RTAI way for user-kernel space on stack args exchange."
>>
>>           default n
>>
>>           help
>>
>>           When it is sure that RTAI syscalls args, to be copied
>> between user-kernel
>>
>>           space, are on the stack, RTAI uses kernel space memcopy,
>> instead of the
>>
>>           various canonical copy from-to user. Such a scheme, quite
>> simple and
>>
>>           effective, has worked for almost two decades without any glitch.
>>
>>           Unfortunately, recent CPUs, likely from Ix-6xxx onward,
>> segfault with it,
>>
>>           forcing RTAI to comply with the canonical way.
>>
>>           The related trouble can be easily spotted at the very first
>> rtai_sched.ko
>>
>>           usage, i.e. when the very first latency calibration is
>> carried out (see
>>
>>           the related parameter helps), when the core dump summary,
>> as read with dmesg,
>>
>>           shows a segfault at lxrt_intercept_syscall.
>>
>>           Nonetheless, such a way can still be a useful choice and
>> may not cause any
>>
>>           trouble with recent ATOMs and CELERONs.
>>
>>           Thus, this parameter, defaulting to a safer no, lets the
>> user have a choice
>>
>>           anyhow.
>>
>> The problem may have surfaced some time before but I did not had a hardware to verify it and remained unexplained to me till a user, after upgrading his hardware to an I5-6500 and seeing the failure of his RTAI application, which run unpaired for quite a bit of time, installed the disk of the running hardware onto the new one and saw it failed to work. Fortunately he was willing to help and we fixed the problem in a short time, As I said, I could not see it because I was using a not so new hardware release. Now I've one and I verified first hand that RTAI-5.1 works on it.
>>
>> The failure was of the type you mentioned, albeit on an Intel cpu. Yours is AMD, but I dare hoping the problem could be the same.
>>
>> Paolo.
>>
>>
>>
>>
>> -Duane
>>
>> From: Rtai [mailto:[email protected]] On Behalf Of Paolo
>> Mantegazza
>> Sent: Tuesday, December 19, 2017 5:36 AM
>> To: [email protected]<mailto:[email protected]>
>> Subject: Re: [Rtai] Problems with a new Ryzen-based PC
>>
>> May I bet that you are not using the latest rtai-5.1-test1? My bet is based on the fact that it fail at rt_lxrt_invoke.
>>
>> Paolo.
>>
>> On 12/18/2017 09:13 PM, Duane AW. Jeffery wrote:
>> Hi all,
>>
>> It's been a while since I wrote to the list, but I'm back with a new problem.  Once again the hardware we use has reached end-of-life, and I'm left to try and get our RTAI system working on some new modern hardware.
>>
>> New system:
>>
>> AMD Ryzen 3 1200
>> Gigabyte A320M-HD2
>> 1TB WD Blue
>> MSI Radeon HD 6450
>>
>> Installing our usual software (Linux 4.1.18, RTAI... 5? I think) on it mostly works, except that our RT-enabled programs freeze the system (mouse and keyboard unresponsive, display frozen) on attempting to start them, and I notice during bootup where we insmod the various RTAI modules that on insmod of rtai_sched, I get a kernel bug:
>>
>> BUG: unable to handle kernel paging request at 00007ffde855f1e0
>>
>> Attached is a photo of the full kernel bug readout.
>>
>> I tried recompiling the kernel, RTAI, and our software on the new machine to no avail.   Manually running the latency tests in the testsuite appears to work and give reasonable latencies.
>>
>> Anybody have any idea where things are falling down?
>>
>> -Duane Jeffery
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>>
>> Rtai mailing list
>>
>> [email protected]<mailto:[email protected]>
>>
>> https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
>>
>>
>>
>>
>>
>>
>>

_______________________________________________
Rtai mailing list
[email protected]
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
_______________________________________________
Rtai mailing list
[email protected]
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai