FW: Problems with a new Ryzen-based PC

-----Original Message-----
From: Duane AW. Jeffery 
Sent: Thursday, January 11, 2018 3:54 PM
To: 'Paolo Mantegazza' <[email protected]>
Subject: RE: [Rtai] Problems with a new Ryzen-based PC

In the thinking that it could be comedi related, I pulled a fresh set of comedi sources from github and tried recompiling them - no change.

comedi_board_info produces the expected output about my board, and comedi_soft_calibrate produces sensible output for my board as well.  comedi_calibrate fails during some setup routine - this is as normal for our systems in past.

Attached is a screen grab of my lsmod and /proc/interrupts.  It appears that comedi has installed itself onto IRQ 35.  (this is with my A/D card present and my kernel module not yet insmod'd)

System definitely crashes on attempting to insmod my kernel module.  Same crash dump as a couple weeks ago if it happens in text mode.  If in X, then it just locks the system solid.

On insmod'ing the RTAI modules, dmesg has a message that says Linux Timer IRQ: 4353.

Any ideas what else I can try?

-Duane

-----Original Message-----
From: Paolo Mantegazza [mailto:[email protected]] 
Sent: Wednesday, December 27, 2017 12:08 PM
To: Duane AW. Jeffery <[email protected]>
Subject: RE: [Rtai] Problems with a new Ryzen-based PC

It seem that it could be comedi related. 
In that view take into account that RTAI is now based on the content of COMEDI gits, even if it is likely that the trouble you witness is not related to the COMEDI support you are using.

You should try first to see what comedi support helper tool say, e.g. using comedi_board_info and comedi_calibrate, which should work independently from RTAI. I've gathered some dust in comedi usage, so you should first try looking at comedi docs for further help.

At the moment the only suggestion I have is a possible conflict in the interrupt number, i.e. comedi uses an interrupt number already in use.

Paolo
________________________________________
From: Duane AW. Jeffery [[email protected]]
Sent: Wednesday, December 27, 2017 5:37 PM
To: Paolo Mantegazza; [email protected]
Subject: RE: [Rtai] Problems with a new Ryzen-based PC

I'm happy to try any suggestions you have or go looking wherever might be useful.  I'm not sure even how to find whose handler might be causing the trouble - the system dies with the photographed kernel panic from December 21st on insmod of my module.

I inherited this module from another developer I no longer have access to, so I'm not sure what interrupts it might be connected to.  It uses Comedi to retrieve A/D data from a National Instruments A/D card.  We have a separate stub version for testing that leaves out the comedi calls and it gets loaded when the system detects the A/D card isn't present on bootup.  This version does not crash the system on insmod.  So I'm guessing its related to the interrupts from the A/D card.

-Duane

-----Original Message-----
From: Paolo Mantegazza [mailto:[email protected]]
Sent: Wednesday, December 27, 2017 11:30 AM
To: Duane AW. Jeffery <[email protected]>; [email protected]
Subject: RE: [Rtai] Problems with a new Ryzen-based PC

Glad that kernel the posix support is fixed. The trouble was due to a rewrite of the user space posix timer interface, whereas I forgot to keep the compatibility with its kernel space counterpart. Thanks for signaling it-

As for the interrupt handler I need to know something more. Can you help  me understanding whose handler is causing the trouble?

Paolo

________________________________________
From: Duane AW. Jeffery [[email protected]]
Sent: Wednesday, December 27, 2017 3:50 PM
To: Paolo Mantegazza; [email protected]
Subject: RE: [Rtai] Problems with a new Ryzen-based PC

Contents of this machine's /usr/realtime/calibration/latencies file:

1221 1345 99999

These latencies appear to be in-line with what I see when running the /usr/realtime/testsuite/kern/latency test, and latencies in the ~1.5 us range are within parameters for my application.

Paolo, the fix you sent me seems to eliminate the compiler whining on #include <rtai_posix.h>.

Not sure my app actually needs to include that file, since it compiled fine without it, but I put it back the way it originally was after installing your new tarball and the complaints stopped, so I think you squished that bug.

Machine still kernel panics on inserting my module with the same interrupt handler complaint though.

-Duane

-----Original Message-----
From: Paolo Mantegazza [mailto:[email protected]]
Sent: Friday, December 22, 2017 12:44 PM
To: Duane AW. Jeffery <[email protected]>; [email protected]
Subject: RE: [Rtai] Problems with a new Ryzen-based PC

RTAI calibrate kernel and user space latencies  the very first time it is run. The related results are written in a file store in the calibration directory of the RTAI install directory and are reused at any subsequent execution of an RTAI application.
 So you simply have to look for the existence of such a file and check if the latencies test show an average latency close to zero when run within a quiet environment.
Latency is a single line file containing the kernel and space latency and the period used for the calibration, in nanosecs.

As for the rest of your problems  I'll have a look to the posix compilation problems ASAP, and see what I can do for the interrupt handler afterward.
Can you provide me a simple kernel test of yours for the posix timer case?

What count more for me, at the moment, is to have it confirmed that the INTEL fixes are OK for recent AMDs also.

Paolo


________________________________________
From: Duane AW. Jeffery [[email protected]]
Sent: Friday, December 22, 2017 4:12 PM
To: Paolo Mantegazza; [email protected]
Subject: RE: [Rtai] Problems with a new Ryzen-based PC

Yes, working in kernel space.  Our module RTAD uses rtai_math.

I am able to run the kernel and user space latency tests without issue, but not sure about the latency calibration file - I've never needed to work with it before.  Where can I find it and how will I know its been created correctly?

-Duane

From: Paolo Mantegazza [mailto:[email protected]]
Sent: Friday, December 22, 2017 4:04 AM
To: Duane AW. Jeffery <[email protected]>; [email protected]
Subject: Re: [Rtai] Problems with a new Ryzen-based PC

The problem is of another kind, i.e. interrupt related. What I would like to know first is if you can run the kernel/user space latency tests and if the latency calibration file is created correctly.
As for the timer the fix should not be a problem, but a small, stand alone, test case showing the problem will be of help.
Mine works, but from the use of rtai_math I can infer that you are working in kernel space. Is that correct?

Paolo.

On 12/21/2017 11:17 PM, Duane AW. Jeffery wrote:
Well, I tried the 5.1-test1, and it seems to fix that problem on rtai_sched insmod.

It created a new problem where my RTAI kernel module portion of my software #include's <rtai_posix.h>, which made it fail to compile on rt_ptimer_create not matching its prototype in rtai_tasklets.h.

I was able to compile my code without #include'ing <rtai_posix.h> to clear that error.

Now I'm having another issue that appeared before when I insmod my kernel module, which uses comedi calls.  It seems to still be rtai_sched that blows up.

Crash attached.

-Duane

From: Paolo Mantegazza [mailto:[email protected]]
Sent: Thursday, December 21, 2017 9:26 AM
To: Duane AW. Jeffery <[email protected]><mailto:[email protected]>; [email protected]<mailto:[email protected]>
Subject: Re: [Rtai] Problems with a new Ryzen-based PC

On 12/21/2017 03:18 PM, Duane AW. Jeffery wrote:
I am indeed not.  I'm running probably the RTAI 5.0-test2 package - Its whatever RTAI was at when 4.1.18 was added to HAL patch support.  Generally speaking I get the system working and then don't update anything after that until forced to.

Should I try RTAI 5.1-test1?  Is there a fix related to rt_lxrt_invoke in 5.1-test1?
I expect so. Here is an explanation I gave to somebody else:




What I pointed out to the list is the problem of the argument passing from user to kernel space, where what has worked unchanged for almost 2 decades, not only on x86 archs, failed to run on recent Ix-6xxx INTEL. Here it is how it is explained in the configuration help:

config RTAI_USE_STACK_ARGS

        bool "Keep using RTAI way for user-kernel space on stack args exchange."

        default n

        help

        When it is sure that RTAI syscalls args, to be copied between user-kernel

        space, are on the stack, RTAI uses kernel space memcopy, instead of the

        various canonical copy from-to user. Such a scheme, quite simple and

        effective, has worked for almost two decades without any glitch.

        Unfortunately, recent CPUs, likely from Ix-6xxx onward, segfault with it,

        forcing RTAI to comply with the canonical way.

        The related trouble can be easily spotted at the very first rtai_sched.ko

        usage, i.e. when the very first latency calibration is carried out (see

        the related parameter helps), when the core dump summary, as read with dmesg,

        shows a segfault at lxrt_intercept_syscall.

        Nonetheless, such a way can still be a useful choice and may not cause any

        trouble with recent ATOMs and CELERONs.

        Thus, this parameter, defaulting to a safer no, lets the user have a choice

        anyhow.

The problem may have surfaced some time before but I did not had a hardware to verify it and remained unexplained to me till a user, after upgrading his hardware to an I5-6500 and seeing the failure of his RTAI application, which run unpaired for quite a bit of time, installed the disk of the running hardware onto the new one and saw it failed to work. Fortunately he was willing to help and we fixed the problem in a short time, As I said, I could not see it because I was using a not so new hardware release. Now I've one and I verified first hand that RTAI-5.1 works on it.

The failure was of the type you mentioned, albeit on an Intel cpu. Yours is AMD, but I dare hoping the problem could be the same.

Paolo.




-Duane

From: Rtai [mailto:[email protected]] On Behalf Of Paolo Mantegazza
Sent: Tuesday, December 19, 2017 5:36 AM
To: [email protected]<mailto:[email protected]>
Subject: Re: [Rtai] Problems with a new Ryzen-based PC

May I bet that you are not using the latest rtai-5.1-test1? My bet is based on the fact that it fail at rt_lxrt_invoke.

Paolo.

On 12/18/2017 09:13 PM, Duane AW. Jeffery wrote:
Hi all,

It's been a while since I wrote to the list, but I'm back with a new problem.  Once again the hardware we use has reached end-of-life, and I'm left to try and get our RTAI system working on some new modern hardware.

New system:

AMD Ryzen 3 1200
Gigabyte A320M-HD2
1TB WD Blue
MSI Radeon HD 6450

Installing our usual software (Linux 4.1.18, RTAI... 5? I think) on it mostly works, except that our RT-enabled programs freeze the system (mouse and keyboard unresponsive, display frozen) on attempting to start them, and I notice during bootup where we insmod the various RTAI modules that on insmod of rtai_sched, I get a kernel bug:

BUG: unable to handle kernel paging request at 00007ffde855f1e0

Attached is a photo of the full kernel bug readout.

I tried recompiling the kernel, RTAI, and our software on the new machine to no avail.   Manually running the latency tests in the testsuite appears to work and give reasonable latencies.

Anybody have any idea where things are falling down?

-Duane Jeffery







_______________________________________________

Rtai mailing list

[email protected]<mailto:[email protected]>

https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai