Re: RTAI 3.8.1 - BUG: scheduling while atomic
Paolo Mantegazza <[email protected]> Mon, 12 Feb 2018 09:19:44 +0100
| Newsgroups | gmane.linux.real-time.rtai |
|---|---|
| Message-ID | <[email protected]> |
This is a multi-part message in MIME format. --===============5310248358626168999== Content-Type: multipart/alternative; boundary="------------75C56E8F0B126B9F677731D2" Content-Language: en-US This is a multi-part message in MIME format. --------------75C56E8F0B126B9F677731D2 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: quoted-printable Is it not a problem of the way the "up" following the "down"=A0 is called= ?=20 I mean, is the up called from a truly Linux schedulable environment? Paolo? On 02/12/2018 12:09 AM, Graeme Foot wrote: > > Hi, > > Let see if this helps make it more clear. > > Kernel module ec_master is insmod'ed. > > - It creates a Linux kthread > > - It creates an RTDM device for RTAI user space function calls > > - It creates a standard character device for a utility that is used to=20 > configure the ec_master module > > - this module uses Linux "struct semaphore"'s for resource protection > > My (user space) application: > > - An RTAI hard task. > > - Calls ec_master functions via RTDM device. > > - These functions do NOT use the Linux "struct semaphore"'s > > - An RTAI soft task. > > - Calls ec_master functions via RTDM device. > > - These functions DO use the Linux "struct semaphore"'s > > The scheduling while atomic bug occurs when the RTAI soft task has=20 > called an ec_master function and it is calling a down() on a Linux=20 > "struct semaphore". > > One thing I didn't mention last time (as I hadn't considered it) is=20 > that the Linux "struct semaphore" that is being used is inside a=20 > virtual network driver component inside the ec_master module.=A0 The=20 > lock is also being used inside an ndo_start_xmit callback.=A0 I'll=20 > investigate this further as it may be a Linux issue rather than an=20 > RTAI one. > > Regards, > > Graeme. > > *From:*Paolo Mantegazza [mailto:[email protected]] > *Sent:* Friday, 9 February 2018 9:53 p.m. > *To:* Graeme Foot <[email protected]>; [email protected] > *Subject:* Re: [Rtai] RTAI 3.8.1 - BUG: scheduling while atomic > > I'm unable to realize a picture of your framework. > For example, when you say kthread do you mean a linux kthread or an=20 > RTAI one, or how can a soft rtai task in user space call down()? > > Moreover, at as far as I know, but it can be not so much, down/up is=20 > for semaphores and mutexes are exactly not the same. So if a linux=20 > kernel sem has to be substituted it should through an RTAI sem. > > In any case if the problem is related to an inappropriate interaction=20 > of soft RTAI enabled tasks I'd try to see what happen by making the=20 > low prio RTAI task using linux hard, letting it switch back and forth=20 > Linux as needed. > > Paolo > > On 02/08/2018 11:22 PM, Graeme Foot wrote: > > Hi, > > I'm adding some functionality to an old system (yes it's old but > I'm not updating it yet): > > - Linux 2.6.32.11 > > - RTAI 3.8.1 > > - Beckhoff CX2020 PC (single core Celeron) > > I have a third party kernel module (Etherlab's EtherCAT master) > which runs a kthread with "struct semaphore" locking.=A0 My > application is running in user space using LXRT and an RTDM > interface to access the EtherCAT functions.=A0 My application has a > hard rtai task thread and a soft rtai task thread. > > - The hard rtai task thread calls EtherCAT functions that do not > use any "struct semaphore" locks (priority 50). > > - The soft rtai task thread calls EtherCAT functions that do use > "struct semaphore" locks (priority 0). > > After a time (10 minutes to 1 hour) I am getting the following > kernel bug: > > [=A0 373.990904] BUG: scheduling while atomic: kmc/988/0x00000200 > > [=A0 374.000014] Modules linked in: ec_e1000 ec_e1000e ec_cx2100 > ec_master rtai_s > > hm rtai_msg rtai_mbx rtai_rtdm rtai_sem rtai_lxrt rtai_hal > > [=A0 374.019325] > > [=A0 374.028624] Pid: 988, comm: kmc Not tainted > (2.6.32.11-kmc-rtai-3.8.1 #1) CX > > 2020-0100 > > [=A0 374.038308] EIP: 0060:[<c011643f>] EFLAGS: 00003282 CPU: 0 > > [=A0 374.047931] EIP is at ipipe_critical_exit+0x5d/0x60 > > [=A0 374.057342] EAX: 00000000 EBX: 00000000 ECX: 00000000 EDX: 000= 00000 > > [=A0 374.066927] ESI: 00000000 EDI: 00000000 EBP: dec5fe1c ESP: c20= 02a90 > > [=A0 374.076260]=A0 DS: 0000 ES: 0000 FS: 0000 GS: 0033 SS: 0068 > > [=A0 374.085493] CR0: 8005003b CR2: b765d6b0 CR3: 1ee11000 CR4: 000= 406d0 > > [=A0 374.094878] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 000= 00000 > > [=A0 374.104062] DR6: ffff0ff0 DR7: 00000400 > > [=A0 374.113062] Call Trace: > > [=A0 374.122082] [<c0146aa2>] ? ktime_get_real+0x0/0x29 > > [=A0 374.131009] [<c0146987>] ? ktime_get+0x0/0x88 > > There's not much of a call stack, but through logging I have > determined this occurs when the soft rtai task thread is calling > down() on a semaphore within the EtherCAT module.=A0 After it occur= s > the /proc/rtai/scheduler output shows the soft rtai task with a > State of 0x0, and the thread is no longer functioning. > > My understanding is that a soft rtai task could call any Linux > function and be pre-empted by other rtai tasks.=A0 The above looks > like soft tasks may still have a problem with atomic operations. > > Do you know if there is a fix for this (maintaining my Linux and > RTAI versions)?=A0 Is it a problem with current RTAI versions? > > One option I'm investigating is to change the EtherCAT modules > locks to RTAI rt_mutex=85 (from rtai_sem.h).=A0 However, from what = I > understand the rt_mutex_lock() call won't block when used by the > EtherCAT modules kthread.=A0 Is this correct?=A0 If so, there is a > showroom example "kern\rsmfromlnxitr" that makes a kthread into a > soft rtai task.=A0 Should I use this method for the EtherCAT kthrea= ds? > > Alternatively, would Linux's rt_mutex=85 function calls solve the > above problem? > > Regards, > > Graeme Foot. > > > > > _______________________________________________ > > Rtai mailing list > > [email protected] <mailto:[email protected]> > > https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai > --------------75C56E8F0B126B9F677731D2 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dwindows-1252"> </head> <body text=3D"#000000" bgcolor=3D"#FFFFFF"> <div class=3D"moz-cite-prefix">Is it not a problem of the way the "up= " following the "down"=A0 is called? I mean, is the up called from a truly Linux schedulable environment? <br> <br> Paolo?<br> <br> On 02/12/2018 12:09 AM, Graeme Foot wrote:<br> </div> <blockquote type=3D"cite" cite=3D"mid:[email protected]"> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dwindows-1252"> <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:Consolas; panose-1:2 11 6 9 2 2 4 3 2 4;} @font-face {font-family:"Courier New \;color\:black"; panose-1:0 0 0 0 0 0 0 0 0 0;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif; color:black; mso-fareast-language:EN-US;} a:link, span.MsoHyperlink {mso-style-priority:99; color:#0563C1; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} p.MsoPlainText, li.MsoPlainText, div.MsoPlainText {mso-style-priority:99; mso-style-link:"Plain Text Char"; margin:0cm; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif; color:black; mso-fareast-language:EN-US;} p {mso-style-priority:99; mso-margin-top-alt:auto; margin-right:0cm; mso-margin-bottom-alt:auto; margin-left:0cm; font-size:12.0pt; font-family:"Times New Roman",serif; color:black;} pre {mso-style-priority:99; mso-style-link:"HTML Preformatted Char"; margin:0cm; margin-bottom:.0001pt; font-size:10.0pt; font-family:"Courier New"; color:black;} span.HTMLPreformattedChar {mso-style-name:"HTML Preformatted Char"; mso-style-priority:99; mso-style-link:"HTML Preformatted"; font-family:Consolas; color:black; mso-fareast-language:EN-US;} p.msonormal0, li.msonormal0, div.msonormal0 {mso-style-name:msonormal; mso-style-priority:99; mso-margin-top-alt:auto; margin-right:0cm; mso-margin-bottom-alt:auto; margin-left:0cm; font-size:12.0pt; font-family:"Times New Roman",serif; color:black;} span.PlainTextChar {mso-style-name:"Plain Text Char"; mso-style-priority:99; mso-style-link:"Plain Text"; font-family:"Calibri",sans-serif;} span.EmailStyle23 {mso-style-type:personal; font-family:"Calibri",sans-serif; color:windowtext;} span.EmailStyle24 {mso-style-type:personal-reply; font-family:"Calibri",sans-serif; color:#1F497D;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt;} @page WordSection1 {size:612.0pt 792.0pt; margin:72.0pt 72.0pt 72.0pt 72.0pt;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--> <div class=3D"WordSection1"> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Hi,<o:p></o:= p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>=A0</o:= p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Let see if t= his helps make it more clear.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>=A0</o:= p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Kernel modul= e ec_master is insmod'ed.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">- It creates= a Linux kthread<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">- It creates= an RTDM device for RTAI user space function calls<o:p></o:p></sp= an></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">- It creates= a standard character device for a utility that is used to configure the ec_master module<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">- this modul= e uses Linux "struct semaphore"'s for resource protection<o:p><= /o:p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>=A0</o:= p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">My (user spa= ce) application:<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">- An RTAI ha= rd task.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 - Calls ec_master functions via RTDM device.<o:p></o:p></span= ></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 - These functions do NOT use the Linux "struct semaphore"'s<o= :p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">- An RTAI so= ft task.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 - Calls ec_master functions via RTDM device.<o:p></o:p></span= ></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 - These functions DO use the Linux "struct semaphore"'s<o:p><= /o:p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>=A0</o:= p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">The scheduli= ng while atomic bug occurs when the RTAI soft task has called an ec_master function and it is calling a down() on a Linux "struct semaphore".<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>=A0</o:= p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>=A0</o:= p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">One thing I didn't mention last time (as I hadn't considered it) is that the Linux "struct semaphore" that is being used is inside a virtual network driver component inside the ec_master module.=A0 The lock is also being used inside an ndo_start_xmit callback.=A0 I'll investigate this further as it may be a Linux issue rather than an RTAI one.<o:p></o:p></= span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>=A0</o:= p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>=A0</o:= p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Regards,<o:p= ></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">Graeme.<o:p>= </o:p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>=A0</o:= p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>=A0</o:= p></span></p> <div> <div style=3D"border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> <p class=3D"MsoNormal"><b><span style=3D"color:windowtext;mso-fareast-language:EN-NZ" lang=3D"EN-US">From:</span></b><span style=3D"color:windowtext;mso-fareast-language:EN-NZ" lang=3D"EN-US"> Paolo Mantegazza [<a class=3D"moz-txt-link-freetext" href=3D"mailto:paolo.= [email protected]">mailto:[email protected]</a>] <br> <b>Sent:</b> Friday, 9 February 2018 9:53 p.m.<br> <b>To:</b> Graeme Foot <a class=3D"moz-txt-link-rfc2396E"= href=3D"mailto:[email protected]"><[email protected]>= ;</a>; <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:rtai= @rtai.org">[email protected]</a><br> <b>Subject:</b> Re: [Rtai] RTAI 3.8.1 - BUG: scheduling while atomic<o:p></o:p></span></p> </div> </div> <p class=3D"MsoNormal"><o:p>=A0</o:p></p> <div> <p class=3D"MsoNormal">I'm unable to realize a picture of your framework.<br> For example, when you say kthread do you mean a linux kthread or an RTAI one, or how can a soft rtai task in user space call down()?<br> <br> Moreover, at as far as I know, but it can be not so much, down/up is for semaphores and mutexes are exactly not the same. So if a linux kernel sem has to be substituted it should through an RTAI sem.<br> <br> In any case if the problem is related to an inappropriate interaction of soft RTAI enabled tasks I'd try to see what happen by making the low prio RTAI task using linux hard, letting it switch back and forth Linux as needed.<br> <br> Paolo<br> <br> On 02/08/2018 11:22 PM, Graeme Foot wrote:<span style=3D"font-size:12.0pt;mso-fareast-language:EN-NZ"><o:p>= </o:p></span></p> </div> <blockquote style=3D"margin-top:5.0pt;margin-bottom:5.0pt"> <p class=3D"MsoNormal">Hi,<o:p></o:p></p> <p class=3D"MsoNormal">=A0<o:p></o:p></p> <p class=3D"MsoNormal">I'm adding some functionality to an old system (yes it's old but I'm not updating it yet):<o:p></o:p>= </p> <p class=3D"MsoNormal">- Linux 2.6.32.11<o:p></o:p></p> <p class=3D"MsoNormal">- RTAI 3.8.1<o:p></o:p></p> <p class=3D"MsoNormal">- Beckhoff CX2020 PC (single core Celeron)<o:p></o:p></p> <p class=3D"MsoNormal">=A0<o:p></o:p></p> <p class=3D"MsoNormal">I have a third party kernel module (Etherlab's EtherCAT master) which runs a kthread with "struct semaphore" locking.=A0 My application is running in user space using LXRT and an RTDM interface to access the EtherCAT functions.=A0 My application has a hard rtai task thread and a soft rtai task thread.<o:p></o:p></p> <p class=3D"MsoNormal">- The hard rtai task thread calls EtherCAT functions that do not use any "struct semaphore" locks (priority 50).<o:p></o:p></p> <p class=3D"MsoNormal">- The soft rtai task thread calls EtherCAT functions that do use "struct semaphore" locks (priority 0).<o:p></o:p></p> <p class=3D"MsoNormal">=A0<o:p></o:p></p> <p class=3D"MsoNormal">After a time (10 minutes to 1 hour) I am getting the following kernel bug:<o:p></o:p></p> <p class=3D"MsoNormal">=A0<o:p></o:p></p> <p class=3D"MsoPlainText"><span style=3D"font-family:"Cour= ier New ;color:black",serif">[=A0 373.990904] BUG: scheduling while atomic: kmc/988/0x00000200</span><o:p></o:= p></p> <p class=3D"MsoPlainText"><span style=3D"font-family:"Cour= ier New ;color:black",serif">[=A0 374.000014] Modules linked in: ec_e1000 ec_e1000e ec_cx2100 ec_master rtai_s</s= pan><o:p></o:p></p> <p class=3D"MsoPlainText"><span style=3D"font-family:"Cour= ier New ;color:black",serif">hm rtai_msg rtai_mbx rtai_rtdm rtai_sem rtai_lxrt rtai_hal</span><o:p></o:p></p> <p class=3D"MsoPlainText"><span style=3D"font-family:"Cour= ier New ;color:black",serif">[=A0 374.019325] </span><o:p></o:p></p> <p class=3D"MsoPlainText"><span style=3D"font-family:"Cour= ier New ;color:black",serif">[=A0 374.028624] Pid: 988, comm: kmc Not tainted (2.6.32.11-kmc-rtai-3.8.1 #1) CX</spa= n><o:p></o:p></p> <p class=3D"MsoPlainText"><span style=3D"font-family:"Cour= ier New ;color:black",serif">2020-0100</span><o:p></o:p></= p> <p class=3D"MsoPlainText"><span style=3D"font-family:"Cour= ier New ;color:black",serif">[=A0 374.038308] EIP: 0060:[<c011643f>] EFLAGS: 00003282 CPU: 0</span><o:p>= </o:p></p> <p class=3D"MsoPlainText"><span style=3D"font-family:"Cour= ier New ;color:black",serif">[=A0 374.047931] EIP is at ipipe_critical_exit+0x5d/0x60</span><o:p></o:p></p> <p class=3D"MsoPlainText"><span style=3D"font-family:"Cour= ier New ;color:black",serif">[=A0 374.057342] EAX: 0000000= 0 EBX: 00000000 ECX: 00000000 EDX: 00000000</span><o:p></o:p>= </p> <p class=3D"MsoPlainText"><span style=3D"font-family:"Cour= ier New ;color:black",serif">[=A0 374.066927] ESI: 0000000= 0 EDI: 00000000 EBP: dec5fe1c ESP: c2002a90</span><o:p></o:p>= </p> <p class=3D"MsoPlainText"><span style=3D"font-family:"Cour= ier New ;color:black",serif">[=A0 374.076260]=A0 DS: 0000 = ES: 0000 FS: 0000 GS: 0033 SS: 0068</span><o:p></o:p></p> <p class=3D"MsoPlainText"><span style=3D"font-family:"Cour= ier New ;color:black",serif">[=A0 374.085493] CR0: 8005003= b CR2: b765d6b0 CR3: 1ee11000 CR4: 000406d0</span><o:p></o:p>= </p> <p class=3D"MsoPlainText"><span style=3D"font-family:"Cour= ier New ;color:black",serif">[=A0 374.094878] DR0: 0000000= 0 DR1: 00000000 DR2: 00000000 DR3: 00000000</span><o:p></o:p>= </p> <p class=3D"MsoPlainText"><span style=3D"font-family:"Cour= ier New ;color:black",serif">[=A0 374.104062] DR6: ffff0ff= 0 DR7: 00000400</span><o:p></o:p></p> <p class=3D"MsoPlainText"><span style=3D"font-family:"Cour= ier New ;color:black",serif">[=A0 374.113062] Call Trace:<= /span><o:p></o:p></p> <p class=3D"MsoPlainText"><span style=3D"font-family:"Cour= ier New ;color:black",serif">[=A0 374.122082]=A0 [<c0146aa2>] ? ktime_get_real+0x0/0x29</span><o:p></o= :p></p> <p class=3D"MsoPlainText"><span style=3D"font-family:"Cour= ier New ;color:black",serif">[=A0 374.131009]=A0 [<c0146987>] ? ktime_get+0x0/0x88</span><o:p></o:p></= p> <p class=3D"MsoNormal">=A0<o:p></o:p></p> <p class=3D"MsoNormal">There's not much of a call stack, but through logging I have determined this occurs when the soft rtai task thread is calling down() on a semaphore within the EtherCAT module.=A0 After it occurs the /proc/rtai/scheduler output shows the soft rtai task with a State of 0x0, and the thread is no longer functioning.<o:p></o:p></p> <p class=3D"MsoNormal">=A0<o:p></o:p></p> <p class=3D"MsoNormal">=A0<o:p></o:p></p> <p class=3D"MsoNormal">My understanding is that a soft rtai tas= k could call any Linux function and be pre-empted by other rtai tasks.=A0 The above looks like soft tasks may still have a problem with atomic operations.<o:p></o:p></p> <p class=3D"MsoNormal">=A0<o:p></o:p></p> <p class=3D"MsoNormal">Do you know if there is a fix for this (maintaining my Linux and RTAI versions)?=A0 Is it a problem with current RTAI versions?<o:p></o:p></p> <p class=3D"MsoNormal">=A0<o:p></o:p></p> <p class=3D"MsoNormal">=A0<o:p></o:p></p> <p class=3D"MsoNormal">One option I'm investigating is to chang= e the EtherCAT modules locks to RTAI rt_mutex=85 (from rtai_sem.h).=A0 However, from what I understand the rt_mutex_lock() call won't block when used by the EtherCAT modules kthread.=A0 Is this correct?=A0 If so, there is a showroom example "kern\rsmfromlnxitr" that makes a kthread into a soft rtai task.=A0 Should I use this method for the EtherCAT kthreads?<o:p></o:p></p> <p class=3D"MsoNormal">=A0<o:p></o:p></p> <p class=3D"MsoNormal">Alternatively, would Linux's rt_mutex=85 function calls solve the above problem?<o:p></o:p></p> <p class=3D"MsoNormal">=A0<o:p></o:p></p> <p class=3D"MsoNormal">=A0<o:p></o:p></p> <p class=3D"MsoNormal">Regards,<o:p></o:p></p> <p class=3D"MsoNormal">Graeme Foot.<o:p></o:p></p> <p class=3D"MsoNormal">=A0<o:p></o:p></p> <p class=3D"MsoNormal">=A0<o:p></o:p></p> <p class=3D"MsoNormal">=A0<o:p></o:p></p> <p class=3D"MsoNormal">=A0<o:p></o:p></p> <p class=3D"MsoNormal"><span style=3D"font-size:12.0pt;font-family:"Times New Roman",serif;mso-fareast-language:EN-NZ"><br> <br> <br> <o:p></o:p></span></p> <pre>_______________________________________________<o:p></o:p>= </pre> <pre>Rtai mailing list<o:p></o:p></pre> <pre><a href=3D"mailto:[email protected]" moz-do-not-send=3D"true">= [email protected]</a><o:p></o:p></pre> <pre><a href=3D"https://mail.rtai.org/cgi-bin/mailman/listinfo/= rtai" moz-do-not-send=3D"true">https://mail.rtai.org/cgi-bin/mailman/list= info/rtai</a><o:p></o:p></pre> </blockquote> <p><o:p>=A0</o:p></p> </div> </blockquote> <p><br> </p> </body> </html> --------------75C56E8F0B126B9F677731D2-- --===============5310248358626168999== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Rtai mailing list [email protected] https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai --===============5310248358626168999==--