Re: RTAI 3.8.1 - BUG: scheduling while atomic
Graeme Foot <[email protected]> Sun, 11 Feb 2018 23:09:24 +0000
| Newsgroups | gmane.linux.real-time.rtai |
|---|---|
| Message-ID | <[email protected]> |
--===============6964333523021101320==
Content-Language: en-US
Content-Type: multipart/alternative;
boundary="_000_b29ca291eaca44f8b8c7c48fc16a90f7touchcutcom_"
--_000_b29ca291eaca44f8b8c7c48fc16a90f7touchcutcom_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Hi,
Let see if this helps make it more clear.
Kernel module ec_master is insmod'ed.
- It creates a Linux kthread
- It creates an RTDM device for RTAI user space function calls
- It creates a standard character device for a utility that is used to conf=
igure the ec_master module
- this module uses Linux "struct semaphore"'s for resource protection
My (user space) application:
- An RTAI hard task.
- Calls ec_master functions via RTDM device.
- These functions do NOT use the Linux "struct semaphore"'s
- An RTAI soft task.
- Calls ec_master functions via RTDM device.
- These functions DO use the Linux "struct semaphore"'s
The scheduling while atomic bug occurs when the RTAI soft task has called a=
n ec_master function and it is calling a down() on a Linux "struct semaphor=
e".
One thing I didn't mention last time (as I hadn't considered it) is that th=
e Linux "struct semaphore" that is being used is inside a virtual network d=
river component inside the ec_master module. The lock is also being used i=
nside an ndo_start_xmit callback. I'll investigate this further as it may =
be a Linux issue rather than an RTAI one.
Regards,
Graeme.
From: Paolo Mantegazza [mailto:[email protected]]
Sent: Friday, 9 February 2018 9:53 p.m.
To: Graeme Foot <[email protected]>; [email protected]
Subject: Re: [Rtai] RTAI 3.8.1 - BUG: scheduling while atomic
I'm unable to realize a picture of your framework.
For example, when you say kthread do you mean a linux kthread or an RTAI on=
e, or how can a soft rtai task in user space call down()?
Moreover, at as far as I know, but it can be not so much, down/up is for se=
maphores and mutexes are exactly not the same. So if a linux kernel sem has=
to be substituted it should through an RTAI sem.
In any case if the problem is related to an inappropriate interaction of so=
ft RTAI enabled tasks I'd try to see what happen by making the low prio RTA=
I task using linux hard, letting it switch back and forth Linux as needed.
Paolo
On 02/08/2018 11:22 PM, Graeme Foot wrote:
Hi,
I'm adding some functionality to an old system (yes it's old but I'm not up=
dating it yet):
- Linux 2.6.32.11
- RTAI 3.8.1
- Beckhoff CX2020 PC (single core Celeron)
I have a third party kernel module (Etherlab's EtherCAT master) which runs =
a kthread with "struct semaphore" locking. My application is running in us=
er space using LXRT and an RTDM interface to access the EtherCAT functions.=
My application has a hard rtai task thread and a soft rtai task thread.
- The hard rtai task thread calls EtherCAT functions that do not use any "s=
truct semaphore" locks (priority 50).
- The soft rtai task thread calls EtherCAT functions that do use "struct se=
maphore" locks (priority 0).
After a time (10 minutes to 1 hour) I am getting the following kernel bug:
[ 373.990904] BUG: scheduling while atomic: kmc/988/0x00000200
[ 374.000014] Modules linked in: ec_e1000 ec_e1000e ec_cx2100 ec_master rt=
ai_s
hm rtai_msg rtai_mbx rtai_rtdm rtai_sem rtai_lxrt rtai_hal
[ 374.019325]
[ 374.028624] Pid: 988, comm: kmc Not tainted (2.6.32.11-kmc-rtai-3.8.1 #1=
) CX
2020-0100
[ 374.038308] EIP: 0060:[<c011643f>] EFLAGS: 00003282 CPU: 0
[ 374.047931] EIP is at ipipe_critical_exit+0x5d/0x60
[ 374.057342] EAX: 00000000 EBX: 00000000 ECX: 00000000 EDX: 00000000
[ 374.066927] ESI: 00000000 EDI: 00000000 EBP: dec5fe1c ESP: c2002a90
[ 374.076260] DS: 0000 ES: 0000 FS: 0000 GS: 0033 SS: 0068
[ 374.085493] CR0: 8005003b CR2: b765d6b0 CR3: 1ee11000 CR4: 000406d0
[ 374.094878] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 374.104062] DR6: ffff0ff0 DR7: 00000400
[ 374.113062] Call Trace:
[ 374.122082] [<c0146aa2>] ? ktime_get_real+0x0/0x29
[ 374.131009] [<c0146987>] ? ktime_get+0x0/0x88
There's not much of a call stack, but through logging I have determined thi=
s occurs when the soft rtai task thread is calling down() on a semaphore wi=
thin the EtherCAT module. After it occurs the /proc/rtai/scheduler output =
shows the soft rtai task with a State of 0x0, and the thread is no longer f=
unctioning.
My understanding is that a soft rtai task could call any Linux function and=
be pre-empted by other rtai tasks. The above looks like soft tasks may st=
ill have a problem with atomic operations.
Do you know if there is a fix for this (maintaining my Linux and RTAI versi=
ons)? Is it a problem with current RTAI versions?
One option I'm investigating is to change the EtherCAT modules locks to RTA=
I rt_mutex... (from rtai_sem.h). However, from what I understand the rt_mu=
tex_lock() call won't block when used by the EtherCAT modules kthread. Is =
this correct? If so, there is a showroom example "kern\rsmfromlnxitr" that=
makes a kthread into a soft rtai task. Should I use this method for the E=
therCAT kthreads?
Alternatively, would Linux's rt_mutex... function calls solve the above pro=
blem?
Regards,
Graeme Foot.
_______________________________________________
Rtai mailing list
[email protected]<mailto:[email protected]>
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
--_000_b29ca291eaca44f8b8c7c48fc16a90f7touchcutcom_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Courier New \;color\:black";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;
mso-fareast-language:EN-US;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;
mso-fareast-language:EN-US;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor=3D"white" lang=3D"EN-NZ" link=3D"#0563C1" vlink=3D"#954F72">
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><span style=3D"color:#1F497D">Hi,<o:p></o:p></span><=
/p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p> </o:p></spa=
n></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D">Let see if this helps =
make it more clear.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p> </o:p></spa=
n></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D">Kernel module ec_maste=
r is insmod'ed.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D">- It creates a Linux k=
thread<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D">- It creates an RTDM d=
evice for RTAI user space function calls<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D">- It creates a standar=
d character device for a utility that is used to configure the ec_master mo=
dule<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D">- this module uses Lin=
ux "struct semaphore"'s for resource protection<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p> </o:p></spa=
n></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D">My (user space) applic=
ation:<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D">- An RTAI hard task.<o=
:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D"> &nbs=
p; - Call=
s ec_master functions via RTDM device.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D"> &nbs=
p; - Thes=
e functions do NOT use the Linux "struct semaphore"'s<o:p></o:p><=
/span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D">- An RTAI soft task.<o=
:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D"> &nbs=
p; - Call=
s ec_master functions via RTDM device.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D"> &nbs=
p; - Thes=
e functions DO use the Linux "struct semaphore"'s<o:p></o:p></spa=
n></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p> </o:p></spa=
n></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D">The scheduling while a=
tomic bug occurs when the RTAI soft task has called an ec_master function a=
nd it is calling a down() on a Linux "struct semaphore".<o:p></o:=
p></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p> </o:p></spa=
n></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p> </o:p></spa=
n></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D">One thing I didn't men=
tion last time (as I hadn't considered it) is that the Linux "struct s=
emaphore" that is being used is inside a virtual network driver compon=
ent inside the ec_master module. The lock is
also being used inside an ndo_start_xmit callback. I'll investigate =
this further as it may be a Linux issue rather than an RTAI one.<o:p></o:p>=
</span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p> </o:p></spa=
n></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p> </o:p></spa=
n></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D">Regards,<o:p></o:p></s=
pan></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D">Graeme.<o:p></o:p></sp=
an></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p> </o:p></spa=
n></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p> </o:p></spa=
n></p>
<div>
<div style=3D"border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm =
0cm 0cm">
<p class=3D"MsoNormal"><b><span lang=3D"EN-US" style=3D"color:windowtext;ms=
o-fareast-language:EN-NZ">From:</span></b><span lang=3D"EN-US" style=3D"col=
or:windowtext;mso-fareast-language:EN-NZ"> Paolo Mantegazza [mailto:paolo.m=
[email protected]]
<br>
<b>Sent:</b> Friday, 9 February 2018 9:53 p.m.<br>
<b>To:</b> Graeme Foot <[email protected]>; [email protected]<br>
<b>Subject:</b> Re: [Rtai] RTAI 3.8.1 - BUG: scheduling while atomic<o:p></=
o:p></span></p>
</div>
</div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<div>
<p class=3D"MsoNormal">I'm unable to realize a picture of your framework.<b=
r>
For example, when you say kthread do you mean a linux kthread or an RTAI on=
e, or how can a soft rtai task in user space call down()?<br>
<br>
Moreover, at as far as I know, but it can be not so much, down/up is for se=
maphores and mutexes are exactly not the same. So if a linux kernel sem has=
to be substituted it should through an RTAI sem.<br>
<br>
In any case if the problem is related to an inappropriate interaction of so=
ft RTAI enabled tasks I'd try to see what happen by making the low prio RTA=
I task using linux hard, letting it switch back and forth Linux as needed.<=
br>
<br>
Paolo<br>
<br>
On 02/08/2018 11:22 PM, Graeme Foot wrote:<span style=3D"font-size:12.0pt;m=
so-fareast-language:EN-NZ"><o:p></o:p></span></p>
</div>
<blockquote style=3D"margin-top:5.0pt;margin-bottom:5.0pt">
<p class=3D"MsoNormal">Hi,<o:p></o:p></p>
<p class=3D"MsoNormal"> <o:p></o:p></p>
<p class=3D"MsoNormal">I'm adding some functionality to an old system (yes =
it's old but I'm not updating it yet):<o:p></o:p></p>
<p class=3D"MsoNormal">- Linux 2.6.32.11<o:p></o:p></p>
<p class=3D"MsoNormal">- RTAI 3.8.1<o:p></o:p></p>
<p class=3D"MsoNormal">- Beckhoff CX2020 PC (single core Celeron)<o:p></o:p=
></p>
<p class=3D"MsoNormal"> <o:p></o:p></p>
<p class=3D"MsoNormal">I have a third party kernel module (Etherlab's Ether=
CAT master) which runs a kthread with "struct semaphore" locking.=
My application is running in user space using LXRT and an RTDM inter=
face to access the EtherCAT functions. My application
has a hard rtai task thread and a soft rtai task thread.<o:p></o:p></p>
<p class=3D"MsoNormal">- The hard rtai task thread calls EtherCAT functions=
that do not use any "struct semaphore" locks (priority 50).<o:p>=
</o:p></p>
<p class=3D"MsoNormal">- The soft rtai task thread calls EtherCAT functions=
that do use "struct semaphore" locks (priority 0).<o:p></o:p></p=
>
<p class=3D"MsoNormal"> <o:p></o:p></p>
<p class=3D"MsoNormal">After a time (10 minutes to 1 hour) I am getting the=
following kernel bug:<o:p></o:p></p>
<p class=3D"MsoNormal"> <o:p></o:p></p>
<p class=3D"MsoPlainText"><span style=3D"font-family:"Courier New ;col=
or:black",serif">[ 373.990904] BUG: scheduling while atomic: kmc=
/988/0x00000200</span><o:p></o:p></p>
<p class=3D"MsoPlainText"><span style=3D"font-family:"Courier New ;col=
or:black",serif">[ 374.000014] Modules linked in: ec_e1000 ec_e1=
000e ec_cx2100 ec_master rtai_s</span><o:p></o:p></p>
<p class=3D"MsoPlainText"><span style=3D"font-family:"Courier New ;col=
or:black",serif">hm rtai_msg rtai_mbx rtai_rtdm rtai_sem rtai_lxrt rta=
i_hal</span><o:p></o:p></p>
<p class=3D"MsoPlainText"><span style=3D"font-family:"Courier New ;col=
or:black",serif">[ 374.019325]
</span><o:p></o:p></p>
<p class=3D"MsoPlainText"><span style=3D"font-family:"Courier New ;col=
or:black",serif">[ 374.028624] Pid: 988, comm: kmc Not tainted (=
2.6.32.11-kmc-rtai-3.8.1 #1) CX</span><o:p></o:p></p>
<p class=3D"MsoPlainText"><span style=3D"font-family:"Courier New ;col=
or:black",serif">2020-0100</span><o:p></o:p></p>
<p class=3D"MsoPlainText"><span style=3D"font-family:"Courier New ;col=
or:black",serif">[ 374.038308] EIP: 0060:[<c011643f>] EFLA=
GS: 00003282 CPU: 0</span><o:p></o:p></p>
<p class=3D"MsoPlainText"><span style=3D"font-family:"Courier New ;col=
or:black",serif">[ 374.047931] EIP is at ipipe_critical_exit+=
;0x5d/0x60</span><o:p></o:p></p>
<p class=3D"MsoPlainText"><span style=3D"font-family:"Courier New ;col=
or:black",serif">[ 374.057342] EAX: 00000000 EBX: 00000000 ECX: =
00000000 EDX: 00000000</span><o:p></o:p></p>
<p class=3D"MsoPlainText"><span style=3D"font-family:"Courier New ;col=
or:black",serif">[ 374.066927] ESI: 00000000 EDI: 00000000 EBP: =
dec5fe1c ESP: c2002a90</span><o:p></o:p></p>
<p class=3D"MsoPlainText"><span style=3D"font-family:"Courier New ;col=
or:black",serif">[ 374.076260] DS: 0000 ES: 0000 FS: 0000 =
GS: 0033 SS: 0068</span><o:p></o:p></p>
<p class=3D"MsoPlainText"><span style=3D"font-family:"Courier New ;col=
or:black",serif">[ 374.085493] CR0: 8005003b CR2: b765d6b0 CR3: =
1ee11000 CR4: 000406d0</span><o:p></o:p></p>
<p class=3D"MsoPlainText"><span style=3D"font-family:"Courier New ;col=
or:black",serif">[ 374.094878] DR0: 00000000 DR1: 00000000 DR2: =
00000000 DR3: 00000000</span><o:p></o:p></p>
<p class=3D"MsoPlainText"><span style=3D"font-family:"Courier New ;col=
or:black",serif">[ 374.104062] DR6: ffff0ff0 DR7: 00000400</span=
><o:p></o:p></p>
<p class=3D"MsoPlainText"><span style=3D"font-family:"Courier New ;col=
or:black",serif">[ 374.113062] Call Trace:</span><o:p></o:p></p>
<p class=3D"MsoPlainText"><span style=3D"font-family:"Courier New ;col=
or:black",serif">[ 374.122082] [<c0146aa2>] ? ktime_=
get_real+0x0/0x29</span><o:p></o:p></p>
<p class=3D"MsoPlainText"><span style=3D"font-family:"Courier New ;col=
or:black",serif">[ 374.131009] [<c0146987>] ? ktime_=
get+0x0/0x88</span><o:p></o:p></p>
<p class=3D"MsoNormal"> <o:p></o:p></p>
<p class=3D"MsoNormal">There's not much of a call stack, but through loggin=
g I have determined this occurs when the soft rtai task thread is calling d=
own() on a semaphore within the EtherCAT module. After it occurs the =
/proc/rtai/scheduler output shows the soft
rtai task with a State of 0x0, and the thread is no longer functioning.<o:=
p></o:p></p>
<p class=3D"MsoNormal"> <o:p></o:p></p>
<p class=3D"MsoNormal"> <o:p></o:p></p>
<p class=3D"MsoNormal">My understanding is that a soft rtai task could call=
any Linux function and be pre-empted by other rtai tasks. The above =
looks like soft tasks may still have a problem with atomic operations.<o:p>=
</o:p></p>
<p class=3D"MsoNormal"> <o:p></o:p></p>
<p class=3D"MsoNormal">Do you know if there is a fix for this (maintaining =
my Linux and RTAI versions)? Is it a problem with current RTAI versio=
ns?<o:p></o:p></p>
<p class=3D"MsoNormal"> <o:p></o:p></p>
<p class=3D"MsoNormal"> <o:p></o:p></p>
<p class=3D"MsoNormal">One option I'm investigating is to change the EtherC=
AT modules locks to RTAI rt_mutex… (from rtai_sem.h). However, =
from what I understand the rt_mutex_lock() call won't block when used by th=
e EtherCAT modules kthread. Is this correct?
If so, there is a showroom example "kern\rsmfromlnxitr" that mak=
es a kthread into a soft rtai task. Should I use this method for the =
EtherCAT kthreads?<o:p></o:p></p>
<p class=3D"MsoNormal"> <o:p></o:p></p>
<p class=3D"MsoNormal">Alternatively, would Linux's rt_mutex… functio=
n calls solve the above problem?<o:p></o:p></p>
<p class=3D"MsoNormal"> <o:p></o:p></p>
<p class=3D"MsoNormal"> <o:p></o:p></p>
<p class=3D"MsoNormal">Regards,<o:p></o:p></p>
<p class=3D"MsoNormal">Graeme Foot.<o:p></o:p></p>
<p class=3D"MsoNormal"> <o:p></o:p></p>
<p class=3D"MsoNormal"> <o:p></o:p></p>
<p class=3D"MsoNormal"> <o:p></o:p></p>
<p class=3D"MsoNormal"> <o:p></o:p></p>
<p class=3D"MsoNormal"><span style=3D"font-size:12.0pt;font-family:"Ti=
mes New Roman",serif;mso-fareast-language:EN-NZ"><br>
<br>
<br>
<o:p></o:p></span></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Rtai mailing list<o:p></o:p></pre>
<pre><a href=3D"mailto:[email protected]">[email protected]</a><o:p></o:p></pre>
<pre><a href=3D"https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai">https:=
//mail.rtai.org/cgi-bin/mailman/listinfo/rtai</a><o:p></o:p></pre>
</blockquote>
<p><o:p> </o:p></p>
</div>
</body>
</html>
--_000_b29ca291eaca44f8b8c7c48fc16a90f7touchcutcom_--
--===============6964333523021101320==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Rtai mailing list
[email protected]
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai
--===============6964333523021101320==--