Re: Re: [PATCH i386] Live Patching Function on 2.6.11.7

This is a multi-part message in MIME format.
--------------010403060600020408010201
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hello, Andrew
Sorry for late reply.
I don't know montavista's implementation, but it seems some 
modifications to legacy application are needed(as long as it is 
implemented as library...).
And there is no open-sourced implementation yet.
I think only the way to realize user space implementation without zero 
modification on legacy application is using ptrace function.

We tested ptrace PEEKDATA performance.
The tested environment is workstation like below.
CPU Xeon 2.8GHz x2,
Memory 1GB,
Kernel 2.6.9

The results are below;
--------------------------------------------------------------------------------
memory-size(byte)	Time-to-write(micro-seconds)
1	9
10	23
50	83
100	167
500	754
1000	1475
5000	7259
10000	14592
50000	73070
100000	146037
500000	727189
1000000	1452708

You can also see liner scaling graph.
If the patch module is almost 1MB, then it takes almost 1.5second...
High-throughput ptrace PEEKDATA seems very reasonable(at least for us).


Corey Minyard wrote:
> Andrew, you are correct.  This can be implemented as a userspace 
> library.  It's not easy, but it can be done.
> 
> There are a few things that cannot be handled with hostile 
> applications.  For instance, if a thread turns off all signals, it 
> cannot be stopped and thus cannot be patched.  However, there are 
> important operations that need to be done that are extremely hard to do 
> in a ptrace environment and are easy with a library.  I doubt the 
> current implementation from NTT has these operations, but they are 
> important for patching.
> 
> I know this because Montavista delivers a product (which I initially 
> wrote) that does exactly this.  It is a set of libraries that the user 
> can link in or can be preloaded on an existing application.  We have 
> many customers using this tool.
> 
> I haven't done any tests, but the library should actually be faster than 
> the ptrace method because of the reduced mm switching.  I don't know why 
> it would be any slower.  Plus, the actual application of the patch is 
> the easy part of this operation; it's all the other stuff you have to do 
> that makes this so hard.
> 
> I'm not sure I agree with the level of hostility this received.  This is 
> an important tool for improving reliability.  I try to stay out of frays 
> like this, perhaps to a fault.
> 
> -Corey
> 
> Andrew Morton wrote:
> 
>> Takashi Ikebe <[email protected]> wrote:
>>  
>>
>>> This patch add function called "Live patching" which is defined on
>>> OSDL's carrier grade linux requiremnt definition to linux 2.6.11.7 
>>> kernel.
>>>   
>>
>>
>> I must say that I'd agree with the hostile reception which this work
>> received on the linux-kernel list.
>>
>> It needs to be exhaustively demonstrated that this functionality 
>> cannot be
>> provided in userspace.  If it can be implemented in userspace then it 
>> would
>> be via a library and would require basically zero modifications to legacy
>> applications.
>>
>> If PTRACE_PEEKTEXT/POKETEXT ends up being a limiting factor then we would
>> certainly support extensions to the ptrace function which enable higher
>> throughput - presumably with scatter/gather lists.
>>
>> To repeat: I think it _can_ be done in userspace - if the 
>> implementation is
>> sufficiently good then the limiting factor would be memory bandwidth.
>>
>> We've seen assertions that a userspace approach would be too slow, but to
>> get support for a patch like this, quite frankly, we'd need to see the
>> too-slow userspace implementation before we'd agree that kernel help is
>> needed.
>>
>>  
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> cgl_discussion mailing list
>> [email protected]
>> http://lists.osdl.org/mailman/listinfo/cgl_discussion
>>  
>>
> 


--------------010403060600020408010201
Content-Type: image/gif;
 name="peekdata.GIF"
Content-Transfer-Encoding: base64
Content-Disposition: inline;
 filename="peekdata.GIF"

R0lGODlhCgLkAfcAAAAAAIAAAACAAICAAAAAgIAAgACAgICAgMDAwP8AAAD/AP//AAAA//8A
/wD//////wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMwAAZgAAmQAAzAAA/wAzAAAzMwAzZgAz
mQAzzAAz/wBmAABmMwBmZgBmmQBmzABm/wCZAACZMwCZZgCZmQCZzACZ/wDMAADMMwDMZgDM
mQDMzADM/wD/AAD/MwD/ZgD/mQD/zAD//zMAADMAMzMAZjMAmTMAzDMA/zMzADMzMzMzZjMz
mTMzzDMz/zNmADNmMzNmZjNmmTNmzDNm/zOZADOZMzOZZjOZmTOZzDOZ/zPMADPMMzPMZjPM
mTPMzDPM/zP/ADP/MzP/ZjP/mTP/zDP//2YAAGYAM2YAZmYAmWYAzGYA/2YzAGYzM2YzZmYz
mWYzzGYz/2ZmAGZmM2ZmZmZmmWZmzGZm/2aZAGaZM2aZZmaZmWaZzGaZ/2bMAGbMM2bMZmbM
mWbMzGbM/2b/AGb/M2b/Zmb/mWb/zGb//5kAAJkAM5kAZpkAmZkAzJkA/5kzAJkzM5kzZpkz
mZkzzJkz/5lmAJlmM5lmZplmmZlmzJlm/5mZAJmZM5mZZpmZmZmZzJmZ/5nMAJnMM5nMZpnM
mZnMzJnM/5n/AJn/M5n/Zpn/mZn/zJn//8wAAMwAM8wAZswAmcwAzMwA/8wzAMwzM8wzZswz
mcwzzMwz/8xmAMxmM8xmZsxmmcxmzMxm/8yZAMyZM8yZZsyZmcyZzMyZ/8zMAMzMM8zMZszM
mczMzMzM/8z/AMz/M8z/Zsz/mcz/zMz///8AAP8AM/8AZv8Amf8AzP8A//8zAP8zM/8zZv8z
mf8zzP8z//9mAP9mM/9mZv9mmf9mzP9m//+ZAP+ZM/+ZZv+Zmf+ZzP+Z///MAP/MM//MZv/M
mf/MzP/M////AP//M///Zv//mf//zP///yH5BAEAABAALAAAAAAKAuQBAAj/AP8JHEiwoMGD
CBMqXMiwocOHECNKnEixosWLGDNq3Mixo8ePIEOKHEmypMmTKFOqXMmypcuXMGPKnEmzps2b
OHPq3Mmzp8+fQIMKHUq0qNGjSJMqXcq0qdOnD++hQPEDqtWrWLNqtehP1T9/lGJtHUu2rNmm
2wQqE3u2rdu3cGva8xG3rt27eDsqo5q3r9+/SdeO9AenEVdiK9KCxaGY0o2BoBLzJGwYoT9D
gDNrhknpFkRKkgvuRUH34L3SFSNvI4wCx797lEin/adsBU+pKDAnvGd7s+/fg8M+tDc19EBl
j2mjJniaIyjGX4klB0p5Ie/ZwLNrv6isasTa2F+r/5gNm+1Af82qbfSHHLKP8D3BMmKobPn2
+/ityzFPm//ryscZRxt0AoFSGjNTqeAfRs8pJp1QYAG42375VWhhQe64RhB62PFGjWjG+YPL
crU9pAwL27iDgmSgrFiQgQKxp+E/xLyn1nQGgTVVLB/GCMdUNgqET2wo4PgPkQIqMx5x421I
ZA+U6Hbkjl4RVOOFWFYIY0H+GCPWdQaBd15hBDGjYEMtrmBKLGDdIMpqlJTG2nQNRvfYPT8S
yKWXapUGJmHQ3XOmVMkRxlaLYumoAh+NgIUanpj5QwwKlb3Zp2hGZqnpZmBJueEl0wgYIHZD
eioofAlJgwJbSs62pUA1zv/WXoFyKjMjl8J9pd4/lgrkznhgmRcrjaiBFZqKbA1L23SwzRfh
QD0OxMwKu25qbWbVmUbhQWL6KGG3DNnT5D/TOojjrP/UKel0tsI51Ypp7XWmQHi++24spxoE
KUGtCmRmog9+RWaMlFQmL6r/iIvwtQzXBZuE513ySpVhhhgnQe701lC59GqsrHIx2jqQsu48
pmNx5CWomByoSrWgywQRx1a+6fpJ6XlRSlucf/assGDDQL/lqL4U3gMIquCmq+er4Y7L8br8
0gkd1GrpiRCC4+H5Mmn6wrGcwv6eSbVUxRZcEJP8gR302nCxZ9+Q5t3jHb8C0lzeQ9LMy5t7
2LX/+xW6Bs6WoWUUu+xoh2zGFrdYiJlXH3njDmssznT5s8pA/axK98Jsd64VM1YLxhzESUta
+ZXDjSsm1WqFpm7AtIlKsG7+zJGWVGeCnhaC9iYnVaDzkis27AgmR9yK2pj9mu3noe7582T5
wzJEuE01t69TQbzQaKtOOhUsuBT3FZHjIQm+ykgiXI2K8I4JJHPv4sgaCige9y4s6Qt0PAqM
XOxPNaMRkPQ4B70COqU+BpwIM+yTwAY2JVgOdAiYIkhBpnSqgruRHQY3SBR3/KyCXeGgCEdI
whKa8IQoTKEKV8jCFrrwhTCMoQxnSMMa2vCGOMyhDnfIwx768IdADKIQ/4dIxCIWxV5ITKIS
l8jEJjrxiVCMohSnSMUqWvGKWMyiFrfIxS56EYsM+6IYx0jGMprxjGhMoxrXWMYwoiAkUxFJ
HOH4Rjra8Y4gmWMe67hHPH5Ej5sCZEcEOUg+/tGQHiHkRhS5SEQWUo6O5AgjMzJJSkbyQpW8
SCY1eUmNbNIin6RIKEXZSUtC8pR99KOmRikRVraylBhxZURk+RBa1hKWnERlKg+py0DiMie2
lEkwYTJMYv4SJ8VcSDL9skyWNNOZx7zJM1cyzZRUsyDXrEs2TbJNbkbTJt0sSThHEs5xtsWc
qgwKOnc5lHXyMp1YcmcivwlOetJEnvM8oj0hgv/PrfTTiDP8Z1YEClAYEvQqBy1oCxMKFYYq
VIUOdUpEH3rCiTLFohQlIUaVstGMcrCjGJmcvuzlGfvxRTTvawhIPVrBlVaEd6KS1LuM4zcE
Hgc6NlXmPlkqRJdaJGPw6QcDX+MzH5V0goZiiE95msClUiRp6Wpf1JpXGnSZTqk7ZeoPnTqR
EsHvXWW7npguNiqdavWsXJUIVNWSPXrBwVMK29dA1JaQtJ6VbXaFCD00ZpAhSSap0koMYPVH
LZ0y8a5APOxT1joQmMm1ddt4bOwI+A8nItaHTYSKVy3TmfGJVTJkhaxZL+vRvJpIgzGyBMCm
c1XWXXW0pKWoaR0C1IT/XKaxRRXYUUMz2LpmNbY3nG1DwCUp2v3hRaVh2paY5lvgyva3OMkc
WE3aGm7tKExtVSl0nRvQ7TLTu9w1KHjzItzwVqi8OkGvee+jXmSOd70rbK803wvfFMq3nvXd
YDnpa5f75tco7+rlKvn73+wEWMBZ8m+B1WlOBc/EwQvuyYHJSeC4QDjC6V3nhY2JYc9NmCQb
fkmIO/xgeY64JScmsYjxmWJqVljFFu5ni1UyYxij5MMnqfGNX2zjs+A4xzx2i457DMkgK2TI
3iQyJg+KZHEaWckITWiTQfxkKD/lxzSuclmmbGVQOpTLCO5yZrDsYjeK+TdkLvO1wHxmh6RZ
/83WYnObDVsTOb9zzn15MzTNjGfydtTO+ezzXfSMYi2TBdCCJnSh+SxouCh60Ws2dKPnK+lX
MnrSZnm0SxAtyUpjOiaa3rSnBzrqTxe61LNEtVU4beVQr/jSpo6yqlMN61hfOa+s9uSsbV1k
oeTalLy+9a4tHelgS1S4v47lsI2t62UTO87OZraXo03rYkv7KK7Gr7WvTZRsaxva3O62fJOd
y3AzmNqkrLW5M4zudG973TvxNjDbjW16w3sg8p63uu9dYnu7G9z8pvSt9x1wUfu7IuT2csHv
GeKEIxyJCzd4Q/M9ZohH3MUHJyXFAePwu25cwhn3dch5/XGQE/ziFP8eebkBjvIdq3zlvmy5
yw/9cp90vLRDvvmzZd7rLdfc5Dzvuc9PHnRO/hzYLC96p4+O9Jgrfelv0Xm1n25Jpuua6FR3
s5ylzk+r97jkSOH6LbM+7f563b1k1/igzz7ftL+S7Y3EutsJAvaliF3rc9c63Dst97nX/aJ7
Z3je6fzdvpP97025u3YHjxDEJz7wJWa8QRz/eMMrnfKVfzfjMZ/5pA+e8513+ualrnisSh70
x7Y8ylGfes0f/u6lh23aWT9w1z+d9rX3/O1jH/sjQ56nuJ+46u8dfOHbXubFN77ukd97fP8e
1M9XaPJXHf1XF3361B/+tbGf/eMXnPvdX/7/95t/EPKXv/o9Nf/k0Q/pi4MfK+rHJvsxG//1
az/W7yf1/T+df/17n9v953/iB4D111z/x2wBqBUFKBAL6EbzF3kHSHINaIADGGwJOBYXZiwI
MxoMxIHYxTWmZ24XiIG/hxt8dRw20h38koIkkoLX03gPKEMjSIKaJSqCwhaAdYM+MjPz0lvn
F24zSINPsVcGQQ+3YlODEzKlkYSX4nsEGINtt1gCFFqTNTSjYoWiRYHGFoQ0V4MFUSrMkTXK
Qy/AkjONNS4wKG1c2IVSGB7ZEjZsMjBw+IbC80EMmFmNtobUp0RQUVuNNYb/kDHj4ylA9TB0
Y1hLlIcT6FuJ2IZO/0IigkWF4IGFk+WEEgiF8RZ9mxUjsOM2fzMjrWVVOZWGtqaHmaaJNtiD
cMCDOLiKZNiKdniHpbiIE7FhRGgQxOAdzkMj3gEKRWAlpZGLIch/tFiLJWgvHZhd/KKMJqU9
P0iMmGhz0ZhkmGaKQjaNTlaNxfhw+xdh1hh12EhlihiO0tiN//WNjkaOYdZq26hs5ghf6Bhj
72he8SiPEUhi9WiPFahi+aiPogdl/eiPA9ZlASmQCaaOa1OQBhlPCAk0CrmQS6ZkDwmRFtKO
PteQAIaRgQZjE6lNGvlIHGmR61iRH7lkJalP8wh8IklhKYlzJ5mRLflcLwmT9whcHZlnM//p
jgt2kzgZkwDFkz1Zk4gFlEG5j6RFlEX5j/S4klnmk+mXk3YHlQq3Xkj5Fw3IlBkplVHplPSn
lVsplBlVlRXHlTsklmMJltKHldZnlGGplmuplGjllm85kJdllr5xlV7pY3LJYWg5RHaJZnm5
cyoZmOEHl21JmIVJl4OpkmQpXogZZY3pQn+5HXjJUpNJmY+Jd4fpcZm5eDI5lJ05jAV1mfhR
mWkZmgJomE8ZW6ZpRKR5XqhpiUX0mrAZmRq1l+XYl90Vm0IIbfaSWLgJcnj5m13JXRbHljVE
mwdpmy0VnAzGnBGknIqpmsHlnCIHnU1lndepmxClnduJnJLpnd//SZ0yKJ7jOZ02JJ0N05rp
aZ7txJuk2J7waXbYmZDuKW716ZD3iZ/c+VH7yZ/gWULq2Tns+UIDSqDzaX8xdKAImp8DlqCF
158OxKAeBqF0Z6FOhqFJiZ729Z8cpaGVBaJ7JKJ+5qD5QaEFVKAmhKIpCqIeGqIkGqEBmp0x
KqPk2Zw1aqMc6p85qqPLKaAvqnw32kAsGp0u2qOphqRWeaQjVKQUpKJGqqQcx6T6FaSpuaM0
aoFU+qRWqoBbOqFd6qUm6mdSepdfakBO2qRnCj1pqqZj6pFlChxQildhOnQSWpp1aqczapLw
Nqfrmaen+KZ6GafstaZ/SqiFKqg0h6iJ/3qnZsqojbqn2tGmLOSnDAmpeKqo8AeoFFmbDthy
lnqinOqRblE9mmNSyQiC8fmgPMdlLSI+A5GE7kAiyTGK8uebQTdlzaEv46I1ZEgerviMB4mp
DHkW7CFVKIgdvtiEsPKCzseq14eNplo2kAgn1SqsJEmsP1oWPcI+tBOsAiENPkOH01ItF6pY
gKmthcqHQgMHkiFZcfVWMYOG59qIjwqq6OoW/oACntFbkwiulaigoqquWBo9wkGJkyiJGtR7
lIpDcuYPluAqQUIssDKxoOCssoinBDukWXEPUmI3wQqyH1R6DZtDTQYKM0MxTZhTnuiJqyqn
o7oZTcY9VsNWuf/xgZ4isDC7sY7qo5pRsmVpqFPKs5oqpD8bszBbtNJItH+qtDwBtPTntOwm
eRcqte7FtEETqumItVkrtHBKtTpbsFsLtmG7rXgBtX7ptdfItQ3as4DHtm0rqXtItloothdJ
t3VrtuAIt8+jtV7Kt32rtv6EtHJ7tGOLt7Jnt5sKuC1qtdZEuI77oecEuZGbFE2GtpYpuMKG
uF1XuULHuZ5ZuO/JuEbquYdEuqXrtk9LuXRqulAHuoKpt5sLu7FbrLJGu8bousqGulWqu2WH
u7mrunXGuo0rvP0GvNxovNDHu26qvCvGvM0ruvUEvdHLsUuLvDopvceLvTCnuMLJvdn/a71T
C77dK7vnRr7h673zRr0o5LdXi75Np77TC7/xa76ry77d6bvrh7/567yfS7/la7vXC8BX57+n
S8BxZ8CFxL8LpbnLi8B8p8DNBsER7JtciLlfh8EaF4QaHJIdXLvCRLwN7LkfTGTu+7gM3F2R
W8IS6cAzR8HwFJHbC8PsJL9UlsLt6bQsTJAunHI0LI4SrF047LA9PKI/nI1B7HtDbLJFvMBH
TI3aq3ZPjMRR/HZTDMXi20hLzEMnPMFXjMU2vLtfDGT1ucPaiJ1mfMZJnMZqXMVKPMZNKcFs
TIyROcd0bMB2fMduLH9bPJtNrHdwLGpOmcf498eEF8jtl8VY/9XHo2nIMMjIjZyShKyl8zjJ
lOyolnzJ0pvJmqzIfIzIw7t/nLyFjjzKpKx9pnzKfZnKqjyjrNzKWfzKsCy/sjzL9ru/oLy6
clfLCKi5vNzLRPfLwPx/wjzMA1jMxkydIkyVQrvMzKxu1ohe5SXNTLdftfaN1IyS2vyeI+mp
zpdx2Qyg3CzOG/mP8RjO40zOQGHN21aP6Hyez6nOFex0f8fONQySd5bP5bzP+MzP8/ygiGfP
+tzPBF3Q/2zQCTzQB32p1yTQ/pzQD13A97zQEI3QEu1LbJTRGr3RHN3RHv3RIJ1EDhjSJF3S
Jn3SKJ3SaJTLLN3SLv3SME3AHkgU3P8zVDO9jKrqEj5406gqE3OC06k6VCdxMsFjs0F9Exr4
gTadUkBdIUloqz6hDCWFILcCMrQhViwIE4gyVYG4HEboK0KNEnvxi1w9qwTx1V3NEjlIUy64
gmqBsS5hgvCRhPRgHzalgslq1dshskFxD+a6F3Gjiqy4g3F9Xb/qVjwIrLEYHKmo2If9GgB7
EroDGZWhgwIz2JdtE0nD169I2OIBi/cxirs4FFJRUoF4hFXFWlA91KByqsqBHajzOJAB1yXR
L6IxsbFtsbQtEjTDHqyi2kuI2pp9grJdIHPDhFYtimENGAiLWj7BMeNTrc1NWSGxFjAT3Zsz
3SshFdoz3dr/nRJ40iRzQTDSrbDUnRLEZd7YfYXqnR29Bd0dBCAXdIaRBYg0kxL+QBfX/d6C
BbA9s9ggAQoskDzZxd+r4d+5Bd5E8rH2XYamQq8xcYueTViJYoa/aoj0rR3wCuHUwTzeUiYK
Qq5FbRL+cAm3c6obHlnyOlccThI6glM3m+IpzhK4QSAiHocS8i81AS4zfuM3rh3+6tw8YQzh
0SyHiA8r7itCDhLSMDOnGuQH/i1L/hGDpYFQToeMRRLSQBcqIhlgGKuJYeQBMtwbArBiIuat
8w9ITohTnhffLRTM4B/e3d4n4bGN9eTm/eZ1nuS0UYblfa3gPS/EUTl5Tucykd7X/zrngJ4d
H7OsQ2EPP6Pcn+jW4K1EhfUxwlgzwMgSlHA9PZMWmN6LyzHaJiGIA/HVMkLpkl4TflixVjI3
kp7qKHgfnB0UzCAhowA5oP3ZE07jrs3ZtY7ebP3Yvioejq0S4DLZlp2Dgj3cRd6r4LrsIdvs
29Gyq80T+/Mu14M6u3ixxg0T1/03lcOyd73cweE742LtxVLuau01aQE3s/3tIxOM5s4Sm3gp
Ljvvrh7v6bLbmzEaOfvcSuQfo6E9BR8T4W6zAQ/wLyFTyKrwOKvTP4ICI37wKHUzNTGtFx/w
tMGMNuuMMR3yIj/yJF/yJl/yuEHWJ69QIQRBK69QsgLgL///Q+M98wq1F/5u8w0kOiFRHbYd
JiPO2xyv81kBFqbdEKCBNEx9Nj6wFwmOMxR/3hsB30Rf9LnSEMcjKrOy2s1BVwWh4xFxDzKP
EBNU9VihDCp/WuHRD7169THSDGkhrgtz3w8RsRaB12ZvFfewLWrhH/dAOkmyNAeSIGzh9SA+
9tIS9BAxQHlvFcj9FemBW7AAItjR3Anx3x1/UsY+095TVdXFPW0/FaxFJLBgrnrd+E3BXF/B
J2VfVh+e+B8ENiUeHmbCB2Ix6ONj4ybuD3WIW7cDBybDJ6c/IFKP+j8x31wCKhp0IpVv4cZ+
NYI1ByNl2rfI9rNhORkO+X1TVFf/b/p0b/xIQYfMwfeUPyZSvjDiMg2Z0vv+ojmvFeew7yNJ
JBby8jNgD/5JgeYEgQ9yMDEbaDEA4ePfwH/KVhBEOFAaChQrqiX8d0/FNoL3UNwayMyhv1EI
JVL8h09OLIgDLaKYmPBjSZYtXb6EGVPmTJo1bd7EmVPnTp49ff4EGnSgP0oCVY6MCAgkQoNL
/4HCsRSU0ZL2JkJ1+o+ZCpIZuQ4tyqwRwq1d/cHpypIZSqdlhb6FG1fuXLp17d7FC1cZVbBp
7/2A2BRhv5QRKaWFKG3FtrNFVBb2pwyHx4ZOV/7zh2tyRZKqKqJA7G5xXtKlTZ9GnVr1apjM
ojJFHHEsydPRBP0RE3ibL0SrFO2BJgv8n73aYHeX9UfSYkp7NzBTMjR0jtOprK1fx55d+/bU
/uRkfXmSIWCmDBm1jMwQ9MLKBNmj2JyQGXhKKJybrI/CqD9V7hgeRMifS2LjrkADD0QwQe72
UnAoUsDjyZ7dGqSwQgsvxJAmoggs8J7ogroHwAxHJLFEEw8k6sMO/4NQpxBbPDFGGWek8S13
OLzOIhF/8kfFGn8EMkghhySySCOPRDJJJZdkskknn4QySimnpLJKK6/E0sqAAAA7
--------------010403060600020408010201
Content-Type: text/plain; charset="iso-8859-1"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

_______________________________________________
cgl_discussion mailing list
[email protected]
http://lists.osdl.org/mailman/listinfo/cgl_discussion

--------------010403060600020408010201--