Unexplainable hangs
Mattias Bergvall <[email protected]> Fri, 27 Oct 2017 13:49:54 +0000
| Newsgroups | gmane.network.lftp.user |
|---|---|
| Message-ID | <HE1PR0601MB2089CF367529990C133ED910F15A0@HE1PR0601MB2089.eurprd06.prod.outlook.com> |
--===============0207926880082855871==
Content-Language: en-US
Content-Type: multipart/alternative;
boundary="_000_HE1PR0601MB2089CF367529990C133ED910F15A0HE1PR0601MB2089_"
--_000_HE1PR0601MB2089CF367529990C133ED910F15A0HE1PR0601MB2089_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Currently running 4.6.0 built from source on Centos 7, but have had this pr=
oblem with earlier versions as well.
Client is running on a virtual machine (vmware).
Every now and then, when sending files with the FTP protocol, lftp (in batc=
h mode) hangs and uses 100% CPU on one core until being killed.
SIGHUP has no effect. SIGINT terminates it.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
384 username 20 0 25080 2860 1932 R 93.0 0.1 108:11.80 lftp
2543 username 20 0 25080 2860 1924 R 33.0 0.1 92:00.84 lftp
Typical batch file looks like this:
set dns:fatal-timeout 30
set net:max-retries 4
set cmd:fail-exit true
open hostname
user username password
put "File1.pdf" -o "/path//File1.pdf"
put "File2.pdf" -o "/path//File2.pdf"
Invoked with "lftp -f {path_to_batchfile} > {path_to_logfile} 2>&1" as a no=
n-root user.
The {path_to_logfile} is 0 byte when the process hangs.
My guess is that it is some kind of timing issue or race condition that fai=
ls to pick up an error somehow, but it is very difficult to provide more in=
formation, since it is quite rare and impossible to reliably provoke it int=
o happening again.
I use lots and lots of lftp calls daily, and this happens only intermittent=
ly and infrequently, so turning on debugging would cause a lot of extra out=
put that is not really needed, so I haven't done that for most transfers.
Any ideas what could cause these hangs?
I have not compiled with debugging info, but if I attach with gdb, continue=
and interrupt many times, it seems to be consistently interrupted inside l=
ibc: (apologies if this is misleading information):
(gdb) where
#0 0x0000000000477c9e in ?? ()
#1 0x0000000000475f5e in ?? ()
#2 0x0000000000473fdf in ?? ()
#3 0x000000000045831d in ?? ()
#4 0x00000000004586a0 in ?? ()
#5 0x0000000000423bc6 in ?? ()
#6 0x0000000000423bfe in ?? ()
#7 0x00000000004115c6 in ?? ()
#8 0x0000000000448721 in ?? ()
#9 0x0000000000448929 in ?? ()
#10 0x000000000040e03d in ?? ()
#11 0x00000000004060b3 in ?? ()
#12 0x00007f2d4d41bb15 in __libc_start_main () from /lib64/libc.so.6
#13 0x0000000000407575 in ?? ()
(gdb) n
Cannot find bounds of current function
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x00007f2d4d441957 in vfprintf () from /lib64/libc.so.6
(gdb) n
Single stepping until exit from function vfprintf,
which has no line number information.
0x00007f2d4d46e589 in vsnprintf () from /lib64/libc.so.6
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x00007f2d4d442ac0 in vfprintf () from /lib64/libc.so.6
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x00007f2d4d420489 in __gconv_transform_ascii_internal () from /lib64/libc.=
so.6
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x00007f2d4d441967 in vfprintf () from /lib64/libc.so.6
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x0000000000478e40 in ?? ()
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x00007f2d4d4418a9 in vfprintf () from /lib64/libc.so.6
/proc/{pid}/io of the lftp process doesn't change:
rchar: 612533
wchar: 131756
syscr: 101
syscw: 28
read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0
/proc/{pid}/sched of the lftp process changes:
lftp ({pid}, #threads: 1)
-------------------------------------------------------------------
se.exec_start : 4812627832.697335 (increa=
ses)
se.vruntime : 5842041293.625487 (increa=
ses)
se.sum_exec_runtime : 6292136.992748 (increa=
ses)
nr_switches : 425117 (increa=
ses)
nr_voluntary_switches : 303
nr_involuntary_switches : 424814 (increa=
ses)
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 34 (increa=
ses)
mm->numa_scan_seq : 0
numa_migrations, 0
numa_faults_memory, 0, 0, 1, 0, -1
numa_faults_memory, 1, 0, 0, 0, -1
I've enabled debugging on some transfers now, hoping to catch another one o=
f these hangs within a month or so.
Kind regards, and keep up the good work!
LFTP is really good
Mattias Bergvall
--_000_HE1PR0601MB2089CF367529990C133ED910F15A0HE1PR0601MB2089_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:8.5in 11.0in;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"SV" link=3D"#0563C1" vlink=3D"#954F72">
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><span lang=3D"EN-US">Currently running 4.6.0 built f=
rom source on Centos 7, but have had this problem with earlier versions as =
well.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Client is running on a virtual =
machine (vmware).<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Every now and then, when sendin=
g files with the FTP protocol, lftp (in batch mode) hangs and uses 100% CPU=
on one core until being killed.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">SIGHUP has no effect. SIGINT te=
rminates it.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US"> PID USER &nbs=
p; PR NI VIRT RES&nbs=
p; SHR S %CPU %MEM TIME+ COMM=
AND<o:p></o:p></span></p>
<p class=3D"MsoNormal"> 384 username 20 0 &nbs=
p; 25080 2860 1932 R 93.0 0.1 108:11.80=
lftp<o:p></o:p></p>
<p class=3D"MsoNormal">2543 username 20 0 250=
80 2860 1924 R 33.0 0.1 92:00.84 =
lftp<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Typical batch file looks like t=
his:<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">set dns:fatal-timeout 30<o:p></=
o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">set net:max-retries 4<o:p></o:p=
></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">set cmd:fail-exit true<o:p></o:=
p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">open hostname<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">user username password<o:p></o:=
p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">put "File1.pdf"=
-o "/path//File1.pdf"<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">put "File2.pdf"=
-o "/path//File2.pdf"<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Invoked with “lftp -f {pa=
th_to_batchfile} > {path_to_logfile} 2>&1” as a non-root us=
er.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">The {path_to_logfile} is 0 byte=
when the process hangs.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">My guess is that it is some kin=
d of timing issue or race condition that fails to pick up an error somehow,=
but it is very difficult to provide more information, since it is quite ra=
re and impossible to reliably provoke
it into happening again.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">I use lots and lots of lftp cal=
ls daily, and this happens only intermittently and infrequently, so turning=
on debugging would cause a lot of extra output that is not really needed, =
so I haven’t done that for most transfers.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Any ideas what could cause thes=
e hangs?<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">I have not compiled with debugg=
ing info, but if I attach with gdb, continue and interrupt many times, it s=
eems to be consistently interrupted inside libc: (apologies if this is misl=
eading information):<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">(gdb) where<o:p></o:p></span></=
p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">#0 0x0000000000477c9e in =
?? ()<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">#1 0x0000000000475f5e in =
?? ()<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">#2 0x0000000000473fdf in =
?? ()<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">#3 0x000000000045831d in =
?? ()<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">#4 0x00000000004586a0 in =
?? ()<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">#5 0x0000000000423bc6 in =
?? ()<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">#6 0x0000000000423bfe in =
?? ()<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">#7 0x00000000004115c6 in =
?? ()<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">#8 0x0000000000448721 in =
?? ()<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">#9 0x0000000000448929 in =
?? ()<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">#10 0x000000000040e03d in ?? ()=
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">#11 0x00000000004060b3 in ?? ()=
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">#12 0x00007f2d4d41bb15 in __lib=
c_start_main () from /lib64/libc.so.6<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">#13 0x0000000000407575 in ?? ()=
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">(gdb) n<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Cannot find bounds of current f=
unction<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">(gdb) c<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Continuing.<o:p></o:p></span></=
p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">^C<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Program received signal SIGINT,=
Interrupt.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">0x00007f2d4d441957 in vfprintf =
() from /lib64/libc.so.6<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">(gdb) n<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Single stepping until exit from=
function vfprintf,<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">which has no line number inform=
ation.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">0x00007f2d4d46e589 in vsnprintf=
() from /lib64/libc.so.6<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">(gdb) c<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Continuing.<o:p></o:p></span></=
p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">^C<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Program received signal SIGINT,=
Interrupt.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">0x00007f2d4d442ac0 in vfprintf =
() from /lib64/libc.so.6<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">(gdb) c<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Continuing.<o:p></o:p></span></=
p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">^C<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Program received signal SIGINT,=
Interrupt.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">0x00007f2d4d420489 in __gconv_t=
ransform_ascii_internal () from /lib64/libc.so.6<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">(gdb) c<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Continuing.<o:p></o:p></span></=
p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">^C<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Program received signal SIGINT,=
Interrupt.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">0x00007f2d4d441967 in vfprintf =
() from /lib64/libc.so.6<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">(gdb) c<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Continuing.<o:p></o:p></span></=
p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">^C<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Program received signal SIGINT,=
Interrupt.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">0x0000000000478e40 in ?? ()<o:p=
></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">(gdb) c<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Continuing.<o:p></o:p></span></=
p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">^C<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Program received signal SIGINT,=
Interrupt.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">0x00007f2d4d4418a9 in vfprintf =
() from /lib64/libc.so.6<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">/proc/{pid}/io of the lftp proc=
ess doesn’t change:<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">rchar: 612533<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">wchar: 131756<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">syscr: 101<o:p></o:p></span></p=
>
<p class=3D"MsoNormal"><span lang=3D"EN-US">syscw: 28<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">read_bytes: 0<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">write_bytes: 0<o:p></o:p></span=
></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">cancelled_write_bytes: 0<o:p></=
o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">/proc/{pid}/sched of the lftp p=
rocess changes:<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">lftp ({pid}, #threads: 1)<o:p><=
/o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">-------------------------------=
------------------------------------<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">se.exec_start =
&nb=
sp; =
: 4812627832.697335 (increases)<o:p></=
o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">se.vruntime &n=
bsp;  =
; &n=
bsp; : 5842041293.625487 (increas=
es)<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">se.sum_exec_runtime =
&nb=
sp; : &nbs=
p; 6292136.992748 (increases)<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">nr_switches &n=
bsp;  =
; &n=
bsp; : &nb=
sp; 425117 (increases)<o:p></o:p></span=
></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">nr_voluntary_switches &nbs=
p; &=
nbsp; : &n=
bsp;  =
; 303<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">nr_involuntary_switches &n=
bsp;  =
; : =
424814 (increases)<o=
:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">se.load.weight  =
; &n=
bsp;  =
; :  =
; 1024<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">policy &=
nbsp; &nbs=
p; &=
nbsp; : &n=
bsp;  =
; 0<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">prio &nb=
sp; =
&nb=
sp; : &nbs=
p; &=
nbsp; 120<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">clock-delta &n=
bsp;  =
; &n=
bsp; : &nb=
sp; 34 (increas=
es)<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">mm->numa_scan_seq  =
; &n=
bsp; &nbs=
p;: =
0<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">numa_migrations, 0<o:p></o:p></=
span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">numa_faults_memory, 0, 0, 1, 0,=
-1<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">numa_faults_memory, 1, 0, 0, 0,=
-1<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">I’ve enabled debugging on=
some transfers now, hoping to catch another one of these hangs within a mo=
nth or so.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Kind regards, and keep up the g=
ood work!<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">LFTP is really good<o:p></o:p><=
/span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US">Mattias Bergvall<o:p></o:p></sp=
an></p>
</div>
</body>
</html>
--_000_HE1PR0601MB2089CF367529990C133ED910F15A0HE1PR0601MB2089_--
--===============0207926880082855871==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
lftp mailing list
[email protected]
http://univ.uniyar.ac.ru/mailman/listinfo/lftp
--===============0207926880082855871==--