Re: Bug in lftp mirror when using xfer:use-temp-file option
akshay gupta <[email protected]> Mon, 27 Oct 2014 17:53:19 +0530
| Newsgroups | gmane.network.lftp.user,gmane.network.lftp.devel |
|---|---|
| Message-ID | <CAB_qc73jArYF1RvGrUM0ATqRM8vCcwcEhaph9B==BjEYXFXZ7w@mail.gmail.com> |
--===============6438424569747366228== Content-Type: multipart/alternative; boundary=047d7b450e06c4865c050666996f --047d7b450e06c4865c050666996f Content-Type: text/plain; charset=UTF-8 now its producing 0 byte files and i am also getting input/output error here are the debug logs: lftp [email protected]:~/Desktop/lftp> mirror --ignore-time ---> PASV <--- 227 Entering Passive Mode (192,168,128,124,248,18) ---- Connecting data socket to (192.168.128.124) port 63506 ---- Data connection established ---> LIST <--- 150 Opening ASCII mode data connection for '/bin/ls'. ---- Got EOF on data connection ---- Closing data socket <--- 226 Transfer complete. ---> TYPE I <--- 200 Type set to I. ---> PASV <--- 227 Entering Passive Mode (192,168,128,124,248,19) ---- Connecting data socket to (192.168.128.124) port 63507 ---- Data connection established ---> RETR test.txt <--- 150 Opening BINARY mode data connection for 'test.txt' (240 bytes). ---- Got EOF on data connection ---- Closing data socket <--- 226 Transfer complete. copy: renaming `/mount/user/tom/test.txt_tmp_' to `test.txt' mirror: rename(/mount/user/tom/test.txt_tmp_, /mount/user/tom/test.txt): Input/output error ---> PASV <--- 227 Entering Passive Mode (192,168,128,124,248,51) ---- Connecting data socket to (192.168.128.124) port 63539 ---- Data connection established ---> RETR .DS_Store <--- 150 Opening BINARY mode data connection for '.DS_Store' (6148 bytes). ---- Got EOF on data connection ---- Closing data socket <--- 226 Transfer complete. copy: renaming `/mount/user/tom/.DS_Store_tmp_' to `.DS_Store' mirror: rename(/mount/user/tom/.DS_Store_tmp_, /mount/user/tom/.DS_Store): Input/output error ---> PASV <--- 227 Entering Passive Mode (192,168,128,124,248,131) ---- Connecting data socket to (192.168.128.124) port 63619 ---- Data connection established ---> RETR settings.xml <--- 150 Opening BINARY mode data connection for 'settings.xml' (1737 bytes). <--- 226 Transfer complete. ---- Got EOF on data connection ---- Closing data socket copy: renaming `/mount/user/tom/settings.xml_tmp_' to `settings.xml' mirror: rename(/mount/user/tom/settings.xml_tmp_, /mount/user/tom/settings.xml): Input/output error ---> PASV <--- 227 Entering Passive Mode (192,168,128,124,248,178) ---- Connecting data socket to (192.168.128.124) port 63666 ---- Data connection established ---> RETR search-2.html <--- 150 Opening BINARY mode data connection for 'search-2.html' (42518 bytes). <--- 226 Transfer complete. ---- Got EOF on data connection ---- Closing data socket copy: renaming `/mount/user/tom/search-2.html_tmp_' to `search-2.html' mirror: rename(/mount/user/tom/search-2.html_tmp_, /mount/user/tom/search-2.html): Input/output error ---> PASV <--- 227 Entering Passive Mode (192,168,128,124,249,146) ---- Connecting data socket to (192.168.128.124) port 63890 ---- Data connection established ---> RETR search-3.html <--- 150 Opening BINARY mode data connection for 'search-3.html' (42569 bytes). <--- 226 Transfer complete. ---- Got EOF on data connection ---- Closing data socket copy: renaming `/mount/user/tom/search-3.html_tmp_' to `search-3.html' mirror: rename(/mount/user/tom/search-3.html_tmp_, /mount/user/tom/search-3.html): Input/output error ---> PASV <--- 227 Entering Passive Mode (192,168,128,124,250,160) ---- Connecting data socket to (192.168.128.124) port 64160 ---- Data connection established ---> RETR search-4.html <--- 150 Opening BINARY mode data connection for 'search-4.html' (42569 bytes). <--- 226 Transfer complete. ---- Got EOF on data connection ---- Closing data socket copy: renaming `/mount/user/tom/search-4.html_tmp_' to `search-4.html' mirror: rename(/mount/user/tom/search-4.html_tmp_, /mount/user/tom/search-4.html): Input/output error ---> PASV <--- 227 Entering Passive Mode (192,168,128,124,251,142) ---- Connecting data socket to (192.168.128.124) port 64398 ---- Data connection established ---> RETR search.html <--- 150 Opening BINARY mode data connection for 'search.html' (42519 bytes). <--- 226 Transfer complete. ---- Got EOF on data connection ---- Closing data socket copy: renaming `/mount/user/tom/search.html_tmp_' to `search.html' mirror: rename(/mount/user/tom/search.html_tmp_, /mount/user/tom/search.html): Input/output error Total: 1 directory, 7 files, 0 symlinks Modified: 7 files, 0 symlinks 178300 bytes transferred in 47 seconds (3.7 KiB/s) To be removed: 0 directories, 7 files, 0 symlinks 7 errors detected On Mon, Oct 27, 2014 at 5:01 PM, Alexander Lukyanov <[email protected]> wrote: > This change should fix the problem. > > https://github.com/lavv17/lftp/commit/b1164d695393cf8ec8429bb2b5caa9f05cfc4d39 > > 2014-10-27 14:20 GMT+03:00 Alexander Lukyanov <[email protected]>: > >> I have found the bug. As a workaround, use "set xfer:clobber yes". >> >> 2014-10-27 13:06 GMT+03:00 akshay gupta <[email protected]>: >> >>> no, I am running as a root user. There is some issue with lftp, when >>> file with same name already exist its not able to rename the temp file to >>> actual file. >>> >>> On Mon, Oct 27, 2014 at 1:44 PM, Alexander V. Lukyanov <[email protected]> >>> wrote: >>> >>>> On Mon, Oct 27, 2014 at 01:30:42PM +0530, akshay gupta wrote: >>>> > ---> RETR test.txt >>>> > <--- 150 Opening BINARY mode data connection for 'test.txt' (202 >>>> bytes). >>>> > ---- Got EOF on data connection >>>> > ---- Closing data socket >>>> > <--- 226 Transfer complete. >>>> >>>> So the file cannot be renamed locally, right? Is there a permissions >>>> problem? >>>> >>>> -- >>>> Alexander. >>>> >>>> > On Mon, Oct 27, 2014 at 1:18 PM, Alexander Lukyanov < >>>> [email protected]> >>>> > wrote: >>>> > >>>> > > Can you provide a debug log? Why lftp cannot replace the old file? >>>> > > >>>> > > 2014-10-27 10:44 GMT+03:00 akshay gupta <[email protected]>: >>>> > > >>>> > >> Hi, >>>> > >> >>>> > >> I am trying to mirror files using lftp, I have tested a use case >>>> where a >>>> > >> file at the source machine gets modified but the file name remains >>>> the >>>> > >> same. After mirroring, lftp is unable to replace the old file and >>>> also >>>> > >> leaves a temporary file. >>>> > >> >>>> > >> For example. If source have file a.txt and its mirrored >>>> successfully at >>>> > >> the destination, now if I update a.txt append some content and >>>> save it with >>>> > >> same name. If I run lftp mirror again then its not able to replace >>>> the >>>> > >> existing a.txt but it also leaves the temp file a.txt_transit_ >>>> behind. And >>>> > >> if i run the mirror multiple times it always downloads the changed >>>> a.txt, i >>>> > >> can see this from the lftp logs. >>>> > >> >>>> > >> Suggestion: Will it make sense to keep versions of the files as >>>> they are >>>> > >> updated at the source. Like if a.txt is changed at source then >>>> lftp mirror >>>> > >> will save it as a.txt_{TIMESTAMP}_. Every time a file is updated >>>> it will be >>>> > >> saved with a new time stamp. And mirror will consider these file >>>> to decide >>>> > >> whether to download or not. >>>> > >> >>>> > >> >>>> > >> Thanks, >>>> > >> Akshay >>>> > >> >>>> > >> >>>> > >> >>>> > >> >>>> > >> >>>> > > >>>> > > >>>> > > -- >>>> > > Alexander. >>>> > > >>>> >>>> > _______________________________________________ >>>> > lftp mailing list >>>> > [email protected] >>>> > http://univ.uniyar.ac.ru/mailman/listinfo/lftp >>>> >>>> >>> >> >> >> -- >> Alexander. >> > > > > -- > Alexander. > --047d7b450e06c4865c050666996f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">now its producing 0 byte files and i am also getting input= /output error here are the debug logs:<div><br>lftp [email protected]= .124:~/Desktop/lftp> mirror --ignore-time<br>---> PASV<br><--- 227= Entering Passive Mode (192,168,128,124,248,18)=C2=A0=C2=A0 <br>---- Connec= ting data socket to (192.168.128.124) port 63506<br>---- Data connection es= tablished<br>---> LIST<br><--- 150 Opening ASCII mode data connection= for '/bin/ls'.<br>---- Got EOF on data connection<br>---- Closing = data socket<br><--- 226 Transfer complete.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 <br>---> TYPE I<br><--- 200 Type set to I.<br>---> PA= SV<br><--- 227 Entering Passive Mode (192,168,128,124,248,19)<br>---- Co= nnecting data socket to (192.168.128.124) port 63507<br>---- Data connectio= n established<br>---> RETR test.txt<br><--- 150 Opening BINARY mode d= ata connection for 'test.txt' (240 bytes).<br>---- Got EOF on data = connection<br>---- Closing data socket<br><--- 226 Transfer complete.=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <br>copy: renaming `/mount/user/= tom/test.txt_tmp_' to `test.txt'<br>mirror: rename(/mount/user/tom/= test.txt_tmp_, /mount/user/tom/test.txt): Input/output error<br><br>--->= PASV<br><--- 227 Entering Passive Mode (192,168,128,124,248,51)<br>----= Connecting data socket to (192.168.128.124) port 63539<br>---- Data connec= tion established<br>---> RETR .DS_Store<br><--- 150 Opening BINARY mo= de data connection for '.DS_Store' (6148 bytes).<br>---- Got EOF on= data connection=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 <br>---- Closing data socket<br><--- 226 Transfer comple= te.<br>copy: renaming `/mount/user/tom/.DS_Store_tmp_' to `.DS_Store= 9;<br>mirror: rename(/mount/user/tom/.DS_Store_tmp_, /mount/user/tom/.DS_St= ore): Input/output error<br><br>---> PASV<br><--- 227 Entering Passiv= e Mode (192,168,128,124,248,131)<br>---- Connecting data socket to (192.168= .128.124) port 63619<br>---- Data connection established<br>---> RETR se= ttings.xml<br><--- 150 Opening BINARY mode data connection for 'sett= ings.xml' (1737 bytes).<br><--- 226 Transfer complete.<br>---- Got E= OF on data connection<br>---- Closing data socket<br>copy: renaming `/mount= /user/tom/settings.xml_tmp_' to `settings.xml'<br>mirror: rename(/m= ount/user/tom/settings.xml_tmp_, /mount/user/tom/settings.xml): Input/outpu= t error<br><br>---> PASV<br><--- 227 Entering Passive Mode (192,168,1= 28,124,248,178)<br>---- Connecting data socket to (192.168.128.124) port 63= 666<br>---- Data connection established<br>---> RETR search-2.html<br>&l= t;--- 150 Opening BINARY mode data connection for 'search-2.html' (= 42518 bytes).<br><--- 226 Transfer complete.=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <br>---- Got EOF on data connection= <br>---- Closing data socket<br>copy: renaming `/mount/user/tom/search-2.ht= ml_tmp_' to `search-2.html'<br>mirror: rename(/mount/user/tom/searc= h-2.html_tmp_, /mount/user/tom/search-2.html): Input/output error<br><br>--= -> PASV<br><--- 227 Entering Passive Mode (192,168,128,124,249,146)<b= r>---- Connecting data socket to (192.168.128.124) port 63890<br>---- Data = connection established<br>---> RETR search-3.html<br><--- 150 Opening= BINARY mode data connection for 'search-3.html' (42569 bytes).<br>= <--- 226 Transfer complete.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 <br>---- Got EOF on data connection=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 <br>---- Closing data socket<br>copy: renaming `/mount/user/to= m/search-3.html_tmp_' to `search-3.html'<br>mirror: rename(/mount/u= ser/tom/search-3.html_tmp_, /mount/user/tom/search-3.html): Input/output er= ror<br><br>---> PASV<br><--- 227 Entering Passive Mode (192,168,128,1= 24,250,160)<br>---- Connecting data socket to (192.168.128.124) port 64160<= br>---- Data connection established<br>---> RETR search-4.html<br><--= - 150 Opening BINARY mode data connection for 'search-4.html' (4256= 9 bytes).<br><--- 226 Transfer complete.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <br>---- Got EOF on data connection=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 <br>---- Closing data socket<br>copy: renaming = `/mount/user/tom/search-4.html_tmp_' to `search-4.html'<br>mirror: = rename(/mount/user/tom/search-4.html_tmp_, /mount/user/tom/search-4.html): = Input/output error<br><br>---> PASV<br><--- 227 Entering Passive Mode= (192,168,128,124,251,142)<br>---- Connecting data socket to (192.168.128.1= 24) port 64398<br>---- Data connection established<br>---> RETR search.h= tml<br><--- 150 Opening BINARY mode data connection for 'search.html= ' (42519 bytes).<br><--- 226 Transfer complete.=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 <br>---- Got EOF on data connection<br>---- Clo= sing data socket<br>copy: renaming `/mount/user/tom/search.html_tmp_' t= o `search.html'<br>mirror: rename(/mount/user/tom/search.html_tmp_, /mo= unt/user/tom/search.html): Input/output error<br><br>Total: 1 directory, 7 = files, 0 symlinks<br>Modified: 7 files, 0 symlinks<br>178300 bytes transfer= red in 47 seconds (3.7 KiB/s)<br>To be removed: 0 directories, 7 files, 0 s= ymlinks<br>7 errors detected<br><br></div><div><br></div><div><br></div></d= iv><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Mon, Oct 27= , 2014 at 5:01 PM, Alexander Lukyanov <span dir=3D"ltr"><<a href=3D"mail= to:[email protected]" target=3D"_blank">[email protected]</a>></span> wr= ote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border= -left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">This change should = fix the problem.<br><a href=3D"https://github.com/lavv17/lftp/commit/b1164d= 695393cf8ec8429bb2b5caa9f05cfc4d39" target=3D"_blank">https://github.com/la= vv17/lftp/commit/b1164d695393cf8ec8429bb2b5caa9f05cfc4d39</a><br></div><div= class=3D"gmail_extra"><div><div class=3D"h5"><br><div class=3D"gmail_quote= ">2014-10-27 14:20 GMT+03:00 Alexander Lukyanov <span dir=3D"ltr"><<a hr= ef=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>>= </span>:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bo= rder-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">I have found th= e bug. As a workaround, use "set xfer:clobber yes".<br></div><div= class=3D"gmail_extra"><div><div><br><div class=3D"gmail_quote">2014-10-27 = 13:06 GMT+03:00 akshay gupta <span dir=3D"ltr"><<a href=3D"mailto:akshay= [email protected]" target=3D"_blank">[email protected]</a>></spa= n>:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-= left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">no, I am running as = a root user. There is some issue with lftp, when file with same name alread= y exist its not able to rename the temp file to actual file.</div><div><div= ><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Mon, Oct 27, = 2014 at 1:44 PM, Alexander V. Lukyanov <span dir=3D"ltr"><<a href=3D"mai= lto:[email protected]" target=3D"_blank">[email protected]</a>></span> wrote:<br><= blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px= #ccc solid;padding-left:1ex"><span>On Mon, Oct 27, 2014 at 01:30:42PM +053= 0, akshay gupta wrote:<br> > ---> RETR test.txt<br> > <--- 150 Opening BINARY mode data connection for 'test.txt'= (202 bytes).<br> > ---- Got EOF on data connection<br> > ---- Closing data socket<br> > <--- 226 Transfer complete.<br> <br> </span>So the file cannot be renamed locally, right? Is there a permissions= problem?<br> <br> --<br> =C2=A0 =C2=A0Alexander.<br> <span><br> > On Mon, Oct 27, 2014 at 1:18 PM, Alexander Lukyanov <<a href=3D"mai= lto:[email protected]" target=3D"_blank">[email protected]</a>><br> > wrote:<br> ><br> > > Can you provide a debug log? Why lftp cannot replace the old file= ?<br> > ><br> > > 2014-10-27 10:44 GMT+03:00 akshay gupta <<a href=3D"mailto:aks= [email protected]" target=3D"_blank">[email protected]</a>>:<= br> > ><br> > >> Hi,<br> > >><br> > >> I am trying to mirror files using lftp, I have tested a use c= ase where a<br> > >> file at the source machine gets modified but the file name re= mains the<br> > >> same. After mirroring, lftp is unable to replace the old file= and also<br> > >> leaves a temporary file.<br> > >><br> > >> For example. If source have file a.txt and its mirrored succe= ssfully at<br> > >> the destination, now if I update a.txt append some content an= d save it with<br> > >> same name. If I run lftp mirror again then its not able to re= place the<br> > >> existing a.txt but it also leaves the temp file a.txt_transit= _ behind. And<br> > >> if i run the mirror multiple times it always downloads the ch= anged a.txt, i<br> > >> can see this from the lftp logs.<br> > >><br> > >> Suggestion: Will it make sense to keep versions of the files = as they are<br> > >> updated at the source. Like if a.txt is changed at source the= n lftp mirror<br> > >> will save it as a.txt_{TIMESTAMP}_. Every time a file is upda= ted it will be<br> > >> saved with a new time stamp. And mirror will consider these f= ile to decide<br> > >> whether to download or not.<br> > >><br> > >><br> > >> Thanks,<br> > >> Akshay<br> > >><br> > >><br> > >><br> > >><br> > >><br> > ><br> > ><br> > > --<br> > >=C2=A0 =C2=A0 Alexander.<br> > ><br> <br> </span>> _______________________________________________<br> > lftp mailing list<br> > <a href=3D"mailto:[email protected]" target=3D"_blank">[email protected].= ru</a><br> > <a href=3D"http://univ.uniyar.ac.ru/mailman/listinfo/lftp" target=3D"_= blank">http://univ.uniyar.ac.ru/mailman/listinfo/lftp</a><br> <br> </blockquote></div><br></div> </div></div></blockquote></div><br><br clear=3D"all"><br></div></div><span>= <font color=3D"#888888">-- <br><div dir=3D"ltr"><div></div>=C2=A0=C2=A0 Ale= xander.<br></div> </font></span></div> </blockquote></div><br><br clear=3D"all"><br></div></div><span class=3D"HOE= nZb"><font color=3D"#888888">-- <br><div dir=3D"ltr"><div></div>=C2=A0=C2= =A0 Alexander.<br></div> </font></span></div> </blockquote></div><br></div> --047d7b450e06c4865c050666996f-- --===============6438424569747366228== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lftp mailing list [email protected] http://univ.uniyar.ac.ru/mailman/listinfo/lftp --===============6438424569747366228==--