Re: Bug in lftp mirror when using xfer:use-temp-file option
Alexander Lukyanov <[email protected]> Tue, 28 Oct 2014 13:51:07 +0400
| Newsgroups | gmane.network.lftp.user,gmane.network.lftp.devel |
|---|---|
| Message-ID | <CANnoEwoFCgj0H6uEbohZr8dBr8qBpkyj3HeVJxPsyZbCLHRf1Q@mail.gmail.com> |
--===============1816448023420363191== Content-Type: multipart/alternative; boundary=001a1134cdee1f46fa05067896cd --001a1134cdee1f46fa05067896cd Content-Type: text/plain; charset=UTF-8 BTW, it looks like mirror should remove the old file before transferring the new one, please run "mirror -vvv" to check that. You can also run strace to check system call sequence. 2014-10-28 11:18 GMT+03:00 Alexander Lukyanov <[email protected]>: > It looks like a bug in the FS (fuse and/or hdfs) which also hits rsync: > > https://groups.google.com/a/cloudera.org/forum/#!msg/cdh-user/R1GR2Asj_lc/f_uuIjG5R1UJ > > Please try this workaround. > > 2014-10-28 10:29 GMT+03:00 akshay gupta <[email protected]>: > >> will giving --delete-first option in lftp mirror help in my case? I think >> its not working in case of use-temp-file option. As per the man page its >> suppose to delete the old version of the file before it start downloading >> the new one. Its not doing anything. Please help. >> >> >> On Mon, Oct 27, 2014 at 11:50 PM, akshay gupta <[email protected]> >> wrote: >> >>> I am using fuse mounted hadoop distributed file system(HDFS). Can you >>> suggest what changes need to be done to make it work as this is a very >>> critical requirement for my project. Will deleting file and then renaming >>> temp file help? how to achieve that? >>> >>> On Mon, Oct 27, 2014 at 7:38 PM, Alexander V. Lukyanov <[email protected]> >>> wrote: >>> >>>> On Mon, Oct 27, 2014 at 05:53:19PM +0530, akshay gupta wrote: >>>> > now its producing 0 byte files and i am also getting input/output >>>> error >>>> > here are the debug logs: >>>> > >>>> > copy: renaming `/mount/user/tom/test.txt_tmp_' to `test.txt' >>>> > mirror: rename(/mount/user/tom/test.txt_tmp_, >>>> /mount/user/tom/test.txt): >>>> > Input/output error >>>> >>>> Which filesystem do you use? Maybe it is necessary to unlink target file >>>> before renaming, although rename should do it automatically. >>>> >>>> If newpath already exists it will be atomically replaced >>>> (subject to a >>>> few conditions; see ERRORS below), so that there is no point >>>> at which >>>> another process attempting to access newpath will find it >>>> missing. >>>> >>>> And man page for rename(2) does not list EIO error. >>>> >>>> -- >>>> Alexander. >>>> >>> >>> >> > > > -- > Alexander. > -- Alexander. --001a1134cdee1f46fa05067896cd Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">BTW, it looks like mirror should remove the old file befor= e transferring the new one, please run "mirror -vvv" to check tha= t. You can also run strace to check system call sequence.<br></div><div cla= ss=3D"gmail_extra"><br><div class=3D"gmail_quote">2014-10-28 11:18 GMT+03:0= 0 Alexander Lukyanov <span dir=3D"ltr"><<a href=3D"mailto:lavv17f@gmail.= com" target=3D"_blank">[email protected]</a>></span>:<br><blockquote cla= ss=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pa= dding-left:1ex"><div dir=3D"ltr"><div>It looks like a bug in the FS (fuse a= nd/or hdfs) which also hits rsync:<br><a href=3D"https://groups.google.com/= a/cloudera.org/forum/#!msg/cdh-user/R1GR2Asj_lc/f_uuIjG5R1UJ" target=3D"_bl= ank">https://groups.google.com/a/cloudera.org/forum/#!msg/cdh-user/R1GR2Asj= _lc/f_uuIjG5R1UJ</a><br><br></div>Please try this workaround.<br></div><div= class=3D"gmail_extra"><div><div class=3D"h5"><br><div class=3D"gmail_quote= ">2014-10-28 10:29 GMT+03:00 akshay gupta <span dir=3D"ltr"><<a href=3D"= mailto:[email protected]" target=3D"_blank">[email protected]= </a>></span>:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0= .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">will gi= ving --delete-first option in lftp mirror help in my case? I think its not = working in case of use-temp-file option. As per the man page its suppose to= delete the old version of the file before it start downloading the new one= . Its not doing anything. Please help.<div><br></div></div><div><div><div c= lass=3D"gmail_extra"><br><div class=3D"gmail_quote">On Mon, Oct 27, 2014 at= 11:50 PM, akshay gupta <span dir=3D"ltr"><<a href=3D"mailto:akshaygupta= [email protected]" target=3D"_blank">[email protected]</a>></span> wr= ote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border= -left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">I am using fuse mou= nted hadoop distributed file system(HDFS). Can you suggest what changes nee= d to be done to make it work as this is a very critical requirement for my = project. Will deleting file and then renaming temp file help? how to achiev= e that?</div><div><div><div class=3D"gmail_extra"><br><div class=3D"gmail_q= uote">On Mon, Oct 27, 2014 at 7:38 PM, Alexander V. Lukyanov <span dir=3D"l= tr"><<a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>&= gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 = 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On Mon, Oct 27, 2= 014 at 05:53:19PM +0530, akshay gupta wrote:<br> > now its producing 0 byte files and i am also getting input/output erro= r<br> > here are the debug logs:<br> ><br> </span><span>> copy: renaming `/mount/user/tom/test.txt_tmp_' to `te= st.txt'<br> > mirror: rename(/mount/user/tom/test.txt_tmp_, /mount/user/tom/test.txt= ):<br> > Input/output error<br> <br> </span>Which filesystem do you use? Maybe it is necessary to unlink target = file<br> before renaming, although rename should do it automatically.<br> <br> =C2=A0 =C2=A0 =C2=A0 =C2=A0If=C2=A0 newpath already exists it will be atomi= cally replaced (subject to a<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0few conditions; see ERRORS below), so that there= is no point=C2=A0 at=C2=A0 which<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0another process attempting to access newpath wil= l find it missing.<br> <br> And man page for rename(2) does not list EIO error.<br> <span><font color=3D"#888888"><br> --<br> =C2=A0 =C2=A0Alexander.<br> </font></span></blockquote></div><br></div> </div></div></blockquote></div><br></div> </div></div></blockquote></div><br><br clear=3D"all"><br></div></div><span = class=3D"HOEnZb"><font color=3D"#888888">-- <br><div dir=3D"ltr"><div></div= >=C2=A0=C2=A0 Alexander.<br></div> </font></span></div> </blockquote></div><br><br clear=3D"all"><br>-- <br><div dir=3D"ltr"><div><= /div>=C2=A0=C2=A0 Alexander.<br></div> </div> --001a1134cdee1f46fa05067896cd-- --===============1816448023420363191== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lftp mailing list [email protected] http://univ.uniyar.ac.ru/mailman/listinfo/lftp --===============1816448023420363191==--