Re: Bug in lftp mirror when using xfer:use-temp-file option
akshay gupta <[email protected]> Tue, 28 Oct 2014 15:51:16 +0530
| Newsgroups | gmane.network.lftp.user,gmane.network.lftp.devel |
|---|---|
| Message-ID | <CAB_qc72P9dpWM4PwX+9wgRqwmrv9EVNVOv1US7oykGyuG1F7+w@mail.gmail.com> |
--===============0232788658605975526== Content-Type: multipart/alternative; boundary=047d7ba9763223449d05067903df --047d7ba9763223449d05067903df Content-Type: text/plain; charset=UTF-8 Thanks for the patch Alexander, files seems to be getting replaced now. Ill do some more testing to make sure every thing works perfectly. Thanks again, Akshay On Tue, Oct 28, 2014 at 3:21 PM, Alexander Lukyanov <[email protected]> wrote: > BTW, it looks like mirror should remove the old file before transferring > the new one, please run "mirror -vvv" to check that. You can also run > strace to check system call sequence. > > 2014-10-28 11:18 GMT+03:00 Alexander Lukyanov <[email protected]>: > >> It looks like a bug in the FS (fuse and/or hdfs) which also hits rsync: >> >> https://groups.google.com/a/cloudera.org/forum/#!msg/cdh-user/R1GR2Asj_lc/f_uuIjG5R1UJ >> >> Please try this workaround. >> >> 2014-10-28 10:29 GMT+03:00 akshay gupta <[email protected]>: >> >>> will giving --delete-first option in lftp mirror help in my case? I >>> think its not working in case of use-temp-file option. As per the man page >>> its suppose to delete the old version of the file before it start >>> downloading the new one. Its not doing anything. Please help. >>> >>> >>> On Mon, Oct 27, 2014 at 11:50 PM, akshay gupta <[email protected] >>> > wrote: >>> >>>> I am using fuse mounted hadoop distributed file system(HDFS). Can you >>>> suggest what changes need to be done to make it work as this is a very >>>> critical requirement for my project. Will deleting file and then renaming >>>> temp file help? how to achieve that? >>>> >>>> On Mon, Oct 27, 2014 at 7:38 PM, Alexander V. Lukyanov <[email protected]> >>>> wrote: >>>> >>>>> On Mon, Oct 27, 2014 at 05:53:19PM +0530, akshay gupta wrote: >>>>> > now its producing 0 byte files and i am also getting input/output >>>>> error >>>>> > here are the debug logs: >>>>> > >>>>> > copy: renaming `/mount/user/tom/test.txt_tmp_' to `test.txt' >>>>> > mirror: rename(/mount/user/tom/test.txt_tmp_, >>>>> /mount/user/tom/test.txt): >>>>> > Input/output error >>>>> >>>>> Which filesystem do you use? Maybe it is necessary to unlink target >>>>> file >>>>> before renaming, although rename should do it automatically. >>>>> >>>>> If newpath already exists it will be atomically replaced >>>>> (subject to a >>>>> few conditions; see ERRORS below), so that there is no point >>>>> at which >>>>> another process attempting to access newpath will find it >>>>> missing. >>>>> >>>>> And man page for rename(2) does not list EIO error. >>>>> >>>>> -- >>>>> Alexander. >>>>> >>>> >>>> >>> >> >> >> -- >> Alexander. >> > > > > -- > Alexander. > --047d7ba9763223449d05067903df Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Thanks for the patch Alexander, =C2=A0files seems to be ge= tting replaced now. Ill do some more testing to make sure every thing works= perfectly.=C2=A0<div><br></div><div>Thanks again,</div><div>Akshay</div></= div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, Oct 2= 8, 2014 at 3:21 PM, Alexander Lukyanov <span dir=3D"ltr"><<a href=3D"mai= lto:[email protected]" target=3D"_blank">[email protected]</a>></span> w= rote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;borde= r-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">BTW, it looks like= mirror should remove the old file before transferring the new one, please = run "mirror -vvv" to check that. You can also run strace to check= system call sequence.<br></div><div class=3D"gmail_extra"><div><div class= =3D"h5"><br><div class=3D"gmail_quote">2014-10-28 11:18 GMT+03:00 Alexander= Lukyanov <span dir=3D"ltr"><<a href=3D"mailto:[email protected]" target= =3D"_blank">[email protected]</a>></span>:<br><blockquote class=3D"gmail= _quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:= 1ex"><div dir=3D"ltr"><div>It looks like a bug in the FS (fuse and/or hdfs)= which also hits rsync:<br><a href=3D"https://groups.google.com/a/cloudera.= org/forum/#!msg/cdh-user/R1GR2Asj_lc/f_uuIjG5R1UJ" target=3D"_blank">https:= //groups.google.com/a/cloudera.org/forum/#!msg/cdh-user/R1GR2Asj_lc/f_uuIjG= 5R1UJ</a><br><br></div>Please try this workaround.<br></div><div class=3D"g= mail_extra"><div><div><br><div class=3D"gmail_quote">2014-10-28 10:29 GMT+0= 3:00 akshay gupta <span dir=3D"ltr"><<a href=3D"mailto:akshaygupta202@gm= ail.com" target=3D"_blank">[email protected]</a>></span>:<br><blo= ckquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #c= cc solid;padding-left:1ex"><div dir=3D"ltr">will giving --delete-first opti= on in lftp mirror help in my case? I think its not working in case of use-t= emp-file option. As per the man page its suppose to delete the old version = of the file before it start downloading the new one. Its not doing anything= . Please help.<div><br></div></div><div><div><div class=3D"gmail_extra"><br= ><div class=3D"gmail_quote">On Mon, Oct 27, 2014 at 11:50 PM, akshay gupta = <span dir=3D"ltr"><<a href=3D"mailto:[email protected]" target=3D= "_blank">[email protected]</a>></span> wrote:<br><blockquote clas= s=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pad= ding-left:1ex"><div dir=3D"ltr">I am using fuse mounted hadoop distributed = file system(HDFS). Can you suggest what changes need to be done to make it = work as this is a very critical requirement for my project. Will deleting f= ile and then renaming temp file help? how to achieve that?</div><div><div><= div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Mon, Oct 27, 20= 14 at 7:38 PM, Alexander V. Lukyanov <span dir=3D"ltr"><<a href=3D"mailt= o:[email protected]" target=3D"_blank">[email protected]</a>></span> wrote:<br><bl= ockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #= ccc solid;padding-left:1ex"><span>On Mon, Oct 27, 2014 at 05:53:19PM +0530,= akshay gupta wrote:<br> > now its producing 0 byte files and i am also getting input/output erro= r<br> > here are the debug logs:<br> ><br> </span><span>> copy: renaming `/mount/user/tom/test.txt_tmp_' to `te= st.txt'<br> > mirror: rename(/mount/user/tom/test.txt_tmp_, /mount/user/tom/test.txt= ):<br> > Input/output error<br> <br> </span>Which filesystem do you use? Maybe it is necessary to unlink target = file<br> before renaming, although rename should do it automatically.<br> <br> =C2=A0 =C2=A0 =C2=A0 =C2=A0If=C2=A0 newpath already exists it will be atomi= cally replaced (subject to a<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0few conditions; see ERRORS below), so that there= is no point=C2=A0 at=C2=A0 which<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0another process attempting to access newpath wil= l find it missing.<br> <br> And man page for rename(2) does not list EIO error.<br> <span><font color=3D"#888888"><br> --<br> =C2=A0 =C2=A0Alexander.<br> </font></span></blockquote></div><br></div> </div></div></blockquote></div><br></div> </div></div></blockquote></div><br><br clear=3D"all"><br></div></div><span>= <font color=3D"#888888">-- <br><div dir=3D"ltr"><div></div>=C2=A0=C2=A0 Ale= xander.<br></div> </font></span></div> </blockquote></div><br><br clear=3D"all"><br></div></div><span class=3D"HOE= nZb"><font color=3D"#888888">-- <br><div dir=3D"ltr"><div></div>=C2=A0=C2= =A0 Alexander.<br></div> </font></span></div> </blockquote></div><br></div> --047d7ba9763223449d05067903df-- --===============0232788658605975526== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lftp mailing list [email protected] http://univ.uniyar.ac.ru/mailman/listinfo/lftp --===============0232788658605975526==--