Re: Bug in lftp mirror when using xfer:use-temp-file option
Alexander Lukyanov <[email protected]> Tue, 28 Oct 2014 12:18:28 +0400
| Newsgroups | gmane.network.lftp.user,gmane.network.lftp.devel |
|---|---|
| Message-ID | <CANnoEwqAGLzc3kVQTBptjZV3tv8b529thOS7W-WJfxRw1QxdHA@mail.gmail.com> |
--001a11c1b66ac224c50506774a00 Content-Type: multipart/alternative; boundary=001a11c1b66ac224c00506774afe --001a11c1b66ac224c00506774afe Content-Type: text/plain; charset=UTF-8 It looks like a bug in the FS (fuse and/or hdfs) which also hits rsync: https://groups.google.com/a/cloudera.org/forum/#!msg/cdh-user/R1GR2Asj_lc/f_uuIjG5R1UJ Please try this workaround. 2014-10-28 10:29 GMT+03:00 akshay gupta <[email protected]>: > will giving --delete-first option in lftp mirror help in my case? I think > its not working in case of use-temp-file option. As per the man page its > suppose to delete the old version of the file before it start downloading > the new one. Its not doing anything. Please help. > > > On Mon, Oct 27, 2014 at 11:50 PM, akshay gupta <[email protected]> > wrote: > >> I am using fuse mounted hadoop distributed file system(HDFS). Can you >> suggest what changes need to be done to make it work as this is a very >> critical requirement for my project. Will deleting file and then renaming >> temp file help? how to achieve that? >> >> On Mon, Oct 27, 2014 at 7:38 PM, Alexander V. Lukyanov <[email protected]> >> wrote: >> >>> On Mon, Oct 27, 2014 at 05:53:19PM +0530, akshay gupta wrote: >>> > now its producing 0 byte files and i am also getting input/output error >>> > here are the debug logs: >>> > >>> > copy: renaming `/mount/user/tom/test.txt_tmp_' to `test.txt' >>> > mirror: rename(/mount/user/tom/test.txt_tmp_, >>> /mount/user/tom/test.txt): >>> > Input/output error >>> >>> Which filesystem do you use? Maybe it is necessary to unlink target file >>> before renaming, although rename should do it automatically. >>> >>> If newpath already exists it will be atomically replaced >>> (subject to a >>> few conditions; see ERRORS below), so that there is no point at >>> which >>> another process attempting to access newpath will find it missing. >>> >>> And man page for rename(2) does not list EIO error. >>> >>> -- >>> Alexander. >>> >> >> > -- Alexander. --001a11c1b66ac224c00506774afe Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>It looks like a bug in the FS (fuse and/or hdfs) whic= h also hits rsync:<br><a href=3D"https://groups.google.com/a/cloudera.org/f= orum/#!msg/cdh-user/R1GR2Asj_lc/f_uuIjG5R1UJ">https://groups.google.com/a/c= loudera.org/forum/#!msg/cdh-user/R1GR2Asj_lc/f_uuIjG5R1UJ</a><br><br></div>= Please try this workaround.<br></div><div class=3D"gmail_extra"><br><div cl= ass=3D"gmail_quote">2014-10-28 10:29 GMT+03:00 akshay gupta <span dir=3D"lt= r"><<a href=3D"mailto:[email protected]" target=3D"_blank">akshay= [email protected]</a>></span>:<br><blockquote class=3D"gmail_quote" sty= le=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div d= ir=3D"ltr">will giving --delete-first option in lftp mirror help in my case= ? I think its not working in case of use-temp-file option. As per the man p= age its suppose to delete the old version of the file before it start downl= oading the new one. Its not doing anything. Please help.<div><br></div></di= v><div class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmail_extra"><br><d= iv class=3D"gmail_quote">On Mon, Oct 27, 2014 at 11:50 PM, akshay gupta <sp= an dir=3D"ltr"><<a href=3D"mailto:[email protected]" target=3D"_b= lank">[email protected]</a>></span> wrote:<br><blockquote class= =3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd= ing-left:1ex"><div dir=3D"ltr">I am using fuse mounted hadoop distributed f= ile system(HDFS). Can you suggest what changes need to be done to make it w= ork as this is a very critical requirement for my project. Will deleting fi= le and then renaming temp file help? how to achieve that?</div><div><div><d= iv class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Mon, Oct 27, 201= 4 at 7:38 PM, Alexander V. Lukyanov <span dir=3D"ltr"><<a href=3D"mailto= :[email protected]" target=3D"_blank">[email protected]</a>></span> wrote:<br><blo= ckquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #c= cc solid;padding-left:1ex"><span>On Mon, Oct 27, 2014 at 05:53:19PM +0530, = akshay gupta wrote:<br> > now its producing 0 byte files and i am also getting input/output erro= r<br> > here are the debug logs:<br> ><br> </span><span>> copy: renaming `/mount/user/tom/test.txt_tmp_' to `te= st.txt'<br> > mirror: rename(/mount/user/tom/test.txt_tmp_, /mount/user/tom/test.txt= ):<br> > Input/output error<br> <br> </span>Which filesystem do you use? Maybe it is necessary to unlink target = file<br> before renaming, although rename should do it automatically.<br> <br> =C2=A0 =C2=A0 =C2=A0 =C2=A0If=C2=A0 newpath already exists it will be atomi= cally replaced (subject to a<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0few conditions; see ERRORS below), so that there= is no point=C2=A0 at=C2=A0 which<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0another process attempting to access newpath wil= l find it missing.<br> <br> And man page for rename(2) does not list EIO error.<br> <span><font color=3D"#888888"><br> --<br> =C2=A0 =C2=A0Alexander.<br> </font></span></blockquote></div><br></div> </div></div></blockquote></div><br></div> </div></div></blockquote></div><br><br clear=3D"all"><br>-- <br><div dir=3D= "ltr"><div></div>=C2=A0=C2=A0 Alexander.<br></div> </div> --001a11c1b66ac224c00506774afe-- --001a11c1b66ac224c50506774a00 Content-Type: application/octet-stream; name=diff Content-Disposition: attachment; filename=diff Content-Transfer-Encoding: base64 X-Attachment-Id: f_i1szfei30 ZGlmZiAtLWdpdCBhL3NyYy9GaWxlQ29weS5jYyBiL3NyYy9GaWxlQ29weS5jYwppbmRleCA1MTI4 ZWVkLi5lODRlYmZlIDEwMDY0NAotLS0gYS9zcmMvRmlsZUNvcHkuY2MKKysrIGIvc3JjL0ZpbGVD b3B5LmNjCkBAIC0xNTE4LDcgKzE1MTgsMTMgQEAgaW50IEZpbGVDb3B5UGVlckZEU3RyZWFtOjpE bygpCiAJICAgIHN0cnVjdCBzdGF0IHN0OwogCSAgICBpZih0ZW1wX2ZpbGUgfHwgKGxzdGF0KG5l d19uYW1lLCZzdCk9PS0xICYmIGVycm5vPT1FTk9FTlQpIHx8IFJlc01ncjo6UXVlcnlCb29sKCJ4 ZmVyOmNsb2JiZXIiLDApKSB7CiAJICAgICAgIGRlYnVnKCg1LCJjb3B5OiByZW5hbWluZyBgJXMn IHRvIGAlcydcbiIsc3RyZWFtLT5mdWxsX25hbWUuZ2V0KCksc3VnZ2VzdGVkX2ZpbGVuYW1lLmdl dCgpKSk7Ci0JICAgICAgIGlmKHJlbmFtZShzdHJlYW0tPmZ1bGxfbmFtZSxuZXdfbmFtZSk9PS0x KSB7CisJICAgICAgIGludCByZXM9cmVuYW1lKHN0cmVhbS0+ZnVsbF9uYW1lLG5ld19uYW1lKTsK KwkgICAgICAgaWYocmVzPT0tMSAmJiBlcnJubz09RUlPKSB7CisJCSAgLy8gRlVTRSB3aXRoIEhh ZG9vcEZTIHdvcmthcm91bmQKKwkJICB1bmxpbmsobmV3X25hbWUpOworCQkgIHJlcz1yZW5hbWUo c3RyZWFtLT5mdWxsX25hbWUsbmV3X25hbWUpOworCSAgICAgICB9CisJICAgICAgIGlmKHJlcz09 LTEpIHsKIAkJICBjb25zdCBjaGFyICplcnI9eHN0cmluZzo6Zm9ybWF0KCJyZW5hbWUoJXMsICVz KTogJXNcbiIsc3RyZWFtLT5mdWxsX25hbWUuZ2V0KCksbmV3X25hbWUsc3RyZXJyb3IoZXJybm8p KTsKIAkJICBpZih0ZW1wX2ZpbGUpCiAJCSAgICAgU2V0RXJyb3IoZXJyKTsK --001a11c1b66ac224c50506774a00 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lftp mailing list [email protected] http://univ.uniyar.ac.ru/mailman/listinfo/lftp --001a11c1b66ac224c50506774a00--