Re: Possible Bug in lftp mirror using mirror:sort-by option

--===============7057211722532419108==
Content-Type: multipart/alternative; boundary=047d7b3a885050250d05081c4449

--047d7b3a885050250d05081c4449
Content-Type: text/plain; charset=UTF-8

We are in desperate need of this feature as without this feature we can
only ensure file order within a folder. There is a huge dependency on file
order for some of our hadoop oozie jobs. By when can we expect this feature
to come in lftp?



On Wed, Nov 12, 2014 at 1:54 PM, Alexander Lukyanov <[email protected]>
wrote:

> I think it is possible and the amount of programming work is moderate. The
> recursive model of mirror will stay but the file transfers will be deferred
> and done on the top level when recursive directory traversal is finished.
>
> 2014-11-12 8:32 GMT+03:00 akshay gupta <[email protected]>:
>
>> Hi Alexander,
>>
>> Are there any plans for redesigning LFTP Mirror algorithm to build the
>> whole recursive list first. How much change would that require if we go
>> that path?
>>
>> Thanks,
>> Akshay
>>
>> On Thu, Oct 30, 2014 at 4:44 PM, akshay gupta <[email protected]>
>> wrote:
>>
>>> lftp have options like --depth-first and --no-recursion I think it will
>>> make sense to have an option like  " --iterative" for iterative file
>>> transfer. This option will fetch the full file list and then fetch them
>>> iteratively.
>>>
>>> On Wed, Oct 29, 2014 at 2:29 PM, akshay gupta <[email protected]>
>>> wrote:
>>>
>>>> I think the functionality of mirror:sort-by option implies that the
>>>> order is for all the files irrespective of the directory structure. Or may
>>>> be it should support both the options and it would be configuration defined
>>>> which one to use.
>>>>
>>>> On Wed, Oct 29, 2014 at 2:09 PM, Alexander Lukyanov <[email protected]>
>>>> wrote:
>>>>
>>>>> Mirror in lftp works directory by directory. It does not build full
>>>>> list of files first, so it cannot sort the whole list. Maybe the mirror
>>>>> algorithm should be redesigned to build the whole recursive list first.
>>>>> What do you think?
>>>>>
>>>>> 2014-10-29 10:07 GMT+03:00 akshay gupta <[email protected]>:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am trying to mirror files using mirror:sort-by option set to
>>>>>> 'date-desc'. My  file structure is like this
>>>>>>
>>>>>> $ ls -l -R
>>>>>> total 3191264
>>>>>> drwxr-xr-x  3 vikasgoyal  biadmin         102 Oct 22 12:35 1
>>>>>> -rw-r--r--  1 vikasgoyal  biadmin           0 Aug  1 01:00 tmp.txt
>>>>>> -rw-r--r--  1 vikasgoyal  biadmin           0 Aug  1 05:31 tmp1.txt
>>>>>> -rw-r--r--  1 vikasgoyal  biadmin   263940738 Oct 28 13:51 abc.csv
>>>>>>
>>>>>> ./1:
>>>>>> total 0
>>>>>> drwxr-xr-x  4 vikasgoyal  biadmin  136 Oct 28 14:19 2
>>>>>>
>>>>>> ./1/2:
>>>>>> total 515512
>>>>>> -rw-r--r--  1 vikasgoyal  biadmin          0 Oct 28 14:19 3
>>>>>> -rw-r--r--  1 vikasgoyal  biadmin  263940732 Oct 28 14:03 bc.csv
>>>>>>
>>>>>>
>>>>>> I expect the file synch order should be
>>>>>>
>>>>>> ~/tmp.txt
>>>>>> ~/tmp1.txt
>>>>>> ~/abc.csv
>>>>>> ~/1/2/bc.csv
>>>>>> ~/1/2/3
>>>>>>
>>>>>>
>>>>>> but the synch order is:
>>>>>> ~/tmp.txt
>>>>>> ~/tmp1.txt
>>>>>> ~/1/2/bc.csv
>>>>>> ~/1/2/3
>>>>>> ~/abc.csv
>>>>>>
>>>>>>
>>>>>> I think its the folder modification time that is causing this issue.
>>>>>> Lftp considers folder synch time to decide the order even though files
>>>>>> contained in the folder are newer then the remaining files in the parent
>>>>>> folder. File synch order should be in order of the modification time
>>>>>> irrespective of the folder structure. Please help.
>>>>>>
>>>>>> Thanks,
>>>>>> Akshay
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>    Alexander.
>>>>>
>>>>
>>>>
>>>
>>
>
>
> --
>    Alexander.
>

--047d7b3a885050250d05081c4449
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">We are in desperate need of this feature as without this f=
eature we can only ensure file order within a folder. There is a huge depen=
dency on file order for some of our hadoop oozie jobs. By when can we expec=
t this feature to come in lftp?<br><br><br></div><div class=3D"gmail_extra"=
><br><div class=3D"gmail_quote">On Wed, Nov 12, 2014 at 1:54 PM, Alexander =
Lukyanov <span dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]" target=
=3D"_blank">[email protected]</a>&gt;</span> wrote:<br><blockquote class=3D=
"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding=
-left:1ex"><div dir=3D"ltr">I think it is possible and the amount of progra=
mming work is moderate. The recursive model of mirror will stay but the fil=
e transfers will be deferred and done on the top level when recursive direc=
tory traversal is finished.<br></div><div class=3D"gmail_extra"><div><div c=
lass=3D"h5"><br><div class=3D"gmail_quote">2014-11-12 8:32 GMT+03:00 akshay=
 gupta <span dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]" ta=
rget=3D"_blank">[email protected]</a>&gt;</span>:<br><blockquote cla=
ss=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pa=
dding-left:1ex"><div dir=3D"ltr"><div>Hi Alexander,<br><br></div><div>Are t=
here any plans for redesigning LFTP Mirror algorithm to build the whole rec=
ursive list first. How much change would that require if we go that path?<b=
r><br></div><div>Thanks,<br></div><div>Akshay<br></div></div><div><div><div=
 class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Thu, Oct 30, 2014 =
at 4:44 PM, akshay gupta <span dir=3D"ltr">&lt;<a href=3D"mailto:akshaygupt=
[email protected]" target=3D"_blank">[email protected]</a>&gt;</span> w=
rote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;borde=
r-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">lftp have options =
like=C2=A0<span style=3D"color:rgb(0,0,0)">--depth-first and=C2=A0</span><s=
pan style=3D"color:rgb(0,0,0)">--no-recursion I think it will make sense to=
 have an option like =C2=A0&quot; --iterative&quot; for iterative file tran=
sfer. This option will fetch the full file list and then fetch them iterati=
vely.=C2=A0</span></div><div><div><div class=3D"gmail_extra"><br><div class=
=3D"gmail_quote">On Wed, Oct 29, 2014 at 2:29 PM, akshay gupta <span dir=3D=
"ltr">&lt;<a href=3D"mailto:[email protected]" target=3D"_blank">aks=
[email protected]</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_q=
uote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1e=
x"><div dir=3D"ltr">I think the functionality of mirror:sort-by option impl=
ies that the order is for all the files irrespective of the directory struc=
ture. Or may be it should support both the options and it would be configur=
ation defined which one to use.</div><div><div><div class=3D"gmail_extra"><=
br><div class=3D"gmail_quote">On Wed, Oct 29, 2014 at 2:09 PM, Alexander Lu=
kyanov <span dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]" target=3D=
"_blank">[email protected]</a>&gt;</span> wrote:<br><blockquote class=3D"gm=
ail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-le=
ft:1ex"><div dir=3D"ltr">Mirror in lftp works directory by directory. It do=
es not build full list of files first, so it cannot sort the whole list. Ma=
ybe the mirror algorithm should be redesigned to build the whole recursive =
list first. What do you think?<br></div><div class=3D"gmail_extra"><div><di=
v><br><div class=3D"gmail_quote">2014-10-29 10:07 GMT+03:00 akshay gupta <s=
pan dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]" target=3D"_=
blank">[email protected]</a>&gt;</span>:<br><blockquote class=3D"gma=
il_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-lef=
t:1ex"><div dir=3D"ltr">Hi,<div><br></div><div>I am trying to mirror files =
using mirror:sort-by option set to &#39;date-desc&#39;. My =C2=A0file struc=
ture is like this=C2=A0</div><div><br></div><div><div>$ ls -l -R</div><div>=
total 3191264</div><div>drwxr-xr-x =C2=A03 vikasgoyal =C2=A0biadmin =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 102 Oct 22 12:35 1</div><div>-rw-r--r-- =C2=A01 vikasg=
oyal =C2=A0biadmin =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 Aug =C2=A01 01:00 t=
mp.txt</div><div>-rw-r--r-- =C2=A01 vikasgoyal =C2=A0biadmin =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 0 Aug =C2=A01 05:31 tmp1.txt</div><div>-rw-r--r-- =C2=
=A01 vikasgoyal =C2=A0biadmin =C2=A0 263940738 Oct 28 13:51 abc.csv</div><d=
iv><br></div><div>./1:</div><div>total 0</div><div>drwxr-xr-x =C2=A04 vikas=
goyal =C2=A0biadmin =C2=A0136 Oct 28 14:19 2</div><div><br></div><div>./1/2=
:</div><div>total 515512</div><div>-rw-r--r-- =C2=A01 vikasgoyal =C2=A0biad=
min =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 Oct 28 14:19 3</div><div>-rw-r--r--=
 =C2=A01 vikasgoyal =C2=A0biadmin =C2=A0263940732 Oct 28 14:03 bc.csv</div>=
</div><div><br></div><div><br></div><div>I expect the file synch order shou=
ld be=C2=A0</div><div><br></div><div>~/tmp.txt</div><div>~/tmp1.txt</div><d=
iv>~/abc.csv</div><div>~/1/2/bc.csv</div><div>~/1/2/3</div><div><br></div><=
div><br></div><div>but the synch order is:</div><div><div>~/tmp.txt</div><d=
iv>~/tmp1.txt</div></div><div>~/1/2/bc.csv</div><div>~/1/2/3</div><div>~/ab=
c.csv</div><div><br></div><div><br></div><div>I think its the folder modifi=
cation time that is causing this issue. Lftp considers folder synch time to=
 decide the order even though files contained in the folder are newer then =
the remaining files in the parent folder. File synch order should be in ord=
er of the modification time irrespective of the folder structure. Please he=
lp.</div><div><br></div><div>Thanks,</div><div>Akshay</div><div><br></div><=
div><br></div><div><br></div><div><br></div><div><br></div></div>
</blockquote></div><br><br clear=3D"all"><br></div></div><span><font color=
=3D"#888888">-- <br><div dir=3D"ltr"><div></div>=C2=A0=C2=A0 Alexander.<br>=
</div>
</font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear=3D"all"><br></div></div><span =
class=3D"HOEnZb"><font color=3D"#888888">-- <br><div><div dir=3D"ltr"><div>=
</div>=C2=A0=C2=A0 Alexander.<br></div></div>
</font></span></div>
</blockquote></div><br></div>

--047d7b3a885050250d05081c4449--

--===============7057211722532419108==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
lftp mailing list
[email protected]
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

--===============7057211722532419108==--