Re: Possible Bug in lftp mirror using mirror:sort-by option

--===============0022449886524777189==
Content-Type: multipart/alternative; boundary=001a11c2b5a869888a0507a2bb8d

--001a11c2b5a869888a0507a2bb8d
Content-Type: text/plain; charset=UTF-8

Hi Alexander,

Are there any plans for redesigning LFTP Mirror algorithm to build the
whole recursive list first. How much change would that require if we go
that path?

Thanks,
Akshay

On Thu, Oct 30, 2014 at 4:44 PM, akshay gupta <[email protected]>
wrote:

> lftp have options like --depth-first and --no-recursion I think it will
> make sense to have an option like  " --iterative" for iterative file
> transfer. This option will fetch the full file list and then fetch them
> iteratively.
>
> On Wed, Oct 29, 2014 at 2:29 PM, akshay gupta <[email protected]>
> wrote:
>
>> I think the functionality of mirror:sort-by option implies that the order
>> is for all the files irrespective of the directory structure. Or may be it
>> should support both the options and it would be configuration defined which
>> one to use.
>>
>> On Wed, Oct 29, 2014 at 2:09 PM, Alexander Lukyanov <[email protected]>
>> wrote:
>>
>>> Mirror in lftp works directory by directory. It does not build full list
>>> of files first, so it cannot sort the whole list. Maybe the mirror
>>> algorithm should be redesigned to build the whole recursive list first.
>>> What do you think?
>>>
>>> 2014-10-29 10:07 GMT+03:00 akshay gupta <[email protected]>:
>>>
>>>> Hi,
>>>>
>>>> I am trying to mirror files using mirror:sort-by option set to
>>>> 'date-desc'. My  file structure is like this
>>>>
>>>> $ ls -l -R
>>>> total 3191264
>>>> drwxr-xr-x  3 vikasgoyal  biadmin         102 Oct 22 12:35 1
>>>> -rw-r--r--  1 vikasgoyal  biadmin           0 Aug  1 01:00 tmp.txt
>>>> -rw-r--r--  1 vikasgoyal  biadmin           0 Aug  1 05:31 tmp1.txt
>>>> -rw-r--r--  1 vikasgoyal  biadmin   263940738 Oct 28 13:51 abc.csv
>>>>
>>>> ./1:
>>>> total 0
>>>> drwxr-xr-x  4 vikasgoyal  biadmin  136 Oct 28 14:19 2
>>>>
>>>> ./1/2:
>>>> total 515512
>>>> -rw-r--r--  1 vikasgoyal  biadmin          0 Oct 28 14:19 3
>>>> -rw-r--r--  1 vikasgoyal  biadmin  263940732 Oct 28 14:03 bc.csv
>>>>
>>>>
>>>> I expect the file synch order should be
>>>>
>>>> ~/tmp.txt
>>>> ~/tmp1.txt
>>>> ~/abc.csv
>>>> ~/1/2/bc.csv
>>>> ~/1/2/3
>>>>
>>>>
>>>> but the synch order is:
>>>> ~/tmp.txt
>>>> ~/tmp1.txt
>>>> ~/1/2/bc.csv
>>>> ~/1/2/3
>>>> ~/abc.csv
>>>>
>>>>
>>>> I think its the folder modification time that is causing this issue.
>>>> Lftp considers folder synch time to decide the order even though files
>>>> contained in the folder are newer then the remaining files in the parent
>>>> folder. File synch order should be in order of the modification time
>>>> irrespective of the folder structure. Please help.
>>>>
>>>> Thanks,
>>>> Akshay
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>    Alexander.
>>>
>>
>>
>

--001a11c2b5a869888a0507a2bb8d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi Alexander,<br><br></div><div>Are there any plans f=
or redesigning LFTP Mirror algorithm to build the whole recursive list firs=
t. How much change would that require if we go that path?<br><br></div><div=
>Thanks,<br></div><div>Akshay<br></div></div><div class=3D"gmail_extra"><br=
><div class=3D"gmail_quote">On Thu, Oct 30, 2014 at 4:44 PM, akshay gupta <=
span dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]" target=3D"=
_blank">[email protected]</a>&gt;</span> wrote:<br><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex"><div dir=3D"ltr">lftp have options like=C2=A0<span style=3D"c=
olor:rgb(0,0,0)">--depth-first and=C2=A0</span><span style=3D"color:rgb(0,0=
,0)">--no-recursion I think it will make sense to have an option like =C2=
=A0&quot; --iterative&quot; for iterative file transfer. This option will f=
etch the full file list and then fetch them iteratively.=C2=A0</span></div>=
<div class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmail_extra"><br><div=
 class=3D"gmail_quote">On Wed, Oct 29, 2014 at 2:29 PM, akshay gupta <span =
dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]" target=3D"_blan=
k">[email protected]</a>&gt;</span> wrote:<br><blockquote class=3D"g=
mail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-l=
eft:1ex"><div dir=3D"ltr">I think the functionality of mirror:sort-by optio=
n implies that the order is for all the files irrespective of the directory=
 structure. Or may be it should support both the options and it would be co=
nfiguration defined which one to use.</div><div><div><div class=3D"gmail_ex=
tra"><br><div class=3D"gmail_quote">On Wed, Oct 29, 2014 at 2:09 PM, Alexan=
der Lukyanov <span dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]" tar=
get=3D"_blank">[email protected]</a>&gt;</span> wrote:<br><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex"><div dir=3D"ltr">Mirror in lftp works directory by directory.=
 It does not build full list of files first, so it cannot sort the whole li=
st. Maybe the mirror algorithm should be redesigned to build the whole recu=
rsive list first. What do you think?<br></div><div class=3D"gmail_extra"><d=
iv><div><br><div class=3D"gmail_quote">2014-10-29 10:07 GMT+03:00 akshay gu=
pta <span dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]" targe=
t=3D"_blank">[email protected]</a>&gt;</span>:<br><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex"><div dir=3D"ltr">Hi,<div><br></div><div>I am trying to mirror=
 files using mirror:sort-by option set to &#39;date-desc&#39;. My =C2=A0fil=
e structure is like this=C2=A0</div><div><br></div><div><div>$ ls -l -R</di=
v><div>total 3191264</div><div>drwxr-xr-x =C2=A03 vikasgoyal =C2=A0biadmin =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 102 Oct 22 12:35 1</div><div>-rw-r--r-- =C2=A01=
 vikasgoyal =C2=A0biadmin =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 Aug =C2=A01 =
01:00 tmp.txt</div><div>-rw-r--r-- =C2=A01 vikasgoyal =C2=A0biadmin =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 0 Aug =C2=A01 05:31 tmp1.txt</div><div>-rw-r--r=
-- =C2=A01 vikasgoyal =C2=A0biadmin =C2=A0 263940738 Oct 28 13:51 abc.csv</=
div><div><br></div><div>./1:</div><div>total 0</div><div>drwxr-xr-x =C2=A04=
 vikasgoyal =C2=A0biadmin =C2=A0136 Oct 28 14:19 2</div><div><br></div><div=
>./1/2:</div><div>total 515512</div><div>-rw-r--r-- =C2=A01 vikasgoyal =C2=
=A0biadmin =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 Oct 28 14:19 3</div><div>-rw=
-r--r-- =C2=A01 vikasgoyal =C2=A0biadmin =C2=A0263940732 Oct 28 14:03 bc.cs=
v</div></div><div><br></div><div><br></div><div>I expect the file synch ord=
er should be=C2=A0</div><div><br></div><div>~/tmp.txt</div><div>~/tmp1.txt<=
/div><div>~/abc.csv</div><div>~/1/2/bc.csv</div><div>~/1/2/3</div><div><br>=
</div><div><br></div><div>but the synch order is:</div><div><div>~/tmp.txt<=
/div><div>~/tmp1.txt</div></div><div>~/1/2/bc.csv</div><div>~/1/2/3</div><d=
iv>~/abc.csv</div><div><br></div><div><br></div><div>I think its the folder=
 modification time that is causing this issue. Lftp considers folder synch =
time to decide the order even though files contained in the folder are newe=
r then the remaining files in the parent folder. File synch order should be=
 in order of the modification time irrespective of the folder structure. Pl=
ease help.</div><div><br></div><div>Thanks,</div><div>Akshay</div><div><br>=
</div><div><br></div><div><br></div><div><br></div><div><br></div></div>
</blockquote></div><br><br clear=3D"all"><br></div></div><span><font color=
=3D"#888888">-- <br><div dir=3D"ltr"><div></div>=C2=A0=C2=A0 Alexander.<br>=
</div>
</font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>

--001a11c2b5a869888a0507a2bb8d--

--===============0022449886524777189==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
lftp mailing list
[email protected]
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

--===============0022449886524777189==--