Re: Possible Bug in lftp mirror using mirror:sort-by option

--===============4504772659737797657==
Content-Type: multipart/alternative; boundary=047d7ba980d0f4a7ce0507a520b5

--047d7ba980d0f4a7ce0507a520b5
Content-Type: text/plain; charset=UTF-8

I think it is possible and the amount of programming work is moderate. The
recursive model of mirror will stay but the file transfers will be deferred
and done on the top level when recursive directory traversal is finished.

2014-11-12 8:32 GMT+03:00 akshay gupta <[email protected]>:

> Hi Alexander,
>
> Are there any plans for redesigning LFTP Mirror algorithm to build the
> whole recursive list first. How much change would that require if we go
> that path?
>
> Thanks,
> Akshay
>
> On Thu, Oct 30, 2014 at 4:44 PM, akshay gupta <[email protected]>
> wrote:
>
>> lftp have options like --depth-first and --no-recursion I think it will
>> make sense to have an option like  " --iterative" for iterative file
>> transfer. This option will fetch the full file list and then fetch them
>> iteratively.
>>
>> On Wed, Oct 29, 2014 at 2:29 PM, akshay gupta <[email protected]>
>> wrote:
>>
>>> I think the functionality of mirror:sort-by option implies that the
>>> order is for all the files irrespective of the directory structure. Or may
>>> be it should support both the options and it would be configuration defined
>>> which one to use.
>>>
>>> On Wed, Oct 29, 2014 at 2:09 PM, Alexander Lukyanov <[email protected]>
>>> wrote:
>>>
>>>> Mirror in lftp works directory by directory. It does not build full
>>>> list of files first, so it cannot sort the whole list. Maybe the mirror
>>>> algorithm should be redesigned to build the whole recursive list first.
>>>> What do you think?
>>>>
>>>> 2014-10-29 10:07 GMT+03:00 akshay gupta <[email protected]>:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am trying to mirror files using mirror:sort-by option set to
>>>>> 'date-desc'. My  file structure is like this
>>>>>
>>>>> $ ls -l -R
>>>>> total 3191264
>>>>> drwxr-xr-x  3 vikasgoyal  biadmin         102 Oct 22 12:35 1
>>>>> -rw-r--r--  1 vikasgoyal  biadmin           0 Aug  1 01:00 tmp.txt
>>>>> -rw-r--r--  1 vikasgoyal  biadmin           0 Aug  1 05:31 tmp1.txt
>>>>> -rw-r--r--  1 vikasgoyal  biadmin   263940738 Oct 28 13:51 abc.csv
>>>>>
>>>>> ./1:
>>>>> total 0
>>>>> drwxr-xr-x  4 vikasgoyal  biadmin  136 Oct 28 14:19 2
>>>>>
>>>>> ./1/2:
>>>>> total 515512
>>>>> -rw-r--r--  1 vikasgoyal  biadmin          0 Oct 28 14:19 3
>>>>> -rw-r--r--  1 vikasgoyal  biadmin  263940732 Oct 28 14:03 bc.csv
>>>>>
>>>>>
>>>>> I expect the file synch order should be
>>>>>
>>>>> ~/tmp.txt
>>>>> ~/tmp1.txt
>>>>> ~/abc.csv
>>>>> ~/1/2/bc.csv
>>>>> ~/1/2/3
>>>>>
>>>>>
>>>>> but the synch order is:
>>>>> ~/tmp.txt
>>>>> ~/tmp1.txt
>>>>> ~/1/2/bc.csv
>>>>> ~/1/2/3
>>>>> ~/abc.csv
>>>>>
>>>>>
>>>>> I think its the folder modification time that is causing this issue.
>>>>> Lftp considers folder synch time to decide the order even though files
>>>>> contained in the folder are newer then the remaining files in the parent
>>>>> folder. File synch order should be in order of the modification time
>>>>> irrespective of the folder structure. Please help.
>>>>>
>>>>> Thanks,
>>>>> Akshay
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>    Alexander.
>>>>
>>>
>>>
>>
>


-- 
   Alexander.

--047d7ba980d0f4a7ce0507a520b5
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I think it is possible and the amount of programming work =
is moderate. The recursive model of mirror will stay but the file transfers=
 will be deferred and done on the top level when recursive directory traver=
sal is finished.<br></div><div class=3D"gmail_extra"><br><div class=3D"gmai=
l_quote">2014-11-12 8:32 GMT+03:00 akshay gupta <span dir=3D"ltr">&lt;<a hr=
ef=3D"mailto:[email protected]" target=3D"_blank">akshaygupta202@gma=
il.com</a>&gt;</span>:<br><blockquote class=3D"gmail_quote" style=3D"margin=
:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><=
div>Hi Alexander,<br><br></div><div>Are there any plans for redesigning LFT=
P Mirror algorithm to build the whole recursive list first. How much change=
 would that require if we go that path?<br><br></div><div>Thanks,<br></div>=
<div>Akshay<br></div></div><div class=3D"HOEnZb"><div class=3D"h5"><div cla=
ss=3D"gmail_extra"><br><div class=3D"gmail_quote">On Thu, Oct 30, 2014 at 4=
:44 PM, akshay gupta <span dir=3D"ltr">&lt;<a href=3D"mailto:akshaygupta202=
@gmail.com" target=3D"_blank">[email protected]</a>&gt;</span> wrote=
:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-le=
ft:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">lftp have options like=
=C2=A0<span style=3D"color:rgb(0,0,0)">--depth-first and=C2=A0</span><span =
style=3D"color:rgb(0,0,0)">--no-recursion I think it will make sense to hav=
e an option like =C2=A0&quot; --iterative&quot; for iterative file transfer=
. This option will fetch the full file list and then fetch them iteratively=
.=C2=A0</span></div><div><div><div class=3D"gmail_extra"><br><div class=3D"=
gmail_quote">On Wed, Oct 29, 2014 at 2:29 PM, akshay gupta <span dir=3D"ltr=
">&lt;<a href=3D"mailto:[email protected]" target=3D"_blank">akshayg=
[email protected]</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote=
" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><=
div dir=3D"ltr">I think the functionality of mirror:sort-by option implies =
that the order is for all the files irrespective of the directory structure=
. Or may be it should support both the options and it would be configuratio=
n defined which one to use.</div><div><div><div class=3D"gmail_extra"><br><=
div class=3D"gmail_quote">On Wed, Oct 29, 2014 at 2:09 PM, Alexander Lukyan=
ov <span dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]" target=3D"_bl=
ank">[email protected]</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_=
quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1=
ex"><div dir=3D"ltr">Mirror in lftp works directory by directory. It does n=
ot build full list of files first, so it cannot sort the whole list. Maybe =
the mirror algorithm should be redesigned to build the whole recursive list=
 first. What do you think?<br></div><div class=3D"gmail_extra"><div><div><b=
r><div class=3D"gmail_quote">2014-10-29 10:07 GMT+03:00 akshay gupta <span =
dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]" target=3D"_blan=
k">[email protected]</a>&gt;</span>:<br><blockquote class=3D"gmail_q=
uote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1e=
x"><div dir=3D"ltr">Hi,<div><br></div><div>I am trying to mirror files usin=
g mirror:sort-by option set to &#39;date-desc&#39;. My =C2=A0file structure=
 is like this=C2=A0</div><div><br></div><div><div>$ ls -l -R</div><div>tota=
l 3191264</div><div>drwxr-xr-x =C2=A03 vikasgoyal =C2=A0biadmin =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 102 Oct 22 12:35 1</div><div>-rw-r--r-- =C2=A01 vikasgoya=
l =C2=A0biadmin =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 Aug =C2=A01 01:00 tmp.=
txt</div><div>-rw-r--r-- =C2=A01 vikasgoyal =C2=A0biadmin =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 0 Aug =C2=A01 05:31 tmp1.txt</div><div>-rw-r--r-- =C2=A01=
 vikasgoyal =C2=A0biadmin =C2=A0 263940738 Oct 28 13:51 abc.csv</div><div><=
br></div><div>./1:</div><div>total 0</div><div>drwxr-xr-x =C2=A04 vikasgoya=
l =C2=A0biadmin =C2=A0136 Oct 28 14:19 2</div><div><br></div><div>./1/2:</d=
iv><div>total 515512</div><div>-rw-r--r-- =C2=A01 vikasgoyal =C2=A0biadmin =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 Oct 28 14:19 3</div><div>-rw-r--r-- =C2=
=A01 vikasgoyal =C2=A0biadmin =C2=A0263940732 Oct 28 14:03 bc.csv</div></di=
v><div><br></div><div><br></div><div>I expect the file synch order should b=
e=C2=A0</div><div><br></div><div>~/tmp.txt</div><div>~/tmp1.txt</div><div>~=
/abc.csv</div><div>~/1/2/bc.csv</div><div>~/1/2/3</div><div><br></div><div>=
<br></div><div>but the synch order is:</div><div><div>~/tmp.txt</div><div>~=
/tmp1.txt</div></div><div>~/1/2/bc.csv</div><div>~/1/2/3</div><div>~/abc.cs=
v</div><div><br></div><div><br></div><div>I think its the folder modificati=
on time that is causing this issue. Lftp considers folder synch time to dec=
ide the order even though files contained in the folder are newer then the =
remaining files in the parent folder. File synch order should be in order o=
f the modification time irrespective of the folder structure. Please help.<=
/div><div><br></div><div>Thanks,</div><div>Akshay</div><div><br></div><div>=
<br></div><div><br></div><div><br></div><div><br></div></div>
</blockquote></div><br><br clear=3D"all"><br></div></div><span><font color=
=3D"#888888">-- <br><div dir=3D"ltr"><div></div>=C2=A0=C2=A0 Alexander.<br>=
</div>
</font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear=3D"all"><br>-- <br><div class=
=3D"gmail_signature"><div dir=3D"ltr"><div></div>=C2=A0=C2=A0 Alexander.<br=
></div></div>
</div>

--047d7ba980d0f4a7ce0507a520b5--

--===============4504772659737797657==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
lftp mailing list
[email protected]
http://univ.uniyar.ac.ru/mailman/listinfo/lftp

--===============4504772659737797657==--