Re: Review of script for syncing HostedFTP and local server
Alexander Lukyanov <[email protected]> Fri, 02 Jun 2017 04:11:29 +0000
| Newsgroups | gmane.network.lftp.user |
|---|---|
| Message-ID | <CANnoEwpDsU0iNOz5zgwk_HCS2bJ0YEffCD4PVofO1c5AedpO9A@mail.gmail.com> |
--===============0176217057572890598== Content-Type: multipart/alternative; boundary="94eb2c04fa92b1f7900550f25987" --94eb2c04fa92b1f7900550f25987 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable The --include-glob options look incorrect. They should not be given an absolute path, but rather a relative one. Even more, the glob pattern is matched against just a few last path components (separated by slashes), depending on the number of slashes in the pattern. The regex patterns are different, they are matched against the complete relative path, starting with the base mirrored directory. =D0=BF=D1=82, 2 =D0=B8=D1=8E=D0=BD=D1=8F 2017, 5:04 Alan Steel <alan@visage= mobile.com>: > Hello, > > > > We are a SaaS company, migrating our colocation environment to AWS. > > As part of that, we are changing how customers will provide us with files > to be processed by our application. > > Currently they SSH to a server in our colo, and place their files in one > of their subdirectories. > > When we migrate to AWS, our customers will instead STFP upload their file= s > to a hosted FTP provider. > > These files will be synchronized with two AWS-based servers, ProdN01IL01 > and ProdN02IL01, each of which will be running an LFTP script. > > > > Description =E2=80=93 > > > > On each of the two AWS servers, each customer=E2=80=99s directories will = be under > /opt/ftpusers/home/ > > Each of these will have 6 subdirectories =E2=80=93 as described below. > > Some of these directories are to receive files from the customer, some ar= e > for our application=E2=80=99s use, and some are for file sharing between = us and our > customers. > > > > At the hosted FTP provider we are going to make the following directory > structure=E2=80=A6 > > /Servers > > /ProdN01IL01 > > /[Customer1] > > /error > > /other > > /reports > > /saved > > /synch > > /[Customer2] > > /error > > /other > > /reports > > /saved > > /synch=E2=80=A6.(and so o= n) > > /ProdN02IL01 > > /[Customer51] > > /error > > /other > > /reports > > /saved > > /synch > > /[Customer52] > > /error > > /other > > /reports > > /saved > > /synch=E2=80=A6.(and so o= n) > > > > Note: We don=E2=80=99t need /inbox subdirectories on HostedFTP, because = they are > only used by our application. > > > > The script -- > > > > Below is the script we are thinking of using. > > We invite comments and suggestions. > > Thank you! > > > > #!/bin/bash > > > > FTPPROTOCOL=3D=E2=80=9Dsftp=E2=80=9D > > FTPHOST=3D=E2=80=9Dourcompany.hostedftp.com=E2=80=9D > > [email protected]=E2=80=9D > > FTPPASS=3D=E2=80=9Dourcompanyftppassword=E2=80=9D > > FTPURL=3D"$FTPPROTOCOL://$FTPUSER:$FTPPASS@$FTPHOST" > > FTPREMOTEFOLDER=3D=E2=80=9D/Servers/ProdN01IL01=E2=80=9D (or ProdN02IL01= , depending on the > server) > > FTPLOCALFOLDER=3D=E2=80=9D/opt/ftpusers/home=E2=80=9D > > FTPLOGFILE=3D=E2=80=9D/var/log/lftpsyncs.log=E2=80=9D > > > > # The =E2=80=9CSynch=E2=80=9D subdirectories -- where customers put files= for processing > by our application > > # Move all files from HostedFTP =E2=80=9CSynch=E2=80=9D directories to lo= cal server Synch > directories, the delete the source files from HostedFTP Synch directories > > # =E2=80=9CRegular=E2=80=9D mirror, with --Remove-source-files > > > > EXCLUDES=3D"--exclude-glob .*" > > EXCLUDES+=3D" --exclude-glob .*/" > > > > INCLUDES=3D"--include-glob /opt/ftpusers/home/*/Synch/*" > > > > lftp -e " > > open '$FTPURL '; > > lcd $FTPLOCALFOLDER; > > cd $FTPREMOTEFOLDER; > > mirror --continue --only-newer --Remove-source-files --verbose=3D3 > --no-perms --parallel=3D8 $EXCLUDES $INCLUDES=E2=80=9D > > > > # The =E2=80=9CSaved=E2=80=9D subdirectories =E2=80=93 where our applicat= ion puts files from > Inbox, after successfully processing them > > # The =E2=80=9CError=E2=80=9D subdirectories -- where our application put= s files from > Inbox, if it fails to process them > > # The =E2=80=9CReports=E2=80=9D subdirectories =E2=80=93 where our applic= ation puts reports > > # Sync-*up* files from local =E2=80=9CSaved=E2=80=9D, =E2=80=9CError=E2= =80=9D and =E2=80=9CReports=E2=80=9D directories > to HostedFTP, deleting HostedFTP files if they are not present on the loc= al > server > > # =E2=80=9CReverse=E2=80=9D mirror, with deleting of remote files not pre= sent locally > > > > EXCLUDES=3D"--exclude-glob .*" > > EXCLUDES+=3D" --exclude-glob .*/" > > > > INCLUDES=3D"--include-glob /opt/ftpusers/home/*/Saved/*" > > INCLUDES+=3D" --include-glob /opt/ftpusers/home/*/Error/*" > > INCLUDES+=3D" --include-glob /opt/ftpusers/home/*/Reports/*" > > > > lftp -e " > > open '$FTPURL '; > > lcd $FTPLOCALFOLDER; > > cd $FTPREMOTEFOLDER; > > mirror --reverse --continue --only-newer --delete --verbose=3D3 --no-perm= s > --parallel=3D8 $EXCLUDES $INCLUDES=E2=80=9D > > > > # The =E2=80=9COther=E2=80=9D subdirectories -- =E2=80=93 where customers= or staff put various > files > > # Sync-*down* files in HostedFTP =E2=80=9COther=E2=80=9D directories to t= he local server > > # =E2=80=9CRegular=E2=80=9D mirror > > > > EXCLUDES=3D"--exclude-glob .*" > > EXCLUDES+=3D" --exclude-glob .*/" > > > > INCLUDES=3D"--include-glob /opt/ftpusers/home/*/Other/*" > > > > lftp -e " > > open '$ FTPURL '; > > lcd $ FTPLOCALFOLDER; > > cd $ FTPREMOTEFOLDER; > > mirror --continue --only-newer --verbose=3D3 --no-perms --parallel=3D8 > $EXCLUDES $INCLUDES=E2=80=9D > > > > > > AGS > _______________________________________________ > lftp mailing list > [email protected] > http://univ.uniyar.ac.ru/mailman/listinfo/lftp > --94eb2c04fa92b1f7900550f25987 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <p dir=3D"ltr">The --include-glob options look incorrect. They should not b= e given an absolute path, but rather a relative one. Even more, the glob pa= ttern is matched against just a few last path components (separated by slas= hes), depending on the number of slashes in the pattern. The regex patterns= are different, they are matched against the complete relative path, starti= ng with the base mirrored directory. </p> <br><div class=3D"gmail_quote"><div dir=3D"ltr">=D0=BF=D1=82, 2 =D0=B8=D1= =8E=D0=BD=D1=8F 2017, 5:04 Alan Steel <<a href=3D"mailto:alan@visagemobi= le.com">[email protected]</a>>:<br></div><blockquote class=3D"gmail_= quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1= ex"> <div lang=3D"EN-US" link=3D"#0563C1" vlink=3D"#954F72"> <div class=3D"m_-7344037388996566610WordSection1"> <p class=3D"MsoNormal">Hello,<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">We are a SaaS company, migrating our colocation envi= ronment to AWS.<u></u><u></u></p> <p class=3D"MsoNormal">As part of that, we are changing how customers will = provide us with files to be processed by our application.<u></u><u></u></p> <p class=3D"MsoNormal">Currently they SSH to a server in our colo, and plac= e their files in one of their subdirectories.<u></u><u></u></p> <p class=3D"MsoNormal">When we migrate to AWS, our customers will instead S= TFP upload their files to a hosted FTP provider.<u></u><u></u></p> <p class=3D"MsoNormal">These files will be synchronized with two AWS-based = servers, ProdN01IL01 and ProdN02IL01, each of which will be running an LFTP= script.<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">Description =E2=80=93<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">On each of the two AWS servers, each customer=E2=80= =99s directories will be under /opt/ftpusers/home/<u></u><u></u></p> <p class=3D"MsoNormal">Each of these will have 6 subdirectories =E2=80=93 a= s described below.<u></u><u></u></p> <p class=3D"MsoNormal">Some of these directories are to receive files from = the customer, some are for our application=E2=80=99s use, and some are for = file sharing between us and our customers.<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">At the hosted FTP provider we are going to make the = following directory structure=E2=80=A6<u></u><u></u></p> <p class=3D"MsoNormal">/Servers<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /ProdN01IL01<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /[Customer1]<u= ></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 /error<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 /other<u></u><u></u></p> <p class=3D"MsoNormal" style=3D"margin-left:1.0in;text-indent:.5in">/report= s<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 /saved<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 /synch<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /[Customer2]<u= ></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 /error<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 /other<u></u><u></u></p> <p class=3D"MsoNormal" style=3D"margin-left:1.0in;text-indent:.5in">/report= s<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 /saved<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 /synch=E2=80=A6.(and so on)<u></u><u></u></p> <p class=3D"MsoNormal" style=3D"text-indent:.5in">/ProdN02IL01<u></u><u></u= ></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /[Customer51]<= u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 /error<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 /other<u></u><u></u></p> <p class=3D"MsoNormal" style=3D"margin-left:1.0in;text-indent:.5in">/report= s<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 /saved<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 /synch<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /[Customer52]<= u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 /error<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 /other<u></u><u></u></p> <p class=3D"MsoNormal" style=3D"margin-left:1.0in;text-indent:.5in">/report= s<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 /saved<u></u><u></u></p> <p class=3D"MsoNormal">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 /synch=E2=80=A6.(and so on)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">Note:=C2=A0 We don=E2=80=99t need /inbox subdirector= ies on HostedFTP, because they are only used by our application.<u></u><u><= /u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">The script --<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">Below is the script we are thinking of using.<u></u>= <u></u></p> <p class=3D"MsoNormal">We invite comments and suggestions.<u></u><u></u></p= > <p class=3D"MsoNormal">Thank you!<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">#!/bin/bash<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">FTPPROTOCOL=3D=E2=80=9Dsftp=E2=80=9D<u></u><u></u></= p> <p class=3D"MsoNormal">FTPHOST=3D=E2=80=9D<a href=3D"http://ourcompany.host= edftp.com" target=3D"_blank">ourcompany.hostedftp.com</a>=E2=80=9D<u></u><u= ></u></p> <p class=3D"MsoNormal">FTPUSER=3D=E2=80=9D<a href=3D"mailto:hftp.itaccounts= @ourcompany.com" target=3D"_blank">[email protected]</a>=E2=80= =9D<u></u><u></u></p> <p class=3D"MsoNormal">FTPPASS=3D=E2=80=9Dourcompanyftppassword=E2=80=9D<u>= </u><u></u></p> <p class=3D"MsoNormal">FTPURL=3D"$FTPPROTOCOL://$FTPUSER:$FTPPASS@$FTP= HOST"<u></u><u></u></p> <p class=3D"MsoNormal">FTPREMOTEFOLDER=3D=E2=80=9D/Servers/ProdN01IL01=E2= =80=9D =C2=A0(or ProdN02IL01, depending on the server)<u></u><u></u></p> <p class=3D"MsoNormal">FTPLOCALFOLDER=3D=E2=80=9D/opt/ftpusers/home=E2=80= =9D<u></u><u></u></p> <p class=3D"MsoNormal">FTPLOGFILE=3D=E2=80=9D/var/log/lftpsyncs.log=E2=80= =9D<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal"># The =E2=80=9CSynch=E2=80=9D subdirectories -- wher= e customers put files for processing by our application<u></u><u></u></p> <p class=3D"MsoNormal"># Move all files from HostedFTP =E2=80=9CSynch=E2=80= =9D directories to local server Synch directories, the delete the source fi= les from HostedFTP Synch directories<u></u><u></u></p> <p class=3D"MsoNormal"># =E2=80=9CRegular=E2=80=9D mirror, with --Remove-so= urce-files<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">EXCLUDES=3D"--exclude-glob .*"<u></u><u></= u></p> <p class=3D"MsoNormal">EXCLUDES+=3D" --exclude-glob .*/"<u></u><u= ></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">INCLUDES=3D"--include-glob /opt/ftpusers/home/*= /Synch/*"<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">lftp -e "<u></u><u></u></p> <p class=3D"MsoNormal">open '$FTPURL ';<u></u><u></u></p> <p class=3D"MsoNormal">lcd $FTPLOCALFOLDER;<u></u><u></u></p> <p class=3D"MsoNormal">cd $FTPREMOTEFOLDER;<u></u><u></u></p> <p class=3D"MsoNormal">mirror --continue --only-newer --Remove-source-files= --verbose=3D3 --no-perms --parallel=3D8 $EXCLUDES $INCLUDES=E2=80=9D<u></u= ><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal"># The =E2=80=9CSaved=E2=80=9D subdirectories =E2=80= =93 where our application puts files from Inbox, after successfully process= ing them<u></u><u></u></p> <p class=3D"MsoNormal"># The =E2=80=9CError=E2=80=9D subdirectories -- wher= e our application puts files from Inbox, if it fails to process them<u></u>= <u></u></p> <p class=3D"MsoNormal"># The =E2=80=9CReports=E2=80=9D subdirectories =E2= =80=93 where our application puts reports<u></u><u></u></p> <p class=3D"MsoNormal">#=C2=A0 Sync-<u>up</u> files from local =E2=80=9CSav= ed=E2=80=9D, =E2=80=9CError=E2=80=9D and =E2=80=9CReports=E2=80=9D director= ies to HostedFTP, deleting HostedFTP files if they are not present on the l= ocal server<u></u><u></u></p> <p class=3D"MsoNormal"># =E2=80=9CReverse=E2=80=9D mirror, with deleting of= remote files not present locally<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">EXCLUDES=3D"--exclude-glob .*"<u></u><u></= u></p> <p class=3D"MsoNormal">EXCLUDES+=3D" --exclude-glob .*/"<u></u><u= ></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">INCLUDES=3D"--include-glob /opt/ftpusers/home/*= /Saved/*"<u></u><u></u></p> <p class=3D"MsoNormal">INCLUDES+=3D" --include-glob /opt/ftpusers/home= /*/Error/*"<u></u><u></u></p> <p class=3D"MsoNormal">INCLUDES+=3D" --include-glob /opt/ftpusers/home= /*/Reports/*"<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">lftp -e "<u></u><u></u></p> <p class=3D"MsoNormal">open '$FTPURL ';<u></u><u></u></p> <p class=3D"MsoNormal">lcd $FTPLOCALFOLDER;<u></u><u></u></p> <p class=3D"MsoNormal">cd $FTPREMOTEFOLDER;<u></u><u></u></p> <p class=3D"MsoNormal">mirror --reverse --continue --only-newer --delete --= verbose=3D3 --no-perms --parallel=3D8 $EXCLUDES $INCLUDES=E2=80=9D<u></u><u= ></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal"># The =E2=80=9COther=E2=80=9D subdirectories -- =E2= =80=93 where customers or staff put various files <u></u><u></u></p> <p class=3D"MsoNormal"># Sync-<u>down</u> files in HostedFTP =E2=80=9COther= =E2=80=9D directories to the local server<u></u><u></u></p> <p class=3D"MsoNormal"># =E2=80=9CRegular=E2=80=9D mirror<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">EXCLUDES=3D"--exclude-glob .*"<u></u><u></= u></p> <p class=3D"MsoNormal">EXCLUDES+=3D" --exclude-glob .*/"<u></u><u= ></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">INCLUDES=3D"--include-glob /opt/ftpusers/home/*= /Other/*"<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">lftp -e "<u></u><u></u></p> <p class=3D"MsoNormal">open '$ FTPURL ';<u></u><u></u></p> <p class=3D"MsoNormal">lcd $ FTPLOCALFOLDER;<u></u><u></u></p> <p class=3D"MsoNormal">cd $ FTPREMOTEFOLDER;<u></u><u></u></p> <p class=3D"MsoNormal">mirror --continue --only-newer --verbose=3D3 --no-pe= rms --parallel=3D8 $EXCLUDES $INCLUDES=E2=80=9D<u></u><u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">AGS<u></u><u></u></p> </div> </div> _______________________________________________<br> lftp mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a= ><br> <a href=3D"http://univ.uniyar.ac.ru/mailman/listinfo/lftp" rel=3D"noreferre= r" target=3D"_blank">http://univ.uniyar.ac.ru/mailman/listinfo/lftp</a><br> </blockquote></div> --94eb2c04fa92b1f7900550f25987-- --===============0176217057572890598== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lftp mailing list [email protected] http://univ.uniyar.ac.ru/mailman/listinfo/lftp --===============0176217057572890598==--