Review of script for syncing HostedFTP and local server
Alan Steel <[email protected]> Fri, 2 Jun 2017 02:04:05 +0000
| Newsgroups | gmane.network.lftp.user |
|---|---|
| Message-ID | <BN6PR01MB2515B092CEE7059971167A42B8F70@BN6PR01MB2515.prod.exchangelabs.com> |
--===============0314867239294704747==
Content-Language: en-US
Content-Type: multipart/alternative;
boundary="_000_BN6PR01MB2515B092CEE7059971167A42B8F70BN6PR01MB2515prod_"
--_000_BN6PR01MB2515B092CEE7059971167A42B8F70BN6PR01MB2515prod_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Hello,
We are a SaaS company, migrating our colocation environment to AWS.
As part of that, we are changing how customers will provide us with files t=
o be processed by our application.
Currently they SSH to a server in our colo, and place their files in one of=
their subdirectories.
When we migrate to AWS, our customers will instead STFP upload their files =
to a hosted FTP provider.
These files will be synchronized with two AWS-based servers, ProdN01IL01 an=
d ProdN02IL01, each of which will be running an LFTP script.
Description -
On each of the two AWS servers, each customer's directories will be under /=
opt/ftpusers/home/
Each of these will have 6 subdirectories - as described below.
Some of these directories are to receive files from the customer, some are =
for our application's use, and some are for file sharing between us and our=
customers.
At the hosted FTP provider we are going to make the following directory str=
ucture...
/Servers
/ProdN01IL01
/[Customer1]
/error
/other
/reports
/saved
/synch
/[Customer2]
/error
/other
/reports
/saved
/synch....(and so on)
/ProdN02IL01
/[Customer51]
/error
/other
/reports
/saved
/synch
/[Customer52]
/error
/other
/reports
/saved
/synch....(and so on)
Note: We don't need /inbox subdirectories on HostedFTP, because they are o=
nly used by our application.
The script --
Below is the script we are thinking of using.
We invite comments and suggestions.
Thank you!
#!/bin/bash
FTPPROTOCOL=3D"sftp"
FTPHOST=3D"ourcompany.hostedftp.com"
FTPUSER=3D"[email protected]"
FTPPASS=3D"ourcompanyftppassword"
FTPURL=3D"$FTPPROTOCOL://$FTPUSER:$FTPPASS@$FTPHOST"
FTPREMOTEFOLDER=3D"/Servers/ProdN01IL01" (or ProdN02IL01, depending on the=
server)
FTPLOCALFOLDER=3D"/opt/ftpusers/home"
FTPLOGFILE=3D"/var/log/lftpsyncs.log"
# The "Synch" subdirectories -- where customers put files for processing by=
our application
# Move all files from HostedFTP "Synch" directories to local server Synch d=
irectories, the delete the source files from HostedFTP Synch directories
# "Regular" mirror, with --Remove-source-files
EXCLUDES=3D"--exclude-glob .*"
EXCLUDES+=3D" --exclude-glob .*/"
INCLUDES=3D"--include-glob /opt/ftpusers/home/*/Synch/*"
lftp -e "
open '$FTPURL ';
lcd $FTPLOCALFOLDER;
cd $FTPREMOTEFOLDER;
mirror --continue --only-newer --Remove-source-files --verbose=3D3 --no-per=
ms --parallel=3D8 $EXCLUDES $INCLUDES"
# The "Saved" subdirectories - where our application puts files from Inbox,=
after successfully processing them
# The "Error" subdirectories -- where our application puts files from Inbox=
, if it fails to process them
# The "Reports" subdirectories - where our application puts reports
# Sync-up files from local "Saved", "Error" and "Reports" directories to H=
ostedFTP, deleting HostedFTP files if they are not present on the local ser=
ver
# "Reverse" mirror, with deleting of remote files not present locally
EXCLUDES=3D"--exclude-glob .*"
EXCLUDES+=3D" --exclude-glob .*/"
INCLUDES=3D"--include-glob /opt/ftpusers/home/*/Saved/*"
INCLUDES+=3D" --include-glob /opt/ftpusers/home/*/Error/*"
INCLUDES+=3D" --include-glob /opt/ftpusers/home/*/Reports/*"
lftp -e "
open '$FTPURL ';
lcd $FTPLOCALFOLDER;
cd $FTPREMOTEFOLDER;
mirror --reverse --continue --only-newer --delete --verbose=3D3 --no-perms =
--parallel=3D8 $EXCLUDES $INCLUDES"
# The "Other" subdirectories -- - where customers or staff put various file=
s
# Sync-down files in HostedFTP "Other" directories to the local server
# "Regular" mirror
EXCLUDES=3D"--exclude-glob .*"
EXCLUDES+=3D" --exclude-glob .*/"
INCLUDES=3D"--include-glob /opt/ftpusers/home/*/Other/*"
lftp -e "
open '$ FTPURL ';
lcd $ FTPLOCALFOLDER;
cd $ FTPREMOTEFOLDER;
mirror --continue --only-newer --verbose=3D3 --no-perms --parallel=3D8 $EXC=
LUDES $INCLUDES"
AGS
--_000_BN6PR01MB2515B092CEE7059971167A42B8F70BN6PR01MB2515prod_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"EN-US" link=3D"#0563C1" vlink=3D"#954F72">
<div class=3D"WordSection1">
<p class=3D"MsoNormal">Hello,<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">We are a SaaS company, migrating our colocation envi=
ronment to AWS.<o:p></o:p></p>
<p class=3D"MsoNormal">As part of that, we are changing how customers will =
provide us with files to be processed by our application.<o:p></o:p></p>
<p class=3D"MsoNormal">Currently they SSH to a server in our colo, and plac=
e their files in one of their subdirectories.<o:p></o:p></p>
<p class=3D"MsoNormal">When we migrate to AWS, our customers will instead S=
TFP upload their files to a hosted FTP provider.<o:p></o:p></p>
<p class=3D"MsoNormal">These files will be synchronized with two AWS-based =
servers, ProdN01IL01 and ProdN02IL01, each of which will be running an LFTP=
script.<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Description –<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">On each of the two AWS servers, each customer’=
s directories will be under /opt/ftpusers/home/<o:p></o:p></p>
<p class=3D"MsoNormal">Each of these will have 6 subdirectories – as =
described below.<o:p></o:p></p>
<p class=3D"MsoNormal">Some of these directories are to receive files from =
the customer, some are for our application’s use, and some are for fi=
le sharing between us and our customers.<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">At the hosted FTP provider we are going to make the =
following directory structure…<o:p></o:p></p>
<p class=3D"MsoNormal">/Servers<o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; /ProdN01IL01<o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; &=
nbsp; /[Customer1]<o:=
p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; /error<o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; /other<o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"margin-left:1.0in;text-indent:.5in">/report=
s<o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; /saved<o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; /synch<o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; &=
nbsp; /[Customer2]<o:=
p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; /error<o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; /other<o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"margin-left:1.0in;text-indent:.5in">/report=
s<o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; /saved<o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; /synch….(and so on)<o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"text-indent:.5in">/ProdN02IL01<o:p></o:p></=
p>
<p class=3D"MsoNormal"> &nbs=
p; &=
nbsp; /[Customer51]<o=
:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; /error<o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; /other<o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"margin-left:1.0in;text-indent:.5in">/report=
s<o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; /saved<o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; /synch<o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; &=
nbsp; /[Customer52]<o=
:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; /error<o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; /other<o:p></o:p></p>
<p class=3D"MsoNormal" style=3D"margin-left:1.0in;text-indent:.5in">/report=
s<o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; /saved<o:p></o:p></p>
<p class=3D"MsoNormal"> &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; /synch….(and so on)
<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Note: We don’t need /inbox subdirectorie=
s on HostedFTP, because they are only used by our application.<o:p></o:p></=
p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">The script --<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Below is the script we are thinking of using.<o:p></=
o:p></p>
<p class=3D"MsoNormal">We invite comments and suggestions.<o:p></o:p></p>
<p class=3D"MsoNormal">Thank you!<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">#!/bin/bash<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">FTPPROTOCOL=3D”sftp”<o:p></o:p></p>
<p class=3D"MsoNormal">FTPHOST=3D”ourcompany.hostedftp.com”<o:p=
></o:p></p>
<p class=3D"MsoNormal">FTPUSER=3D”[email protected]̶=
1;<o:p></o:p></p>
<p class=3D"MsoNormal">FTPPASS=3D”ourcompanyftppassword”<o:p></=
o:p></p>
<p class=3D"MsoNormal">FTPURL=3D"$FTPPROTOCOL://$FTPUSER:$FTPPASS@$FTP=
HOST"<o:p></o:p></p>
<p class=3D"MsoNormal">FTPREMOTEFOLDER=3D”/Servers/ProdN01IL01”=
(or ProdN02IL01, depending on the server)<o:p></o:p></p>
<p class=3D"MsoNormal">FTPLOCALFOLDER=3D”/opt/ftpusers/home”<o:=
p></o:p></p>
<p class=3D"MsoNormal">FTPLOGFILE=3D”/var/log/lftpsyncs.log”<o:=
p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"># The “Synch” subdirectories -- where cu=
stomers put files for processing by our application<o:p></o:p></p>
<p class=3D"MsoNormal"># Move all files from HostedFTP “Synch” =
directories to local server Synch directories, the delete the source files =
from HostedFTP Synch directories<o:p></o:p></p>
<p class=3D"MsoNormal"># “Regular” mirror, with --Remove-source=
-files<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">EXCLUDES=3D"--exclude-glob .*"<o:p></o:p><=
/p>
<p class=3D"MsoNormal">EXCLUDES+=3D" --exclude-glob .*/"<o:p>=
</o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">INCLUDES=3D"--include-glob /opt/ftpusers/home/*=
/Synch/*"<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">lftp -e "<o:p></o:p></p>
<p class=3D"MsoNormal">open '$FTPURL ';<o:p></o:p></p>
<p class=3D"MsoNormal">lcd $FTPLOCALFOLDER;<o:p></o:p></p>
<p class=3D"MsoNormal">cd $FTPREMOTEFOLDER;<o:p></o:p></p>
<p class=3D"MsoNormal">mirror --continue --only-newer --Remove-source-files=
--verbose=3D3 --no-perms --parallel=3D8 $EXCLUDES $INCLUDES”<o:p></o=
:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"># The “Saved” subdirectories – whe=
re our application puts files from Inbox, after successfully processing the=
m<o:p></o:p></p>
<p class=3D"MsoNormal"># The “Error” subdirectories -- where ou=
r application puts files from Inbox, if it fails to process them<o:p></o:p>=
</p>
<p class=3D"MsoNormal"># The “Reports” subdirectories – w=
here our application puts reports<o:p></o:p></p>
<p class=3D"MsoNormal"># Sync-<u>up</u> files from local “Saved=
”, “Error” and “Reports” directories to Hoste=
dFTP, deleting HostedFTP files if they are not present on the local server<=
o:p></o:p></p>
<p class=3D"MsoNormal"># “Reverse” mirror, with deleting of rem=
ote files not present locally<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">EXCLUDES=3D"--exclude-glob .*"<o:p></o:p><=
/p>
<p class=3D"MsoNormal">EXCLUDES+=3D" --exclude-glob .*/"<o:p>=
</o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">INCLUDES=3D"--include-glob /opt/ftpusers/home/*=
/Saved/*"<o:p></o:p></p>
<p class=3D"MsoNormal">INCLUDES+=3D" --include-glob /opt/ftpusers/=
home/*/Error/*"<o:p></o:p></p>
<p class=3D"MsoNormal">INCLUDES+=3D" --include-glob /opt/ftpusers/=
home/*/Reports/*"<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">lftp -e "<o:p></o:p></p>
<p class=3D"MsoNormal">open '$FTPURL ';<o:p></o:p></p>
<p class=3D"MsoNormal">lcd $FTPLOCALFOLDER;<o:p></o:p></p>
<p class=3D"MsoNormal">cd $FTPREMOTEFOLDER;<o:p></o:p></p>
<p class=3D"MsoNormal">mirror --reverse --continue --only-newer --delete --=
verbose=3D3 --no-perms --parallel=3D8 $EXCLUDES $INCLUDES”<o:p></o:p>=
</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"># The “Other” subdirectories -- – =
where customers or staff put various files
<o:p></o:p></p>
<p class=3D"MsoNormal"># Sync-<u>down</u> files in HostedFTP “Other&#=
8221; directories to the local server<o:p></o:p></p>
<p class=3D"MsoNormal"># “Regular” mirror<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">EXCLUDES=3D"--exclude-glob .*"<o:p></o:p><=
/p>
<p class=3D"MsoNormal">EXCLUDES+=3D" --exclude-glob .*/"<o:p>=
</o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">INCLUDES=3D"--include-glob /opt/ftpusers/home/*=
/Other/*"<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">lftp -e "<o:p></o:p></p>
<p class=3D"MsoNormal">open '$ FTPURL ';<o:p></o:p></p>
<p class=3D"MsoNormal">lcd $ FTPLOCALFOLDER;<o:p></o:p></p>
<p class=3D"MsoNormal">cd $ FTPREMOTEFOLDER;<o:p></o:p></p>
<p class=3D"MsoNormal">mirror --continue --only-newer --verbose=3D3 --no-pe=
rms --parallel=3D8 $EXCLUDES $INCLUDES”<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">AGS<o:p></o:p></p>
</div>
</body>
</html>
--_000_BN6PR01MB2515B092CEE7059971167A42B8F70BN6PR01MB2515prod_--
--===============0314867239294704747==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
lftp mailing list
[email protected]
http://univ.uniyar.ac.ru/mailman/listinfo/lftp
--===============0314867239294704747==--