Is there a better way to transfer data that doesn't use so much cache?

Rob Campbell via rsync <[email protected]>
Newsgroups gmane.network.rsync.general
Message-ID <CAHej=pVNXebp9fOc=Nn8x01R3LUjcgv2HT++a76Drg_vog7WuQ@mail.gmail.com>
I've created a script that syncs (and removes) data from as many as 4
places and puts them all in one of 2 directories.  The commands are:

rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.nef'
-f'+ *.jp*g' -f'+ *.tif' -f'+ *.xmp' -f'+ /*' -f'- *'
"$D850/DCIM/100ND850/" $STAGINGP/ | tee -a $LOG

rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.nef'
-f'+ *.jp*g' -f'+ *.tif' -f'+ *.xmp' -f'+ /*' -f'- *' "$Z9/DCIM/100NCZ_9/"
$STAGINGP/ | tee -a $LOG

rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4'
-f'+ /*' -f'- *' "$DASHCAM/CARDV/VIDEO/" $STAGINGV/ | tee -a $LOG

rsync -avt --progress --remove-source-files --info=progress2 -f'-
Screenshots/' -f'+ *.nef' -f'+ *.jpg' -f'+ *.jp*g' -f'+ *.png' -f'+ *.dng'
-f'+ *.gif' -f'- *.thumbnails' -f'- *.android' -f'+ */' -f'+ DCIM/*' -f'+
Snapbridge/*' -f'+ Pictures/*' -f'+ Download/*' -f'+ Textgram/*' -f'- *'
$PHONE/ $STAGINGP/ | tee -a $LOG

rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4'
-f'+ *.avi' -f'+ *.mov' -f'+ *.mp*g' -f'+ *.3gp' -f'- *'
--files-from=<(find $PHONE -type f ! -path "*Download*" ! -path
"*.trashed*" ! -iname ????????????????????????????????.mp4 ! -iname
'*.mp4\.*')/ $STAGINGV/ | tee -a $LOG

rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4'
-f'+ *.avi' -f'+ *.mov' -f'+ *.mp*g' -f'+ *.3gp' -f'+ Movies/*' -f'+
*Recordings/*' -f'+ DCIM/*' -f'+ Snapbridge/*' -f'- */' -f'- *' $PHONE/
$STAGINGV/ | tee -a $LOG

rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4'
-f'- *' --files-from=<(find $PHONE -iname
????????????????????????????????.mp4) / $STAGINGV/TIKTOK/ | tee -a $LOG

rsync -avt --progress --remove-source-files --info=progress2 -f'+ *'
$PHONE/Downloads/ $COMPUTER/Downloads/

rsync -avt --progress --remove-source-files --info=progress2 -f'-
screenshot*' -'f- Screenshot*' -f'- Boondocks/' -f'- Dilbert/' -f'+ *.png'
-f'+ *.jp*g' -f'+ *.dng' -f'+ *.gif' -f'- *20*/' -f'- *' -f'+ */' -f'-
$STAGINGP/' $MYPICS/ $STAGINGP/ | tee -a $LOG

rsync -avt --progress --remove-source-files --info=progress2 -f'+
Screenshot*.png' -f'- Staging/' -f'- *' $MYPICS/ $STAGINGP/Screenshots/ |
tee -a $LOG

rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.3gpp'
-f'+ *.mp4' -f'+ *.mp*g' -f'+ *.avi' -f'+ *.asf' -f'+ *.wmv' -f'- *'
$HOME/Downloads $STAGINGV/ | tee -a $LOG

rsync -avt --progress --remove-source-files --info=progress2 -f'+ *.mp4'
-f'+ *.mp*g' -f'+ *.avi' -f'+ *.asf' -f'+ *.wmv' -f'+ *.3gpp' -f'- *'
$MYVIDEOS/ $STAGINGV/ | tee -a $LOG


The problem isn't that there are many syncs because the problem happens on
the first one that runs.  Before any of them run I run:

sudo free -w -h;sync && echo 1 > /proc/sys/vm/drop_caches;free -w -h

I do not run this before each one because it sometimes takes a while to
/proc/sys/vm/drop_caches

Is there something in the logic that can be done to make this perform
better or should I use something other than rsync or is what I am getting
as good as it will get regardless of what I use?

Some of these directories can be over a gig.  Most of these are media files
and should have exif data that has the timestamp so maybe I can get rid of
-t but it is easier to keep the timestamp of the file rather than running
exiftool to also use the create date to "touch" the file but maybe using
exiftool is a faster way?

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In all things, Be Intentional.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.