lftp -u user,pwd -e "set sftp:connect-program 'ssh -a -x -T -c arcfour -o Compression=no'; mirror -v -c --loop --use-pget-n=3 -P 2 /remote/dir/ /local/dir/; quit" sftp://remotehost:22
Fastest segmented parallel sync of a remote directory over ssh
Mirror a remote directory using some tricks to maximize network speed.
lftp:: coolest file transfer tool ever
-u: username and password (pwd is merely a placeholder if you have ~/.ssh/id_rsa)
-e: execute internal lftp commands
set sftp:connect-program: use some specific command instead of plain ssh
-a -x -T: disable useless things
-c arcfour: use the most efficient cipher specification
-o Compression=no: disable compression to save CPU
mirror: copy remote dir subtree to local dir
-v: be verbose (cool progress bar and speed meter, one for each file in parallel)
-c: continue interrupted file transfers if possible
–loop: repeat mirror until no differences found
–use-pget-n=3: transfer each file with 3 independent parallel TCP connections
-P 2: transfer 2 files in parallel (totalling 6 TCP connections)
sftp://remotehost:22: use sftp protocol on port 22 (you can give any other port if appropriate)
You can play with values for –use-pget-n and/or -P to achieve maximum speed depending on the particular network.
If the files are compressible removing “-o Compression=n” can be beneficial.
Better create an alias for the command.