Links2Linux Wiki

SecureCopy (scp) is often used for copying files between unix machines nowadays. What if you have a really huge file and such a transfer got interrupted?

ftp and http both can do partial downloads of files so that you can continue downloading a file where it got interupted to save bandwidth and time.

But with SSH? Hm. SSH/SCP do not have someting like that.

The follwoing little shellscript can do it. It supports downloading, uploading and even transferring a file from one remote host to another!

Call it like “scp+ user@example.com:/home/foo/mybigfile /my/local/file” to download a file using ssh.

If the file exists on the destination and is smaller than the source file file, the remaining bytes will be appended to the destination file.

#!/bin/bash
# scp+.sh
# get/put files via ssh and
# continue partial downloads/uploads or host-to-host transfers
#
# Copyright (c) 2005-2008 by Marc Schiffbauer
# Version 2.0
#

SSH_OPTS="-o ServerAliveInterval=5 -o ServerAliveCountMax=3"
BLOCK_SIZE="1500" # Bytes

# URL like "user@host:path/to/file"
SRC_URL="$1"
SRC_URL_HOST="${SRC_URL%%:*}"
SRC_URL_FILE="${SRC_URL##*:}"

DST_URL="${2:-$(basename $SRC_URL_FILE)}"
DST_URL_HOST="${DST_URL%%:*}"
DST_URL_FILE="${DST_URL##*:}"

if [[ -z "$SRC_URL" || -z "$DST_URL" ]]; then
  echo ""
  echo "  Syntax: $(basename $0) <src> [[<dst>]]"
  echo ""
  echo "    src/dst: [[[user@]]host:]/path/to/file"
  exit 1
fi

if [[ $SRC_URL_HOST == $SRC_URL_FILE ]]; then
  SRC_CMD="sh -c"
else
  SRC_CMD="ssh $SSH_OPTS $SRC_URL_HOST"
fi
if [[ $DST_URL_HOST == $DST_URL_FILE ]]; then
  DST_CMD="sh -c"
else
  DST_CMD="ssh $SSH_OPTS $DST_URL_HOST"
fi

DST_SIZE=$($DST_CMD "if [[sshpartialdownload|-f '$DST_URL_FILE' ]]; then find '$DST_URL_FILE' -printf '%s'; else echo 0; fi")
PADDING=$((BLOCK_SIZE-DST_SIZE%BLOCK_SIZE))

$SRC_CMD "if [[sshpartialdownload|$PADDING -ne $BLOCK_SIZE ]]; then dd if='$SRC_URL_FILE' ibs=1 skip=$DST_SIZE status=noxfer count=$PADDING 2>/dev/null; PADDING=$PADDING; \
  else PADDING=0; fi; dd if='$SRC_URL_FILE' bs=$BLOCK_SIZE skip=\$((($DST_SIZE+PADDING)/$BLOCK_SIZE))" | $DST_CMD "cat >> $DST_URL_FILE"
exit $?