Links2Linux Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
SecureCopy (scp) is often used for copying files between unix machines nowadays. 
What if you have a really huge file and such a transfer got interrupted?

ftp and http both can do partial downloads of files so that you can continue
downloading a file where it got interupted to save bandwidth and time.

But with SSH? Hm. SSH/SCP do not have someting like that.

The follwoing little shellscript can do it. It supports downloading, uploading and 
even transferring a file from one remote host to another!

Call it like "scp+ user@example.com:/home/foo/mybigfile /my/local/file"
to download a file using ssh. 

If the file exists on the destination and is smaller
than the source file file, the remaining bytes will be appended to the destination file.

<code>
#!/bin/bash
# scp+.sh
# get/put files via ssh and
# continue partial downloads/uploads or host-to-host transfers
#
# Copyright (c) 2005-2008 by Marc Schiffbauer
# Version 2.0
#

SSH_OPTS="-o ServerAliveInterval=5 -o ServerAliveCountMax=3"
BLOCK_SIZE="1500" # Bytes

# URL like "user@host:path/to/file"
SRC_URL="$1"
SRC_URL_HOST="${SRC_URL%%:*}"
SRC_URL_FILE="${SRC_URL##*:}"

DST_URL="${2:-$(basename $SRC_URL_FILE)}"
DST_URL_HOST="${DST_URL%%:*}"
DST_URL_FILE="${DST_URL##*:}"

if [[ -z "$SRC_URL" || -z "$DST_URL" ]]; then
  echo ""
  echo "  Syntax: $(basename $0) <src> [[<dst>]]"
  echo ""
  echo "    src/dst: [[[user@]]host:]/path/to/file"
  exit 1
fi

if [[ $SRC_URL_HOST == $SRC_URL_FILE ]]; then
  SRC_CMD="sh -c"
else
  SRC_CMD="ssh $SSH_OPTS $SRC_URL_HOST"
fi
if [[ $DST_URL_HOST == $DST_URL_FILE ]]; then
  DST_CMD="sh -c"
else
  DST_CMD="ssh $SSH_OPTS $DST_URL_HOST"
fi

DST_SIZE=$($DST_CMD "if [[sshpartialdownload|-f '$DST_URL_FILE' ]]; then find '$DST_URL_FILE' -printf '%s'; else echo 0; fi")
PADDING=$((BLOCK_SIZE-DST_SIZE%BLOCK_SIZE))

$SRC_CMD "if [[sshpartialdownload|$PADDING -ne $BLOCK_SIZE ]]; then dd if='$SRC_URL_FILE' ibs=1 skip=$DST_SIZE status=noxfer count=$PADDING 2>/dev/null; PADDING=$PADDING; \
  else PADDING=0; fi; dd if='$SRC_URL_FILE' bs=$BLOCK_SIZE skip=\$((($DST_SIZE+PADDING)/$BLOCK_SIZE))" | $DST_CMD "cat >> $DST_URL_FILE"
exit $?
</code>
Links2Linux Wiki

User Tools

Site Tools

Page Tools
