·9 min read

SFTP Automation and Scripting: Batch File Operations

Automate repetitive file transfers with SFTP scripting. Learn batch uploads, scheduled syncs, and integration with CI/CD pipelines.

SFTPAutomationScriptingDevOpsFile Transfer

SFTP (SSH File Transfer Protocol) is secure and reliable, but manual file transfers are tedious. Automation lets you batch uploads, schedule syncs, and integrate file transfers into deployment pipelines.

Why Automate SFTP?

Manual SFTP is slow:

  • Drag-and-drop one file at a time
  • Remember remote paths
  • Repeat same operations daily
  • Error-prone for bulk operations

Automated SFTP is fast:

  • Transfer hundreds of files in one command
  • Schedule recurring transfers
  • Integrate with CI/CD pipelines
  • Consistent and repeatable

Shell Script Automation

Basic Upload Script

#!/bin/bash
# upload.sh - Upload files to remote server

HOST="server.example.com"
USER="deploy"
REMOTE_DIR="/var/www/html"
LOCAL_DIR="./dist"

# Upload all files in dist/
sftp $USER@$HOST <<EOF
cd $REMOTE_DIR
put $LOCAL_DIR/*
bye
EOF

echo "Upload complete"

Batch Upload with Progress

#!/bin/bash
# batch-upload.sh - Upload with progress tracking

HOST="server.example.com"
USER="deploy"
REMOTE_DIR="/uploads"

# Create batch file
cat > /tmp/sftp-batch.txt <<EOF
cd $REMOTE_DIR
put file1.txt
put file2.txt
put file3.txt
bye
EOF

# Execute batch
sftp -b /tmp/sftp-batch.txt $USER@$HOST

# Cleanup
rm /tmp/sftp-batch.txt

Download with Pattern Matching

#!/bin/bash
# download-logs.sh - Download log files

HOST="server.example.com"
USER="admin"
REMOTE_DIR="/var/log"
LOCAL_DIR="./logs"

mkdir -p $LOCAL_DIR

sftp $USER@$HOST <<EOF
cd $REMOTE_DIR
mget *.log $LOCAL_DIR/
bye
EOF

echo "Downloaded $(ls $LOCAL_DIR/*.log | wc -l) log files"

Sync Directory (rsync-like)

#!/bin/bash
# sync.sh - Sync local to remote

HOST="server.example.com"
USER="deploy"
REMOTE_DIR="/app"
LOCAL_DIR="./build"

# Get remote file list
sftp $USER@$HOST <<EOF > /tmp/remote-files.txt
cd $REMOTE_DIR
ls -1
bye
EOF

# Upload only new/changed files
for file in $LOCAL_DIR/*; do
    filename=$(basename $file)
    if ! grep -q "$filename" /tmp/remote-files.txt; then
        echo "Uploading $filename"
        sftp $USER@$HOST <<EOF
cd $REMOTE_DIR
put $file
bye
EOF
    fi
done

rm /tmp/remote-files.txt

Error Handling

#!/bin/bash
# safe-upload.sh - Upload with error handling

HOST="server.example.com"
USER="deploy"
REMOTE_DIR="/uploads"
LOCAL_FILE="data.csv"

# Test connection first
if ! ssh -o ConnectTimeout=5 -o BatchMode=yes $USER@$HOST exit 2>/dev/null; then
    echo "Error: Cannot connect to $HOST"
    exit 1
fi

# Upload with error check
sftp $USER@$HOST <<EOF
cd $REMOTE_DIR
put $LOCAL_FILE
bye
EOF

if [ $? -eq 0 ]; then
    echo "Upload successful"
else
    echo "Upload failed"
    exit 1
fi

Python Automation

Basic Upload with paramiko

#!/usr/bin/env python3
# upload.py - Upload files using paramiko

import paramiko
import os

HOST = 'server.example.com'
USER = 'deploy'
REMOTE_DIR = '/uploads'
LOCAL_DIR = './dist'

# Connect
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(HOST, username=USER, key_filename='~/.ssh/id_ed25519')

# Open SFTP
sftp = ssh.open_sftp()
sftp.chdir(REMOTE_DIR)

# Upload files
for filename in os.listdir(LOCAL_DIR):
    local_path = os.path.join(LOCAL_DIR, filename)
    if os.path.isfile(local_path):
        print(f'Uploading {filename}')
        sftp.put(local_path, filename)

# Cleanup
sftp.close()
ssh.close()
print('Upload complete')

Download with Progress

#!/usr/bin/env python3
# download.py - Download with progress callback

import paramiko

def progress(transferred, total):
    percent = (transferred / total) * 100
    print(f'\rDownloading: {percent:.1f}%', end='')

HOST = 'server.example.com'
USER = 'admin'
REMOTE_FILE = '/data/large-dataset.zip'
LOCAL_FILE = './large-dataset.zip'

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(HOST, username=USER)

sftp = ssh.open_sftp()
sftp.get(REMOTE_FILE, LOCAL_FILE, callback=progress)

print('\nDownload complete')
sftp.close()
ssh.close()

Directory Sync

#!/usr/bin/env python3
# sync.py - Sync directories

import paramiko
import os
import hashlib

def file_hash(filepath):
    """Calculate MD5 hash of file"""
    hash_md5 = hashlib.md5()
    with open(filepath, "rb") as f:
        for chunk in iter(lambda: f.read(4096), b""):
            hash_md5.update(chunk)
    return hash_md5.hexdigest()

def sync_directories(ssh, local_dir, remote_dir):
    sftp = ssh.open_sftp()
    
    # Get remote files
    remote_files = {}
    for filename in sftp.listdir(remote_dir):
        remote_path = os.path.join(remote_dir, filename)
        try:
            stat = sftp.stat(remote_path)
            if stat.st_size >= 0:  # Is file
                remote_files[filename] = stat.st_size
        except:
            pass
    
    # Upload new/changed files
    for filename in os.listdir(local_dir):
        local_path = os.path.join(local_dir, filename)
        if os.path.isfile(local_path):
            local_size = os.path.getsize(local_path)
            
            if filename not in remote_files or remote_files[filename] != local_size:
                remote_path = os.path.join(remote_dir, filename)
                print(f'Uploading {filename}')
                sftp.put(local_path, remote_path)
    
    sftp.close()

# Usage
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('server.example.com', username='deploy')

sync_directories(ssh, './build', '/app')
ssh.close()

Batch Operations

#!/usr/bin/env python3
# batch-operations.py - Multiple SFTP operations

import paramiko

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('server.example.com', username='deploy')
sftp = ssh.open_sftp()

# Create directory
try:
    sftp.mkdir('/uploads/2026')
    print('Created directory /uploads/2026')
except IOError:
    print('Directory already exists')

# Upload multiple files
files = ['report.pdf', 'data.csv', 'image.png']
for filename in files:
    sftp.put(f'./{filename}', f'/uploads/2026/{filename}')
    print(f'Uploaded {filename}')

# List files
print('\nFiles in /uploads/2026:')
for filename in sftp.listdir('/uploads/2026'):
    stat = sftp.stat(f'/uploads/2026/{filename}')
    print(f'  {filename}: {stat.st_size} bytes')

# Delete old files
old_files = ['report-old.pdf', 'data-backup.csv']
for filename in old_files:
    try:
        sftp.remove(f'/uploads/2026/{filename}')
        print(f'Deleted {filename}')
    except IOError:
        pass

sftp.close()
ssh.close()

Scheduled Transfers

Cron Job (Linux/macOS)

# Edit crontab
crontab -e

# Upload files every day at 2 AM
0 2 * * * /path/to/upload.sh >> /var/log/sftp-upload.log 2>&1

# Sync every hour
0 * * * * /path/to/sync.sh >> /var/log/sftp-sync.log 2>&1

Launchd (macOS)

<!-- ~/Library/LaunchAgents/com.foxterm.sftp-upload.plist -->
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.foxterm.sftp-upload</string>
    <key>ProgramArguments</key>
    <array>
        <string>/path/to/upload.sh</string>
    </array>
    <key>StartInterval</key>
    <integer>3600</integer>
    <key>StandardOutPath</key>
    <string>/tmp/sftp-upload.log</string>
    <key>StandardErrorPath</key>
    <string>/tmp/sftp-upload-error.log</string>
</dict>
</plist>

Load with:

launchctl load ~/Library/LaunchAgents/com.foxterm.sftp-upload.plist

CI/CD Integration

GitHub Actions

# .github/workflows/deploy.yml
name: Deploy to Server

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Build
        run: npm run build
      
      - name: Upload via SFTP
        uses: appleboy/scp-action@master
        with:
          host: ${{ secrets.SERVER_HOST }}
          username: ${{ secrets.SERVER_USER }}
          key: ${{ secrets.SSH_KEY }}
          source: "./dist/*"
          target: "/var/www/html"

GitLab CI

# .gitlab-ci.yml
deploy:
  stage: deploy
  image: alpine:latest
  before_script:
    - apk add --no-cache openssh-client
  script:
    - mkdir -p ~/.ssh
    - echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_ed25519
    - chmod 600 ~/.ssh/id_ed25519
    - ssh-keyscan $SERVER_HOST >> ~/.ssh/known_hosts
    - sftp -b batch.txt $SERVER_USER@$SERVER_HOST
  only:
    - main

FoxTerm SFTP Automation

FoxTerm makes SFTP automation visual and intuitive:

Visual Batch Operations

  • Select multiple files with Shift/Cmd-click
  • Drag-and-drop to upload/download
  • Progress bars for all transfers
  • Pause/resume support

Saved Connections

  • Store server credentials securely
  • One-click connect
  • Recent files list
  • Quick access to favorite directories

Transfer Queue

  • Queue multiple transfers
  • Priority ordering
  • Bandwidth throttling
  • Automatic retry on failure

Integration

  • Open remote files in local editors
  • Sync local and remote directories
  • Watch folders for automatic upload
  • Webhook notifications on completion

Best Practices

  1. Use SSH keys — Never hardcode passwords in scripts
  2. Test connections — Verify connectivity before transfers
  3. Handle errors — Check return codes and log failures
  4. Limit bandwidth — Don't saturate network during business hours
  5. Validate transfers — Check file sizes or checksums after upload
  6. Secure credentials — Use environment variables or secret managers
  7. Monitor transfers — Log all operations for auditing

Conclusion

SFTP automation transforms tedious manual transfers into reliable, repeatable workflows. Whether you use shell scripts for simple tasks or Python for complex logic, automation saves time and reduces errors.

FoxTerm complements scripted automation with a visual interface for ad-hoc transfers and monitoring.

FoxTerm Pro includes advanced SFTP automation features. Download at foxterm.app.