Welcome, guest | Sign In | My Account | Store | Cart

This script copies files in unattended mode over SSH using a glob pattern. It uses the paramiko module behind the scenes. It operates as an actual SSH client, and does not rely on any command line utilities, such as scp.

It first tries to connect using a key from a private key file or from an SSH agent. If RSA authentication fails, it will try to authenticate with a password if passwords are allowed on the SSH server. Assumes the rsa_private_key was generated with an empty passphrase.

On most linux/unix-like systems paramiko can be installed with sudo easy_install paramiko or pip install paramiko or using the built-in system package manager.

See also http://www.paramiko.org/installing.html

Python, 140 lines
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
#!/usr/bin/env python

## Copy files unattended over SSH using a glob pattern.
## It tries first to connect using a private key from a private key file
## or provided by an SSH agent. If RSA authentication fails, then 
## password login is attempted.

##
## DEPENDENT MODULES:
##      * paramiko, install it with `easy_install paramiko`

## NOTE: 1. The script assumes that the files on the source
##       computer are *always* newer that on the target;
##       2. no timestamps or file size comparisons are made
##       3. use at your own risk

hostname = '10.0.43.4' # remote hostname where SSH server is running
port = 22
username = 'myssh-username'
password = 'myssh-password'
rsa_private_key = r"/home/paramikouser/.ssh/rsa_private_key"

dir_local='/home/paramikouser/local_data'
dir_remote = "remote_machine_folder/subfolder"
glob_pattern='*.*'

import os
import glob
import paramiko
import md5

def agent_auth(transport, username):
    """
    Attempt to authenticate to the given transport using any of the private
    keys available from an SSH agent or from a local private RSA key file (assumes no pass phrase).
    """
    try:
        ki = paramiko.RSAKey.from_private_key_file(rsa_private_key)
    except Exception, e:
        print 'Failed loading' % (rsa_private_key, e)

    agent = paramiko.Agent()
    agent_keys = agent.get_keys() + (ki,)
    if len(agent_keys) == 0:
        return

    for key in agent_keys:
        print 'Trying ssh-agent key %s' % key.get_fingerprint().encode('hex'),
        try:
            transport.auth_publickey(username, key)
            print '... success!'
            return
        except paramiko.SSHException, e:
            print '... failed!', e

# get host key, if we know one
hostkeytype = None
hostkey = None
files_copied = 0
try:
    host_keys = paramiko.util.load_host_keys(os.path.expanduser('~/.ssh/known_hosts'))
except IOError:
    try:
        # try ~/ssh/ too, e.g. on windows
        host_keys = paramiko.util.load_host_keys(os.path.expanduser('~/ssh/known_hosts'))
    except IOError:
        print '*** Unable to open host keys file'
        host_keys = {}

if host_keys.has_key(hostname):
    hostkeytype = host_keys[hostname].keys()[0]
    hostkey = host_keys[hostname][hostkeytype]
    print 'Using host key of type %s' % hostkeytype

# now, connect and use paramiko Transport to negotiate SSH2 across the connection
try:
    print 'Establishing SSH connection to:', hostname, port, '...'
    t = paramiko.Transport((hostname, port))
    t.start_client()

    agent_auth(t, username)

    if not t.is_authenticated():
        print 'RSA key auth failed! Trying password login...'
        t.connect(username=username, password=password, hostkey=hostkey)
    else:
        sftp = t.open_session()
    sftp = paramiko.SFTPClient.from_transport(t)

    # dirlist on remote host
#    dirlist = sftp.listdir('.')
#    print "Dirlist:", dirlist

    try:
        sftp.mkdir(dir_remote)
    except IOError, e:
        print '(assuming ', dir_remote, 'exists)', e

#    print 'created ' + dir_remote +' on the hostname'

    # BETTER: use the get() and put() methods
    # for fname in os.listdir(dir_local):

    for fname in glob.glob(dir_local + os.sep + glob_pattern):
        is_up_to_date = False
        if fname.lower().endswith('xml'):
            local_file = os.path.join(dir_local, fname)
            remote_file = dir_remote + '/' + os.path.basename(fname)

            #if remote file exists
            try:
                if sftp.stat(remote_file):
                    local_file_data = open(local_file, "rb").read()
                    remote_file_data = sftp.open(remote_file).read()
                    md1 = md5.new(local_file_data).digest()
                    md2 = md5.new(remote_file_data).digest()
                    if md1 == md2:
                        is_up_to_date = True
                        print "UNCHANGED:", os.path.basename(fname)
                    else:
                        print "MODIFIED:", os.path.basename(fname),
            except:
                print "NEW: ", os.path.basename(fname),

            if not is_up_to_date:
                print 'Copying', local_file, 'to ', remote_file
                sftp.put(local_file, remote_file)
                files_copied += 1
    t.close()

except Exception, e:
    print '*** Caught exception: %s: %s' % (e.__class__, e)
    try:
        t.close()
    except:
        pass
print '=' * 60
print 'Total files copied:',files_copied
print 'All operations complete!'
print '=' * 60

I have looked in the demos folder in paramiko, but the examples did not provide a full solution for me, so here is a working version of what I wanted to achieve - copy a bunch of files to a remote server running SSH, using either a private RSA key file, an SSH key agent, or a password. Should be able to work with pageant.exe on win32.

If you have an SSH key generated with puttygen, then make sure that you have exported the key to OpenSSH format.

3 comments

Rob 9 years, 7 months ago  # | flag

Works like charm, but a small problem with it. If the authentication doesn't happen with the RSA forcing a username/password to be used, python throws an exception of "thread already in use".

Rob 9 years, 7 months ago  # | flag

For what it's worth, I was able to fix this by changing line 85:

t.connect(username=username, password=password, hostkey=hostkey)

to

t.auth_password(username=username, password=password)

This is because a thread is already started from transport and if your RSA key doesn't connect, this program will resort to authenticate using password. And since t.connect starts a new thread, an exception will be raised.

However, t.auth_password utilizes the existing thread to authenticate via password. I tested this and it works.

fahadkalis 9 years, 4 months ago  # | flag

can you plz guide me what should I added at line 70 AND plz check correct me if anything wrong in below data? Thank You

hostname = '192.168.2.34' # remote hostname where SSH server is running port = 22 username = 'pi' password = 'raspberry' rsa_private_key = r"/home/paramikouser/.ssh/rsa_private_key"

dir_local='/home/tansen/python' dir_remote = "/home/pi/python" glob_pattern='.'

Created by ccpizza on Tue, 16 Jun 2009 (MIT)
Python recipes (4591)
ccpizza's recipes (18)
Network (1)

Required Modules

Other Information and Tasks