ttr.aws.utils.s3 0.2.3 (experimental)

Some command lines utilities to interact with files stored in AWS S3 incl. versioned ones.

by easy_install or pip

$ pip install ttr.aws.utils.s3

or:

$ easy_install ttr.aws.utils.s3

Using setup.py

1. Unpack the package
2. go to directory where is setup.py located
3. run: python setup.py install

Resulting scripts are then located in Python Script directory.

Using buildout

being in root of source directory:

$ python bootstrap.py

$ bin/buildout

And you get your scripts in directory bin/

s3lsvers

List versions of some feed. Could output into CSV file (-list-file) and/or html chart (-html-file).

$ s3lsvers.exe -h
usage: s3lsvers-script.py [-h] [-from None] [-to None] [-list-file None]
                          [-html-file None] [-version-id None]
                          bucket_name key_name

Lists all versions of given key, possibly filtering by from - to range for version last_modified time.
    Allows to put the listing into csv file and or into html chart.

        Listing shows:
          key_name
            "file name". Can repeat if the file has more versions

          version_id
            unique identifier for given version on given bucket. Has form of string and not a number. identifiers are "random", do not
            expect that they are sorten alphabetically.

          size
            size of file in bytes

          last_modified
            ISO 8601 formated time of file modification, e.g. `2011-06-22T03:05:09.000Z`

          age
            difference between last_modified or given version and preceding version.
            It is sort of current update interval for that version.

        Sample use:
        Lists to the screen all versions of file keynme in the bucketname bucket::

            $ s3lsvers bucketname keyname

        Lists all versions younger then given time (from given time till now)::

            $ s3lsvers -from 2011-07-19T12:00:00 bucketname keyname

        Lists all versions older then given time (from very first versin till given date)::

            $ s3lsvers -to 2011-07-19T12:00:00 bucketname keyname

        Lists all versions in period betwen from and to time::

            $ s3lsvers -from 2010-01-01 -to 2011-07-19T12:00:00 bucketname keyname

        Lists all versions and writes them into csv file named versions.csv::

            $ s3lsvers -list-file versions.csv bucketname keyname

        Lists all versions and writes them into html chart file named chart.html::

            $ s3lsvers -html-file chart.html bucketname keyname

        Prints to screen, writes to csv, creates html chart and this all for versions in given time period.::

            $ s3lsvers -from 2010-01-01 -to 2011-07-19T12:00:00 -list-file versions.csv -html-file chart.html bucketname keyname



positional arguments:
  bucket_name           name of AWS S3 bucket, which is searched.
  key_name              name of key to list. Typically it is complete name of
                        a key, but also truncated name can be set, in this
                        case all keys, sharing this prefix, will be listed.

optional arguments:
  -h, --help            show this help message and exit
  -from None, --from-time None
                        Modification time of oldest expected version expressed
                        in ISO 8601 format. Can be truncated. (default: goes
                        to the oldest version)
  -to None, --to-time None
                        Modification time of youngest expected version
                        expressed in ISO 8601 format. Can be truncated.
                        (default: goes to the latest version)
  -list-file None       Name of file, where is result written in csv format.
                        If set, the file is always overwritten.
  -html-file None       Name of file, where is result written in html format
                        (as a chart). If set, the file is always overwritten.
  -version-id None      Optional version-id. If specified, listing does not
                        start from the freshest version, but starts searching
                        from given VERSION_ID and continues searching older
                        and older versions. This could speed up listng in
                        case, you need rather older files and you know
                        VERSION_ID which came somehow later then is the time
                        scope you are going to list.

s3getvers

$ s3getvers -h
usage: s3getvers-script.py [-h] [-output-version-id-names] [-no-decompression]
                           bucket_name csv_version_file

Fetch file versions as listed in provided csv file

    Typical csv file (as by default produced by s3lsvers) is:

        my/versioned/feed.xml;OrUr6XO8KSKEHbd8mQ.MloGcGlsh7Sir;191345;2012-05-23T20:45:10.000Z;39
        my/versioned/feed.xml;xhkVOy.dJfjSfUwse8tsieqjDicp0owq;192790;2012-05-23T20:44:31.000Z;62
        my/versioned/feed.xml;oKneK.N2wS8pW8.EmLqjldYlgcFwxN3V;193912;2012-05-23T20:43:29.000Z;58

    and has columns:
    :key_name: name of the feed (not containing the bucket name itself)
    :version_id: string, identifying unique version. Any following columns can contain anything.
    :size: size in bytes. This column is not used and can be missing.
    :last_modified: date, when the version was posted. This column is not used and can be missing.

    Typical use (assuming, above csv file is available under name verlist.csv)::

        $ s3getvers-script.py yourbucketname verlist.csv

    What will create following files in current directory:

    * my/versioned/feed.xml.2012-05-23T20_45_10.xml
    * my/versioned/feed.xml.2012-05-23T20_44_31.xml
    * my/versioned/feed.xml.2012-05-23T20_43_29.xml

    Even though these files are gzipped on server, they will be decompressed on local disk.



positional arguments:
  bucket_name           bucket name (default: None)
  csv_version_file      name of CSV file with version_id

optional arguments:
  -h, --help            show this help message and exit
  -output-version-id-names
                        Resulting file names shall use version_id to become
                        distinguished (default is to use timestamp of file
                        creation)
  -no-decompression     Keeps the files as they come, do not decompress, if
                        they come compressed

0.2.3

Release date: 2012-05-28

• command s3lsvers - to list key versions
• command s3getvers - to fetch versions listed in csv file