iw.recipe.backup 0.2.1

Zc buildout recipe that provides a backup script

trunk (2009-11-25)

• xxx [Ingeniweb]

0.2.1 (2008-11-24)

• added 'base-path' option [fdupre]

if set, use it instead buildout-dir, really usefull if datas are stored in another place in the system (NAS, NFS mount...).

0.2.0 (2008-05-08)

• added 'archive-before-backup' option. [tdesvenain]

True by default. If set to 0, buildout is not backuped before a restore

• added 'include-folders' option, that allows to limit backup

on some directories [tdesvenain]

• create backups dir if not exists [tdesvenain]
• add format option instead of archive-root-name

[gawel]

• fixing test so it passes on all machines

[tarek]

0.1.2 (2008-04-11)

• Added the fs-location option, that allows

doing a safe backup of a running Data.fs. [tarek]

0.1.1 (2008-03-21)

• Added the exclude-folders option
• Now asks a question before restoring.

It is more a conveniency since everything is backed up in any case. [tarek]

0.1.0 (2008-03-12)

• Initial implementation.

[tarek] - Created recipe with ZopeSkel [Ingeniweb].

Supported options

The recipe supports the following options:

backup-script-name Name of the backup script. Default: backup

restore-script-name Name of the restore script. Default: restore

format Formated name of the archive. Default to %(name)s-%(year)s-%(month)s-%(day)s-%(hour)s-%(minute)s where name is the section name

exclude-folders Names of folder to avoid backing up. Relative to buildout root.

include-folders If set, back up those folders only. Relative to buildout root.

fs-location If given, indicates the path of the file system storage. The recipe will carefully copy it when doing a backup, by using a transaction-level read of the file. This means that you can launch a backup without stopping the Zope server.

target-folder Folder where the archives are stored. Mandatory

log-file File where all calls are recorded. Mandatory

archive-before-restore If not set to 0, do a complete backup before a restore

prompt-before-restore If not set to 0, prompt before a restore

base-path If set, backup this path instead base on instance path

Example usage

We'll start by creating a buildout that uses the recipe:

>>> import os
>>> root =  os.path.split(sample_buildout)[0]
>>> if root == '':

... root = '.'

Let's copy a real Data.fs in our buildout:

>>> import shutil
>>> data_fs = os.path.join(test_dir, 'Data.fs')
>>> shutil.copyfile(data_fs, join(sample_buildout, 'Data.fs'))

>>> write('buildout.cfg',
... """
... [buildout]
... parts = backup
... index = http://pypi.python.org/simple
...
... [backup]
... recipe = iw.recipe.backup
...
... format = %(format)s
... target-folder = %(root)s
... log-file = %(root)s/backup.log
... fs-location = ${buildout:directory}/Data.fs
... """ % {'root': root, 'format':'%(name)s-%(year)s-%(month)s-%(day)s'})

Let's run the buildout:

>>> null = system(buildout)

Let's see what we got in the backup script:

>>> print open(join(sample_buildout, 'bin', 'backup')).read()

#!... <BLANKLINE> import sys sys.path[0:0] = [ ... ] <BLANKLINE> import iw.recipe.backup.archive <BLANKLINE> if __name__ == '__main__': iw.recipe.backup.archive.archive_buildout(('/sample-buildout', '%(name)s-%(year)s-%(month)s-%(day)s', '..._TEST_', '/backup.log', [], [], '/sample-buildout/Data.fs', {'name': 'backup'})) <BLANKLINE>

Ok, let's call it to backup the current buildout:

>>> print system(join(sample_buildout, 'bin', 'backup'))

Starting the backup... Archived in /backup-XXXX-XX-XX.tar.gz ... <BLANKLINE>

We should have a log file generated as well:

>>> print open(join(root, 'backup.log')).read()

20... Starting the backup... 20... INFO Archived in /backup-XXXX-XX-XX.tar.gz

We also have a restore feature:

>>> print system(join(sample_buildout, 'bin', 'restore'))

Usage: ...restore archive_name <BLANKLINE>

Let's set the user input:

>>> from iw.recipe.backup.testing import set_input
>>> set_input('Y')

Oh right, the restore script takes the name of the archive:

>>> import glob
>>> arc = glob.glob('%s/*.tar.gz' % root)[0]
>>> print system(join(sample_buildout, 'bin', 'restore %s' % arc))

Are you sure you want to restore ? Every data will be lost ! (y/N) Y Archiving current folder before restoring Starting the backup... Archived in /before-restore-XXXX-XX-XX-XX-XX.tar.gz Starting the restore... Archive restored in /sample-buildout ... <BLANKLINE>

And a restore always makes an archive on the current folder before it is applied, to make sure nothing is never lost.

There's also something quite important: make sure the archive and log files are not located in the buildout !:

>>> write('buildout.cfg',

... """ ... [buildout] ... parts = backup ... index = http://pypi.python.org/simple ... ... [backup] ... recipe = iw.recipe.backup ... ... target-folder = %(root)s ... log-file = %(root)s/backup.log ... """ % {'root': sample_buildout})

>>> print system(buildout)
Uninstalling backup.
Installing backup.
While:
Installing backup.
<BLANKLINE>
An internal error occured due to a bug in either zc.buildout or in a
recipe being used:
...
ValueError: Cannot backup within the buildout ! Check your values
<BLANKLINE>

A bit of cleaning:

>>> import glob
>>> arc = glob.glob('%s/*.tar.gz' % root)
>>> for f in arc:

... os.remove(f)

We can also exclude some folders from being archived:

>>> os.mkdir(join(sample_buildout, 'not'))
>>> open(join(sample_buildout, 'not', 'f'), 'w').write('me file')

>>> os.mkdir(join(sample_buildout, 'neh'))

>>> write('buildout.cfg',
... """
... [buildout]
... parts = backup
... index = http://pypi.python.org/simple
...
... [backup]
... recipe = iw.recipe.backup
...
... target-folder = %(root)s
... log-file = %(root)s/backup.log
... fs-location = ${buildout:directory}/Data.fs
... exclude-folders =
...     %(sample_buildout)s/not
...     %(sample_buildout)s/neh
... """ % {'root': root, 'sample_buildout': sample_buildout})

Running the buildout again:

>>> print system(buildout+' -D')

Installing backup. Generated script '...backup'. Generated script '...restore'. <BLANKLINE>

Let's backup:

>>> print system(join(sample_buildout, 'bin', 'backup'))

Starting the backup... Archived in /backup-XXXX-XX-XX-XX-XX.tar.gz ... <BLANKLINE>

Let's remove the folder and the Data.fs:

>>> import shutil
>>> shutil.rmtree(join(sample_buildout, 'not'))
>>> os.rmdir(join(sample_buildout, 'neh'))
>>> os.remove(join(sample_buildout, 'Data.fs'))

Let's restore:

>>> arc = glob.glob('%s/*.tar.gz' % root)[0]
>>> print system(join(sample_buildout, 'bin', 'restore %s' % arc))

Are you sure you want to restore ? Every data will be lost ! (y/N) Y ... <BLANKLINE>

And make sure the not folder is not back !

>>> os.path.exists(join(sample_buildout, 'not'))
False
>>> os.path.exists(join(sample_buildout, 'neh'))
False

And the Data.fs is back, so we're OK:

>>> os.path.exists(join(sample_buildout, 'Data.fs'))

True

We can also restrict archive on some folders

>>> os.mkdir(join(sample_buildout, 'not'))
>>> open(join(sample_buildout, 'not', 'f'), 'w').write('me file')

>>> os.mkdir(join(sample_buildout, 'neh'))
>>> open(join(sample_buildout, 'neh', 'f'), 'w').write('me file')

>>> os.mkdir(join(sample_buildout, 'neh/not'))
>>> open(join(sample_buildout, 'neh/not', 'f'), 'w').write('me file')

>>> write('buildout.cfg',
... """
... [buildout]
... parts = backup
... index = http://pypi.python.org/simple
...
... [backup]
... recipe = iw.recipe.backup
...
... target-folder = %(root)s
... log-file = %(root)s/backup.log
... fs-location = ${buildout:directory}/Data.fs
... exclude-folders =
...     %(sample_buildout)s/not
...     %(sample_buildout)s/neh/not
... include-folders =
...     %(sample_buildout)s/neh
... prompt-before-restore = 0
... archive-before-restore = 0
... """ % {'root': root, 'sample_buildout': sample_buildout})

Running the buildout again:

>>> print system(buildout+' -D')

Uninstalling backup. Installing backup. Generated script '...backup'. Generated script '...restore'. <BLANKLINE>

Let's see what we got in the backup script:

>>> print open(join(sample_buildout, 'bin', 'backup')).read()

#!... <BLANKLINE> import sys sys.path[0:0] = [ ... ] <BLANKLINE> import iw.recipe.backup.archive <BLANKLINE> if __name__ == '__main__': iw.recipe.backup.archive.archive_buildout(('/sample-buildout', '...', '..._TEST_', '/backup.log', ['/sample-buildout/not', '/sample-buildout/neh/not'], ['/sample-buildout/neh'], '/sample-buildout/Data.fs', {'name': 'backup'})) <BLANKLINE>

Let's backup:

>>> print system(join(sample_buildout, 'bin', 'backup'))

Starting the backup... Archived in /backup-XXXX-XX-XX-XX-XX.tar.gz <BLANKLINE>

Let's remove the folder and the Data.fs:

>>> import shutil
>>> shutil.rmtree(join(sample_buildout, 'not'))
>>> shutil.rmtree(join(sample_buildout, 'neh'))
>>> os.remove(join(sample_buildout, 'Data.fs'))

Let's restore:

>>> arc = glob.glob('%s/*.tar.gz' % root)[0]
>>> rest = os.popen(join(sample_buildout, 'bin', 'restore %s' % arc))

And make sure the not folder and 'neh/not' are not back and neh is back ! >>> os.path.exists(join(sample_buildout, 'not')) False >>> os.path.exists(join(sample_buildout, 'neh/not')) False >>> os.path.exists(join(sample_buildout, 'neh')) True