iw.fss 2.8.0rc5

Archetypes storage for storing fields raw values on the file system.

4.1   From FSS 2.5.x and older

The way we configure the storage and backup paths as well as the storage strategies HAVE CHANGED IN DEPTH. There's no automatic migration from this version. Please create a configuration file that clones your old configlet preferences as indicated in FSS main configuration.

4.2   From iw.fss 2.7.1

From this version and later version, iw.fss registers upgrade steps in GenericSetup. When upgrading to a newer version of iw.fss, just have a look at the "Upgrade" tab of the portal_setup tool in your Plone site.

Warning, due to the removal of the portal_fss tool upgrading from 2.7.1 does not retain the RDF options (activated and specific script). If you used RDF with 2.7.1, you need to redo the same RDF setting in the FileSystemStorage config panel.

5.1   With buildout

This example speaks of itself:

[buildout]
...
eggs =
  ...
  iw.fss
  ...

[instance]
...
zcml =
  ...
  iw.fss
  iw.fss-meta
  ...

You may also use the iw.recipe.fss recipe egg to configure fully an instance.

5.2   From SVN reposo

• Inflate iw.fss package into your zope instance lib/python directory. I'm sure it's already done.

• Add $INSTANCE_HOME/etc/package-includes/iw.fss-meta.zcml file with this line:

  <include package="iw.fss" file="meta.zcml"/>
  

• Add $INSTANCE_HOME/etc/package-includes/iw.fss-configure.zcml file with this line:

  <include package="iw.fss" />
  

5.3   Configuring your Plone sites

• Read carefully Storage strategies and FSS main configuration below and configure your instance in plone-filesystemstorage.conf accordingly. You may use the iw.recipe.fss recipe in your buildout.cfg to do this.
• Start your Zope instance
• In ZMI, go into your plone site and install the product using portal_quickinstaller
• Optionally, tweak some - self explained - preferences in the FileSystemStorage configuration panel of your Plone site.

6.1   Strategies overview

FSS comes with 4 storage strategies. Each strategy requires two directories:

• The storage directory stores the values of relevant fields in files according to the selected strategy.
• The backup directory that keeps the values of the fields of deleted contents for "Undo" purpose.
• Only one storage strategy can be selected on a given Plone site, but different Plone site in the same Zope instance may have distinct storage strategies.
• You need the best performances: choose the Flat storage strategy or Directory storage strategy.
• You need a storage that looks like your Plone site (in example to publish the storage directory in a read only samba or NFS share): choose the Site storage strategy 1 or the Site storage strategy 2.
• If you care about CMFEdition support (means that one of the content types using FSS is versioned with CMFEditions), you should not use the Site storage strategy 1 or the Site storage strategy 2.

See FSS main configuration to configure the best suited strategy on your Zope instance and Plone sites.

6.2   Flat storage strategy

All field values are stored in a flat structure. This strategy is the default one.

• Filename of these stored values: <uid of content>_<field name>
• Filename of rdf file: <uid of content>_<field name>.rdf
• Filename of backup values: <uid of content>_<field name>.bak

Rdf files are not backed up. They are automatically generated.

Example of storage:

fssDirectory
|
|- f42ad00adb7d4580f929d734bf1ed3ab_image
|
|- f42ad00adb7d4580f929d734bf1ed3ab_image.rdf
|
|- f42ad00adb7d4580f929d734bf1ed3ab_file
|
|- f42ad00adb7d4580f929d734bf1ed3ab_file.rdf

fssBackupDirectory
|
|- 9efeb7638fb35f5f9b9955da3e2dbfec_file.bak

6.3   Directory storage strategy

All field values are stored in a directory structure. Sub directories are defined on two level. First level of directory uses the 2 characters of content uid. Second level of directory uses the 4 characters of content uid. Backup files are stored in a flat structure.

• Filename of these stored values: <uid of content>_<field name>
• Filename of rdf file: <uid of content>_<field name>.rdf
• Filename of backup values: <uid of content>_<field name>.bak

RDF files are not backed up. They are automatically generated

Example of storage:

fssDirectory
|
|- f42
   |
   |- f42ad
      |
      |- f42ad00adb7d4580f929d734bf1ed3ab_image
      |
      |- f42ad00adb7d4580f929d734bf1ed3ab_image.rdf
      |
      |- f42ad00adb7d4580f929d734bf1ed3ab_file
      |
      |- f42ad00adb7d4580f929d734bf1ed3ab_file.rdf

fssBackupDirectory
|
|- 9efeb7638fb35f5f9b9955da3e2dbfec_file.bak

Depending in one hand on your file system performances facing a huge amount of files in the same directory, and in the other hand on the number of contents relying on FSS, you might choose this strategy or the Flat storage strategy.

6.4   Site storage strategy 1

All field values are stored in a directory structure mirroring structure of PloneSite. Backup files are stored in a flat structure.

• Filename of these stored values: Filename of field value or field name if not defined
• Filename of rdf file: <field name>.rdf
• Filename of backup values: <uid of content>_<field name>.bak

Rdf files are not backed up. They are automatically generated

Example of storage:

fssDirectory
|
|- members
   |
   |- john
      |
      |- dummy-document
         |
         |- image
         |  |
         |  |- moutain.jpg
         |  |
         |  |- image.rdf
         |
         |- file
            |
            |- diary.odt
            |
            |- file.rdf

fssBackupDirectory
|
|- 9efeb7638fb35f5f9b9955da3e2dbfec_file.bak

6.5   Site storage strategy 2

All field values are stored in a directory structure mirroring structure of PloneSite. Backup files are stored in a flat structure.

• Filename of these stored values: Filename of field value or field name if not defined
• Filename of rdf file: <field filename>.rdf
• Filename of backup values: <uid of content>_<field name>.bak

Rdf files are not backed up. They are automatically generated

Example of storage:

fssDirectory
|
|- members
   |
   |- john
      |
      |- dummy-document
         |
         |- fss.cfg
         |
         |- moutain.jpg
         |
         |- mountain.jpg.rdf
         |
         |- diary.odt
         |
         |- diary.odt.rdf

fssBackupDirectory
|
|- 9efeb7638fb35f5f9b9955da3e2dbfec_file.bak

8.1   FSS and RDF

As mentioned above in Storage strategies, FSS can optionally store RDF files in the back-end storage. These RDF files conform the DCMES-XML standard on XML expression of DublinCore elements.

If you select the Flat storage strategy, the RDF files may be used to build a files tree as close as possible of the Plone tree structure from the storage back-end, with the build_fs_tree.py utility:

$ cd /path/to/iw/fss/bin
$ python build_fs_tree.py --help

8.2   FSS maintenance

The "Maintenance" tab of FSS configuration panel shows some statistics about the amount of files managed by FSS.

In addition 2 additional buttons are provided:

• Update FSS files : Cleans up the back-end storage directories. Files not referenced from a content object are removed.
• Update RDF files : Builds all RDF files, in the event you chose to select the generetion of RDF files after having inserted relevant contents.

11.1   Using FSS in your content types

We assume that creating AT based content types is a familiar activity.

When creating the schema of a content type, you usually don't mention the storage of your fields since AT provides a default ZODB storage. You can choose another storage, either another one that ships in the Archetypes bundle, or a third party storage like FSS.

Here is a small example that doesn't need much comments:

# Usual Zope/CMF/Plone/Archetypes imports
...
from iw.fss.FileSystemStorage import FileSystemStorage
...
my_schema = Schema((
    FileField('file',
              ...
              storage=FileSystemStorage(),
              widget=FileWidget(...)
              ),
    ...
    ))
 ...

You may have a look at examples/FSSItem.py to see a demo content type using FSS. If you want to play with this content type:

$ $INSTANCE_HOME/bin/zopectl stop
$ export FSS_INSTALL_EXAMPLE_TYPES=1
$ $INSTANCE_HOME/bin/zopectl start

11.2   Monkey patching a third party content type

...
from iw.fss import zcml as fss_utils
from Products.ATContentTypes import atct
...
fss_utils.patchATType(atct.ATFile, ['file'])
...

<configure
  ...
  xmlns:zcml="http://namespaces.zope.org/zcml"
  xmlns:fss="http://namespaces.ingeniweb.com/filesystemstorage"
  ...>
  ...
  <fss:typeWithFSS
    zcml:condition="installed iw.fss"
    class="Products.ATContentTypes.atct.ATFile"
    fields="file" />
  ...
</configure>

11.3   Storing Plone standard content types with FSS

Include this ZCML element either in your instance package-includes directory or in any personal configure.zcml in your instance packages:

<include package="iw.fss" file="atct.zcml" />

This ZCML file wires FSS with the relevant content types that ship with Plone bundle:

• File
• Image
• News Item

11.4   Customize RDF info set

You can add data to the RDF info set via a FSS dedicated - and optional - hook.

Open the FSS configuration panel and give the name of your personal RDF data script, let's call it fss_custom_rdf for this example. This script will add or change the RDF default data set.

<fss:rdf_date>2007-05-13 13:21:00</fss:rdf_date>

## Script (Python) "fss_custom_rdf"
##bind container=container
##bind context=context
##bind namespace=
##bind script=script
##bind subpath=traverse_subpath
##parameters=name, instance, properties, default_rdf
##title=
##
"""We add the date of RDF file generation"""

rdf_args = default_rdf.copy()

# Add the date of RDF generation
new_props = []
new_props.extend(rdf_args['field_props'])
new_props.append({'id': 'fss:rdf_date', 'value': DateTime().ISO()})
rdf_args['field_props'] = new_props

return rdf_args

## Script (Python) "fss_custom_rdf"
##bind container=container
##bind context=context
##bind namespace=
##bind script=script
##bind subpath=traverse_subpath
##parameters=name, instance, properties, default_rdf
##title=
##
from Products.CMFCore.utils import getToolByName

rdf_args = {}

# Set RDF charset
ptool = getToolByName(instance, 'portal_properties')
rdf_args['charset'] = ptool.site_properties.default_charset

# Set RDF namespaces
rdf_args['namespaces'] = (
    {'id': 'xmlns:rdf', 'value': 'http://www.w3.org/1999/02/22-rdf-syntax-ns#'},
    {'id': 'xmlns:dc', 'value': 'http://purl.org/dc/elements/1.1/'},
    {'id': 'xmlns:fss', 'value': 'http://namespace.ingeniweb.com/fss'},)

# Set field url
utool = getToolByName(instance, 'portal_url')
portal_path = utool.getPortalObject().getPhysicalPath()
portal_path_len = len(portal_path)
rel_path = '/'.join(instance.getPhysicalPath()[portal_path_len:])
fss_path = '%s/%s' % (rel_path, name)
rdf_args['field_url'] = fss_path

# Set RDF properties
props = (
   {'id': 'dc:title', 'value': instance.title_or_id()},
   {'id': 'dc:description', 'value': instance.Description()},
   {'id': 'dc:language', 'value': instance.Language()},
   {'id': 'dc:creator', 'value': instance.Creator()},
   {'id': 'dc:date', 'value': instance.modified()},
   {'id': 'dc:format', 'value': properties.get('mimetype', 'text/plain')},
   )
rdf_args['field_props'] = props

# Returns the RDF structure
return rdf_args

12.1   Backup your production site

Yes, at this early stage, we didn't test migration with lots and lots of content types. Only the basic OTB content types have been tested. But this should work with any other. Whatever, you should always backup a site before proceeding to a deep migration like this one.

12.2   Add the iw.fss settings

Install iw.fss and the ZCML settings from your develoment site on your production site. Note that at this step, it's safer to deny access to the users (use Apache settings or whatever).

Start your Zope instance in forground mode (recommanded if you want to follow the migration logging).

12.3   Do the migration

Open the FileSystemStorage configlet. Check that the main control panel reports the expected settings (storage paths and strategies, supported content types and fields).

Click the "migration" tab of this control panel, check the options, and proceed to migrations.

12.4   Pack the ZODB

After a successful migration, you may pack immediately the ZODB to appreciate the effect of using iw.fss.

13.1   Migrating between strategies

Use the bin/strategymigrator.py shell utility that ships with FSS. Get more info on this utility using the --help option.

Note that only flat -> directory and directory -> flat are available today. Sponsorship is welcome - as indicated in Support and feedback - if you need other strategy migrations.

13.2   Where are the files for this item?

Sometimes we need to know what files store the fields of a content item. For this, login as site manager and browse to http://my.site.tld/where/is/my/content/@@fss_where_are_my_files

20.1   2.8.0rc5

• Fixed a critical issue where an image would be attempted restored at every request. [malthe]
• Fixed an issue where the AT tool for scaling images would pass a unicode file path which would result in a bug because the code was only looking for a regular string (else deciding that it's a stream object). [malthe]
• Fixed an issue with HTTP range leading to a missing file size. With some clients (seen in Firefox 3.6), the end value is trivial and must be explicitly set to the file size before returning the iterator. [malthe]

20.2   2.8.0rc4

Plone 4 support

• "from Globals import InitializeClass" <- "from App.class_init ..." closes deprecation warning.
• AT moved from Zope2 style to Zope3 style interfaces since the old Zope 2 "Interfaces" package has gone away with Zope 2.12. We have a workaround that supports both styles. [glenfant]
• (Deprecated) tabindex removed from all templates. [glenfant]
• Added a view for content items that allows managers to see where in the file system the fields are stored. [glenfant]

20.3   2.8.0rc3

• Change the behaviour of upload on fss. No more memory consuption for that now. We use always use fileiterator when inspecting big file. To achieve to that, VirtualDate override _read_data of OFS.Image.File to construct an clean FSSPdata. [yboussard]
• Monkey patching with collective.monkeypatcher. All monkey patches should be made that way. [glenfant]
• Using z3c.autoniclude when available. [glenfant]
• Change behaviour of VirtualData class. Now VirtualData work as OFS.image ie to acces data you MUST pass by a chained list (FSSPdata) that contain only a block of data. This block of data is getting by a filestream iterator. So no more memmory consuption (max 65536 , a block of data). This is an important change because when Products.Archetype.Field.FileField get metatada of a File old version of fss get all file data for this (ie myField.getFilename() -> you have your all data in memory !!). Also when you invoke Field.download(instance, no_output = True) you get now a filestream iterator (There is a patch to this method because of dowload must set content-length header for the publisher) [yboussard]
• fix content_type retreving for migration process. [yboussard]

20.4   2.8.0rc2

• Fixed crappy error logging on migration [glenfant]

20.5   2.8.0rc1

• Worked around (and cancel migration) when AT tries (and fails) stupidly to guess the charset of a binary data. [glenfant]
• Spanish translation is up to date [dmunicio]

20.6   2.8.0b2

• Fixed bad mime type setting when migrating. [glenfant]
• Fixed loss of text when migrating News items with image [glenfant]
• Fixed logging on transaction when migrating and show the total migrated items in the log too. [glenfant]

20.7   2.8.0b1

• Upgraded translations (.pot and french translation) [glenfant]
• Fixed issues reported by test users. [glenfant]
• Faster and safer method to guess the mime type when migrating. [glenfant]

20.8   2.8.0a3

• Fixed a NameError in migrations that could not be raised by unit tests. [glenfant]

20.9   2.8.0a2

• Fixed an issue on optional (non "required") fields: getting an empty field must raise an AttributeError and not return an empty string. [glenfant]
• ATCT settings disabled [glenfant]

20.10   2.8.0a1

• Adding migration utility (XStorage -> FSS) [glenfant]

20.11   2.7.6

• Safety belt against GS irrelevant upgrades. See https://dev.plone.org/plone/ticket/8507 [glenfant]
• fix i18n initialization under win32 [bmathieu]

20.12   2.7.5

• removed fss_get in VirtualBinary urls: this old script has been removed with skins. This caused problems with content such as plone news item calling image.tag(). [b_mathieu]
• run tests without shell variable ZOPE_TESTCASE [b_mathieu]

20.13   2.7.4

• Fix a bug on versioning : previous version of FSS was unable to deal with content having archetypes fields with no storages [clebeaupin]

20.14   2.7.3

• GenericSetup upgrade step (any version) -> 1.0 [glenfant]
• Red warnings when using the default common config in config panels [glenfant]
• Config panel enhancements (config details view, documentation) [glenfant]
• Code cleanup (useless stuffs) [glenfant]
• Fixed test cases [cbosse] [glenfant]
• CMFEditions compatibility [glenfant]
• Annotations replacement for the attributes (FSSInfo), and some cosmetics. [mehdi]
• CMF skins layer removed in favour of views. [wid]
• Add a test verifying that the storage path contains one item after adding a file, and zero items after deleting it. [maikroeder]
• "site1" and "site2" strategies are forfidden as global strategies [glenfant]
• 100% GenericSetup installation [clebeaupin]

20.15   2.7.2

• Fix the bobo traverse mess: we should not honour Plone 2.5(-) compatibility (remove patches.py) and fix the traverse deprecation warning. [glenfant]
• pyflakes everything. [glenfant]
• Removed Plone 2.5 (and previous) compatibility. [glenfant]
• Protect with appropriate permission ZMI views/methods. [glenfant]

20.16   2.7.1 - 2008/03/17

• Test case for FSS registration. [glenfant]
• Hint in README.txt to set FSS storage only if iw.fss is installed. [glenfant]
• The ZMI "Overview" tab shows the various types changed to use FSS. [glenfant]
• I18N domain is now "iw.fss" being consistent with new styles components. [glenfant]
• Translations moved to zope 3 style in place of PTS style. As a consequence, the config panel title is not translated any more ("plone" domain). [glenfant]
• Public function to patch a type and registering patched types for migration purposes [glenfant]
• Renamed flat2directory.py into strategymigrator.py since future other strategy migrations will go there. [glenfant]
• Added bin/flat2directory.py that migrates "flat" strategy storage to "directory" strategy storage. [glenfant]
• Added discriminator in ZCML support that warns on duplicate config of the same content type. [glenfant]

20.17   2.7.0 - 2008/02/26

• Convert FileSystemStorage Zope 2 product to iw.fss python package [tarek]
• Added ZCML namespace/directive to wire fields of content types with FSS. [glenfant]

20.18   2.6.2 - 2007/12/07

• Fixed range header processing: if one range - specific file stream iterator if multiple range - OFS.image traitment (1680760 sourceforge tracker) Add some unit test to test range header

20.19   2.6.1 - 2007/10/29

• ZConfig bug that lowercases some config strings: we test path in lowercase too (ED)
• Remove the OSError when moving file into backup
• Plone 3.0 compatibility [glenfant]
• Do not override Content-Disposition HTTP header if already set in VirtualBinary.index_html (zegor)
• filestream_iterator is not "range proof". Add a new range_filestream_iterator to negociate Range request

20.20   2.6.0 RC1 - 2007/06/15

• Changed in depth the way we configure FSS with the ZConfig way. Changing the storage and backup path, and the storage strategy TTW is disabled [glenfant]
• As a consequence, changing the storage strategy doesn't require to hack config.py any more [glenfant]
• As a consequence, each Plone site in a Zope instance can have its own storage strategy [glenfant]
• Configuration view and README.txt view in ZMI [glenfant]
• Fixed documentation issues [glenfant]
• SiteStorageStrategy and SiteStorageStrategy2 didn't work under Win32 [glenfant]
• Make FSS compatible with Plone 2.0 : just rewrite some testcases

20.21   2.5.6 - 2007/06/15

• When renaming a folder containing FSS content, FSSInfo attribute was stolen one each content of the folder. Provides a solution to avoid this problem (CL)

20.22   2.5.5 - 2007/06/13

• Make FSS works when calling _setUID method on an AT object (CL)
• Add a strategy mark in the storage in a '.strategy' file such we could forbid strategy changes of storages at Zope startup. (i.e., using a 'flat' storage strategy in a storage filled using a 'directory' strategy)

20.23   2.5.4 - 2007/03/13

• Fix a bug on Plone 2.5 when moving a folder containing items using FSS storage. Those items were losing their FSS content. (CL)

20.24   2.5.3 - 2007/02/12

• Don't monkey patch BaseObject __bobo_traverse__ under Plone 2.5, but register ITraversable adapter for this. This change is required as of Archetypes 1.4.1 (see AT HISTORY.txt, search '__bobo_traverse__')

20.25   2.5.2 - 2006/10/27

• Fix bug on maintenance tab: update files and backup, and update RDF (ED)
• Fix some renaming bugs (ED)
• Extend index_html method of VirtualFile and VirtualImage to use a filestream iterator. FSS is now compliant with CacheFu
• Fix bug in FSSTool:updateFSS - replace restoreBackup method by restoreValueFile
• Add new strategy: SiteStorageStrategy2

20.26   2.5.1 - 2006/07/28

• Check existence of backup and storage paths in configlet
• Fix bug on maintenance tab

20.27   2.5 - 2006/07/19

• Update tests to use all strategies
• Add storage strategies : FlatStorageStrategy, DirectoryStorageStrategy, SiteStorageStrategy
• Patch BaseObject to make it possible to access images like this mysite/myobj/image or mysite/myobj/image_mini
• Added a docstring on VirtualData class and subclasses. This should fix http://plone.org/products/atphoto/issues/28 : VirtualImage can be published directly by ATImage

20.28   2.5RC1

• Add a hook to customize RDF files
• Add script to build filesystem tree from FSS files
• Add RDF implementation

20.29   2.1RC2

• Remove use of ComputedAttribute to avoid strange errors
• If file is empty don't create an empty file on filesystem

20.30   2.1RC1

• Add Content-Disposition header in fss_get python script
• Make FSS compliant with AT 1.3.4 and Plone 2.1

20.31   2.0RC3

• Fix bug on delete action (In previous version, it was impossible to delete non required field value)

20.32   2.0RC2

• Add translations
• Fix bug in configlet when no files are stored on filesystem.

20.33   2.0RC1

• First public release