Welcome, guest | Sign In | My Account | Store | Cart

Notice! PyPM is being replaced with the ActiveState Platform, which enhances PyPM’s build and deploy capabilities. Create your free Platform account to download ActivePython or customize Python with the packages you require and get automatic updates.

Download
ActivePython
INSTALL>
pypm install products.csvreplicata

How to install Products.csvreplicata

  1. Download and install ActivePython
  2. Open Command Prompt
  3. Type pypm install products.csvreplicata
 Python 2.7Python 3.2Python 3.3
Windows (32-bit)
1.1.13 Available View build log
1.1.7 Available View build log
Windows (64-bit)
1.1.13 Available View build log
1.1.7 Available View build log
Mac OS X (10.5+)
1.1.13 Available View build log
1.1.7 Available View build log
Linux (32-bit)
1.1.13 Available View build log
1.1.7 Available View build log
Linux (64-bit)
1.1.13 Available View build log
1.1.7 Available View build log
 
Author
License
GPL
Lastest release
version 1.1.13 on Jun 5th, 2013

Introduction

Allows to export and import AT objects contents and hierarchy.

It supports also other contents or objects via plugins (ZCA). This part mainly targeted for a developper audiance, so see interfaces.py / adapters.py.

We have plans to integrate the export/import code with transmogrifier, [ also ] but we have no ETA For now.

Repository: git

A Makina Corpus add-on

makinacom

Products.csvreplicata Installation

To install Products.csvreplicata into the global Python environment (or a workingenv), using a traditional Zope 2 instance, you can do this:

  • When you're reading this you have probably already run easy_install Products.csvreplicata. Find out how to install setuptools (and EasyInstall) here: http://peak.telecommunity.com/DevCenter/EasyInstall

  • If you are using Zope 2.9 (not 2.10), get pythonproducts and install it via:

    python setup.py install --home /path/to/instance
    

    into your Zope instance.

  • Create a file called Products.csvreplicata-configure.zcml in the /path/to/instance/etc/package-includes directory. The file should only contain this:

    <include package="Products.csvreplicata" />
    

Alternatively, if you are using zc.buildout and the plone.recipe.zope2instance recipe to manage your project, you can do this:

  • Add Products.csvreplicata to the list of eggs to install, e.g.
  [buildout]
  ...
  eggs =
      ...
      Products.csvreplicata

* Tell the plone.recipe.zope2instance recipe to install a ZCML slug
  [instance]
  recipe = plone.recipe.zope2instance
  ...
  zcml =
      Products.csvreplicata

* Re-run buildout, e.g. with:

  $ ./bin/buildout

You can skip the ZCML slug if you are going to explicitly include the package from another package's configure.zcml file.

Because its top level Python namespace package is called Products, this package can also be installed in Zope 2 as an old style Zope 2 Product.

For that, move (or symlink) the csvreplicata folder of this project (Products.csvreplicata/Products/csvreplicata) into the Products directory of the Zope instance it has to be installed for, and restart the server.

You can also skip the ZCML slug if you install this package the Zope 2 Product way.

Detailled documentation

Interfaces

import interfaces and classes

>>> from zope.interface.verify import verifyClass
>>> from zope.interface import implements
>>> from Products.csvreplicata.handlers.base import CSVdefault
>>> from Products.csvreplicata.handlers.file import CSVFile
>>> from Products.csvreplicata.interfaces import ICSVDefault, ICSVFile

Verify implementation

>>> verifyClass(ICSVDefault, CSVdefault)
True
>>> verifyClass(ICSVFile, CSVFile)
True
Export / Import in plain format
Export

here we export folders and documents

>>> self.setRoles(['Manager'])
>>> id=self.folder.invokeFactory('Document'      , id='doc1'   , title="Document 1")
>>> id=self.folder.invokeFactory('Document'      , id='doc2'   , title="Document 2")
>>> id=self.folder.invokeFactory('News Item'     , id='news1'  , title="news 'super' 3")
>>> id=self.folder.invokeFactory('Document'      , id='doc4'   , title="Document 4")
>>> id=self.folder.invokeFactory('Folder'        , id='sub1'   , title="mytest")
>>> id=self.folder.sub1.invokeFactory('Document' , id='doc11'  , title="Document 1 du dossier 1")
>>> id=self.folder.sub1.invokeFactory('News Item', id='news21' , title="news 1 du dossier 1")
>>> id=self.folder.sub1.invokeFactory('Document' , id='doc31'  , title="Document 2 du dossier 1")
>>> self.portal.portal_csvreplicatatool.getPlainFormat()
False

>>> self.portal.portal_csvreplicatatool.getEncoding()
'UTF-8'
>>> self.portal.portal_csvreplicatatool.getDelimiter()
';'
>>> self.portal.portal_csvreplicatatool.getStringdelimiter()
'"'
>>> self.portal.portal_csvreplicatatool.replicabletypes =  {'Document': ['default'], 'Folder': ['default'], 'News': ['Default']}
>>> from Products.csvreplicata.interfaces import Icsvreplicata
>>> repl = Icsvreplicata(self.folder)
>>> print repl.csvexport(exportable_content_types=['Document', 'Folder', 'News Item'], depth=0).getvalue()
"/plone/Members/test_user_1_";"..."
"parent";"id";"type";"title";"description";"text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"Title";"label_summary";"label_body_text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"";"doc1";"Document";"Document 1";"";"";"simple_publication_workflow";"private";"";""
"";"doc2";"Document";"Document 2";"";"";"simple_publication_workflow";"private";"";""
"parent";"id";"type";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"";"news1";"News Item";"simple_publication_workflow";"private"
"parent";"id";"type";"title";"description";"text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"Title";"label_summary";"label_body_text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"";"doc4";"Document";"Document 4";"";"";"simple_publication_workflow";"private"
"parent";"id";"type";"title";"description";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"Title";"label_description";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"";"sub1";"Folder";"mytest";"";"simple_publication_workflow";"private"
"parent";"id";"type";"title";"description";"text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"Title";"label_summary";"label_body_text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"sub1";"doc11";"Document";"Document 1 du dossier 1";"";"";"simple_publication_workflow";"private"
"parent";"id";"type";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"sub1";"news21";"News Item";"simple_publication_workflow";"private"
"parent";"id";"type";"title";"description";"text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"Title";"label_summary";"label_body_text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"sub1";"doc31";"Document";"Document 2 du dossier 1";"";"";"simple_publication_workflow";"private"
<BLANKLINE>

This output is not that good to deal with CSV apis, we will try to export it to a flat structure.

>>> from Products.csvreplicata import replicator
>>> self.portal.portal_csvreplicatatool.setPlainFormat(True)
>>> self.portal.portal_csvreplicatatool.getPlainFormat()
True
>>> repl = Icsvreplicata(self.folder)
>>> print Icsvreplicata(self.folder).csvexport(exportable_content_types=['Document', 'Folder', 'News Item'], depth=0).getvalue() # doctest: +ELLIPSIS
"startpoint";"replicata_export_date";"parent";"id";"type";"title";"description";...
"Start point";"Export Date";"Parent folder";"Identifier";"Content type";"Title";...
"/plone/Members/test_user_1_";"...";"";"doc1";"Document";"Document 1";...
"/plone/Members/test_user_1_";"...";"";"doc2";"Document";"Document 2";...
"/plone/Members/test_user_1_";"...";"";"news1";"News Item";...
"/plone/Members/test_user_1_";"...";"";"doc4";"Document";"Document 4";...
"/plone/Members/test_user_1_";"...";"";"sub1";"Folder";"mytest";...
"/plone/Members/test_user_1_";"...";"sub1";"doc11";"Document";"Document 1 du dossier 1";...
"/plone/Members/test_user_1_";"...";"sub1";"news21";"News Item";...
"/plone/Members/test_user_1_";"...";"sub1";"doc31";"Document";"Document 2 du dossier 1";...
<BLANKLINE>

Redo the export but with a plugin that find the title on the object.

>>> from Products.csvreplicata import adapters
>>> class CustomExporter(adapters.CSVReplicataExportImportPluginAbstract):
...    def __init__(self, *args, **kwargs):
...        adapters.CSVReplicataExportImportPluginAbstract.__init__(self, *args, **kwargs)
...        self.ids.append('title')
...    def fill_values(self, row, row_ids):
...        """."""
...        for id in row_ids:
...            if id.replace(self.prefix, '') in self.ids:
...                index = row_ids.index(id)
...                if index < len(row):
...                    row[index] = self.context.Title()
...    def set_values(self, row, row_ids):
...        """."""
...        print "plugin.setValue called with %s <-> %s" % (row, row_ids)
...
>>> provideAdapter(CustomExporter, (interfaces.Icsvreplicata, zope.interface.Interface), interfaces.ICSVReplicataExportImportPlugin, name ='fooplugin' )
>>> from csv import DictReader
>>> content = Icsvreplicata(self.folder).csvexport(exportable_content_types=['Document', 'Folder', 'News Item'], depth=None).getvalue()
>>> items = [item for item in DictReader(StringIO(content), delimiter=";", quotechar='"')]
>>> keys = items[0].keys();keys.sort()

As we cant predict order of the keys, doing some magic to order them before testing.

>>> pprint([[(key, item[key]) for key in keys if not 'date' in key]for item in items], width=130) # doctest: +REPORT_NDIFF
[[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain',
   'ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain'),
  ('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state',
   'ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state'),
  ('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title',
   'ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title'),
  ('description', 'label_...'),
  ('id', 'Identifier'),
  ('parent', 'Parent folder'),
  ('startpoint', 'Start point'),
  ('text', 'label_body_text'),
  ('title', 'Title'),
  ('type', 'Content type')],
 [('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
  ('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
  ('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'Document 1'),
  ('description', ''),
  ('id', 'doc1'),
  ('parent', ''),
  ('startpoint', '/plone/Members/test_user_1_'),
  ('text', ''),
  ('title', 'Document 1'),
  ('type', 'Document')],
 [('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
  ('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
  ('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'Document 2'),
  ('description', ''),
  ('id', 'doc2'),
  ('parent', ''),
  ('startpoint', '/plone/Members/test_user_1_'),
  ('text', ''),
  ('title', 'Document 2'),
  ('type', 'Document')],
 [('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
  ('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
  ('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', "news 'super' 3"),
  ('description', ''),
  ('id', 'news1'),
  ('parent', ''),
  ('startpoint', '/plone/Members/test_user_1_'),
  ('text', ''),
  ('title', ''),
  ('type', 'News Item')],
 [('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
  ('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
  ('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'Document 4'),
  ('description', ''),
  ('id', 'doc4'),
  ('parent', ''),
  ('startpoint', '/plone/Members/test_user_1_'),
  ('text', ''),
  ('title', 'Document 4'),
  ('type', 'Document')],
 [('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
  ('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
  ('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'mytest'),
  ('description', ''),
  ('id', 'sub1'),
  ('parent', ''),
  ('startpoint', '/plone/Members/test_user_1_'),
  ('text', ''),
  ('title', 'mytest'),
  ('type', 'Folder')],
 [('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
  ('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
  ('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'Document 1 du dossier 1'),
  ('description', ''),
  ('id', 'doc11'),
  ('parent', 'sub1'),
  ('startpoint', '/plone/Members/test_user_1_'),
  ('text', ''),
  ('title', 'Document 1 du dossier 1'),
  ('type', 'Document')],
 [('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
  ('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
  ('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'news 1 du dossier 1'),
  ('description', ''),
  ('id', 'news21'),
  ('parent', 'sub1'),
  ('startpoint', '/plone/Members/test_user_1_'),
  ('text', ''),
  ('title', ''),
  ('type', 'News Item')],
 [('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
  ('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
  ('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'Document 2 du dossier 1'),
  ('description', ''),
  ('id', 'doc31'),
  ('parent', 'sub1'),
  ('startpoint', '/plone/Members/test_user_1_'),
  ('text', ''),
  ('title', 'Document 2 du dossier 1'),
  ('type', 'Document')]]
Import

Then, now that we got a working export, what about importing it

>>> id = self.folder.invokeFactory('Folder'      , id='fa'   , title="tests import")
>>> id = self.folder.invokeFactory('Folder'      , id='fb'   , title="tests import")
>>> fa = self.folder.fa; fb = self.folder.fb

Import in CSVReplicata plain format, an entry per line without contextual type hinting

>>> CSV = StringIO("""\
... "startpoint";"replicata_export_date";"parent";"id";"type";"title";"description";"ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title";"text"
... "Start point";"Export Date";"Parent folder";"Identifier";"Content type";"Title";"label_...";"ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title";"label_body_text"
... "/plone/Members/test_user_1_";20090101010101;"";"doc1";"Document";"Document 1";"";"Document 1";""
... "/plone/Members/test_user_1_";20090101010101;"";"doc2";"Document";"Document 2";"";"Document 2";""
... "/plone/Members/test_user_1_";20090101010101;"";"news1";"News Item";"";"";"news 'super' 3";""
... "/plone/Members/test_user_1_";20090101010101;"";"doc4";"Document";"Document 4";"";"Document 4";""
... "/plone/Members/test_user_1_";20090101010101;"";"sub1";"Folder";"mytest";"";"mytest";""
... "/plone/Members/test_user_1_";20090101010101;"sub1";"doc11";"Document";"Document 1 du dossier 1";"";"Document 1 du dossier 1";""
... "/plone/Members/test_user_1_";20090101010101;"sub1";"news21";"News Item";"";"";"news 1 du dossier 1";""
... "/plone/Members/test_user_1_";20090101010101;"sub1";"doc31";"Document";"Document 2 du dossier 1";"";"Document 2 du dossier 1";""
... """)
>>> print Icsvreplicata(self.folder.fa).csvimport(CSV, datetimeformat='%d%m%Y', delimiter=";", stringdelimiter='"', plain_format=True)
p...
(8, 0, ..., [])
>>> self.folder.fa.objectIds()
['doc1', 'doc2', 'news1', 'doc4', 'sub1']
>>> self.folder.fa.sub1.objectIds()
['doc11', 'news21', 'doc31']

Import in CSVReplicata orignal format, an entry per line with contextual type hinting

>>> CSV = StringIO("""\
... "/plone/Members/test_user_1_";20100101010101
... "parent";"id";"type";"title";"description";"text"
... "Parent folder";"Identifier";"Content type";"Title";"label_...";"label_body_text"
... "";"doc1";"Document";"Document 1";"";""
... "";"doc2";"Document";"Document 2";"";""
... "parent";"id";"type"
... "Parent folder";"Identifier";"Content type"
... "";"news1";"News Item"
... "parent";"id";"type";"title";"description";"text"
... "Parent folder";"Identifier";"Content type";"Title";"label_...";"label_body_text"
... "";"doc4";"Document";"Document 4";"";""
... "parent";"id";"type";"title";"description"
... "Parent folder";"Identifier";"Content type";"Title";"label_..."
... "";"sub1";"Folder";"mytest";""
... "parent";"id";"type";"title";"description";"text"
... "Parent folder";"Identifier";"Content type";"Title";"label_...";"label_body_text"
... "sub1";"doc11";"Document";"Document 1 du dossier 1";"";""
... "parent";"id";"type"
... "Parent folder";"Identifier";"Content type"
... "sub1";"news21";"News Item"
... "parent";"id";"type";"title";"description";"text"
... "Parent folder";"Identifier";"Content type";"Title";"label_...";"label_body_text"
... "sub1";"doc31";"Document";"Document 2 du dossier 1";"";""
... """)
>>> print Icsvreplicata(self.folder.fb).csvimport(CSV, datetimeformat='%d%m%Y', delimiter=";", stringdelimiter='"', plain_format=False)
p...
(8, 0, ..., [])
>>> self.folder.fb.objectIds()
['doc1', 'doc2', 'news1', 'doc4', 'sub1']
>>> self.folder.fb.sub1.objectIds()
['doc11', 'news21', 'doc31']
Default handlers

Verify the default handlers provided for Archetypes fields:

>>> handlersDict = self.portal.portal_csvreplicatatool.getHandlers()
>>> handlers = handlersDict.keys();handlers.sort()
>>> pprint(handlers)
['Products.Archetypes.Field.BooleanField',
 'Products.Archetypes.Field.DateTimeField',
 'Products.Archetypes.Field.FileField',
 'Products.Archetypes.Field.FloatField',
 'Products.Archetypes.Field.ImageField',
 'Products.Archetypes.Field.IntegerField',
 'Products.Archetypes.Field.LinesField',
 'Products.Archetypes.Field.ReferenceField',
 'Products.Archetypes.Field.StringField',
 'Products.Archetypes.Field.TextField',
 'Products.AttachmentField.AttachmentField.AttachmentField',
 'default_handler',
 ...]
>>> pprint([(h, handlersDict[h]['handler_class']) for h in handlers], width=130)
[('Products.Archetypes.Field.BooleanField', <Products.csvreplicata.handlers.base.CSVBoolean object at ...>),
 ('Products.Archetypes.Field.DateTimeField', <Products.csvreplicata.handlers.base.CSVDateTime object at ...>),
 ('Products.Archetypes.Field.FileField', <Products.csvreplicata.handlers.file.CSVFile object at ...>),
 ('Products.Archetypes.Field.FloatField', <Products.csvreplicata.handlers.base.CSVFloat object at ...>),
 ('Products.Archetypes.Field.ImageField', <Products.csvreplicata.handlers.file.CSVFile object at ...>),
 ('Products.Archetypes.Field.IntegerField', <Products.csvreplicata.handlers.base.CSVInteger object at ...>),
 ('Products.Archetypes.Field.LinesField', <Products.csvreplicata.handlers.base.CSVLines object at ...>),
 ('Products.Archetypes.Field.ReferenceField', <Products.csvreplicata.handlers.reference.CSVReference object at ...>),
 ('Products.Archetypes.Field.StringField', <Products.csvreplicata.handlers.base.CSVString object at ...>),
 ('Products.Archetypes.Field.TextField', <Products.csvreplicata.handlers.base.CSVText object at ...>),
 ('Products.AttachmentField.AttachmentField.AttachmentField', <Products.csvreplicata.handlers.file.CSVFile object at ...>),
 ('default_handler', <Products.csvreplicata.handlers.base.CSVdefault object at ...>),
 (...

What happened with csvreplicata during import/export if MyField is not in tool's handlers. replicator.py apllies default_handler on it:

{'default_handler':
 {'handler_class': base.CSVdefault(),'file': False}}
The replicator exporter/downloader
Export in normal mode

here we export folders and documents

>>> from Products.csvreplicata.browser import manager
>>> self.portal.portal_csvreplicatatool.replicabletypes = \
... {'Document':['default'], 'Folder':['default'],
... 'News Item': ['default'], 'File': ['default'] }
>>> import re
>>> self.setRoles(['Manager'])
>>> id=self.folder.invokeFactory('Document'      , id='doc1'   , title="Document 1")
>>> id=self.folder.invokeFactory('Document'      , id='doc2'   , title="Document 2")
>>> params = {"datetimeformat": '%d/%m/%Y %H:%M:%S',
... "vocabularyvalue": "No",
... "encoding": "UTF-8",
... "delimiter": ";",
... "stringdelimiter": '"',
... "exportable_content_types": ["News Items", "Document", "Folder"],
... }
>>> req = make_request('/'.join(self.folder.getPhysicalPath())+'@@csvreplicata', **params)
>>> repl = manager.ReplicationManager(self.folder, req)
>>> from Products.csvreplicata import replicator
>>> print ''.join([a for a in repl.doExport()])
"/plone/Members/test_user_1_";...
"parent";"id";"type";"title";"description";"text";...
"Parent folder";"Identifier";"Content type";"Title";"label_summary";...
"";"doc1";"Document";"Document 1";"";...
"";"doc2";"Document";"Document 2";"";...
<BLANKLINE>

>>> items = list(req.response._headers.iteritems());items.sort();pprint(items)
[('content-disposition', ['attachment; filename=export.csv']),
 ('content-length', ['...']),
 ('content-type', ['text/csv;charset=UTF-8'])]
Export as zip when there are files out there and we want them

Now adding and exporting a file:

>>> params['exportfiles'] = 'Yes'
>>> params["exportable_content_types"].append('File')
>>> id=self.folder.invokeFactory('File', id='f1'  , title="File 1")
>>> req = make_request('/'.join(self.folder.getPhysicalPath())+'@@csvreplicata', **params)
>>> f1 = self.folder.f1
>>> f1.getFile().data = 'foo'
>>> f1.setFilename('bar')
>>> repl = manager.ReplicationManager(self.folder, req)
>>> from StringIO import StringIO
>>> import zipfile
>>> content = StringIO([a for a in repl.doExport()][0])
>>> zip = zipfile.ZipFile(content)
>>> print zip.read('export.csv')
"/plone/Members/test_user_1_";...
"parent";"id";"type";"title";"description";"...
"Parent folder";"Identifier";"Content type";"Title";"label_summary";...
"";"doc1";"Document";"Document 1";...
"";"doc2";"Document";"Document 2";...
"parent";"id";"type";"title";"description";...
"Parent folder";"Identifier";"Content ...
"";"f1";"File";"File 1";"";...
<BLANKLINE>


>>> zip.printdir()
File Name...
bar...
export.csv...
>>> items = list(req.response._headers.iteritems());items.sort();pprint(items)
[('content-disposition', ['attachment; filename=export.zip']),
 ('content-length', ['...']),
 ('content-type', ['application/zip'])]
The File Stream Iterator

This object returns a generator to read our big files!:

>>> from Products.csvreplicata.browser.manager import FileStreamIterator, EphemeralStreamIterator
>>> import tempfile
>>> fpath = tempfile.mkstemp('foo')[1]
>>> fobj = open(fpath, 'w');fobj.write('foo');fobj.close()
>>> len(FileStreamIterator(fpath))
3

We can play with chunks to divide rendering into small parts

>>> [[a for a in FileStreamIterator(fpath, chunk=chunk)] for chunk in [1,2, 3,4]]
[['f', 'o', 'o'], ['fo', 'o'], ['foo'], ['foo']]

>>> os.unlink(fpath)
The Ephemeral Stream Iterator

This object returns a generator to read our big files but delete them when they are read!:

>>> from Products.csvreplicata.browser.manager import FileStreamIterator, EphemeralStreamIterator
>>> import tempfile
>>> fdir = tempfile.mkdtemp(); fdir2 = tempfile.mkdtemp(dir=fdir);fpath = os.path.join(fdir2, 'foo')
>>> fobj = open(fpath, 'w');fobj.write('foo');fobj.close()

Files are there, we can ask to not delete parents (default):

>>> [os.path.exists(p) for p in fdir,fdir2, fpath]
[True, True, True]
>>> len(EphemeralStreamIterator(fpath, delete_parent=False, delete_grand_parent=False))
3
>>> [a for a in EphemeralStreamIterator(fpath, delete_parent=False, delete_grand_parent=False)]
['foo']
>>> [os.path.exists(p) for p in fdir,fdir2, fpath]
[True, True, False]

We have read it, the file and the parent are deleted:

>>> fobj = open(fpath, 'w');fobj.write('foo');fobj.close()
>>> [a for a in EphemeralStreamIterator(fpath, delete_parent=True, delete_grand_parent=True)]
['foo']
>>> [os.path.exists(p) for p in fdir2, fdir, fpath]
[False, False, False]

Changelog

1.1.13 (2013-06-04)
  • fix release
1.1.12 (2013-04-12)
  • Support for Products.ATExtensions formattable names field [kiorky]
  • Again, better error traces [kiorky]

>>>>>>> af6cc49cb43f53f13b1630d7860c198288392dc8 1.1.11 (2012-12-21) -------------------

  • fix rst markup
1.1.10 (2012-12-21)
  • Better non exportable types [kiorky]
  • better references restore [kiorky]

System Message: WARNING/2 (<string>, line 602)

Bullet list ends without a blank line; unexpected unindent.

<<<<<<< HEAD - fixed error (invalid literal for int) [potar] - fixed compatibility with SmartFolder [potar] ======= >>>>>>> af6cc49cb43f53f13b1630d7860c198288392dc8

1.1.8 (2012-05-28)
  • fix blob files export/import [kiorky]
  • handle years < 999 [kiorky]
  • let commits be more incremental [kiorky]
  • fix export order [kiorky]
  • improve retry policy [kiorky]
1.1.7 (2010-08-20)
  • Adding comments exporter/searcher [kiorky]
1.1.6 (2010-07-30)
  • Fix importObject (obj referenced before assignment) [danjacka]
1.1.5 (2010-07-29)
  • Fix unittest for plone4 msgid changes [kiorky]
  • Implement import plugins [kiorky]
  • Add a meta.zcml with some basic plugins, like for wf state export/import [kiorky]
  • Rename objects for consistency purpose [kiorky]
  • Fix some Acquisition bugs [kiorky]
  • Support blobstorage [kiorky]
1.1.4 (2010-03-05)
  • fix some encoding issues with file handler [kiorky]
  • better plain format import [kiorky]
  • Fix setSubject [kiorky]
1.1.3 (2010-01-27)
  • Fixed missing import of interface.implements in browser.manager [fRiSi]
1.1.2 (2010-01-27)
  • Fix silly empty list bug in replicator.csvimport prototype. [kiorky]
  • fixed missing import of interface.implements in browser.manager [fRiSi]
  • Fix unicode error with reference fields [kiorky]
  • Add support for temporary export path [kiorky]
  • Add support for flattened CSV Files [kiorky]
  • Add tests, and tests infrastructure [kiorky]
  • Add Stream (Files) Iterators not to overhead the RAM [kiorky]
1.1.1 - Unreleased
  • remove five:traversable directive (deprecated in plone3) [toutpt]
  • Add default config for importcsvStep [toutpt]
  • importcsvStep now depend on plone-final step [toutpt]
  • option ignore_content_errors added to import step config (set it to true) and to the replicator methods. It allows to log errors when setting fields instead of raising exceptions and stop. [kiorky]
1.1 (2009-10-17)
  • A new import step to use csvreplicata to import contents [toutpt]
  • Fix creationflag issue [toutpt]
  • plugins system. Export other things than AT. take a look at interfaces and adpaters [kiorky]
  • some encoding bugs fixed [kiorky]
  • datetime format settings in the tool [kiorky]
  • images/files created with folder structure in export [kiorky]
  • Plone 2.5 compatibility back [kiorky]
1.0.7 (2009-07-15)
  • Prevent fixTools from running outside of the context of Products.csvreplicata [pigeonflight]
  • Fix export when data value is not ascii [yboussard]
1.0.6
  • support vocabulary defined as object method [Jim BAACK]
1.0.5
  • fix bug crc32 with large media files
  • when you set a CSVHandledTypesSchematas default is automatically preselected for user convenience [Jean-Philippe CAMGUILHEM]
1.0.4
  • Uninstall problem fix

    IMPORTANT NOTE: when migrating from previous versions, you need to uninstall csvreplicata and then reinstall csvreplicata. [Jean-Philippe CAMGUILHEM]

1.0.3
  • Custom handlers can be now implemented outside the product, and dynamically declared to the csvreplicata tool. [Jean-Philippe CAMGUILHEM]
1.0.2
  • Initial release.[Eric BREHAULT / Christophe SAUTHIER]
1.0
  • Unreleased

Subscribe to package updates

Last updated Jun 5th, 2013

Download Stats

Last month:5

What does the lock icon mean?

Builds marked with a lock icon are only available via PyPM to users with a current ActivePython Business Edition subscription.

Need custom builds or support?

ActivePython Enterprise Edition guarantees priority access to technical support, indemnification, expert consulting and quality-assured language builds.

Plan on re-distributing ActivePython?

Get re-distribution rights and eliminate legal risks with ActivePython OEM Edition.