Welcome, guest | Sign In | My Account | Store | Cart

Notice! PyPM is being replaced with the ActiveState Platform, which enhances PyPM’s build and deploy capabilities. Create your free Platform account to download ActivePython or customize Python with the packages you require and get automatic updates.

Download
ActivePython

zope.kgs 1.2.0
Known-Good-Set (KGS) Support

INSTALL>
pypm install zope.kgs

How to install zope.kgs

  1. Download and install ActivePython
  2. Open Command Prompt
  3. Type pypm install zope.kgs
 Python 2.7Python 3.2Python 3.3
Windows (32-bit)
1.2.0 Available View build log
Windows (64-bit)
1.2.0 Available View build log
Mac OS X (10.5+)
1.2.0 Available View build log
Linux (32-bit)
1.2.0 Available View build log
Linux (64-bit)
1.2.0 Available View build log
 
Links
Author
License
ZPL 2.1
Imports
Lastest release
version 1.2.0 on Jan 5th, 2011

Zope 3 Controlled Package Index

This package has been developed to support the maintenance of a stable set of Zope project distributions. It manages the controlled packages configuration file and supports the generation of buildout configuration files that can be used by developers.

Another use of this package is to use it for testing new distributions against the index. Here is the workflow for testing a new package against stable set:

  1. Install the correct version of this package.

(a) Download the version of this package that manages the stable set that you are interested in. Currently only the trunk exists, which manages the Zope 3.4 release:

$ svn co svn://svn.zope.org/repos/main/zope.release/trunk zope3.4
$ cd zope3.4

  1. Bootstrap the checkout:

    System Message: WARNING/2 (<string>, line 24)

    Literal block expected; none found.

$ python ./bootstrap.py

  1. Run buildout to create the scripts:

    System Message: WARNING/2 (<string>, line 28)

    Literal block expected; none found.

$ ./bin/buildout

(d) Run the buildout.cfg generation script to build a configuration file that can be used for testing:

$ ./bin/generate-buildout

2. From the generated configuration file, you can now build a testing environment.

  1. Enter the test directory and create a buildout:

$ cd test $ python ../bootstrap.py $ ./bin/buildout

  1. Run all the tests to verify that all tests are initially passing:

$ ./bin/test -vpc1

3. Modify the buildout.cfg to look for your the new distribution to be tested:

(a) Comment out the "index" option. This needs to be done, so that the new package is going to be picked up.

(b) Change the version number of the package of interest in the "versions" section.

Alternative:

  1. Check out the new distribution from SVN.

(b) Add a "develop path/to/my/package" line in the "buildout" section of buildout.cfg.

  1. Run the tests, making sure that they all pass.

5. Modify controlled-packages.cfg by adding the new version of the package to the package's version list.

  1. Generate all files again and upload them:

$ cd .. $ ./bin/generate-buildout $ ./bin/generate-versions $ ./bin/upload

Once the files are uploaded, a crontab-job, running every minute, will detect the changes in controlled-pages.cfg and will generate the new controlled package pages.

Note: I think the process is still a tiny bit too long. I probably write a script that makes testing a new version of a package easier, but let's see whether this process is workable first.

Known Good Sets

This package provides a set of scripts and tools to manage Good-Known-Sets, or short KGSs. A KGS is a set of package distributions that are known to work well together. You can verify this, for example, by running all the tests of all the packages at once.

Let me show you how a typical controlled packages configuration file looks like:

>>> import tempfile
>>> cfgFile = tempfile.mktemp('-cp.cfg')
>>> open(cfgFile, 'w').write('''\
... [DEFAULT]
... tested = true
...
... [KGS]
... name = zope-dev
... version = 1.2.0
... date = 2009-01-01
... changelog = CHANGES.txt
... announcement = ANNOUNCEMENT.txt
... files =
...     zope-dev-1.2.0.tgz
...     zope-dev-1.2.0.zip
...     zope-dev-1.2.0.exe
...
... [packageA]
... versions = 1.0.0
...            1.0.1
...
... [packageB]
... versions = 1.2.3
... test-extras = test
...
... [packageC]
... # Do not test this package.
... tested = false
... versions = 4.3.1
... ''')

As you can see, this file uses an INI-style format. The "DEFAULT" section is special, as it will insert the specified options into all other sections as default. The "KGS" section specifies some global information about the KGS, such as the name of the KGS. Since this section references several external files, we should quickly create those.

>>> import os
>>> dir = os.path.dirname(cfgFile)

>>> open(os.path.join(dir, 'CHANGES.txt'), 'w').write('''\
... =======
... Changes
... =======
...
... packageA
... ========
...
... Version 1.0.0
... -------------
...
... * Initial Release
... ''')

>>> open(os.path.join(dir, 'ANNOUNCEMENT.txt'), 'w').write('''\
... =======================
... zope-dev 1.2.0 Released
... =======================
...
... The announcement text!
... ''')

>>> open(os.path.join(dir, 'zope-dev-1.2.0.tgz'), 'w').write('tgz')
>>> open(os.path.join(dir, 'zope-dev-1.2.0.exe'), 'w').write('exe')

All other sections refer to package names. Currently each package section supports two options. The "versions" option lists all versions that are known to work in the KGS. Those versions should always only be bug fixes to the first listed version. The second option, "tested", specifies whether the package should be part of the KGS test suite. By default, we want all packages to be tested, but some packages require very specific test setups that cannot be easily reproduced _[1], so we turn off those tests.

You can also stack controlled package configurations on top of each other. Base configurations can be specified using the extends option:

>>> import tempfile
>>> cfgFile2 = tempfile.mktemp('-cp.cfg')
>>> open(cfgFile2, 'w').write('''\
... [DEFAULT]
... tested = true
...
... [KGS]
... name = grok-dev
... version = 0.1.0
... extends = %s
...
... [packageA]
... versions = 1.0.2
...
... [packageD]
... versions = 2.2.3
...            2.2.4
... ''' %cfgFile)

As you can see, you can completely override another package's version specification as well.

Generating the configuration file and managing it is actually the hard part. Let's now see what we can do with it.

System Message: WARNING/2 (<string>, line 199)

Explicit markup ends without a blank line; unexpected unindent.

files not containing the correct reference to their PY file.

Generate Versions

One of the easiest scripts, is the version generation. This script will generate a "versions" section that is compatible with buildout.

>>> versionsFile = tempfile.mktemp('-versions.cfg')

>>> from zope.kgs import version
>>> version.main((cfgFile, versionsFile))

>>> print open(versionsFile, 'r').read()
[versions]
packageA = 1.0.1
packageB = 1.2.3
packageC = 4.3.1

Let's now ensure that the versions also work for the extended configuration:

>>> versionsFile2 = tempfile.mktemp('-versions.cfg')

>>> version.main((cfgFile2, versionsFile2))

>>> print open(versionsFile2, 'r').read()
[versions]
packageA = 1.0.2
packageB = 1.2.3
packageC = 4.3.1
packageD = 2.2.4
Generate Buildout

In order to be able to test the KGS, you can also generate a full buildout file that will create and install a testrunner over all packages for you:

>>> buildoutFile = tempfile.mktemp('-buildout.cfg')

>>> from zope.kgs import buildout
>>> buildout.main((cfgFile, buildoutFile))

>>> print open(buildoutFile, 'r').read()
[buildout]
parts = test
versions = versions
<BLANKLINE>
[test]
recipe = zc.recipe.testrunner
eggs = packageA
packageB [test]
<BLANKLINE>
[versions]
packageA = 1.0.1
packageB = 1.2.3
packageC = 4.3.1
<BLANKLINE>

Let's make sure that the buildout generation also honors the extensions:

>>> buildoutFile2 = tempfile.mktemp('-buildout.cfg')

>>> buildout.main((cfgFile2, buildoutFile2))

>>> print open(buildoutFile2, 'r').read()
[buildout]
parts = test
versions = versions
<BLANKLINE>
[test]
recipe = zc.recipe.testrunner
eggs = packageA
packageB [test]
packageD
<BLANKLINE>
[versions]
packageA = 1.0.2
packageB = 1.2.3
packageC = 4.3.1
packageD = 2.2.4
<BLANKLINE>
PPIX Support

You can also use the KGS to limit the available packages in a package index generated zc.mirrorcheeseshopslashsimple. This script also uses PyPI to look up distribution file, so wave to use the real configuration file again.

Let's create the pages:

>>> indexDir = tempfile.mkdtemp('-ppix')

>>> from zope.kgs import ppix
>>> ppix.main((cfgFileReal, indexDir))

The index contains one directory per package. So let's have a look:

>>> import os
>>> sorted(os.listdir(indexDir))
['PIL', 'z3c.formdemo', 'zope.component', 'zope.interface']

Each directory contains a single "index.html" file with the download links:

>>> pkgDir = os.path.join(indexDir, 'zope.component')
>>> sorted(os.listdir(pkgDir))
['index.html']

>>> pkgIndex = os.path.join(pkgDir, 'index.html')
>>> print open(pkgIndex, 'r').read()
<html>
<head>
<title>Links for "zope.component"</title>
</head>
<body>
<h1>Links for "zope.component"</h1>
<a href="http://pypi.python.org/packages/2.4/z/zope.component/zope.component-3.4.0-py2.4.egg#md5=c0763e94912e4a8ac1e321a068c916ba">zope.component-3.4.0-py2.4.egg</a><br/>
<a href="http://pypi.python.org/packages/source/z/zope.component/zope.component-3.4.0.tar.gz#md5=94afb57dfe605d7235ff562d1eaa3bed">zope.component-3.4.0.tar.gz</a><br/>
</body>
</html>

PIL is an interesting case, because it does not upload its distribution files yet, at least not for version 1.1.6:

>>> pkgIndex = os.path.join(indexDir, 'PIL', 'index.html')
>>> print open(pkgIndex, 'r').read()
<html><head><title>Links for PIL</title></head><body><h1>Links for PIL</h1><a href='http://www.pythonware.com/products/pil' rel="homepage">1.1.5 home_page</a><br/>
<a href='http://effbot.org/zone/pil-changes-115.htm' rel="download">1.1.5 download_url</a><br/>
<a href='http://www.pythonware.com/products/pil' rel="homepage">1.1.5a2 home_page</a><br/>
<a href='http://effbot.org/zone/pil-changes-115.htm' rel="download">1.1.5a2 download_url</a><br/>
<a href='http://www.pythonware.com/products/pil' rel="homepage">1.1.5a1 home_page</a><br/>
<a href='http://effbot.org/zone/pil-changes-115.htm' rel="download">1.1.5a1 download_url</a><br/>
<a href='http://www.pythonware.com/products/pil/' rel="homepage">1.1.4 home_page</a><br/>
<a href='http://www.pythonware.com/products/pil/' rel="homepage">1.1.3 home_page</a><br/>
<a href='http://www.pythonware.com/downloads/Imaging-1.1.3.tar.gz' rel="download">1.1.3 download_url</a><br/>
<a href='http://www.pythonware.com/products/pil' rel="homepage">1.1.6 home_page</a><br/>
<a href='http://effbot.org/downloads/#Imaging' rel="download">1.1.6 download_url</a><br/>
</body></html>

Optionally, you can also specify the -i option to generate an overview:

>>> ppix.main(('-i', cfgFileReal, indexDir))

>>> sorted(os.listdir(indexDir))
['PIL', 'index.html', 'z3c.formdemo', 'zope.component', 'zope.interface']

Let's now look at the file:

>>> indexPage = os.path.join(indexDir, 'index.html')
>>> print open(indexPage, 'r').read()
<html>
<head>
<title>Simple Index for the "zope-dev" KGS (version 3.4.0b2)</title>
</head>
<body>
<h1>Simple Index for the "zope-dev" KGS (version 3.4.0b2)</h1>
<a href="PIL">PIL</a><br/>
<a href="z3c.formdemo">z3c.formdemo</a><br/>
<a href="zope.component">zope.component</a><br/>
<a href="zope.interface">zope.interface</a><br/>
</body>
</html>

Allowing exisitng package pages to be overwritten and making the main index page an optional feature makes it possible to use this script for two use cases: (1) Merge the constraints into a PPIX index created by zc.mirrorcheeseshopslashsimple, and (2) create a standalone index which only provides the packages of the KGS.

Getting the Latest Versions

When updating the KGS, it is often useful to know for which packages have new releases.

>>> from zope.kgs import latest
>>> latest.main((cfgFileReal,))
z3c.formdemo: 1.1.1, 1.1.2, 1.2.0, 1.3.0, 1.3.0b1, 1.4.0, ...

However, it is often desired only to show new minor versions; in this case, we can pass an option to exclude all versions that have a different major version:

>>> latest.main(('-m', cfgFileReal))
z3c.formdemo: 1.1.1, 1.1.2

Sometimes you're only interested in changes that apply to a single package, and you won't want to wait for the script to query all of the others

>>> latest.main(('-m', cfgFileReal, 'zope.app.server'))

>>> latest.main(('-m', cfgFileReal, 'z3c.formdemo'))
z3c.formdemo: 1.1.1, 1.1.2
Extracting Change Information

When releasing a version of the KGS, it is desirable to produce a list of changes since the last release. Changes are commonly compared to an older version.

>>> cfgFileRealOrig = tempfile.mktemp('-cp.cfg')
>>> open(cfgFileRealOrig, 'w').write('''\
... [DEFAULT]
... tested = true
...
... [KGS]
... name = zope-dev
... version = 3.4.0b1
...
... [PIL]
... versions = 1.1.6
...
... [zope.component]
... versions = 3.4.0
...
... [zope.interface]
... versions = 3.4.0
... ''')

Let's now produce the changes:

>>> from zope.kgs import change
>>> change.main((cfgFileReal, cfgFileRealOrig))
Processing ('PIL', '1.1.6')
Processing ('z3c.formdemo', '1.1.0')
Processing ('zope.component', '3.4.0')
Processing ('zope.interface', '3.4.1')
===
PIL
===
<BLANKLINE>
No changes or information not found.
<BLANKLINE>
============
z3c.formdemo
============
<BLANKLINE>
1.1.0 (unknown)
---------------
<BLANKLINE>
- Feature: New "SQL Message" demo shows how ``z3c.form`` can be used with
non-object data. Specificically, this small application demonstrates using a
Gadfly database using pure SQL calls without any ORM.
<BLANKLINE>
- Feature: New "Address Book" demo that demonstrates more complex use cases,
such as subforms, composite widgets, and mappings/lists
<BLANKLINE>
<BLANKLINE>
==============
zope.component
==============
<BLANKLINE>
3.4.0 (2007-09-29)
------------------
<BLANKLINE>
No further changes since 3.4.0a1.
<BLANKLINE>
<BLANKLINE>
==============
zope.interface
==============
<BLANKLINE>
3.4.1 (unknown)
---------------
<BLANKLINE>
Fixed a setup bug that prevented installation from source on systems
without setuptools.
<BLANKLINE>
3.4.0 (unknown)
---------------
<BLANKLINE>
Final release for 3.4.0.
<BLANKLINE>
<BLANKLINE>

You can also create the changes without an original file, in which case only the versions listed in the current KGS are considered.

>>> change.main((cfgFileReal,))
Processing ('PIL', '1.1.6')
Processing ('z3c.formdemo', '1.1.0')
Processing ('zope.component', '3.4.0')
Processing ('zope.interface', '3.4.1')
===
PIL
===
<BLANKLINE>
No changes or information not found.
<BLANKLINE>
============
z3c.formdemo
============
<BLANKLINE>
1.1.0 (unknown)
---------------
<BLANKLINE>
- Feature: New "SQL Message" demo shows how ``z3c.form`` can be used with
non-object data. Specificically, this small application demonstrates using a
Gadfly database using pure SQL calls without any ORM.
<BLANKLINE>
- Feature: New "Address Book" demo that demonstrates more complex use cases,
such as subforms, composite widgets, and mappings/lists
<BLANKLINE>
<BLANKLINE>
==============
zope.component
==============
<BLANKLINE>
3.4.0 (2007-09-29)
------------------
<BLANKLINE>
No further changes since 3.4.0a1.
<BLANKLINE>
<BLANKLINE>
==============
zope.interface
==============
<BLANKLINE>
3.4.1 (unknown)
---------------
<BLANKLINE>
Fixed a setup bug that prevented installation from source on systems
without setuptools.
<BLANKLINE>
3.4.0 (unknown)
---------------
<BLANKLINE>
Final release for 3.4.0.
<BLANKLINE>
<BLANKLINE>
The Site Generator

The easiest way to publish the KGS is via a directory published by a Web server. Whenever a new controlled-packages.cfg file is uploaded, a script is run that generates all the files. I usually set up a crontab job to do this. The site generator script acts upon a directory, in which it assumes a controlled-packages.cfg file was placed:

>>> siteDir = tempfile.mkdtemp()
>>> cfgFileSite = os.path.join(siteDir, 'controlled-packages.cfg')

>>> import shutil
>>> shutil.copy(cfgFileReal, cfgFileSite)

>>> from zope.kgs import site
>>> site.main(['-s', siteDir])

Let's have a look at the generated files:

>>> from pprint import pprint
>>> pprint(sorted(os.listdir(siteDir)))
['3.4.0b2', 'index.html', 'intro.html', 'resources']

>>> sorted(os.listdir(os.path.join(siteDir, '3.4.0b2')))
['ANNOUNCEMENT.html', 'CHANGES.html',
'buildout.cfg', 'controlled-packages.cfg', 'index', 'index.html',
'links.html', 'minimal', 'versions.cfg']

>>> sorted(os.listdir(os.path.join(siteDir, '3.4.0b2', 'minimal')))
['PIL', 'index.html', 'z3c.formdemo', 'zope.component', 'zope.interface']

If you try to generate the site again without adding the controlled packages config file to the site directory again, it will simply return:

>>> site.main(['-s', siteDir])
Basic Parser API

The kgs.py module provides a simple class that parses the KGS configuration file and provides all data in an object-oriented manner.

>>> from zope.kgs import kgs

The class is simply instnatiated using the path to the config file:

>>> myKGS = kgs.KGS(cfgFile)
>>> myKGS
<KGS 'zope-dev'>

The name, version and date of the KGS is available via:

>>> myKGS.name
'zope-dev'
>>> myKGS.version
'1.2.0'
>>> myKGS.date
datetime.date(2009, 1, 1)

When the changelog and/or announcement files are available, the KGS references the absolute path:

>>> myKGS.changelog
'.../CHANGES.txt'
>>> myKGS.announcement
'.../ANNOUNCEMENT.txt'

The same is true for other release-related files:

>>> myKGS.files
('.../zope-dev-1.2.0.tgz',
'.../zope-dev-1.2.0.exe')

The packages are available under packages:

>>> myKGS.packages
[<Package 'packageA'>, <Package 'packageB'>, <Package 'packageC'>]

Each package is also an object:

>>> pkgA = myKGS.packages[0]
>>> pkgA
<Package 'packageA'>

>>> pkgA.name
'packageA'
>>> pkgA.versions
['1.0.0', '1.0.1']
>>> pkgA.tested
True

As we have seen in the scripts above, the KGS class also supports the entends option. Thus, let's load the KGS for the config file 2:

>>> myKGS2 = kgs.KGS(cfgFile2)
>>> myKGS2
<KGS 'grok-dev'>

>>> myKGS2.name
'grok-dev'

>>> myKGS2.packages
[<Package 'packageA'>,
<Package 'packageB'>,
<Package 'packageC'>,
<Package 'packageD'>]

CHANGES

1.2.0 (2009-07-24)
  • Add ability to specify the extra-includes for running tests.
1.1.0 (2009-02-01)
  • Added '--no-links', '--no-index', and '--no-minimal-index' options to the

System Message: WARNING/2 (<string>, line 719)

Bullet list ends without a blank line; unexpected unindent.

site generation sctipt to make it faster.

  • Generating list of latest versions does not fail anymore, if downloading and

System Message: WARNING/2 (<string>, line 722)

Bullet list ends without a blank line; unexpected unindent.

parsing of the index page fails.

  • Update links to PyPI web sites.
1.0.1 (2009-01-29)
  • Fix documentation in all scripts, fixing missing imports and incorrect

System Message: WARNING/2 (<string>, line 730)

Bullet list ends without a blank line; unexpected unindent.

wording.

  • The package should depend on python-dateutl and not datetutil, since the

System Message: WARNING/2 (<string>, line 733)

Bullet list ends without a blank line; unexpected unindent.

latter is not available in PyPI anymore.

1.0.0 (2009-01-29)
  • Initial version as zope.kgs.
  • A script that extracts the relevant part of the changelog of each package

System Message: WARNING/2 (<string>, line 741)

Bullet list ends without a blank line; unexpected unindent.

in the KGS.

  • A script that lists all versions of a package released after the latest

System Message: WARNING/2 (<string>, line 744)

Bullet list ends without a blank line; unexpected unindent.

version listed in the KGS.

  • A script that manages the generation of the entire KGS site.
  • Generates generic and version-specific pages.
  • Page generation is template-based for easy customization.
  • Generate links.html file which lists all controlled packages files.
  • Features copied from zope.release:
  • Parser for KGS configuration files.
  • Generate versions.cfg and buildout.cfg script.
  • Features copied from zc.mirrorcheeseshopslashsimple:
  • Generate new index pages for the controlled packages.

Subscribe to package updates

Last updated Jan 5th, 2011

Download Stats

Last month:1

What does the lock icon mean?

Builds marked with a lock icon are only available via PyPM to users with a current ActivePython Business Edition subscription.

Need custom builds or support?

ActivePython Enterprise Edition guarantees priority access to technical support, indemnification, expert consulting and quality-assured language builds.

Plan on re-distributing ActivePython?

Get re-distribution rights and eliminate legal risks with ActivePython OEM Edition.