Download 
How to install collective.funkload
- Download and install ActivePython
- Open Command Prompt
- Type
pypm install collective.funkload
Lastest release
Complex functional load testing and benchmarking
collective.funkload provides some extensions of Funkload, a web performance testing and reporting tool. These extensions provide flexible yet simple ways to:
- run benchmarks of multiple test scenarios
- run these benchmarks against multiple setups
- generate comparisons between those setups
All of the console scripts provided by collective.funkload provide a "--help" option which documents the command and it's options.
Table of Contents
The collective.funkload Workflow
1. Develop test scenarios in Python eggs as with unittest
Funkload test cases can be generated using the funkload test recorder and then placed in a Python egg's "tests" package as with normal unittest test cases.
These test cases should be developed which reflect all of the application's supported usage patterns. Take care to balance between separating tests by usage scenario (anonymous visitors, read access, write access, etc.) and keeping the number of tests low enough to scan results.
2. Benchmark the baseline setup
If there is no single baseline, such as when comparing multiple setups to each other, then the term baseline here might be slightly inaccurate. The important part, however, is to establish that the test cases successfully cover the usage scenarios.
Use the fl-run-bench provided by collective.funkload using zope.testing.testrunner semantics to specify the tests to run and the "--label" option to specify a label indicating the benchmark it the baseline (or some other label if baseline isn't appropriate):
$ fl-run-bench -s foo.loadtests --label=baseline
Running zope.testing.testrunner.layer.UnitTests tests: Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds. ======================================================================== Benching FooTestCase.test_FooTest ======================================================================== ... Bench status: SUCCESS Ran # tests with 0 failures and 0 errors in # minutes #.### seconds. Tearing down left over layers: Tear down zope.testing.testrunner.layer.UnitTests in 0.000 seconds.
Use "fl-build-report --html" to build an HTML report from the XML generated by running the benchmark above:
$ fl-build-report --html FooTest-bench-YYYYMMDDThhmmss.xml
Creating html report ...done: file:///.../test_ReadOnly-YYYYMMDDThhmmss-baseline/index.html
Examine the report details. If the test cases don't sufficiently cover the application's supported useage patterns, repeat steps 1 and 2 until the test cases provide sufficient coverage.
3. Benchmark the other setups
In turn, deploy each of the setups. This procedure will be dictated by the application. If using different buildout configurations, for example, deploy each configuration:
$ ...
Then use the same fl-run-bench command as before (or adusted as needed for the setup) giving a differen "--label" option dsignating the setup:
$ fl-run-bench -s foo.loadtests --label=foo-setup
Running zope.testing.testrunner.layer.UnitTests tests: Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds. ======================================================================== Benching FooTestCase.test_FooTest ======================================================================== ... Bench status: SUCCESS Ran # tests with 0 failures and 0 errors in # minutes #.### seconds. Tearing down left over layers: Tear down zope.testing.testrunner.layer.UnitTests in 0.000 seconds.
Repeat this step for each setup.
4. Build the HTML and differential reports and the matrix index
Use the "fl-build-label-reports" command with the "--x-label" and "--y-label" options to automatically build all the HTML reports, the differential reports based on the labels, and an index matrix to the HTML and differential reports. The "fl-build-label-reports" script will use a default title and sub-title based on the labels but may specified using the "--title" and "--sub-title" options. Arbitrary text or HTML may also be included on stdin or using the "--input" option:
$ echo "When deciding which setup to use..." | \
fl-build-label-reports --x-label=baseline --y-label=foo-setup --y-label=bar-setup --title="Setup Comparison" --sub-title="Compare setups foo and bar against baseline" Creating html report ...done: file:///.../test_ReadOnly-YYYYMMDDThhmmss-baseline/index.html Creating html report ...done: file:///.../test_ReadOnly-YYYYMMDDThhmmss-foo-label/index.html Creating html report ...done: file:///.../test_ReadOnly-YYYYMMDDThhmmss-bar-label/index.html Creating diff report ...done: /.../diff_ReadOnly-YYYYMMDDT_hhmmss-foo-label_vs_hhmmss-baseline/index.html Creating diff report ...done: /.../diff_ReadOnly-YYYYMMDDT_hhmmss-bar-label_vs_hhmmss-baseline/index.html Creating report index ...done: file:///.../index.html
Both the "--x-label" and "--y-label" options may be given multiple times or may use Python regular expressions to create an MxN matrix of differential reports. See the "fl-build-label-reports --help" documentation for more details.
5. Examine the results using the matrix index
Open the index.html generated by the last command to survey the HTML reports and differential reports.
6. Repeat as changes are made
As changes are made in your application or setups or to test new setups, repeat steps 3 and 4. When step 4 is repeated by running "fl-build-label-reports" adjusting the "--x-label" and "--y-label" options as appropriate, new HTML and differential reports will be generated as appropriate for the new load test benchmark results and the matrix index will be updated.
fl-run-bench
The scripts that Funkload installs generally require that they be executed from the directory where the test modules live. While this is appropriate for generating test cases with the Funkload recorder, it's often not the desirable behavior when running load test benchmarks. Additionally, the argument handling for the benchmark runner doesn't allow for specifying which tests to benchmark using the zope.testing.testrunner semantics, such as specifying modules and packages with dotted paths, as one is often wont to do when working with setuptools and eggs.
To accommodate this usage pattern, the collective.funkload package provides a wrapper around the Funkload benchmark runner that handles dotted path arguments gracefully. Specifically, rather than pass *.py file and TestCase.test_method arguments, the "fl-bench-runner" provided by collective.funkload supports zope.testing.testrunner semantics for finding tests with "-s", "-m" and "-t".
>>> from collective.funkload import bench >>> bench.run(defaults, ( ... 'test.py -s foo -t test_foo ' ... '--cycles 1 --url http://bar.com').split()) t... Benching FooTestCase.test_foo... * Server: http://bar.com... * Cycles: [1]...
fl-build-label-reports
The fl-build-label-reports script builds HTML (fl-build-report --html) and differential (fl-build-report --diff) reports for multiple bench results at once based on the bench result labels. Labels are selected for the X and Y axes to be compared against each other using the "--x-label" and "--y-label" options. These options accept the same regular expression filters as the zope.testing.testrunner --module and --test options and like those options maybe given multiple times.
The direction or polarity of the differential reports, which report is the reference and which report is the challenger, is determined by sorting the labels involved. This avoids confusion that could occur if differential reports of both directions are included the same matrix, one showing green and the other read. As such, labels should be specified such that their sort order will reflect the desired differential polarity. The "--reverse" option can also be used to reverse the sort order for polarity only without affecting the sort order used on the axes. IOW, when the polarity of the differentials should be the reverse of the order of the axes, use "--reverse".
The title and sub-title rendered on the matrix index may be specified using the "--title" and "--sub-title" options. If not specified a default title will be used and a sub-title will be generated based on the labels on the X and Y axes. Arbitrary text or HTML may also be included on stdin or using the "--input" option. If provided, it will be rendered beneath the sub-title and above the matrix.
In the examples below, load tests have been run to measure read, write, and add performance under Python 2.4, 2.5, and 2.6. There are three different tests to measure read, write and add performance. Labels are used to designate which Python version the load tests have been run under. Thus fl-build-label-reports can be used to quickly generate reports which can be used to evaluate any performance trade offs the various python versions might have for the application being tested.
Start with some bench result XML files.
>>> import os >>> from collective.funkload import testing >>> testing.setUpReports(reports_dir) >>> sorted(os.listdir(reports_dir), reverse=True) ['write-bench-20081211T071242.xml', 'write-bench-20081211T071242.log', 'read-bench-20081211T071242.xml', 'read-bench-20081211T071242.log', 'read-bench-20081211T071241.xml', 'read-bench-20081211T071241.log', 'read-bench-20081210T071243.xml', 'read-bench-20081210T071243.log', 'read-bench-20081210T071241.xml', 'read-bench-20081210T071241.log', 'add-bench-20081211T071243.xml', 'add-bench-20081211T071243.log', 'add-bench-20081211T071242.xml', 'add-bench-20081211T071242.log']
These bench results cover multiple tests and have multiple labels. Some labels are applied to bench results for multiple tests.
>>> import pprint
>>> pprint.pprint(testing.listReports(reports_dir))
[(u'python-2.4',
[(u'test_add',
[(u'2008-12-11T07:12:43.000000',
Bench(path='add-bench-20081211T071243.xml', diffs={}))]),
(u'test_read',
[(u'2008-12-11T07:12:42.000000',
Bench(path='read-bench-20081211T071242.xml', diffs={})),
(u'2008-12-10T07:12:43.000000',
Bench(path='read-bench-20081210T071243.xml', diffs={}))])]),
(u'python-2.5',
[(u'test_read',
[(u'2008-12-10T07:12:41.000000',
Bench(path='read-bench-20081210T071241.xml', diffs={}))])]),
(u'python-2.6',
[(u'test_add',
[(u'2008-12-11T07:12:42.000000',
Bench(path='add-bench-20081211T071242.xml', diffs={}))]),
(u'test_read',
[(u'2008-12-11T07:12:41.000000',
Bench(path='read-bench-20081211T071241.xml', diffs={}))])])]
When labels are specified for the X or Y axes, HTML reports are generated for the latest bench result XML file for each combination of the specified label and each test for which there are bench results available. Then differential reports are generated between the X and Y axes forming a grid of reports. Finally, an index.html file is generated providing clear and easy access to the generated reports. Generate reports and comparisons for python-2.4 vs python-2.6. Also specify the "--reverse" option so that the differential polarity will be the reverse of the axes label order.
>>> from collective.funkload import label
>>> input_ = os.path.join(reports_dir, 'input.html')
>>> open(input_, 'w').write('<a href="http://foo.com">foo</a>')
>>> args = (
... '-o %s --x-label python-2.4 --y-label !.*-2.5 --reverse'
... % reports_dir).split() + [
... '--title', 'Python 2.6 vs Python 2.4',
... '--sub-title', 'Comparing Python versions',
... '--input', input_]
>>> options, _ = label.parser.parse_args(args=args)
>>> label.run(options)
Creating html report ...done:
.../reports/test_add-20081211T071242-python-2.6/index.html
Creating html report ...done:
.../reports/test_read-20081211T071241-python-2.6/index.html
Creating html report ...done:
.../reports/test_add-20081211T071243-python-2.4/index.html
Creating diff report ...done:
.../reports/diff_add-20081211T_071242-python-2.6_vs_071243-python-2.4/index.html
Creating html report ...done:
.../reports/test_read-20081211T071242-python-2.4/index.html
Creating diff report ...done:
.../reports/diff_read-20081211T_071241-python-2.6_vs_071242-python-2.4/index.html
Creating report index ...done:
.../reports/index.html
'.../reports/index.html'
The report index renders a table with links to the HTML reports on the X and Y axes and links to the differential reports in the table cells. In this case there's only one HTML report on the X axis and four reports on the Y axis. Note that report links aren't included in the column headers for the X axis to conserve space and avoid duplication. When using only one label for the X axis, it may be useful to include it in the Y axis even though the differential report cells will be empty in order to include the links to the non-differential test reports for each test.
>>> print open(os.path.join(reports_dir, 'index.html')).read() <... <title>Python 2.6 vs Python 2.4</title>... <h1 class="title">Python 2.6 vs Python 2.4</h1> <h2 class="subtitle">Comparing Python versions</h2> <a href="http://foo.com">foo</a> <table class="docutils"> <thead> <tr class="field"> <th class="field-name" colspan="2"> </th> <th class="field-name" colspan="1"> Label </th> </tr> <tr class="field"> <th class="field-name">Label</th> <th class="field-name">Test</th> <th class="field-name">python-2.4</th> </tr> </thead> <tbody> <tr> <th class="field-name" rowspan="2">python-2.4</th> <th class="field-name"> <a href="test_add-20081211T071243-python-2.4/index.html"> <img alt="foo.sampletests.FooTestCase.test_add" src="test_add-20081211T071243-python-2.4/tests.png" height="120" width="120"/> <div>test_add</div> </a> </th> <td class="field-body"> </td> </tr> <tr> <th class="field-name"> <a href="test_read-20081211T071242-python-2.4/index.html"> <img alt="foo.sampletests.FooTestCase.test_read" src="test_read-20081211T071242-python-2.4/tests.png" height="120" width="120"/> <div>test_read</div> </a> </th> <td class="field-body"> </td> </tr> <tr> <th class="field-name" rowspan="2">python-2.6</th> <th class="field-name"> <a href="test_add-20081211T071242-python-2.6/index.html"> <img alt="foo.sampletests.FooTestCase.test_add" src="test_add-20081211T071242-python-2.6/tests.png" height="120" width="120"/> <div>test_add</div> </a> </th> <td class="field-body"> <a href="diff_add-20081211T_071242-python-2.6_vs_071243-python-2.4/index.html"> <img alt="diff of python-2.6 vs python-2.4 for test_add" src="diff_add-20081211T_071242-python-2.6_vs_071243-python-2.4/spps_diff.png" height="95" width="160"/> <div>python-2.6 vs python-2.4</div> </a> </td> </tr> <tr> <th class="field-name"> <a href="test_read-20081211T071241-python-2.6/index.html"> <img alt="foo.sampletests.FooTestCase.test_read" src="test_read-20081211T071241-python-2.6/tests.png" height="120" width="120"/> <div>test_read</div> </a> </th> <td class="field-body"> <a href="diff_read-20081211T_071241-python-2.6_vs_071242-python-2.4/index.html"> <img alt="diff of python-2.6 vs python-2.4 for test_read" src="diff_read-20081211T_071241-python-2.6_vs_071242-python-2.4/spps_diff.png" height="95" width="160"/> <div>python-2.6 vs python-2.4</div> </a> </td> </tr> </tbody> <tfooter> <tr class="field"> <th class="field-name">Label</th> <th class="field-name">Test</th> <th class="field-name">python-2.4</th> </tr> <tr class="field"> <th class="field-name" colspan="2"> </th> <th class="field-name" colspan="1"> Label </th> </tr> </tfooter> </table>...
If no labels are specified for the X or Y axes then all labels are selected for both the X and Y axes for a full NxN comparison. Both HTML and differential reports are only generated if they haven't been already. IOW, existing reports will be re-used. Reports or results without labels will be ignored. Since the HTML report contains the bench run XML results file, the original is removed and any corresponding log file is moved into the HTML report directory.
>>> open(input_, 'w').write('')
>>> args = ('-o %s' % reports_dir).split()+['--input', input_]
>>> options, _ = label.parser.parse_args(args=args)
>>> label.run(options)
Creating html report ...done:
.../reports/test_read-20081210T071241-python-2.5/index.html
Creating diff report ...done:
.../reports/diff_read_20081211T071242-python-2.4_vs_20081210T071241-python-2.5/index.html
Creating diff report ...done:
.../reports/diff_read_20081210T071241-python-2.5_vs_20081211T071241-python-2.6/index.html
Creating report index ...done:
.../reports/index.html
'.../reports/index.html'
>>> pprint.pprint(sorted(os.listdir(reports_dir), reverse=True)) ['write-bench-20081211T071242.xml', 'write-bench-20081211T071242.log', 'test_read-20081211T071242-python-2.4', 'test_read-20081211T071241-python-2.6', 'test_read-20081210T071241-python-2.5', 'test_add-20081211T071243-python-2.4', 'test_add-20081211T071242-python-2.6', 'read-bench-20081210T071243.xml', 'read-bench-20081210T071243.log', 'input.html', 'index.html', 'diff_read_20081211T071242-python-2.4_vs_20081210T071241-python-2.5', 'diff_read_20081210T071241-python-2.5_vs_20081211T071241-python-2.6', 'diff_read-20081211T_071242-python-2.4_vs_071241-python-2.6', 'diff_read-20081211T_071241-python-2.6_vs_071242-python-2.4', 'diff_add-20081211T_071243-python-2.4_vs_071242-python-2.6', 'diff_add-20081211T_071242-python-2.6_vs_071243-python-2.4'] >>> os.path.isfile(os.path.join( ... reports_dir, 'test_read-20081211T071242-python-2.4', ... 'funkload.log')) True >>> os.path.isfile(os.path.join( ... reports_dir, 'test_read-20081211T071242-python-2.4', ... 'funkload.xml')) True
The HTML report index will be updated to reflect the newly included results and reports.
>>> print open(os.path.join(reports_dir, 'index.html')).read() <... <title> collective.funkload label matrix report </title>... <h1 class="title"> <a href="http://pypi.python.org/pypi/collective.funkload"> collective.funkload label matrix report </a> </h1> <h2 class="subtitle">python-2.4, python-2.5, python-2.6 vs python-2.4, python-2.5, python-2.6</h2> <table class="docutils"> <thead> <tr class="field"> <th class="field-name" colspan="2"> </th> <th class="field-name" colspan="3"> Label </th> </tr> <tr class="field"> <th class="field-name">Label</th> <th class="field-name">Test</th> <th class="field-name">python-2.4</th> <th class="field-name">python-2.5</th> <th class="field-name">python-2.6</th> </tr> </thead> <tbody> <tr> <th class="field-name" rowspan="2">python-2.4</th> <th class="field-name"> <a href="test_add-20081211T071243-python-2.4/index.html"> <img alt="foo.sampletests.FooTestCase.test_add" src="test_add-20081211T071243-python-2.4/tests.png" height="120" width="120"/> <div>test_add</div> </a> </th> <td class="field-body"> </td> <td class="field-body"> </td> <td class="field-body"> <a href="diff_add-20081211T_071243-python-2.4_vs_071242-python-2.6/index.html"> <img alt="diff of python-2.4 vs python-2.6 for test_add" src="diff_add-20081211T_071243-python-2.4_vs_071242-python-2.6/spps_diff.png" height="95" width="160"/> <div>python-2.4 vs python-2.6</div> </a> </td> </tr> <tr> <th class="field-name"> <a href="test_read-20081211T071242-python-2.4/index.html"> <img alt="foo.sampletests.FooTestCase.test_read" src="test_read-20081211T071242-python-2.4/tests.png" height="120" width="120"/> <div>test_read</div> </a> </th> <td class="field-body"> </td> <td class="field-body"> <a href="diff_read_20081211T071242-python-2.4_vs_20081210T071241-python-2.5/index.html"> <img alt="diff of python-2.4 vs python-2.5 for test_read" src="diff_read_20081211T071242-python-2.4_vs_20081210T071241-python-2.5/spps_diff.png" height="95" width="160"/> <div>python-2.4 vs python-2.5</div> </a> </td> <td class="field-body"> <a href="diff_read-20081211T_071242-python-2.4_vs_071241-python-2.6/index.html"> <img alt="diff of python-2.4 vs python-2.6 for test_read" src="diff_read-20081211T_071242-python-2.4_vs_071241-python-2.6/spps_diff.png" height="95" width="160"/> <div>python-2.4 vs python-2.6</div> </a> </td> </tr> <tr> <th class="field-name" rowspan="1">python-2.5</th> <th class="field-name"> <a href="test_read-20081210T071241-python-2.5/index.html"> <img alt="foo.sampletests.FooTestCase.test_read" src="test_read-20081210T071241-python-2.5/tests.png" height="120" width="120"/> <div>test_read</div> </a> </th> <td class="field-body"> <a href="diff_read_20081211T071242-python-2.4_vs_20081210T071241-python-2.5/index.html"> <img alt="diff of python-2.4 vs python-2.5 for test_read" src="diff_read_20081211T071242-python-2.4_vs_20081210T071241-python-2.5/spps_diff.png" height="95" width="160"/> <div>python-2.4 vs python-2.5</div> </a> </td> <td class="field-body"> </td> <td class="field-body"> <a href="diff_read_20081210T071241-python-2.5_vs_20081211T071241-python-2.6/index.html"> <img alt="diff of python-2.5 vs python-2.6 for test_read" src="diff_read_20081210T071241-python-2.5_vs_20081211T071241-python-2.6/spps_diff.png" height="95" width="160"/> <div>python-2.5 vs python-2.6</div> </a> </td> </tr> <tr> <th class="field-name" rowspan="2">python-2.6</th> <th class="field-name"> <a href="test_add-20081211T071242-python-2.6/index.html"> <img alt="foo.sampletests.FooTestCase.test_add" src="test_add-20081211T071242-python-2.6/tests.png" height="120" width="120"/> <div>test_add</div> </a> </th> <td class="field-body"> <a href="diff_add-20081211T_071243-python-2.4_vs_071242-python-2.6/index.html"> <img alt="diff of python-2.4 vs python-2.6 for test_add" src="diff_add-20081211T_071243-python-2.4_vs_071242-python-2.6/spps_diff.png" height="95" width="160"/> <div>python-2.4 vs python-2.6</div> </a> </td> <td class="field-body"> </td> <td class="field-body"> </td> </tr> <tr> <th class="field-name"> <a href="test_read-20081211T071241-python-2.6/index.html"> <img alt="foo.sampletests.FooTestCase.test_read" src="test_read-20081211T071241-python-2.6/tests.png" height="120" width="120"/> <div>test_read</div> </a> </th> <td class="field-body"> <a href="diff_read-20081211T_071242-python-2.4_vs_071241-python-2.6/index.html"> <img alt="diff of python-2.4 vs python-2.6 for test_read" src="diff_read-20081211T_071242-python-2.4_vs_071241-python-2.6/spps_diff.png" height="95" width="160"/> <div>python-2.4 vs python-2.6</div> </a> </td> <td class="field-body"> <a href="diff_read_20081210T071241-python-2.5_vs_20081211T071241-python-2.6/index.html"> <img alt="diff of python-2.5 vs python-2.6 for test_read" src="diff_read_20081210T071241-python-2.5_vs_20081211T071241-python-2.6/spps_diff.png" height="95" width="160"/> <div>python-2.5 vs python-2.6</div> </a> </td> <td class="field-body"> </td> </tr> </tbody> <tfooter> <tr class="field"> <th class="field-name">Label</th> <th class="field-name">Test</th> <th class="field-name">python-2.4</th> <th class="field-name">python-2.5</th> <th class="field-name">python-2.6</th> </tr> <tr class="field"> <th class="field-name" colspan="2"> </th> <th class="field-name" colspan="3"> Label </th> </tr> </tfooter> </table>...
The "fl-list" script prints out the labeled XML bench result files, HTML report directories, and differential report directories that meet the criteria of the given options. The "--old" option lists everything for which there is an equivalent with a newer time stamp.
>>> from collective.funkload import report
>>> options, _ = report.list_parser.parse_args(
... args=('-o %s --old' % reports_dir).split())
>>> list(report.run(**options.__dict__))
['read-bench-20081210T071243.xml']
Changelog
0.3 - 2010-04-19
- add new TestCase class: PloneFLTestCase with two helper methods:
plone_login and addContent. Please check collective.recipe.funkload for example usage [amleczko]
0.2 - 2010-04-16
- Add custom version of RecorderProgram to make usage
of our custom Script tpl [amleczko]
0.1.1 - 2009-08-09
- Only run funkload tests when invoking bench with -m, -s, -t [evilbungle]
0.1 - 2009-08-09
- Initial release, mainly a snapshot from trunk to compliment the release of
collective.recipe.funkload
successful builds