subunit

Merge lp:~lifeless/subunit/docs into lp:~subunit/subunit/trunk

  1. docs
  2. Merge into trunk
Proposed by Robert Collins
Status: Merged
Merged at revision: not available
Proposed branch: lp:~lifeless/subunit/docs
Merge into: lp:~subunit/subunit/trunk
Diff against target: None lines
To merge this branch: bzr merge lp:~lifeless/subunit/docs
Reviewer Review Type Date Requested Status
Jelmer Vernooij Approve
Review via email: mp+5392@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Robert Collins (lifeless) wrote :

This makes README at least vaguely up to date and interesting to read.

lp:~lifeless/subunit/docs updated
64. By Robert Collins

Missed a bit of dross.

Revision history for this message
Jelmer Vernooij (jelmer) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'README'
--- README 2009-04-07 21:28:28 +0000
+++ README 2009-04-09 11:44:18 +0000
@@ -1,5 +1,5 @@
11
2 subunit: extensions to Python unittest to get test results from subprocesses.2 subunit: A streaming protocol for test results
3 Copyright (C) 2005 Robert Collins <robertc@robertcollins.net>3 Copyright (C) 2005 Robert Collins <robertc@robertcollins.net>
44
5 This program is free software; you can redistribute it and/or modify5 This program is free software; you can redistribute it and/or modify
@@ -20,24 +20,27 @@
20Subunit20Subunit
21-------21-------
2222
23Subunit is attempting to extend unittest with a clean and simple api to23Subunit is a streaming protocol for test results. The protocol is human
24run arbitrary external test suites and return the results to standard24readable and easily generated and parsed. By design all the components of
25Python unittest.25the protocol conceptually fit into the xUnit TestCase->TestResult interaction.
2626
27Subunit comes in three parts:27Subunit comes with command line filters to process a subunit stream and
28 * Protocol writers (clients)28language bindings for python, C, C++ and shell. Bindings are easy to write
29 * Protocol readers (servers)29for other languages.
30 * Filters30
31 31A number of useful things can be done easily with subunit:
32A reader component acts as a test suite in the language that it is written32 * Test aggregation: Tests run separately can be combined and then
33for. Currently subunit only provides a Python protocol reader.33 reported/displayed together. For instance, tests from different languages
3434 can be shown as a seamless whole.
35Writers output test results in subunit form. Writers are typically35 * Test archiving: A test run may be recorded and replayed later.
36test suite runners or test suite result objects in specific languages.36 * Test isolation: Tests that may crash or otherwise interact badly with each
37Currently subunit provides writers for Python, C, C++, and shell.37 other can be run seperately and then aggregated, rather than interfering
3838 with each other.
39Filters provide translation filtering capabilities and can be used to modify a39 * Grid testing: subunit can act as the necessary serialisation and
40stream on-the-fly. Currently subunit provides:40 deserialiation to get test runs on distributed machines to be reported in
41 real time.
42
43Subunit supplies the following filters:
41 * tap2subunit - convert perl's TestAnythingProtocol to subunit.44 * tap2subunit - convert perl's TestAnythingProtocol to subunit.
42 * subunit2pyunit - convert a subunit stream to pyunit test results.45 * subunit2pyunit - convert a subunit stream to pyunit test results.
43 * subunit-filter - filter out tests from a subunit stream.46 * subunit-filter - filter out tests from a subunit stream.
@@ -45,96 +48,107 @@
45 * subunit-stats - generate a summary of a subunit stream.48 * subunit-stats - generate a summary of a subunit stream.
46 * subunit-tags - add or remove tags from a stream.49 * subunit-tags - add or remove tags from a stream.
4750
51Integration with other tools
52----------------------------
53
54Subunit's language bindings act as integration with various test runners like
55'check', 'cppunit', Python's 'unittest'. Beyond that a small amount of glue
56(typically a few lines) will allow Subunit to be used in more sophisticated
57ways.
58
59Python
60======
61
62As a TestResult, Subunit can translate method calls from a test run into a
63Subunit stream::
64
65 # Get a TestSuite or TestCase to run
66 suite = make_suite()
67 # Create a stream (any object with a 'write' method)
68 stream = file('tests.log', 'wb')
69 # Create a subunit result object which will output to the stream
70 result = subunit.TestProtocolClient(stream)
71 # Run the test suite reporting to the subunit result object
72 suite.run(result)
73 # Close the stream.
74 stream.close()
75
76As a TestCase, subunit can read from a stream and inform a TestResult
77of the activity from the stream::
78
79 # Get a stream (any object with a readline() method), in this case example the
80 # stream output by the example before.
81 stream = file('tests.log', 'rb')
82 # Create a subunit ProtocolTestCase which will read from the stream and emit
83 # activity to a result when run() is called.
84 suite = subunit.ProtocolTestCase(stream)
85 # Create a result object to show the contents of the stream.
86 result = unittest._TextTestResult(sys.stdout)
87 # 'run' the tests - process the stream and feed its contents to result.
88 suite.run(result)
89 stream.close()
90
91Subunit has support for non-blocking usage too, for use with asyncore or
92Twisted. See the TestProtocolServer class for more details.
93
94Building on these foundations, Subunit also offers some convenience tools.
95
96The ``IsolatedTestSuite`` class is a decorator that will fork() before running
97the decorated item, and gather the results from the child process via subunit.
98This is useful for handlings tests that mutate global state, or are testing C
99extensions that could crash the VM.
100
101Similarly, ``IsolatedTestCase`` is a base class which can be subclassed to get
102tests that will fork() before the test is run.
103
104Finally, ``ExecTestCase`` is a convenience wrapper for running an external
105program to get a subunit stream and then report that back to an arbitrary
106result object::
107
108 class AggregateTests(subunit.ExecTestCase):
109
110 def test_script_one(self):
111 """./bin/script_one"""
112
113 def test_script_two(self):
114 """./bin/script_two"""
115
116 # Normally your normal test loading would take of this automatically,
117 # It is only spelt out in detail here for clarity.
118 suite = unittest.TestSuite([AggregateTests("test_script_one"),
119 AggregateTests("test_script_two")])
120 # Create any TestResult class you like.
121 result = unittest._TextTestResult(sys.stdout)
122 # And run your suite as normal, subunit will exec each external script as
123 # needed and report to your result object.
124 suite.run(result)
125
126C
127=
128
129Subunit has C bindings to emit the protocol, and comes with a patch for 'check'
130which has been nominally accepted by the 'check' developers. See 'c/README' for
131more details.
132
133C++
134===
135
136C++ uses the C bindings and includes a patch for cppunit. See 'c++/README' for
137details.
138
48The subunit code is organised at the top level by directories for language139The subunit code is organised at the top level by directories for language
49bindings, and additionally the filters directory for filters.140bindings, and additionally the filters directory for filters.
50141
51Using subunit in Python142shell
52-----------------------143=====
53144
541) As a runner for external tests (potentially in other languages)145Similar to C, the shell bindings consist of simple functions to output protocol
552) As a process boundary for unittest TestCases to prevent them fiddling with146elements, and a patch for adding subunit output to the 'ShUnit' shell test
56 in-process state (i.e. singletons).147runner. See 'shell/README' for details.
573) As a wrapper around a TestCase (or Suite) to run a group of tests148
58 externally.149
59150The protocol
601) As a runner for external tests151------------
61=================================
62This is supported on all platforms with Python 2.4.
63For each test script you want to run, declare a ExecTestCase with one
64or more tests whose docstring defines the script to run:
65
66import subunit
67import unittest
68class TestCProgram(subunit.ExecTestCase):
69
70 def test_script_one(self):
71 """./bin/script_one"""
72
73 def test_script_two(self):
74 """./bin/script_two"""
75
76# Yes, the test prefix on the methods matters.
77# Then run this in whatever normal way you would run Python unittests.
78# If you don't have a specific test runner, you can run it using the
79# default one in unittest.py:
80
81if __name__ == '__main__':
82 unittest.main()
83
842) As a process boundary for unittest TestCases
85===============================================
86This is currently supported only on platforms
87that support os.fork(), which allows us to
88transparently introduce a process boundary
89without affecting manual test parameterisation.
90*** TODO explain in more detail and sketch out
91*** a solution for win32
92Just import subunit and derive your test cases
93from subunit.IsolatedTestCase:
94
95import subunit
96
97class TestFoo(subunit.IsolatedTestCase):
98
99 def test_something_globally(self):
100 SomethingGlobal.do()
101 self.assertEqual(SomethingGlobal.value(), 3)
102 # the run() method of IsolatedTestCase will intercept the
103 # test execution, fork() Python to create a new process,
104 # then run the test and report the results back to the parent
105 # process.
106
107# you run this in the normal way you run test cases.
108
1093) As a wrapper around a TestCase to run a group of tests externally.
110=====================================================================
111
112import subunit
113import unittest
114
115class TestFoo(unittest.TestCase):
116
117 def test_foo(self):
118 ...
119
120
121def test_suite():
122 result = subunit.IsolatedTestSuite()
123 loader = unittest.TestLoader()
124 result.addTest(loader.loadTestsFromName(__name__))
125 return result
126
127# you can test the result of test_suite() as follows (or in any normal Python
128# manner.
129runner = unittest.TextTestRunner(verbosity=2)
130runner.run(test_suite())
131# enjoy.
132
133
134Some requirements:
135 The shape of the external unittest should not need to be known a-priori.
136 After the test has run, tests should still exist as discrete objects, so that
137 anything taking a reference to them doesn't get 50 copies of the same object.
138152
139Sample subunit wire contents153Sample subunit wire contents
140----------------------------154----------------------------
@@ -163,7 +177,7 @@
163177
164178
165Subunit protocol description179Subunit protocol description
166----------------------------180============================
167test|testing|test:|testing: test label181test|testing|test:|testing: test label
168success|success:|successful|successful: test label182success|success:|successful|successful: test label
169success|success:|successful|successful: test label [183success|success:|successful|successful: test label [
@@ -211,10 +225,3 @@
211represented as a successful test in Python.225represented as a successful test in Python.
212In future, skip and xfail results will be represented semantically in Python,226In future, skip and xfail results will be represented semantically in Python,
213but some discussion is underway on the right way to do this.227but some discussion is underway on the right way to do this.
214
215
216TODO:
217def run:
218 do a fork,
219 this process runs server
220 child runs client and calls self.run() with a SubprocessTestResult
221228
=== modified file 'c/README'
--- c/README 2007-04-21 07:10:18 +0000
+++ c/README 2009-04-09 11:39:07 +0000
@@ -20,12 +20,9 @@
20This subtree contains an implementation of the subunit child protocol.20This subtree contains an implementation of the subunit child protocol.
21Currently I have no plans to write a test runner in C, so I have not written21Currently I have no plans to write a test runner in C, so I have not written
22an implementation of the parent protocol. [but will happily accept patches].22an implementation of the parent protocol. [but will happily accept patches].
23This implementation is build using SCons and tested via 'check'.23This implementation is built using SCons and tested via 'check'.
24See the tests/ directory for the test programs.24See the tests/ directory for the test programs.
25You can use `make check` or `scons check` to run the tests. I plan to write a25You can use `make check` or `scons check` to run the tests.
26'check' runner which uses these bindings to provide subunit output, at which
27point creating a trivial python test_c.py script which uses the pyunit gui to
28will be added to me todo list.
2926
30The C protocol consists of four functions which you can use to output test27The C protocol consists of four functions which you can use to output test
31metadata trivially. See lib/subunit_child.[ch] for details.28metadata trivially. See lib/subunit_child.[ch] for details.
@@ -34,11 +31,11 @@
34instance] managing assertions, cleaning up on errors etc. You can look at31instance] managing assertions, cleaning up on errors etc. You can look at
35'check' (http://check.sourceforge.net/) or32'check' (http://check.sourceforge.net/) or
36'gunit' (https://garage.maemo.org/projects/gunit) for C unit test33'gunit' (https://garage.maemo.org/projects/gunit) for C unit test
37frameworks. I plan to write ui layers for both of these that use the subunit34frameworks.
38bindings for reporting. There is a patch for 'check'35There is a patch for 'check' (check-subunit-*.patch) in this source tree.
39(check-subunit-0.9.3.patch, and check-subunit-0.9.5.patch) in this source tree.
40Its also available as request ID #1470750 in the sourceforge request tracker36Its also available as request ID #1470750 in the sourceforge request tracker
41http://sourceforge.net/tracker/index.php.37http://sourceforge.net/tracker/index.php. The 'check' developers have indicated
38they will merge this during the current release cycle.
4239
43If you are a test environment maintainer - either homegrown, or 'check' or40If you are a test environment maintainer - either homegrown, or 'check' or
44'gunit' or some other, you will to know how the subunit calls should be used.41'gunit' or some other, you will to know how the subunit calls should be used.

Subscribers

People subscribed via source and target branches