subunit

Merge lp:~lifeless/subunit/docs into lp:~subunit/subunit/trunk

  1. docs
  2. Merge into trunk
Proposed by Robert Collins
Status: Merged
Merged at revision: not available
Proposed branch: lp:~lifeless/subunit/docs
Merge into: lp:~subunit/subunit/trunk
Diff against target: None lines
To merge this branch: bzr merge lp:~lifeless/subunit/docs
Reviewer Review Type Date Requested Status
Jelmer Vernooij Approve
Review via email: mp+5392@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Robert Collins (lifeless) wrote :

This makes README at least vaguely up to date and interesting to read.

lp:~lifeless/subunit/docs updated
64. By Robert Collins

Missed a bit of dross.

Revision history for this message
Jelmer Vernooij (jelmer) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'README'
2--- README 2009-04-07 21:28:28 +0000
3+++ README 2009-04-09 11:44:18 +0000
4@@ -1,5 +1,5 @@
5
6- subunit: extensions to Python unittest to get test results from subprocesses.
7+ subunit: A streaming protocol for test results
8 Copyright (C) 2005 Robert Collins <robertc@robertcollins.net>
9
10 This program is free software; you can redistribute it and/or modify
11@@ -20,24 +20,27 @@
12 Subunit
13 -------
14
15-Subunit is attempting to extend unittest with a clean and simple api to
16-run arbitrary external test suites and return the results to standard
17-Python unittest.
18-
19-Subunit comes in three parts:
20- * Protocol writers (clients)
21- * Protocol readers (servers)
22- * Filters
23-
24-A reader component acts as a test suite in the language that it is written
25-for. Currently subunit only provides a Python protocol reader.
26-
27-Writers output test results in subunit form. Writers are typically
28-test suite runners or test suite result objects in specific languages.
29-Currently subunit provides writers for Python, C, C++, and shell.
30-
31-Filters provide translation filtering capabilities and can be used to modify a
32-stream on-the-fly. Currently subunit provides:
33+Subunit is a streaming protocol for test results. The protocol is human
34+readable and easily generated and parsed. By design all the components of
35+the protocol conceptually fit into the xUnit TestCase->TestResult interaction.
36+
37+Subunit comes with command line filters to process a subunit stream and
38+language bindings for python, C, C++ and shell. Bindings are easy to write
39+for other languages.
40+
41+A number of useful things can be done easily with subunit:
42+ * Test aggregation: Tests run separately can be combined and then
43+ reported/displayed together. For instance, tests from different languages
44+ can be shown as a seamless whole.
45+ * Test archiving: A test run may be recorded and replayed later.
46+ * Test isolation: Tests that may crash or otherwise interact badly with each
47+ other can be run seperately and then aggregated, rather than interfering
48+ with each other.
49+ * Grid testing: subunit can act as the necessary serialisation and
50+ deserialiation to get test runs on distributed machines to be reported in
51+ real time.
52+
53+Subunit supplies the following filters:
54 * tap2subunit - convert perl's TestAnythingProtocol to subunit.
55 * subunit2pyunit - convert a subunit stream to pyunit test results.
56 * subunit-filter - filter out tests from a subunit stream.
57@@ -45,96 +48,107 @@
58 * subunit-stats - generate a summary of a subunit stream.
59 * subunit-tags - add or remove tags from a stream.
60
61+Integration with other tools
62+----------------------------
63+
64+Subunit's language bindings act as integration with various test runners like
65+'check', 'cppunit', Python's 'unittest'. Beyond that a small amount of glue
66+(typically a few lines) will allow Subunit to be used in more sophisticated
67+ways.
68+
69+Python
70+======
71+
72+As a TestResult, Subunit can translate method calls from a test run into a
73+Subunit stream::
74+
75+ # Get a TestSuite or TestCase to run
76+ suite = make_suite()
77+ # Create a stream (any object with a 'write' method)
78+ stream = file('tests.log', 'wb')
79+ # Create a subunit result object which will output to the stream
80+ result = subunit.TestProtocolClient(stream)
81+ # Run the test suite reporting to the subunit result object
82+ suite.run(result)
83+ # Close the stream.
84+ stream.close()
85+
86+As a TestCase, subunit can read from a stream and inform a TestResult
87+of the activity from the stream::
88+
89+ # Get a stream (any object with a readline() method), in this case example the
90+ # stream output by the example before.
91+ stream = file('tests.log', 'rb')
92+ # Create a subunit ProtocolTestCase which will read from the stream and emit
93+ # activity to a result when run() is called.
94+ suite = subunit.ProtocolTestCase(stream)
95+ # Create a result object to show the contents of the stream.
96+ result = unittest._TextTestResult(sys.stdout)
97+ # 'run' the tests - process the stream and feed its contents to result.
98+ suite.run(result)
99+ stream.close()
100+
101+Subunit has support for non-blocking usage too, for use with asyncore or
102+Twisted. See the TestProtocolServer class for more details.
103+
104+Building on these foundations, Subunit also offers some convenience tools.
105+
106+The ``IsolatedTestSuite`` class is a decorator that will fork() before running
107+the decorated item, and gather the results from the child process via subunit.
108+This is useful for handlings tests that mutate global state, or are testing C
109+extensions that could crash the VM.
110+
111+Similarly, ``IsolatedTestCase`` is a base class which can be subclassed to get
112+tests that will fork() before the test is run.
113+
114+Finally, ``ExecTestCase`` is a convenience wrapper for running an external
115+program to get a subunit stream and then report that back to an arbitrary
116+result object::
117+
118+ class AggregateTests(subunit.ExecTestCase):
119+
120+ def test_script_one(self):
121+ """./bin/script_one"""
122+
123+ def test_script_two(self):
124+ """./bin/script_two"""
125+
126+ # Normally your normal test loading would take of this automatically,
127+ # It is only spelt out in detail here for clarity.
128+ suite = unittest.TestSuite([AggregateTests("test_script_one"),
129+ AggregateTests("test_script_two")])
130+ # Create any TestResult class you like.
131+ result = unittest._TextTestResult(sys.stdout)
132+ # And run your suite as normal, subunit will exec each external script as
133+ # needed and report to your result object.
134+ suite.run(result)
135+
136+C
137+=
138+
139+Subunit has C bindings to emit the protocol, and comes with a patch for 'check'
140+which has been nominally accepted by the 'check' developers. See 'c/README' for
141+more details.
142+
143+C++
144+===
145+
146+C++ uses the C bindings and includes a patch for cppunit. See 'c++/README' for
147+details.
148+
149 The subunit code is organised at the top level by directories for language
150 bindings, and additionally the filters directory for filters.
151
152-Using subunit in Python
153------------------------
154-
155-1) As a runner for external tests (potentially in other languages)
156-2) As a process boundary for unittest TestCases to prevent them fiddling with
157- in-process state (i.e. singletons).
158-3) As a wrapper around a TestCase (or Suite) to run a group of tests
159- externally.
160-
161-1) As a runner for external tests
162-=================================
163-This is supported on all platforms with Python 2.4.
164-For each test script you want to run, declare a ExecTestCase with one
165-or more tests whose docstring defines the script to run:
166-
167-import subunit
168-import unittest
169-class TestCProgram(subunit.ExecTestCase):
170-
171- def test_script_one(self):
172- """./bin/script_one"""
173-
174- def test_script_two(self):
175- """./bin/script_two"""
176-
177-# Yes, the test prefix on the methods matters.
178-# Then run this in whatever normal way you would run Python unittests.
179-# If you don't have a specific test runner, you can run it using the
180-# default one in unittest.py:
181-
182-if __name__ == '__main__':
183- unittest.main()
184-
185-2) As a process boundary for unittest TestCases
186-===============================================
187-This is currently supported only on platforms
188-that support os.fork(), which allows us to
189-transparently introduce a process boundary
190-without affecting manual test parameterisation.
191-*** TODO explain in more detail and sketch out
192-*** a solution for win32
193-Just import subunit and derive your test cases
194-from subunit.IsolatedTestCase:
195-
196-import subunit
197-
198-class TestFoo(subunit.IsolatedTestCase):
199-
200- def test_something_globally(self):
201- SomethingGlobal.do()
202- self.assertEqual(SomethingGlobal.value(), 3)
203- # the run() method of IsolatedTestCase will intercept the
204- # test execution, fork() Python to create a new process,
205- # then run the test and report the results back to the parent
206- # process.
207-
208-# you run this in the normal way you run test cases.
209-
210-3) As a wrapper around a TestCase to run a group of tests externally.
211-=====================================================================
212-
213-import subunit
214-import unittest
215-
216-class TestFoo(unittest.TestCase):
217-
218- def test_foo(self):
219- ...
220-
221-
222-def test_suite():
223- result = subunit.IsolatedTestSuite()
224- loader = unittest.TestLoader()
225- result.addTest(loader.loadTestsFromName(__name__))
226- return result
227-
228-# you can test the result of test_suite() as follows (or in any normal Python
229-# manner.
230-runner = unittest.TextTestRunner(verbosity=2)
231-runner.run(test_suite())
232-# enjoy.
233-
234-
235-Some requirements:
236- The shape of the external unittest should not need to be known a-priori.
237- After the test has run, tests should still exist as discrete objects, so that
238- anything taking a reference to them doesn't get 50 copies of the same object.
239+shell
240+=====
241+
242+Similar to C, the shell bindings consist of simple functions to output protocol
243+elements, and a patch for adding subunit output to the 'ShUnit' shell test
244+runner. See 'shell/README' for details.
245+
246+
247+The protocol
248+------------
249
250 Sample subunit wire contents
251 ----------------------------
252@@ -163,7 +177,7 @@
253
254
255 Subunit protocol description
256-----------------------------
257+============================
258 test|testing|test:|testing: test label
259 success|success:|successful|successful: test label
260 success|success:|successful|successful: test label [
261@@ -211,10 +225,3 @@
262 represented as a successful test in Python.
263 In future, skip and xfail results will be represented semantically in Python,
264 but some discussion is underway on the right way to do this.
265-
266-
267-TODO:
268-def run:
269- do a fork,
270- this process runs server
271- child runs client and calls self.run() with a SubprocessTestResult
272
273=== modified file 'c/README'
274--- c/README 2007-04-21 07:10:18 +0000
275+++ c/README 2009-04-09 11:39:07 +0000
276@@ -20,12 +20,9 @@
277 This subtree contains an implementation of the subunit child protocol.
278 Currently I have no plans to write a test runner in C, so I have not written
279 an implementation of the parent protocol. [but will happily accept patches].
280-This implementation is build using SCons and tested via 'check'.
281+This implementation is built using SCons and tested via 'check'.
282 See the tests/ directory for the test programs.
283-You can use `make check` or `scons check` to run the tests. I plan to write a
284-'check' runner which uses these bindings to provide subunit output, at which
285-point creating a trivial python test_c.py script which uses the pyunit gui to
286-will be added to me todo list.
287+You can use `make check` or `scons check` to run the tests.
288
289 The C protocol consists of four functions which you can use to output test
290 metadata trivially. See lib/subunit_child.[ch] for details.
291@@ -34,11 +31,11 @@
292 instance] managing assertions, cleaning up on errors etc. You can look at
293 'check' (http://check.sourceforge.net/) or
294 'gunit' (https://garage.maemo.org/projects/gunit) for C unit test
295-frameworks. I plan to write ui layers for both of these that use the subunit
296-bindings for reporting. There is a patch for 'check'
297-(check-subunit-0.9.3.patch, and check-subunit-0.9.5.patch) in this source tree.
298+frameworks.
299+There is a patch for 'check' (check-subunit-*.patch) in this source tree.
300 Its also available as request ID #1470750 in the sourceforge request tracker
301-http://sourceforge.net/tracker/index.php.
302+http://sourceforge.net/tracker/index.php. The 'check' developers have indicated
303+they will merge this during the current release cycle.
304
305 If you are a test environment maintainer - either homegrown, or 'check' or
306 'gunit' or some other, you will to know how the subunit calls should be used.

Subscribers

People subscribed via source and target branches