Bazaar

Merge lp:~mbp/bzr/597791-http-tests into lp:bzr

  1. 597791-http-tests
  2. Merge into bzr.dev
Proposed by Martin Pool
Status: Merged
Approved by: Vincent Ladeuil
Approved revision: no longer in the source branch.
Merged at revision: 5483
Proposed branch: lp:~mbp/bzr/597791-http-tests
Merge into: lp:bzr
Diff against target: 689 lines (+326/-100)
6 files modified
NEWS (+6/-0)
bzrlib/tests/__init__.py (+14/-1)
bzrlib/tests/scenarios.py (+61/-0)
bzrlib/tests/test_http.py (+115/-89)
bzrlib/tests/test_scenarios.py (+110/-0)
doc/developers/testing.txt (+20/-10)
To merge this branch: bzr merge lp:~mbp/bzr/597791-http-tests
Reviewer Review Type Date Requested Status
Robert Collins (community) Approve
Vincent Ladeuil Approve
Review via email: mp+37941@code.launchpad.net

Commit message

cleanup test_http to use scenarios; add load_tests_from_scenarios

Description of the change

TestCase classes can now just have a `variations` attribute to describe how they should be multiplied. This lets us entirely delete the load_tests function complained of in bug 597791 and to my mind it makes the code _much_ nicer by eliminating spooky action at a distance. Now each class says what kind of variation it wants, without needing to know all the specific values, which may be dynamically determined.

I tested this doesn't change the parameterization by running 'selftest --list' on this and trunk, and ignoring ordering they're the same.

It could be nice to move or copy this into testscenarios.

I haven't yet rolled this out to other modules but I expect in many cases we could delete their custom load_tests methods in favour of just using load_tests_from_their_variations. (The name is a bit explicit because the behaviour's a bit magical.) I think this may also help with cases where we've used subclassing to parameterize because the official way was a bit hard.

To post a comment you must log in.
Revision history for this message
Martin Pool (mbp) wrote :

After some discussion with Robert and bug 658044, I think I can make this converge a bit more with what's already done in testscenarios:

 * name the attribute 'scenarios'
 * make it an iterable of scenarios directly, rather than a list of TestVariations to multiply
 * generate the cross product by calling a function rather than implicitly; to work well with scenario generators that come from registries this function should be quite lazy
 * change the variations to generators or functions rather than classes

Also stylistically
 * clearer tests
 * remove extra VWS
 * module docstring

Revision history for this message
Vincent Ladeuil (vila) wrote :

Lovely result :)

I had a look at that strange assertionError and... it looks like an uncaught regression *long* ago.
Or even worse, this looks like a test that has *never* failed :-(
I'll look at it again tomorrow.

One question: why is it required to do:

  load_tests = load_tests_from_scenarios

Can't that be triggered automatically if a 'scenarios' attribute is defined on the test class ?

It's true that load_tests is generally required when you want to parametrize but... is it still possible to define a load_tests function and call load_tests_from_scenarios for some classes only ?

I can't put my finger on it as I write this but I somehow feel there is a hole somewhere... imbw, you may just have fill the hole instead...

review: Approve
Revision history for this message
Martin Pool (mbp) wrote :

I added (obviously) the raise AssertionError() just to make it really obvious that the thing does indeed fail. My guess is that an error here is sent back to the client as an http 500 error, and it accepts that as a reasonable response.

Is load_tests needed? I think Robert added that as a semi-standard extension point where the test loader calls back in to our code. If we want the test suite to work with other runners, we can't hardcode a check for scenarios. Perhaps we can simplify it more. otoh explicit is better than implicit, and now it's only one line (plus an import...)

Revision history for this message
Robert Collins (lifeless) wrote :

load_tests is in python 2.7 (with a slightly different signature); it would be nice to adapt to that.

Doing load_tests = load_tests_from_scenarios (perhaps it would be better to say 'load_tests_apply_scenarios', but - *blue*) is pretty pithy, explicit and not terrible.

Doing a global is fine to - you could simply use generate_scenarios in bzrlib.test_suite()

381 + # XXX: this is strange; the 'random' name below seems undefined and
382 + # yet the tests pass -- mbp 2010-10-11
383 + raise AssertionError()

Looks like you can't land it like this?

This looks lovely. I would love to see a patch with the missing bits put into testscenarios.

review: Approve
Revision history for this message
Martin Pool (mbp) wrote :

> load_tests is in python 2.7 (with a slightly different signature); it would be
> nice to adapt to that.

Right, bug 607412.

> Doing load_tests = load_tests_from_scenarios (perhaps it would be better to
> say 'load_tests_apply_scenarios', but - *blue*) is pretty pithy, explicit and
> not terrible.
>
> Doing a global is fine to - you could simply use generate_scenarios in
> bzrlib.test_suite()
>
>
> 381 + # XXX: this is strange; the 'random' name below seems
> undefined and
> 382 + # yet the tests pass -- mbp 2010-10-11
> 383 + raise AssertionError()
>
> Looks like you can't land it like this?

I can; the bug (bug 658773) was latent before I came here, and I just noticed it by running pyflakes.

>
> This looks lovely. I would love to see a patch with the missing bits put into
> testscenarios.

Right, I'll do that under bug 658044.

Revision history for this message
Martin Pool (mbp) wrote :

sent to pqm by email

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'NEWS'
--- NEWS 2010-10-11 15:18:38 +0000
+++ NEWS 2010-10-11 22:43:41 +0000
@@ -158,6 +158,12 @@
158Testing158Testing
159*******159*******
160160
161* Add a new simpler way to generate multiple test variations, by setting
162 the `scenarios` attribute of a test class to a list of scenarios
163 descriptions, then using `load_tests_apply_scenarios`. (See the testing
164 guide and `bzrlib.tests.scenarios`.) Simplify `test_http` using this.
165 (Martin Pool, #597791)
166
161* Add ``tests/ssl_certs/ca.crt`` to the required test files list. Test167* Add ``tests/ssl_certs/ca.crt`` to the required test files list. Test
162 involving the pycurl https test server fail otherwise when running168 involving the pycurl https test server fail otherwise when running
163 selftest from an installed version. (Vincent Ladeuil, #651706)169 selftest from an installed version. (Vincent Ladeuil, #651706)
164170
=== modified file 'bzrlib/tests/__init__.py'
--- bzrlib/tests/__init__.py 2010-10-08 10:16:12 +0000
+++ bzrlib/tests/__init__.py 2010-10-11 22:43:41 +0000
@@ -3809,6 +3809,7 @@
3809 'bzrlib.tests.test_rio',3809 'bzrlib.tests.test_rio',
3810 'bzrlib.tests.test_rules',3810 'bzrlib.tests.test_rules',
3811 'bzrlib.tests.test_sampler',3811 'bzrlib.tests.test_sampler',
3812 'bzrlib.tests.test_scenarios',
3812 'bzrlib.tests.test_script',3813 'bzrlib.tests.test_script',
3813 'bzrlib.tests.test_selftest',3814 'bzrlib.tests.test_selftest',
3814 'bzrlib.tests.test_serializer',3815 'bzrlib.tests.test_serializer',
@@ -3991,7 +3992,19 @@
3991 return suite3992 return suite
39923993
39933994
3994def multiply_scenarios(scenarios_left, scenarios_right):3995def multiply_scenarios(*scenarios):
3996 """Multiply two or more iterables of scenarios.
3997
3998 It is safe to pass scenario generators or iterators.
3999
4000 :returns: A list of compound scenarios: the cross-product of all
4001 scenarios, with the names concatenated and the parameters
4002 merged together.
4003 """
4004 return reduce(_multiply_two_scenarios, map(list, scenarios))
4005
4006
4007def _multiply_two_scenarios(scenarios_left, scenarios_right):
3995 """Multiply two sets of scenarios.4008 """Multiply two sets of scenarios.
39964009
3997 :returns: the cartesian product of the two sets of scenarios, that is4010 :returns: the cartesian product of the two sets of scenarios, that is
39984011
=== added file 'bzrlib/tests/scenarios.py'
--- bzrlib/tests/scenarios.py 1970-01-01 00:00:00 +0000
+++ bzrlib/tests/scenarios.py 2010-10-11 22:43:41 +0000
@@ -0,0 +1,61 @@
1# Copyright (C) 2005-2010 Canonical Ltd
2#
3# This program is free software; you can redistribute it and/or modify
4# it under the terms of the GNU General Public License as published by
5# the Free Software Foundation; either version 2 of the License, or
6# (at your option) any later version.
7#
8# This program is distributed in the hope that it will be useful,
9# but WITHOUT ANY WARRANTY; without even the implied warranty of
10# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
11# GNU General Public License for more details.
12#
13# You should have received a copy of the GNU General Public License
14# along with this program; if not, write to the Free Software
15# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
16
17
18"""Generate multiple variations in different scenarios.
19
20For a class whose tests should be repeated in varying scenarios, set a
21`scenarios` member to a list of scenarios where it should be repeated.
22
23This is similar to the interface provided by
24<http://launchpad.net/testscenarios/>.
25"""
26
27
28from bzrlib.tests import (
29 iter_suite_tests,
30 multiply_scenarios,
31 multiply_tests,
32 )
33
34
35def load_tests_apply_scenarios(standard_tests, module, loader):
36 """Multiply tests depending on their 'scenarios' attribute.
37
38 This can be assigned to 'load_tests' in any test module to make this
39 automatically work across tests in the module.
40 """
41 result = loader.suiteClass()
42 multiply_tests_by_their_scenarios(standard_tests, result)
43 return result
44
45
46def multiply_tests_by_their_scenarios(some_tests, into_suite):
47 """Multiply the tests in the given suite by their declared scenarios.
48
49 Each test must have a 'scenarios' attribute which is a list of
50 (name, params) pairs.
51
52 :param some_tests: TestSuite or Test.
53 :param into_suite: A TestSuite into which the resulting tests will be
54 inserted.
55 """
56 for test in iter_suite_tests(some_tests):
57 scenarios = getattr(test, 'scenarios', None)
58 if scenarios is None:
59 into_suite.addTest(test)
60 else:
61 multiply_tests(test, test.scenarios, into_suite)
062
=== modified file 'bzrlib/tests/test_http.py'
--- bzrlib/tests/test_http.py 2010-08-24 13:03:18 +0000
+++ bzrlib/tests/test_http.py 2010-10-11 22:43:41 +0000
@@ -23,10 +23,7 @@
23# TODO: Should be renamed to bzrlib.transport.http.tests?23# TODO: Should be renamed to bzrlib.transport.http.tests?
24# TODO: What about renaming to bzrlib.tests.transport.http ?24# TODO: What about renaming to bzrlib.tests.transport.http ?
2525
26from cStringIO import StringIO
27import httplib26import httplib
28import os
29import select
30import SimpleHTTPServer27import SimpleHTTPServer
31import socket28import socket
32import sys29import sys
@@ -42,7 +39,6 @@
42 tests,39 tests,
43 transport,40 transport,
44 ui,41 ui,
45 urlutils,
46 )42 )
47from bzrlib.tests import (43from bzrlib.tests import (
48 features,44 features,
@@ -50,6 +46,10 @@
50 http_utils,46 http_utils,
51 test_server,47 test_server,
52 )48 )
49from bzrlib.tests.scenarios import (
50 load_tests_apply_scenarios,
51 multiply_scenarios,
52 )
53from bzrlib.transport import (53from bzrlib.transport import (
54 http,54 http,
55 remote,55 remote,
@@ -64,17 +64,11 @@
64 from bzrlib.transport.http._pycurl import PyCurlTransport64 from bzrlib.transport.http._pycurl import PyCurlTransport
6565
6666
67def load_tests(standard_tests, module, loader):67load_tests = load_tests_apply_scenarios
68 """Multiply tests for http clients and protocol versions."""68
69 result = loader.suiteClass()69
7070def vary_by_http_client_implementation():
71 # one for each transport implementation71 """Test the two libraries we can use, pycurl and urllib."""
72 t_tests, remaining_tests = tests.split_suite_by_condition(
73 standard_tests, tests.condition_isinstance((
74 TestHttpTransportRegistration,
75 TestHttpTransportUrls,
76 Test_redirected_to,
77 )))
78 transport_scenarios = [72 transport_scenarios = [
79 ('urllib', dict(_transport=_urllib.HttpTransport_urllib,73 ('urllib', dict(_transport=_urllib.HttpTransport_urllib,
80 _server=http_server.HttpServer_urllib,74 _server=http_server.HttpServer_urllib,
@@ -85,85 +79,48 @@
85 ('pycurl', dict(_transport=PyCurlTransport,79 ('pycurl', dict(_transport=PyCurlTransport,
86 _server=http_server.HttpServer_PyCurl,80 _server=http_server.HttpServer_PyCurl,
87 _url_protocol='http+pycurl',)))81 _url_protocol='http+pycurl',)))
88 tests.multiply_tests(t_tests, transport_scenarios, result)82 return transport_scenarios
8983
90 protocol_scenarios = [84
91 ('HTTP/1.0', dict(_protocol_version='HTTP/1.0')),85def vary_by_http_protocol_version():
92 ('HTTP/1.1', dict(_protocol_version='HTTP/1.1')),86 """Test on http/1.0 and 1.1"""
93 ]87 return [
9488 ('HTTP/1.0', dict(_protocol_version='HTTP/1.0')),
95 # some tests are parametrized by the protocol version only89 ('HTTP/1.1', dict(_protocol_version='HTTP/1.1')),
96 p_tests, remaining_tests = tests.split_suite_by_condition(90 ]
97 remaining_tests, tests.condition_isinstance((91
98 TestAuthOnRedirected,92
99 )))93def vary_by_http_proxy_auth_scheme():
100 tests.multiply_tests(p_tests, protocol_scenarios, result)94 return [
101
102 # each implementation tested with each HTTP version
103 tp_tests, remaining_tests = tests.split_suite_by_condition(
104 remaining_tests, tests.condition_isinstance((
105 SmartHTTPTunnellingTest,
106 TestDoCatchRedirections,
107 TestHTTPConnections,
108 TestHTTPRedirections,
109 TestHTTPSilentRedirections,
110 TestLimitedRangeRequestServer,
111 TestPost,
112 TestProxyHttpServer,
113 TestRanges,
114 TestSpecificRequestHandler,
115 )))
116 tp_scenarios = tests.multiply_scenarios(transport_scenarios,
117 protocol_scenarios)
118 tests.multiply_tests(tp_tests, tp_scenarios, result)
119
120 # proxy auth: each auth scheme on all http versions on all implementations.
121 tppa_tests, remaining_tests = tests.split_suite_by_condition(
122 remaining_tests, tests.condition_isinstance((
123 TestProxyAuth,
124 )))
125 proxy_auth_scheme_scenarios = [
126 ('basic', dict(_auth_server=http_utils.ProxyBasicAuthServer)),95 ('basic', dict(_auth_server=http_utils.ProxyBasicAuthServer)),
127 ('digest', dict(_auth_server=http_utils.ProxyDigestAuthServer)),96 ('digest', dict(_auth_server=http_utils.ProxyDigestAuthServer)),
128 ('basicdigest',97 ('basicdigest',
129 dict(_auth_server=http_utils.ProxyBasicAndDigestAuthServer)),98 dict(_auth_server=http_utils.ProxyBasicAndDigestAuthServer)),
130 ]99 ]
131 tppa_scenarios = tests.multiply_scenarios(tp_scenarios,100
132 proxy_auth_scheme_scenarios)101
133 tests.multiply_tests(tppa_tests, tppa_scenarios, result)102def vary_by_http_auth_scheme():
134103 return [
135 # auth: each auth scheme on all http versions on all implementations.
136 tpa_tests, remaining_tests = tests.split_suite_by_condition(
137 remaining_tests, tests.condition_isinstance((
138 TestAuth,
139 )))
140 auth_scheme_scenarios = [
141 ('basic', dict(_auth_server=http_utils.HTTPBasicAuthServer)),104 ('basic', dict(_auth_server=http_utils.HTTPBasicAuthServer)),
142 ('digest', dict(_auth_server=http_utils.HTTPDigestAuthServer)),105 ('digest', dict(_auth_server=http_utils.HTTPDigestAuthServer)),
143 ('basicdigest',106 ('basicdigest',
144 dict(_auth_server=http_utils.HTTPBasicAndDigestAuthServer)),107 dict(_auth_server=http_utils.HTTPBasicAndDigestAuthServer)),
145 ]108 ]
146 tpa_scenarios = tests.multiply_scenarios(tp_scenarios,109
147 auth_scheme_scenarios)110
148 tests.multiply_tests(tpa_tests, tpa_scenarios, result)111def vary_by_http_activity():
149
150 # activity: on all http[s] versions on all implementations
151 tpact_tests, remaining_tests = tests.split_suite_by_condition(
152 remaining_tests, tests.condition_isinstance((
153 TestActivity,
154 )))
155 activity_scenarios = [112 activity_scenarios = [
156 ('urllib,http', dict(_activity_server=ActivityHTTPServer,113 ('urllib,http', dict(_activity_server=ActivityHTTPServer,
157 _transport=_urllib.HttpTransport_urllib,)),114 _transport=_urllib.HttpTransport_urllib,)),
158 ]115 ]
159 if tests.HTTPSServerFeature.available():116 if tests.HTTPSServerFeature.available():
160 activity_scenarios.append(117 activity_scenarios.append(
161 ('urllib,https', dict(_activity_server=ActivityHTTPSServer,118 ('urllib,https', dict(_activity_server=ActivityHTTPSServer,
162 _transport=_urllib.HttpTransport_urllib,)),)119 _transport=_urllib.HttpTransport_urllib,)),)
163 if features.pycurl.available():120 if features.pycurl.available():
164 activity_scenarios.append(121 activity_scenarios.append(
165 ('pycurl,http', dict(_activity_server=ActivityHTTPServer,122 ('pycurl,http', dict(_activity_server=ActivityHTTPServer,
166 _transport=PyCurlTransport,)),)123 _transport=PyCurlTransport,)),)
167 if tests.HTTPSServerFeature.available():124 if tests.HTTPSServerFeature.available():
168 from bzrlib.tests import (125 from bzrlib.tests import (
169 ssl_certs,126 ssl_certs,
@@ -181,16 +138,8 @@
181138
182 activity_scenarios.append(139 activity_scenarios.append(
183 ('pycurl,https', dict(_activity_server=ActivityHTTPSServer,140 ('pycurl,https', dict(_activity_server=ActivityHTTPSServer,
184 _transport=HTTPS_pycurl_transport,)),)141 _transport=HTTPS_pycurl_transport,)),)
185142 return activity_scenarios
186 tpact_scenarios = tests.multiply_scenarios(activity_scenarios,
187 protocol_scenarios)
188 tests.multiply_tests(tpact_tests, tpact_scenarios, result)
189
190 # No parametrization for the remaining tests
191 result.addTests(remaining_tests)
192
193 return result
194143
195144
196class FakeManager(object):145class FakeManager(object):
@@ -401,6 +350,8 @@
401class TestHttpTransportUrls(tests.TestCase):350class TestHttpTransportUrls(tests.TestCase):
402 """Test the http urls."""351 """Test the http urls."""
403352
353 scenarios = vary_by_http_client_implementation()
354
404 def test_abs_url(self):355 def test_abs_url(self):
405 """Construction of absolute http URLs"""356 """Construction of absolute http URLs"""
406 t = self._transport('http://bazaar-vcs.org/bzr/bzr.dev/')357 t = self._transport('http://bazaar-vcs.org/bzr/bzr.dev/')
@@ -413,7 +364,7 @@
413364
414 def test_invalid_http_urls(self):365 def test_invalid_http_urls(self):
415 """Trap invalid construction of urls"""366 """Trap invalid construction of urls"""
416 t = self._transport('http://bazaar-vcs.org/bzr/bzr.dev/')367 self._transport('http://bazaar-vcs.org/bzr/bzr.dev/')
417 self.assertRaises(errors.InvalidURL,368 self.assertRaises(errors.InvalidURL,
418 self._transport,369 self._transport,
419 'http://http://bazaar-vcs.org/bzr/bzr.dev/')370 'http://http://bazaar-vcs.org/bzr/bzr.dev/')
@@ -475,6 +426,11 @@
475class TestHTTPConnections(http_utils.TestCaseWithWebserver):426class TestHTTPConnections(http_utils.TestCaseWithWebserver):
476 """Test the http connections."""427 """Test the http connections."""
477428
429 scenarios = multiply_scenarios(
430 vary_by_http_client_implementation(),
431 vary_by_http_protocol_version(),
432 )
433
478 def setUp(self):434 def setUp(self):
479 http_utils.TestCaseWithWebserver.setUp(self)435 http_utils.TestCaseWithWebserver.setUp(self)
480 self.build_tree(['foo/', 'foo/bar'], line_endings='binary',436 self.build_tree(['foo/', 'foo/bar'], line_endings='binary',
@@ -525,6 +481,8 @@
525class TestHttpTransportRegistration(tests.TestCase):481class TestHttpTransportRegistration(tests.TestCase):
526 """Test registrations of various http implementations"""482 """Test registrations of various http implementations"""
527483
484 scenarios = vary_by_http_client_implementation()
485
528 def test_http_registered(self):486 def test_http_registered(self):
529 t = transport.get_transport('%s://foo.com/' % self._url_protocol)487 t = transport.get_transport('%s://foo.com/' % self._url_protocol)
530 self.assertIsInstance(t, transport.Transport)488 self.assertIsInstance(t, transport.Transport)
@@ -533,6 +491,11 @@
533491
534class TestPost(tests.TestCase):492class TestPost(tests.TestCase):
535493
494 scenarios = multiply_scenarios(
495 vary_by_http_client_implementation(),
496 vary_by_http_protocol_version(),
497 )
498
536 def test_post_body_is_received(self):499 def test_post_body_is_received(self):
537 server = RecordingServer(expect_body_tail='end-of-body',500 server = RecordingServer(expect_body_tail='end-of-body',
538 scheme=self._url_protocol)501 scheme=self._url_protocol)
@@ -585,6 +548,11 @@
585 Daughter classes are expected to override _req_handler_class548 Daughter classes are expected to override _req_handler_class
586 """549 """
587550
551 scenarios = multiply_scenarios(
552 vary_by_http_client_implementation(),
553 vary_by_http_protocol_version(),
554 )
555
588 # Provide a useful default556 # Provide a useful default
589 _req_handler_class = http_server.TestingHTTPRequestHandler557 _req_handler_class = http_server.TestingHTTPRequestHandler
590558
@@ -841,7 +809,7 @@
841 t = self.get_readonly_transport()809 t = self.get_readonly_transport()
842 # force transport to issue multiple requests810 # force transport to issue multiple requests
843 t._get_max_size = 2811 t._get_max_size = 2
844 l = list(t.readv('a', ((0, 1), (1, 1), (2, 4), (6, 4))))812 list(t.readv('a', ((0, 1), (1, 1), (2, 4), (6, 4))))
845 # The server should have issued 3 requests813 # The server should have issued 3 requests
846 self.assertEqual(3, server.GET_request_nb)814 self.assertEqual(3, server.GET_request_nb)
847 self.assertEqual('0123456789', t.get_bytes('a'))815 self.assertEqual('0123456789', t.get_bytes('a'))
@@ -924,6 +892,8 @@
924 def get_multiple_ranges(self, file, file_size, ranges):892 def get_multiple_ranges(self, file, file_size, ranges):
925 self.send_response(206)893 self.send_response(206)
926 self.send_header('Accept-Ranges', 'bytes')894 self.send_header('Accept-Ranges', 'bytes')
895 # XXX: this is strange; the 'random' name below seems undefined and
896 # yet the tests pass -- mbp 2010-10-11 bug 658773
927 boundary = "%d" % random.randint(0,0x7FFFFFFF)897 boundary = "%d" % random.randint(0,0x7FFFFFFF)
928 self.send_header("Content-Type",898 self.send_header("Content-Type",
929 "multipart/byteranges; boundary=%s" % boundary)899 "multipart/byteranges; boundary=%s" % boundary)
@@ -1055,6 +1025,11 @@
1055class TestLimitedRangeRequestServer(http_utils.TestCaseWithWebserver):1025class TestLimitedRangeRequestServer(http_utils.TestCaseWithWebserver):
1056 """Tests readv requests against a server erroring out on too much ranges."""1026 """Tests readv requests against a server erroring out on too much ranges."""
10571027
1028 scenarios = multiply_scenarios(
1029 vary_by_http_client_implementation(),
1030 vary_by_http_protocol_version(),
1031 )
1032
1058 # Requests with more range specifiers will error out1033 # Requests with more range specifiers will error out
1059 range_limit = 31034 range_limit = 3
10601035
@@ -1134,6 +1109,11 @@
1134 to the file names).1109 to the file names).
1135 """1110 """
11361111
1112 scenarios = multiply_scenarios(
1113 vary_by_http_client_implementation(),
1114 vary_by_http_protocol_version(),
1115 )
1116
1137 # FIXME: We don't have an https server available, so we don't1117 # FIXME: We don't have an https server available, so we don't
1138 # test https connections. --vila toolongago1118 # test https connections. --vila toolongago
11391119
@@ -1236,6 +1216,11 @@
1236class TestRanges(http_utils.TestCaseWithWebserver):1216class TestRanges(http_utils.TestCaseWithWebserver):
1237 """Test the Range header in GET methods."""1217 """Test the Range header in GET methods."""
12381218
1219 scenarios = multiply_scenarios(
1220 vary_by_http_client_implementation(),
1221 vary_by_http_protocol_version(),
1222 )
1223
1239 def setUp(self):1224 def setUp(self):
1240 http_utils.TestCaseWithWebserver.setUp(self)1225 http_utils.TestCaseWithWebserver.setUp(self)
1241 self.build_tree_contents([('a', '0123456789')],)1226 self.build_tree_contents([('a', '0123456789')],)
@@ -1281,6 +1266,11 @@
1281class TestHTTPRedirections(http_utils.TestCaseWithRedirectedWebserver):1266class TestHTTPRedirections(http_utils.TestCaseWithRedirectedWebserver):
1282 """Test redirection between http servers."""1267 """Test redirection between http servers."""
12831268
1269 scenarios = multiply_scenarios(
1270 vary_by_http_client_implementation(),
1271 vary_by_http_protocol_version(),
1272 )
1273
1284 def setUp(self):1274 def setUp(self):
1285 super(TestHTTPRedirections, self).setUp()1275 super(TestHTTPRedirections, self).setUp()
1286 self.build_tree_contents([('a', '0123456789'),1276 self.build_tree_contents([('a', '0123456789'),
@@ -1349,6 +1339,11 @@
1349 -- vila 200702121339 -- vila 20070212
1350 """1340 """
13511341
1342 scenarios = multiply_scenarios(
1343 vary_by_http_client_implementation(),
1344 vary_by_http_protocol_version(),
1345 )
1346
1352 def setUp(self):1347 def setUp(self):
1353 if (features.pycurl.available()1348 if (features.pycurl.available()
1354 and self._transport == PyCurlTransport):1349 and self._transport == PyCurlTransport):
@@ -1399,6 +1394,11 @@
1399class TestDoCatchRedirections(http_utils.TestCaseWithRedirectedWebserver):1394class TestDoCatchRedirections(http_utils.TestCaseWithRedirectedWebserver):
1400 """Test transport.do_catching_redirections."""1395 """Test transport.do_catching_redirections."""
14011396
1397 scenarios = multiply_scenarios(
1398 vary_by_http_client_implementation(),
1399 vary_by_http_protocol_version(),
1400 )
1401
1402 def setUp(self):1402 def setUp(self):
1403 super(TestDoCatchRedirections, self).setUp()1403 super(TestDoCatchRedirections, self).setUp()
1404 self.build_tree_contents([('a', '0123456789'),],)1404 self.build_tree_contents([('a', '0123456789'),],)
@@ -1446,6 +1446,12 @@
1446class TestAuth(http_utils.TestCaseWithWebserver):1446class TestAuth(http_utils.TestCaseWithWebserver):
1447 """Test authentication scheme"""1447 """Test authentication scheme"""
14481448
1449 scenarios = multiply_scenarios(
1450 vary_by_http_client_implementation(),
1451 vary_by_http_protocol_version(),
1452 vary_by_http_auth_scheme(),
1453 )
1454
1449 _auth_header = 'Authorization'1455 _auth_header = 'Authorization'
1450 _password_prompt_prefix = ''1456 _password_prompt_prefix = ''
1451 _username_prompt_prefix = ''1457 _username_prompt_prefix = ''
@@ -1655,6 +1661,12 @@
1655class TestProxyAuth(TestAuth):1661class TestProxyAuth(TestAuth):
1656 """Test proxy authentication schemes."""1662 """Test proxy authentication schemes."""
16571663
1664 scenarios = multiply_scenarios(
1665 vary_by_http_client_implementation(),
1666 vary_by_http_protocol_version(),
1667 vary_by_http_proxy_auth_scheme(),
1668 )
1669
1658 _auth_header = 'Proxy-authorization'1670 _auth_header = 'Proxy-authorization'
1659 _password_prompt_prefix = 'Proxy '1671 _password_prompt_prefix = 'Proxy '
1660 _username_prompt_prefix = 'Proxy '1672 _username_prompt_prefix = 'Proxy '
@@ -1716,6 +1728,11 @@
17161728
1717class SmartHTTPTunnellingTest(tests.TestCaseWithTransport):1729class SmartHTTPTunnellingTest(tests.TestCaseWithTransport):
17181730
1731 scenarios = multiply_scenarios(
1732 vary_by_http_client_implementation(),
1733 vary_by_http_protocol_version(),
1734 )
1735
1719 def setUp(self):1736 def setUp(self):
1720 super(SmartHTTPTunnellingTest, self).setUp()1737 super(SmartHTTPTunnellingTest, self).setUp()
1721 # We use the VFS layer as part of HTTP tunnelling tests.1738 # We use the VFS layer as part of HTTP tunnelling tests.
@@ -1810,6 +1827,8 @@
18101827
1811class Test_redirected_to(tests.TestCase):1828class Test_redirected_to(tests.TestCase):
18121829
1830 scenarios = vary_by_http_client_implementation()
1831
1813 def test_redirected_to_subdir(self):1832 def test_redirected_to_subdir(self):
1814 t = self._transport('http://www.example.com/foo')1833 t = self._transport('http://www.example.com/foo')
1815 r = t._redirected_to('http://www.example.com/foo',1834 r = t._redirected_to('http://www.example.com/foo',
@@ -2061,6 +2080,11 @@
20612080
2062class TestActivity(tests.TestCase, TestActivityMixin):2081class TestActivity(tests.TestCase, TestActivityMixin):
20632082
2083 scenarios = multiply_scenarios(
2084 vary_by_http_activity(),
2085 vary_by_http_protocol_version(),
2086 )
2087
2064 def setUp(self):2088 def setUp(self):
2065 TestActivityMixin.setUp(self)2089 TestActivityMixin.setUp(self)
20662090
@@ -2087,6 +2111,8 @@
2087class TestAuthOnRedirected(http_utils.TestCaseWithRedirectedWebserver):2111class TestAuthOnRedirected(http_utils.TestCaseWithRedirectedWebserver):
2088 """Test authentication on the redirected http server."""2112 """Test authentication on the redirected http server."""
20892113
2114 scenarios = vary_by_http_protocol_version()
2115
2090 _auth_header = 'Authorization'2116 _auth_header = 'Authorization'
2091 _password_prompt_prefix = ''2117 _password_prompt_prefix = ''
2092 _username_prompt_prefix = ''2118 _username_prompt_prefix = ''
20932119
=== added file 'bzrlib/tests/test_scenarios.py'
--- bzrlib/tests/test_scenarios.py 1970-01-01 00:00:00 +0000
+++ bzrlib/tests/test_scenarios.py 2010-10-11 22:43:41 +0000
@@ -0,0 +1,110 @@
1# Copyright (C) 2005-2010 Canonical Ltd
2#
3# This program is free software; you can redistribute it and/or modify
4# it under the terms of the GNU General Public License as published by
5# the Free Software Foundation; either version 2 of the License, or
6# (at your option) any later version.
7#
8# This program is distributed in the hope that it will be useful,
9# but WITHOUT ANY WARRANTY; without even the implied warranty of
10# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
11# GNU General Public License for more details.
12#
13# You should have received a copy of the GNU General Public License
14# along with this program; if not, write to the Free Software
15# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
16
17
18"""Tests for generating multiple tests for scenarios."""
19
20from bzrlib.tests import (
21 TestCase,
22 TestLoader,
23 iter_suite_tests,
24 multiply_tests,
25 )
26
27from bzrlib.tests.scenarios import (
28 load_tests_apply_scenarios,
29 multiply_scenarios,
30 multiply_tests_by_their_scenarios,
31 )
32
33
34# There aren't any actually parameterized tests here, but this exists as a
35# demonstration; so that you can interactively observe them being multiplied;
36# and so that we check everything hooks up properly.
37load_tests = load_tests_apply_scenarios
38
39
40def vary_by_color():
41 """Very simple static variation example"""
42 for color in ['red', 'green', 'blue']:
43 yield (color, {'color': color})
44
45
46def vary_named_attribute(attr_name):
47 """More sophisticated: vary a named parameter"""
48 yield ('a', {attr_name: 'a'})
49 yield ('b', {attr_name: 'b'})
50
51
52def get_generated_test_attributes(suite, attr_name):
53 """Return the `attr_name` attribute from all tests in the suite"""
54 return sorted([
55 getattr(t, attr_name) for t in iter_suite_tests(suite)])
56
57
58class TestTestScenarios(TestCase):
59
60 def test_multiply_tests(self):
61 loader = TestLoader()
62 suite = loader.suiteClass()
63 multiply_tests(
64 self,
65 vary_by_color(),
66 suite)
67 self.assertEquals(
68 ['blue', 'green', 'red'],
69 get_generated_test_attributes(suite, 'color'))
70
71 def test_multiply_scenarios_from_generators(self):
72 """It's safe to multiply scenarios that come from generators"""
73 s = multiply_scenarios(
74 vary_named_attribute('one'),
75 vary_named_attribute('two'),
76 )
77 self.assertEquals(
78 2*2,
79 len(s),
80 s)
81
82 def test_multiply_tests_by_their_scenarios(self):
83 loader = TestLoader()
84 suite = loader.suiteClass()
85 test_instance = PretendVaryingTest('test_nothing')
86 multiply_tests_by_their_scenarios(
87 test_instance,
88 suite)
89 self.assertEquals(
90 ['a', 'a', 'b', 'b'],
91 get_generated_test_attributes(suite, 'value'))
92
93 def test_multiply_tests_no_scenarios(self):
94 """Tests with no scenarios attribute aren't multiplied"""
95 suite = TestLoader().suiteClass()
96 multiply_tests_by_their_scenarios(self,
97 suite)
98 self.assertLength(1, list(iter_suite_tests(suite)))
99
100
101class PretendVaryingTest(TestCase):
102
103 scenarios = multiply_scenarios(
104 vary_named_attribute('value'),
105 vary_named_attribute('other'),
106 )
107
108 def test_nothing(self):
109 """This test exists just so it can be multiplied"""
110 pass
0111
=== modified file 'doc/developers/testing.txt'
--- doc/developers/testing.txt 2010-10-07 07:51:54 +0000
+++ doc/developers/testing.txt 2010-10-11 22:43:41 +0000
@@ -811,17 +811,11 @@
811whether a test should be added for that particular implementation,811whether a test should be added for that particular implementation,
812or for all implementations of the interface.812or for all implementations of the interface.
813813
814The multiplication of tests for different implementations is normally
815accomplished by overriding the ``load_tests`` function used to load tests
816from a module. This function typically loads all the tests, then applies
817a TestProviderAdapter to them, which generates a longer suite containing
818all the test variations.
819
820See also `Per-implementation tests`_ (above).814See also `Per-implementation tests`_ (above).
821815
822816
823Test scenarios817Test scenarios and variations
824--------------818-----------------------------
825819
826Some utilities are provided for generating variations of tests. This can820Some utilities are provided for generating variations of tests. This can
827be used for per-implementation tests, or other cases where the same test821be used for per-implementation tests, or other cases where the same test
@@ -832,8 +826,24 @@
832values to which the test should be applied. The test suite should then826values to which the test should be applied. The test suite should then
833also provide a list of scenarios in which to run the tests.827also provide a list of scenarios in which to run the tests.
834828
835Typically ``multiply_tests_from_modules`` should be called from the test829A single *scenario* is defined by a `(name, parameter_dict)` tuple. The
836module's ``load_tests`` function.830short string name is combined with the name of the test method to form the
831test instance name. The parameter dict is merged into the instance's
832attributes.
833
834For example::
835
836 load_tests = load_tests_apply_scenarios
837
838 class TestCheckout(TestCase):
839
840 variations = multiply_scenarios(
841 VaryByRepositoryFormat(),
842 VaryByTreeFormat(),
843 )
844
845The `load_tests` declaration or definition should be near the top of the
846file so its effect can be seen.
837847
838848
839Test support849Test support