Bazaar

Merge lp:~lifeless/bzr/test-speed into lp:~bzr/bzr/trunk-old

  1. test-speed
  2. Merge into trunk-old
Proposed by Robert Collins
Status: Merged
Merged at revision: not available
Proposed branch: lp:~lifeless/bzr/test-speed
Merge into: lp:~bzr/bzr/trunk-old
Diff against target: 466 lines
To merge this branch: bzr merge lp:~lifeless/bzr/test-speed
Reviewer Review Type Date Requested Status
Andrew Bennetts Approve
Review via email: mp+10633@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Robert Collins (lifeless) wrote :

This little branch makes it possible to lsprof exactly what a single
test is doing; this can be very useful without requiring the use of a
time() section - though those sections are still supported.

-Rob

Revision history for this message
Andrew Bennetts (spiv) wrote :

Looks fine.

bzrlib/tests/__init__.py
------------------------

ProfileResult's docstring has a typo: "be be" should be "it be".

review: Approve
Revision history for this message
John A Meinel (jameinel) wrote :

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Collins wrote:
> Robert Collins has proposed merging lp:~lifeless/bzr/test-speed into lp:bzr.
>
> Requested reviews:
> bzr-core (bzr-core)
>
> This little branch makes it possible to lsprof exactly what a single
> test is doing; this can be very useful without requiring the use of a
> time() section - though those sections are still supported.
>
> -Rob
>
>

I don't really like this as an overload on top of 'lsprof-timed'. I
realize why you might want it, so I wouldn't really block it. But it
does seem like we are getting a few too many options:

bzr --lsprof --lsprof-file foo selftest --lsprof-timed --lsprof-tests -v
...

I personally understand the difference between all of them, but maybe it
would be better to have it as a data field.

bzr --lsprof=filename,overall
bzr --lsprof=filename,tests
bzr --lsprof=filename,benchmark

...

I haven't worked out a great name, but the current situation seems
pretty ad-hoc and not really where we want to be.

I also would say that writing to a file should be the preferred method,
and you can set '-' as the file if you want to write to the screen. It
seems like it would help reduce clutter.

Of course, I should also say that our option parsing is a bit weak as:
  bzr --lsprof-file foo
works, but
  bzr --lsprof-file=foo
does not.

Anyway, if you find this helpful, I would be okay having it. I just
think it takes us down the road of clutter.

(I did not review the actual content, so I'm not voting yet.)

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkqT6ZwACgkQJdeBCYSNAANMWgCgxa9cIJnDbdG3B6IpQpV8mpRN
5dMAoL3MxE5LCkEgfu2yDCT7Yk5Zth3x
=g0Zd
-----END PGP SIGNATURE-----

Revision history for this message
Robert Collins (lifeless) wrote :

Hi John, I totally agree with all your comments.

In my brain-food slices I'm working on test infrastructure to fix the
test suite performance - its got to get fixed if we're to be productive.

So this branch was aimed towards that goal.

I think there are a few other issues :- namely that it would be good to
output kcachegrind files, it might be good to spin on tests (make
identical scenarios for say 100 copies) to get more timing data, it
would be good to be able to remote this, to have ec2 generate the
traces; and it would be great to fold this into test-coverage maps as
well.

Some of these are just glue, others need more thought because we don't
have canned approaches to solving them and I'd like to fit in with
python's core unittest values and facilities more (though that does
require fixing them upstream as well).

Cheers,
Rob

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'NEWS'
--- NEWS 2009-08-24 18:28:46 +0000
+++ NEWS 2009-08-24 23:35:12 +0000
@@ -67,9 +67,16 @@
67Internals67Internals
68*********68*********
6969
70* The ``bzrlib.lsprof`` module has a new class ``BzrProfiler`` which makes
71 profiling in some situations like callbacks and generators easier.
72 (Robert Collins)
73
70Testing74Testing
71*******75*******
7276
77* Passing ``--lsprof-tests -v`` to bzr selftest will cause lsprof output to
78 be output for every test. Note that this is very verbose! (Robert Collins)
79
73bzr 1.1880bzr 1.18
74########81########
7582
7683
=== modified file 'bzrlib/builtins.py'
--- bzrlib/builtins.py 2009-08-11 18:00:37 +0000
+++ bzrlib/builtins.py 2009-08-24 23:35:12 +0000
@@ -3369,6 +3369,8 @@
3369 Option('lsprof-timed',3369 Option('lsprof-timed',
3370 help='Generate lsprof output for benchmarked'3370 help='Generate lsprof output for benchmarked'
3371 ' sections of code.'),3371 ' sections of code.'),
3372 Option('lsprof-tests',
3373 help='Generate lsprof output for each test.'),
3372 Option('cache-dir', type=str,3374 Option('cache-dir', type=str,
3373 help='Cache intermediate benchmark output in this '3375 help='Cache intermediate benchmark output in this '
3374 'directory.'),3376 'directory.'),
@@ -3415,7 +3417,7 @@
3415 first=False, list_only=False,3417 first=False, list_only=False,
3416 randomize=None, exclude=None, strict=False,3418 randomize=None, exclude=None, strict=False,
3417 load_list=None, debugflag=None, starting_with=None, subunit=False,3419 load_list=None, debugflag=None, starting_with=None, subunit=False,
3418 parallel=None):3420 parallel=None, lsprof_tests=False):
3419 from bzrlib.tests import selftest3421 from bzrlib.tests import selftest
3420 import bzrlib.benchmarks as benchmarks3422 import bzrlib.benchmarks as benchmarks
3421 from bzrlib.benchmarks import tree_creator3423 from bzrlib.benchmarks import tree_creator
@@ -3455,6 +3457,7 @@
3455 "transport": transport,3457 "transport": transport,
3456 "test_suite_factory": test_suite_factory,3458 "test_suite_factory": test_suite_factory,
3457 "lsprof_timed": lsprof_timed,3459 "lsprof_timed": lsprof_timed,
3460 "lsprof_tests": lsprof_tests,
3458 "bench_history": benchfile,3461 "bench_history": benchfile,
3459 "matching_tests_first": first,3462 "matching_tests_first": first,
3460 "list_only": list_only,3463 "list_only": list_only,
34613464
=== modified file 'bzrlib/lsprof.py'
--- bzrlib/lsprof.py 2009-03-08 06:18:06 +0000
+++ bzrlib/lsprof.py 2009-08-24 23:35:12 +0000
@@ -13,45 +13,74 @@
1313
14__all__ = ['profile', 'Stats']14__all__ = ['profile', 'Stats']
1515
16_g_threadmap = {}
17
18
19def _thread_profile(f, *args, **kwds):
20 # we lose the first profile point for a new thread in order to trampoline
21 # a new Profile object into place
22 global _g_threadmap
23 thr = thread.get_ident()
24 _g_threadmap[thr] = p = Profiler()
25 # this overrides our sys.setprofile hook:
26 p.enable(subcalls=True, builtins=True)
27
28
29def profile(f, *args, **kwds):16def profile(f, *args, **kwds):
30 """Run a function profile.17 """Run a function profile.
3118
32 Exceptions are not caught: If you need stats even when exceptions are to be19 Exceptions are not caught: If you need stats even when exceptions are to be
33 raised, passing in a closure that will catch the exceptions and transform20 raised, pass in a closure that will catch the exceptions and transform them
34 them appropriately for your driver function.21 appropriately for your driver function.
3522
36 :return: The functions return value and a stats object.23 :return: The functions return value and a stats object.
37 """24 """
38 global _g_threadmap25 profiler = BzrProfiler()
39 p = Profiler()26 profiler.start()
40 p.enable(subcalls=True)
41 threading.setprofile(_thread_profile)
42 try:27 try:
43 ret = f(*args, **kwds)28 ret = f(*args, **kwds)
44 finally:29 finally:
45 p.disable()30 stats = profiler.stop()
46 for pp in _g_threadmap.values():31 return ret, stats
32
33
34class BzrProfiler(object):
35 """Bzr utility wrapper around Profiler.
36
37 For most uses the module level 'profile()' function will be suitable.
38 However profiling when a simple wrapped function isn't available may
39 be easier to accomplish using this class.
40
41 To use it, create a BzrProfiler and call start() on it. Some arbitrary
42 time later call stop() to stop profiling and retrieve the statistics
43 from the code executed in the interim.
44 """
45
46 def start(self):
47 """Start profiling.
48
49 This hooks into threading and will record all calls made until
50 stop() is called.
51 """
52 self._g_threadmap = {}
53 self.p = Profiler()
54 self.p.enable(subcalls=True)
55 threading.setprofile(self._thread_profile)
56
57 def stop(self):
58 """Stop profiling.
59
60 This unhooks from threading and cleans up the profiler, returning
61 the gathered Stats object.
62
63 :return: A bzrlib.lsprof.Stats object.
64 """
65 self.p.disable()
66 for pp in self._g_threadmap.values():
47 pp.disable()67 pp.disable()
48 threading.setprofile(None)68 threading.setprofile(None)
69 p = self.p
70 self.p = None
71 threads = {}
72 for tid, pp in self._g_threadmap.items():
73 threads[tid] = Stats(pp.getstats(), {})
74 self._g_threadmap = None
75 return Stats(p.getstats(), threads)
4976
50 threads = {}77 def _thread_profile(self, f, *args, **kwds):
51 for tid, pp in _g_threadmap.items():78 # we lose the first profile point for a new thread in order to
52 threads[tid] = Stats(pp.getstats(), {})79 # trampoline a new Profile object into place
53 _g_threadmap = {}80 thr = thread.get_ident()
54 return ret, Stats(p.getstats(), threads)81 self._g_threadmap[thr] = p = Profiler()
82 # this overrides our sys.setprofile hook:
83 p.enable(subcalls=True, builtins=True)
5584
5685
57class Stats(object):86class Stats(object):
5887
=== modified file 'bzrlib/tests/__init__.py'
--- bzrlib/tests/__init__.py 2009-08-24 05:23:11 +0000
+++ bzrlib/tests/__init__.py 2009-08-24 23:35:12 +0000
@@ -579,13 +579,22 @@
579 bench_history=None,579 bench_history=None,
580 list_only=False,580 list_only=False,
581 strict=False,581 strict=False,
582 result_decorators=None,
582 ):583 ):
584 """Create a TextTestRunner.
585
586 :param result_decorators: An optional list of decorators to apply
587 to the result object being used by the runner. Decorators are
588 applied left to right - the first element in the list is the
589 innermost decorator.
590 """
583 self.stream = unittest._WritelnDecorator(stream)591 self.stream = unittest._WritelnDecorator(stream)
584 self.descriptions = descriptions592 self.descriptions = descriptions
585 self.verbosity = verbosity593 self.verbosity = verbosity
586 self._bench_history = bench_history594 self._bench_history = bench_history
587 self.list_only = list_only595 self.list_only = list_only
588 self._strict = strict596 self._strict = strict
597 self._result_decorators = result_decorators or []
589598
590 def run(self, test):599 def run(self, test):
591 "Run the given test case or test suite."600 "Run the given test case or test suite."
@@ -600,6 +609,9 @@
600 bench_history=self._bench_history,609 bench_history=self._bench_history,
601 strict=self._strict,610 strict=self._strict,
602 )611 )
612 run_result = result
613 for decorator in self._result_decorators:
614 run_result = decorator(run_result)
603 result.stop_early = self.stop_on_failure615 result.stop_early = self.stop_on_failure
604 result.report_starting()616 result.report_starting()
605 if self.list_only:617 if self.list_only:
@@ -614,13 +626,13 @@
614 try:626 try:
615 import testtools627 import testtools
616 except ImportError:628 except ImportError:
617 test.run(result)629 test.run(run_result)
618 else:630 else:
619 if isinstance(test, testtools.ConcurrentTestSuite):631 if isinstance(test, testtools.ConcurrentTestSuite):
620 # We need to catch bzr specific behaviors632 # We need to catch bzr specific behaviors
621 test.run(BZRTransformingResult(result))633 test.run(BZRTransformingResult(run_result))
622 else:634 else:
623 test.run(result)635 test.run(run_result)
624 run = result.testsRun636 run = result.testsRun
625 actionTaken = "Ran"637 actionTaken = "Ran"
626 stopTime = time.time()638 stopTime = time.time()
@@ -2762,7 +2774,9 @@
2762 strict=False,2774 strict=False,
2763 runner_class=None,2775 runner_class=None,
2764 suite_decorators=None,2776 suite_decorators=None,
2765 stream=None):2777 stream=None,
2778 result_decorators=None,
2779 ):
2766 """Run a test suite for bzr selftest.2780 """Run a test suite for bzr selftest.
27672781
2768 :param runner_class: The class of runner to use. Must support the2782 :param runner_class: The class of runner to use. Must support the
@@ -2785,6 +2799,7 @@
2785 bench_history=bench_history,2799 bench_history=bench_history,
2786 list_only=list_only,2800 list_only=list_only,
2787 strict=strict,2801 strict=strict,
2802 result_decorators=result_decorators,
2788 )2803 )
2789 runner.stop_on_failure=stop_on_failure2804 runner.stop_on_failure=stop_on_failure
2790 # built in decorator factories:2805 # built in decorator factories:
@@ -3131,7 +3146,7 @@
3131 return result3146 return result
31323147
31333148
3134class BZRTransformingResult(unittest.TestResult):3149class ForwardingResult(unittest.TestResult):
31353150
3136 def __init__(self, target):3151 def __init__(self, target):
3137 unittest.TestResult.__init__(self)3152 unittest.TestResult.__init__(self)
@@ -3143,6 +3158,21 @@
3143 def stopTest(self, test):3158 def stopTest(self, test):
3144 self.result.stopTest(test)3159 self.result.stopTest(test)
31453160
3161 def addSkip(self, test, reason):
3162 self.result.addSkip(test, reason)
3163
3164 def addSuccess(self, test):
3165 self.result.addSuccess(test)
3166
3167 def addError(self, test, err):
3168 self.result.addError(test, err)
3169
3170 def addFailure(self, test, err):
3171 self.result.addFailure(test, err)
3172
3173
3174class BZRTransformingResult(ForwardingResult):
3175
3146 def addError(self, test, err):3176 def addError(self, test, err):
3147 feature = self._error_looks_like('UnavailableFeature: ', err)3177 feature = self._error_looks_like('UnavailableFeature: ', err)
3148 if feature is not None:3178 if feature is not None:
@@ -3158,12 +3188,6 @@
3158 else:3188 else:
3159 self.result.addFailure(test, err)3189 self.result.addFailure(test, err)
31603190
3161 def addSkip(self, test, reason):
3162 self.result.addSkip(test, reason)
3163
3164 def addSuccess(self, test):
3165 self.result.addSuccess(test)
3166
3167 def _error_looks_like(self, prefix, err):3191 def _error_looks_like(self, prefix, err):
3168 """Deserialize exception and returns the stringify value."""3192 """Deserialize exception and returns the stringify value."""
3169 import subunit3193 import subunit
@@ -3181,6 +3205,38 @@
3181 return value3205 return value
31823206
31833207
3208class ProfileResult(ForwardingResult):
3209 """Generate profiling data for all activity between start and success.
3210
3211 The profile data is appended to the test's _benchcalls attribute and can
3212 be accessed by the forwarded-to TestResult.
3213
3214 While it might be cleaner do accumulate this in stopTest, addSuccess is
3215 where our existing output support for lsprof is, and this class aims to
3216 fit in with that: while it could be moved it's not necessary to accomplish
3217 test profiling, nor would be be dramatically cleaner.
3218 """
3219
3220 def startTest(self, test):
3221 self.profiler = bzrlib.lsprof.BzrProfiler()
3222 self.profiler.start()
3223 ForwardingResult.startTest(self, test)
3224
3225 def addSuccess(self, test):
3226 stats = self.profiler.stop()
3227 try:
3228 calls = test._benchcalls
3229 except AttributeError:
3230 test._benchcalls = []
3231 calls = test._benchcalls
3232 calls.append(((test.id(), "", ""), stats))
3233 ForwardingResult.addSuccess(self, test)
3234
3235 def stopTest(self, test):
3236 ForwardingResult.stopTest(self, test)
3237 self.profiler = None
3238
3239
3184# Controlled by "bzr selftest -E=..." option3240# Controlled by "bzr selftest -E=..." option
3185# Currently supported:3241# Currently supported:
3186# -Eallow_debug Will no longer clear debug.debug_flags() so it3242# -Eallow_debug Will no longer clear debug.debug_flags() so it
@@ -3208,6 +3264,7 @@
3208 runner_class=None,3264 runner_class=None,
3209 suite_decorators=None,3265 suite_decorators=None,
3210 stream=None,3266 stream=None,
3267 lsprof_tests=False,
3211 ):3268 ):
3212 """Run the whole test suite under the enhanced runner"""3269 """Run the whole test suite under the enhanced runner"""
3213 # XXX: Very ugly way to do this...3270 # XXX: Very ugly way to do this...
@@ -3239,6 +3296,9 @@
3239 if starting_with:3296 if starting_with:
3240 # But always filter as requested.3297 # But always filter as requested.
3241 suite = filter_suite_by_id_startswith(suite, starting_with)3298 suite = filter_suite_by_id_startswith(suite, starting_with)
3299 result_decorators = []
3300 if lsprof_tests:
3301 result_decorators.append(ProfileResult)
3242 return run_suite(suite, 'testbzr', verbose=verbose, pattern=pattern,3302 return run_suite(suite, 'testbzr', verbose=verbose, pattern=pattern,
3243 stop_on_failure=stop_on_failure,3303 stop_on_failure=stop_on_failure,
3244 transport=transport,3304 transport=transport,
@@ -3252,6 +3312,7 @@
3252 runner_class=runner_class,3312 runner_class=runner_class,
3253 suite_decorators=suite_decorators,3313 suite_decorators=suite_decorators,
3254 stream=stream,3314 stream=stream,
3315 result_decorators=result_decorators,
3255 )3316 )
3256 finally:3317 finally:
3257 default_transport = old_transport3318 default_transport = old_transport
32583319
=== modified file 'bzrlib/tests/blackbox/test_selftest.py'
--- bzrlib/tests/blackbox/test_selftest.py 2009-08-24 05:23:11 +0000
+++ bzrlib/tests/blackbox/test_selftest.py 2009-08-24 23:35:12 +0000
@@ -172,3 +172,7 @@
172 outputs_nothing(['selftest', '--list-only', '--exclude', 'selftest'])172 outputs_nothing(['selftest', '--list-only', '--exclude', 'selftest'])
173 finally:173 finally:
174 tests.selftest = original_selftest174 tests.selftest = original_selftest
175
176 def test_lsprof_tests(self):
177 params = self.get_params_passed_to_core('selftest --lsprof-tests')
178 self.assertEqual(True, params[1]["lsprof_tests"])
175179
=== modified file 'bzrlib/tests/test_lsprof.py'
--- bzrlib/tests/test_lsprof.py 2009-03-23 14:59:43 +0000
+++ bzrlib/tests/test_lsprof.py 2009-08-24 23:35:12 +0000
@@ -92,3 +92,22 @@
92 self.stats.save(f)92 self.stats.save(f)
93 data1 = cPickle.load(open(f))93 data1 = cPickle.load(open(f))
94 self.assertEqual(type(data1), bzrlib.lsprof.Stats)94 self.assertEqual(type(data1), bzrlib.lsprof.Stats)
95
96
97class TestBzrProfiler(tests.TestCase):
98
99 _test_needs_features = [LSProfFeature]
100
101 def test_start_call_stuff_stop(self):
102 profiler = bzrlib.lsprof.BzrProfiler()
103 profiler.start()
104 try:
105 def a_function():
106 pass
107 a_function()
108 finally:
109 stats = profiler.stop()
110 stats.freeze()
111 lines = [str(data) for data in stats.data]
112 lines = [line for line in lines if 'a_function' in line]
113 self.assertLength(1, lines)
95114
=== modified file 'bzrlib/tests/test_selftest.py'
--- bzrlib/tests/test_selftest.py 2009-08-24 05:35:28 +0000
+++ bzrlib/tests/test_selftest.py 2009-08-24 23:35:12 +0000
@@ -687,6 +687,25 @@
687 self.assertEqual(url, t.clone('..').base)687 self.assertEqual(url, t.clone('..').base)
688688
689689
690class TestProfileResult(tests.TestCase):
691
692 def test_profiles_tests(self):
693 terminal = unittest.TestResult()
694 result = tests.ProfileResult(terminal)
695 class Sample(tests.TestCase):
696 def a(self):
697 self.sample_function()
698 def sample_function(self):
699 pass
700 test = Sample("a")
701 test.attrs_to_keep = test.attrs_to_keep + ('_benchcalls',)
702 test.run(result)
703 self.assertLength(1, test._benchcalls)
704 # We must be able to unpack it as the test reporting code wants
705 (_, _, _), stats = test._benchcalls[0]
706 self.assertTrue(callable(stats.pprint))
707
708
690class TestTestResult(tests.TestCase):709class TestTestResult(tests.TestCase):
691710
692 def check_timing(self, test_case, expected_re):711 def check_timing(self, test_case, expected_re):
@@ -1031,6 +1050,20 @@
1031 '\n'1050 '\n'
1032 'OK \\(known_failures=1\\)\n')1051 'OK \\(known_failures=1\\)\n')
10331052
1053 def test_result_decorator(self):
1054 # decorate results
1055 calls = []
1056 class LoggingDecorator(tests.ForwardingResult):
1057 def startTest(self, test):
1058 tests.ForwardingResult.startTest(self, test)
1059 calls.append('start')
1060 test = unittest.FunctionTestCase(lambda:None)
1061 stream = StringIO()
1062 runner = tests.TextTestRunner(stream=stream,
1063 result_decorators=[LoggingDecorator])
1064 result = self.run_test_runner(runner, test)
1065 self.assertLength(1, calls)
1066
1034 def test_skipped_test(self):1067 def test_skipped_test(self):
1035 # run a test that is skipped, and check the suite as a whole still1068 # run a test that is skipped, and check the suite as a whole still
1036 # succeeds.1069 # succeeds.
@@ -1103,10 +1136,6 @@
1103 self.assertContainsRe(out.getvalue(),1136 self.assertContainsRe(out.getvalue(),
1104 r'(?m)^ this test never runs')1137 r'(?m)^ this test never runs')
11051138
1106 def test_not_applicable_demo(self):
1107 # just so you can see it in the test output
1108 raise tests.TestNotApplicable('this test is just a demonstation')
1109
1110 def test_unsupported_features_listed(self):1139 def test_unsupported_features_listed(self):
1111 """When unsupported features are encountered they are detailed."""1140 """When unsupported features are encountered they are detailed."""
1112 class Feature1(tests.Feature):1141 class Feature1(tests.Feature):
@@ -1480,6 +1509,7 @@
1480 self.assertEqual((time.sleep, (0.003,), {}), self._benchcalls[1][0])1509 self.assertEqual((time.sleep, (0.003,), {}), self._benchcalls[1][0])
1481 self.assertIsInstance(self._benchcalls[0][1], bzrlib.lsprof.Stats)1510 self.assertIsInstance(self._benchcalls[0][1], bzrlib.lsprof.Stats)
1482 self.assertIsInstance(self._benchcalls[1][1], bzrlib.lsprof.Stats)1511 self.assertIsInstance(self._benchcalls[1][1], bzrlib.lsprof.Stats)
1512 del self._benchcalls[:]
14831513
1484 def test_knownFailure(self):1514 def test_knownFailure(self):
1485 """Self.knownFailure() should raise a KnownFailure exception."""1515 """Self.knownFailure() should raise a KnownFailure exception."""
@@ -1817,6 +1847,19 @@
1817 self.assertNotContainsRe("Test.b", output.getvalue())1847 self.assertNotContainsRe("Test.b", output.getvalue())
1818 self.assertLength(2, output.readlines())1848 self.assertLength(2, output.readlines())
18191849
1850 def test_lsprof_tests(self):
1851 calls = []
1852 class Test(object):
1853 def __call__(test, result):
1854 test.run(result)
1855 def run(test, result):
1856 self.assertIsInstance(result, tests.ForwardingResult)
1857 calls.append("called")
1858 def countTestCases(self):
1859 return 1
1860 self.run_selftest(test_suite_factory=Test, lsprof_tests=True)
1861 self.assertLength(1, calls)
1862
1820 def test_random(self):1863 def test_random(self):
1821 # test randomising by listing a number of tests.1864 # test randomising by listing a number of tests.
1822 output_123 = self.run_selftest(test_suite_factory=self.factory,1865 output_123 = self.run_selftest(test_suite_factory=self.factory,