Merge lp:~allenap/launchpad/isolate-tests into lp:launchpad

Proposed by Gavin Panella
Status: Merged
Approved by: Gavin Panella
Approved revision: no longer in the source branch.
Merged at revision: not available
Proposed branch: lp:~allenap/launchpad/isolate-tests
Merge into: lp:launchpad
Diff against target: 319 lines (+217/-15)
4 files modified
lib/lp/services/job/runner.py (+3/-2)
lib/lp/services/job/tests/test_runner.py (+15/-13)
lib/lp/testing/__init__.py (+68/-0)
lib/lp/testing/tests/test_zope_test_in_subprocess.py (+131/-0)
To merge this branch: bzr merge lp:~allenap/launchpad/isolate-tests
Reviewer Review Type Date Requested Status
Jonathan Lange (community) Approve
Björn Tillenius (community) Abstain
Review via email: mp+20916@code.launchpad.net

Commit message

New class, ZopeTestInSubProcess, a mixin to run tests in forked sub-processes.

Description of the change

Implement a new class - ZopeIsolatedTestCase - that can be used as a mixin with more interesting TestCase subclasses. It runs each test method in a forked sub-process, passing the results back to the parent process with subunit. This is needed to safely run tests that need to, for example, start the Twisted reactor, which cannot be safely restarted in the same process once stopped.

This branch also uses ZopeIsolatedTestCase with a couple of job runner tests. Additionally, one of these tests, TestJobCronScript, has been fixed to override the command line arguments passed to the script class (otherwise sys.argv is used, which is stuffed with arguments passed to bin/test).

To post a comment you must log in.
Revision history for this message
Björn Tillenius (bjornt) wrote :
Download full text (5.2 KiB)

On Mon, Mar 08, 2010 at 06:14:32PM -0000, Gavin Panella wrote:
> Gavin Panella has proposed merging lp:~allenap/launchpad/isolate-tests into lp:launchpad/devel.
>
> Requested reviews:
> Canonical Launchpad Engineering (launchpad): code
> Related bugs:
> #491870 Use Twisted's thread support instead of the threading module in checkwatches
> https://bugs.launchpad.net/bugs/491870
>
>
> Implement a new class - ZopeIsolatedTestCase - that can be used as a
> mixin with more interesting TestCase subclasses. It runs each test
> method in a forked sub-process, passing the results back to the parent
> process with subunit. This is needed to safely run tests that need to,
> for example, start the Twisted reactor, which cannot be safely restarted
> in the same process once stopped.
>
> This branch also uses ZopeIsolatedTestCase with a couple of job runner
> tests. Additionally, one of these tests, TestJobCronScript, has been
> fixed to override the command line arguments passed to the script class
> (otherwise sys.argv is used, which is stuffed with arguments passed to
> bin/test).

    vote needsinfo

If the job runner tests ran fine previously, why is this
ZopeIsolatedTestCase needed?

> === modified file 'lib/lp/services/job/tests/test_runner.py'
> --- lib/lp/services/job/tests/test_runner.py 2010-02-24 19:52:01 +0000
> +++ lib/lp/services/job/tests/test_runner.py 2010-03-08 18:14:32 +0000

> @@ -300,7 +301,7 @@
> self.entries.append(input)
>
>
> -class TestTwistedJobRunner(TestCaseWithFactory):
> +class TestTwistedJobRunner(ZopeIsolatedTestCase, TestCaseWithFactory):
>
> layer = LaunchpadZopelessLayer
>
> @@ -325,7 +326,7 @@
> self.assertIn('Job ran too long.', oops.value)
>
>
> -class TestJobCronScript(TestCaseWithFactory):
> +class TestJobCronScript(ZopeIsolatedTestCase, TestCaseWithFactory):

Will we ever use ZopeIsolatedTestCase without mixing in
TestCaseWithFactory?

> === modified file 'lib/lp/testing/__init__.py'
> --- lib/lp/testing/__init__.py 2010-02-17 20:54:36 +0000
> +++ lib/lp/testing/__init__.py 2010-03-08 18:14:32 +0000
> @@ -586,6 +591,56 @@
> self.client.open(url=u'http://launchpad.dev:8085')
>
>
> +class ZopeIsolatedTestCase(unittest.TestCase):
> + """Run tests in a sub-process, respecting Zope idiosyncrasies.
> +
> + Use this as a baseclass for test cases, or as a mixin with an
> + interesting `TestCase` subclass.
> +
> + This is basically a reimplementation of subunit's
> + `IsolatedTestCase`, but adjusted to work with Zope. In particular,
> + Zope's TestResult object is responsible for calling testSetUp()
> + and testTearDown() on the selected layer.
> + """

I can see why you named it ZopeIsolatedTestCase, but is that really a
good name? I mean, don't TestCase classes in general run their tests in
an isolated environment?

> +
> + def run(self, result):
> + assert isinstance(result, ZopeTestResult), (
> + "result must be a Zope result object, not %r." % (result,))
> + pread, pwrite = os.pipe()
> + pid = os.fork()
> + if pid == 0:
> + # Child.
> + os.close(pread)
> + ...

Read more...

review: Needs Information
Revision history for this message
Gavin Panella (allenap) wrote :
Download full text (7.4 KiB)

> On Mon, Mar 08, 2010 at 06:14:32PM -0000, Gavin Panella wrote:
> > Gavin Panella has proposed merging lp:~allenap/launchpad/isolate-tests into
> lp:launchpad/devel.
> >
> > Requested reviews:
> > Canonical Launchpad Engineering (launchpad): code
> > Related bugs:
> > #491870 Use Twisted's thread support instead of the threading module in
> checkwatches
> > https://bugs.launchpad.net/bugs/491870
> >
> >
> > Implement a new class - ZopeIsolatedTestCase - that can be used as a
> > mixin with more interesting TestCase subclasses. It runs each test
> > method in a forked sub-process, passing the results back to the parent
> > process with subunit. This is needed to safely run tests that need to,
> > for example, start the Twisted reactor, which cannot be safely restarted
> > in the same process once stopped.
> >
> > This branch also uses ZopeIsolatedTestCase with a couple of job runner
> > tests. Additionally, one of these tests, TestJobCronScript, has been
> > fixed to override the command line arguments passed to the script class
> > (otherwise sys.argv is used, which is stuffed with arguments passed to
> > bin/test).
>
> vote needsinfo
>
> If the job runner tests ran fine previously, why is this
> ZopeIsolatedTestCase needed?

Jono once told me - and an IRC discussion between Aaron, Jono and
myself yesterday reconfirmed this - that reactor restarting is not
supported, and is considered by Twisted developers to not be a working
feature right now. They want it to work, but it's not high on the
list. Jono suggested subunit.IsolatedTestCase as a possible solution.

Apart from that, I discovered than one of my new tests for
checkwatches ran fine on its own, or when run alongside many other
tests, but hung indefinitely when run in the same process as the job
runner tests.

> > === modified file 'lib/lp/services/job/tests/test_runner.py'
> > --- lib/lp/services/job/tests/test_runner.py 2010-02-24 19:52:01 +0000
> > +++ lib/lp/services/job/tests/test_runner.py 2010-03-08 18:14:32 +0000
>
> > @@ -300,7 +301,7 @@
> > self.entries.append(input)
> >
> >
> > -class TestTwistedJobRunner(TestCaseWithFactory):
> > +class TestTwistedJobRunner(ZopeIsolatedTestCase, TestCaseWithFactory):
> >
> > layer = LaunchpadZopelessLayer
> >
> > @@ -325,7 +326,7 @@
> > self.assertIn('Job ran too long.', oops.value)
> >
> >
> > -class TestJobCronScript(TestCaseWithFactory):
> > +class TestJobCronScript(ZopeIsolatedTestCase, TestCaseWithFactory):
>
> Will we ever use ZopeIsolatedTestCase without mixing in
> TestCaseWithFactory?

Yes, possibly. I've already made some changes to this branch since the
merge proposal went out, one of which was to not inherit from
anything. The reason is that it can be mixed in with TestSuite (and
its subclasses) too, without alteration, to provide process isolation
for a whole suite. I've updated the docs to explain the behaviour.

Full diff of changes since the proposal:
  http://paste.ubuntu.com/391864/

> > === modified file 'lib/lp/testing/__init__.py'
> > --- lib/lp/testing/__init__.py 2010-02-17 20:54:36 +0000
> > +++ lib/lp/testing/__init__.py 2010-03-08 18:14:32 +0000
> > @@ -586,6 ...

Read more...

Revision history for this message
Jonathan Lange (jml) wrote :

2010/3/9 Gavin Panella <email address hidden>:
>> On Mon, Mar 08, 2010 at 06:14:32PM -0000, Gavin Panella wrote:
>> > Gavin Panella has proposed merging lp:~allenap/launchpad/isolate-tests into
>> lp:launchpad/devel.
>> >
>> > Requested reviews:
>> >   Canonical Launchpad Engineering (launchpad): code
>> > Related bugs:
>> >   #491870 Use Twisted's thread support instead of the threading module in
>> checkwatches
>> >   https://bugs.launchpad.net/bugs/491870
>> >
>> >
>> > Implement a new class - ZopeIsolatedTestCase - that can be used as a
>> > mixin with more interesting TestCase subclasses. It runs each test
>> > method in a forked sub-process, passing the results back to the parent
>> > process with subunit. This is needed to safely run tests that need to,
>> > for example, start the Twisted reactor, which cannot be safely restarted
>> >    in the same process once stopped.
>> >
>> > This branch also uses ZopeIsolatedTestCase with a couple of job runner
>> > tests. Additionally, one of these tests, TestJobCronScript, has been
>> > fixed to override the command line arguments passed to the script class
>> > (otherwise sys.argv is used, which is stuffed with arguments passed to
>> > bin/test).
>>
>>     vote needsinfo
>>
>> If the job runner tests ran fine previously, why is this
>> ZopeIsolatedTestCase needed?
>
> Jono once told me - and an IRC discussion between Aaron, Jono and
> myself yesterday reconfirmed this - that reactor restarting is not
> supported, and is considered by Twisted developers to not be a working
> feature right now. They want it to work, but it's not high on the
> list. Jono suggested subunit.IsolatedTestCase as a possible solution.
>

To be clear, this kind of solution (run tests in their own
subprocesses) is hardly ever used within the Twisted world, since it's
considered good practice there for asynchronous APIs to return
Deferreds, and to test things at that level. It's extremely rare to
have application code that needs to start and stop the reactor.

jml

Revision history for this message
Björn Tillenius (bjornt) wrote :
Download full text (6.1 KiB)

On Tue, Mar 09, 2010 at 04:23:44PM -0000, Gavin Panella wrote:
> > > -class TestJobCronScript(TestCaseWithFactory):
> > > +class TestJobCronScript(ZopeIsolatedTestCase, TestCaseWithFactory):
> >
> > Will we ever use ZopeIsolatedTestCase without mixing in
> > TestCaseWithFactory?
>
> Yes, possibly. I've already made some changes to this branch since the
> merge proposal went out, one of which was to not inherit from
> anything. The reason is that it can be mixed in with TestSuite (and
> its subclasses) too, without alteration, to provide process isolation
> for a whole suite. I've updated the docs to explain the behaviour.
>
> Full diff of changes since the proposal:
> http://paste.ubuntu.com/391864/

In the future, could you please attach the changes instead? Makes it
easier to review and comment.

>
> > > === modified file 'lib/lp/testing/__init__.py'
> > > --- lib/lp/testing/__init__.py 2010-02-17 20:54:36 +0000
> > > +++ lib/lp/testing/__init__.py 2010-03-08 18:14:32 +0000
> > > @@ -586,6 +591,56 @@
> > > self.client.open(url=u'http://launchpad.dev:8085')
> > >
> > >
> > > +class ZopeIsolatedTestCase(unittest.TestCase):
> > > + """Run tests in a sub-process, respecting Zope idiosyncrasies.
> > > +
> > > + Use this as a baseclass for test cases, or as a mixin with an
> > > + interesting `TestCase` subclass.
> > > +
> > > + This is basically a reimplementation of subunit's
> > > + `IsolatedTestCase`, but adjusted to work with Zope. In particular,
> > > + Zope's TestResult object is responsible for calling testSetUp()
> > > + and testTearDown() on the selected layer.
> > > + """
> >
> > I can see why you named it ZopeIsolatedTestCase, but is that really a
> > good name? I mean, don't TestCase classes in general run their tests in
> > an isolated environment?
>
> ZopeIsolatedTestCase is now known as ZopeIsolatedTest. The original
> name form came from subunit.IsolatedTestCase, which does a very
> similar job. I'm happy to rename it though.

I would probably have SubProcess, or Fork, in the name, to make it more
clear in what way it provides the isolation.

> > > + def run(self, result):
> > > + assert isinstance(result, ZopeTestResult), (
> > > + "result must be a Zope result object, not %r." % (result,))
> > > + pread, pwrite = os.pipe()
> > > + pid = os.fork()
> > > + if pid == 0:
> > > + # Child.
> > > + os.close(pread)
> > > + fdwrite = os.fdopen(pwrite, 'w', 1)
> > > + # Send results to both the Zope result object (so that
> > > + # layer setup and teardown are done properly, etc.) and to
> > > + # the subunit stream client so that the parent process can
> > > + # obtain the result.
> > > + result = testtools.MultiTestResult(
> > > + result, subunit.TestProtocolClient(fdwrite))
> > > + super(ZopeIsolatedTestCase, self).run(result)
> > > + fdwrite.flush()
> > > + sys.stdout.flush()
> > > + sys.stderr.flush()
> > > + # Exit hard.
> > > + os._exit(0)
> > > + else:
> > > +...

Read more...

Revision history for this message
Gavin Panella (allenap) wrote :

On Wed, 10 Mar 2010 14:15:31 -0000
Björn Tillenius <email address hidden> wrote:

...
> > ZopeIsolatedTestCase is now known as ZopeIsolatedTest. The original
> > name form came from subunit.IsolatedTestCase, which does a very
> > similar job. I'm happy to rename it though.
>
> I would probably have SubProcess, or Fork, in the name, to make it more
> clear in what way it provides the isolation.

It's now ZopeTestInSubProcess.

...
> > > I'm a bit interested in how this works with layers. Would you run away
> > > screaming if I asked you to add some tests demonstrating this? :)

Have a look at the new unit test I've added.

Basically, layer setUp() and tearDown() get called in the parent,
testSetUp() and testTearDown() in the child. For the test case,
__init__() is called in the parent, setUp(), tearDown() and test
methods are called in the child.

> > > It'd be interesting to see in which process the setUp(), testSetUp(),
> > > tearDown(), and testTearDown() methods of the layer are executed. I'm
> > > wondering if we can see subtle errors due to doing the wrong things in
> > > there.

Yes, there probably will be incompatibilities with some layers :-/

...
> > If it's mixed-in with a TestSuite I think it gets more complicated.

I haven't written tests for this yet. I may not do so (and restrict
ZopeTestInSubProcess to just TestCase for now).

...
> > > Also, how this this TestClass related to TwistedLayer? Oh right, that
> > > leads me to another question. You say that the reactor can't be
> > > restarted easily. But can it be shared between tests, just like the
> > > component architecture?
> >
> > That's an interesting idea. reactor.run() blocks, but I don't know of
> > any reason why it couldn't be started in a thread. I might get sent to
> > hell for it, but at least it's warmer than here.
>
> If it could, than maybe it would be better to use a layer instead, since
> that what we use elsewhere.

I haven't thought about this yet, but I will.

> Some other questions popped into my mind. What happens if you specify
> -D, and a test in the forked process fails?

This works as normal; you get a pdb prompt in the right place in the
child process.

> How well does this play with profiling? Do you still get all profiling
> information if you specify --profile=cProfile?

Not tried this yet. Not actually sure how to test this; it's a very
long time since I did any profiling.

So, I still have some work to do!

Gavin.

=== modified file 'lib/lp/services/job/tests/test_runner.py'
--- lib/lp/services/job/tests/test_runner.py 2010-03-09 14:28:11 +0000
+++ lib/lp/services/job/tests/test_runner.py 2010-03-11 16:36:43 +0000
@@ -27,7 +27,7 @@
27from lp.services.job.model.job import Job27from lp.services.job.model.job import Job
28from lp.services.job.runner import (28from lp.services.job.runner import (
29 BaseRunnableJob, JobCronScript, JobRunner, TwistedJobRunner)29 BaseRunnableJob, JobCronScript, JobRunner, TwistedJobRunner)
30from lp.testing import TestCaseWithFactory, ZopeIsolatedTest30from lp.testing import TestCaseWithFactory, ZopeTestInSubProcess
31from lp.testing.mail_helpers import pop_notifications31from lp.testing.mail_helpers import pop_notifications
3232
3333
@@ -301,7 +301,7 @@
301 self.entries.append(input)301 self.entries.append(input)
302302
303303
304class TestTwistedJobRunner(ZopeIsolatedTest, TestCaseWithFactory):304class TestTwistedJobRunner(ZopeTestInSubProcess, TestCaseWithFactory):
305305
306 layer = LaunchpadZopelessLayer306 layer = LaunchpadZopelessLayer
307307
@@ -326,7 +326,7 @@
326 self.assertIn('Job ran too long.', oops.value)326 self.assertIn('Job ran too long.', oops.value)
327327
328328
329class TestJobCronScript(ZopeIsolatedTest, TestCaseWithFactory):329class TestJobCronScript(ZopeTestInSubProcess, TestCaseWithFactory):
330330
331 layer = LaunchpadZopelessLayer331 layer = LaunchpadZopelessLayer
332332
333333
=== modified file 'lib/lp/testing/__init__.py'
--- lib/lp/testing/__init__.py 2010-03-09 14:28:11 +0000
+++ lib/lp/testing/__init__.py 2010-03-11 16:36:43 +0000
@@ -29,7 +29,7 @@
29 'validate_mock_class',29 'validate_mock_class',
30 'WindmillTestCase',30 'WindmillTestCase',
31 'with_anonymous_login',31 'with_anonymous_login',
32 'ZopeIsolatedTest',32 'ZopeTestInSubProcess',
33 ]33 ]
3434
35import copy35import copy
@@ -590,7 +590,7 @@
590 self.client.open(url=u'http://launchpad.dev:8085')590 self.client.open(url=u'http://launchpad.dev:8085')
591591
592592
593class ZopeIsolatedTest:593class ZopeTestInSubProcess:
594 """Run tests in a sub-process, respecting Zope idiosyncrasies.594 """Run tests in a sub-process, respecting Zope idiosyncrasies.
595595
596 Use this as a mixin with an interesting `TestCase` or `TestSuite`596 Use this as a mixin with an interesting `TestCase` or `TestSuite`
@@ -631,7 +631,7 @@
631 # obtain the result.631 # obtain the result.
632 result = testtools.MultiTestResult(632 result = testtools.MultiTestResult(
633 result, subunit.TestProtocolClient(fdwrite))633 result, subunit.TestProtocolClient(fdwrite))
634 super(ZopeIsolatedTest, self).run(result)634 super(ZopeTestInSubProcess, self).run(result)
635 fdwrite.flush()635 fdwrite.flush()
636 sys.stdout.flush()636 sys.stdout.flush()
637 sys.stderr.flush()637 sys.stderr.flush()
638638
=== added file 'lib/lp/testing/tests/test_zope_test_in_subprocess.py'
--- lib/lp/testing/tests/test_zope_test_in_subprocess.py 1970-01-01 00:00:00 +0000
+++ lib/lp/testing/tests/test_zope_test_in_subprocess.py 2010-03-11 21:36:45 +0000
@@ -0,0 +1,118 @@
1# Copyright 2010 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""Test `lp.testing.ZopeTestInSubProcess`."""
5
6__metaclass__ = type
7
8import functools
9import os
10import unittest
11
12from lp.testing import ZopeTestInSubProcess
13
14
15def record_pid(method):
16 """Decorator that records the pid at method invocation.
17
18 Will probably only DTRT with class methods or bound instance
19 methods.
20 """
21 @functools.wraps(method)
22 def wrapper(self, *args, **kwargs):
23 setattr(self, 'pid_in_%s' % method.__name__, os.getpid())
24 return method(self, *args, **kwargs)
25 return wrapper
26
27
28class TestZopeTestInSubProcessLayer:
29 """Helper to test `ZopeTestInSubProcess`.
30
31 Asserts that layers are set up and torn down in the expected way,
32 namely that setUp() and tearDown() are called in the parent
33 process, and testSetUp() and testTearDown() are called in the
34 child process.
35
36 The assertions for tearDown() and testTearDown() must be done here
37 because the test case runs before these methods are called. In the
38 interests of symmetry and clarity, the assertions for setUp() and
39 testSetUp() are done here too.
40 """
41
42 @record_pid
43 def __init__(self, test):
44 self.test = test
45 # These are needed to satisfy the requirements of the
46 # byzantine Zope layer machinery.
47 self.__name__ = self.__class__.__name__
48 self.__bases__ = self.__class__.__bases__
49
50 @record_pid
51 def setUp(self):
52 # Runs in the parent process.
53 assert self.pid_in___init__ == self.pid_in_setUp, (
54 "layer.setUp() not called in parent process.")
55
56 @record_pid
57 def tearDown(self):
58 # Runs in the parent process.
59 assert self.pid_in___init__ == self.pid_in_tearDown, (
60 "layer.tearDown() not called in parent process.")
61
62 @record_pid
63 def testSetUp(self):
64 # Runs in the child process.
65 assert self.pid_in___init__ != self.pid_in_testSetUp, (
66 "layer.testSetUp() called in parent process.")
67
68 @record_pid
69 def testTearDown(self):
70 # Runs in the child process.
71 assert self.pid_in_testSetUp == self.pid_in_testTearDown, (
72 "layer.testTearDown() not called in same process as testSetUp().")
73
74
75class TestZopeTestInSubProcess(ZopeTestInSubProcess, unittest.TestCase):
76 """Test `ZopeTestInSubProcess`.
77
78 Assert that setUp(), test() and tearDown() are called in the child
79 process.
80 """
81
82 @record_pid
83 def __init__(self, method_name='runTest'):
84 # Runs in the parent process.
85 super(TestZopeTestInSubProcess, self).__init__(method_name)
86 self.layer = TestZopeTestInSubProcessLayer(self)
87
88 @record_pid
89 def setUp(self):
90 # Runs in the child process.
91 super(TestZopeTestInSubProcess, self).setUp()
92 self.failUnlessEqual(
93 self.layer.pid_in_testSetUp, self.pid_in_setUp,
94 "setUp() not called in same process as layer.testSetUp().")
95
96 @record_pid
97 def test(self):
98 # Runs in the child process.
99 self.failUnlessEqual(
100 self.pid_in_setUp, self.pid_in_test,
101 "test method not run in same process as setUp().")
102
103 @record_pid
104 def tearDown(self):
105 # Runs in the child process.
106 super(TestZopeTestInSubProcess, self).tearDown()
107 self.failUnlessEqual(
108 self.pid_in_setUp, self.pid_in_tearDown,
109 "tearDown() not run in same process as setUp().")
110
111
112def test_suite():
113 suite = unittest.TestSuite()
114 loader = unittest.TestLoader()
115 suite.addTests(
116 loader.loadTestsFromTestCase(
117 TestZopeTestInSubProcess))
118 return suite
Revision history for this message
Gavin Panella (allenap) wrote :

> On Wed, 10 Mar 2010 14:15:31 -0000
> Björn Tillenius <email address hidden> wrote:
>
> ...
> > > If it's mixed-in with a TestSuite I think it gets more complicated.
>
> I haven't written tests for this yet. I may not do so (and restrict
> ZopeTestInSubProcess to just TestCase for now).

I've removed the suggestion that it can be mixed-in with a TestSuite,
with the following commit message:

  Don't advertise ZopeTestInSubProcess's use with a TestSuite. When
  one suite is added to another with addTests() the *tests* are
  copied, not the suite, so behaviour in a custom TestSuite is
  lost. This is too fragile and likely to cause confusion.

> ...
> > > > Also, how this this TestClass related to TwistedLayer? Oh right, that
> > > > leads me to another question. You say that the reactor can't be
> > > > restarted easily. But can it be shared between tests, just like the
> > > > component architecture?
> > >
> > > That's an interesting idea. reactor.run() blocks, but I don't know of
> > > any reason why it couldn't be started in a thread. I might get sent to
> > > hell for it, but at least it's warmer than here.
> >
> > If it could, than maybe it would be better to use a layer instead, since
> > that what we use elsewhere.
>
> I haven't thought about this yet, but I will.

Now that this works, and seems to work well in the situations I'm
using it (I am also using it in a dependent branch for checkwatches),
I'm reluctant to spend more time on figuring out how to do it in a
layer.

Both the job runner code and my checkwatches code start and stop the
reactor as part of their operation, so trying to fudge them to use an
already-started reactor skews the test and means more moving parts.

Forking is cheap as chips so, although I warn in the docstring that it
will slow down tests, I doubt it will make much of a difference,
especially when testing network code, database code, etc. I haven't
measured it though.

...
> > How well does this play with profiling? Do you still get all profiling
> > information if you specify --profile=cProfile?
>
> Not tried this yet. Not actually sure how to test this; it's a very
> long time since I did any profiling.

It looks like no profiling information is collected from the
sub-process. I'm not sure how to fix this. How important is this to
you? Can this branch be landed without that support, with a bug filed
to record that it needs to be added?

Gavin.

Revision history for this message
Björn Tillenius (bjornt) wrote :

On Fri, Mar 12, 2010 at 11:27:26AM -0000, Gavin Panella wrote:
> > On Wed, 10 Mar 2010 14:15:31 -0000
> > Björn Tillenius <email address hidden> wrote:
> >
> > ...
> > > > If it's mixed-in with a TestSuite I think it gets more complicated.
> >
> > I haven't written tests for this yet. I may not do so (and restrict
> > ZopeTestInSubProcess to just TestCase for now).
>
> I've removed the suggestion that it can be mixed-in with a TestSuite,
> with the following commit message:
>
> Don't advertise ZopeTestInSubProcess's use with a TestSuite. When
> one suite is added to another with addTests() the *tests* are
> copied, not the suite, so behaviour in a custom TestSuite is
> lost. This is too fragile and likely to cause confusion.

Ok.

> > > > > Also, how this this TestClass related to TwistedLayer? Oh right, that
> > > > > leads me to another question. You say that the reactor can't be
> > > > > restarted easily. But can it be shared between tests, just like the
> > > > > component architecture?
> > > >
> > > > That's an interesting idea. reactor.run() blocks, but I don't know of
> > > > any reason why it couldn't be started in a thread. I might get sent to
> > > > hell for it, but at least it's warmer than here.
> > >
> > > If it could, than maybe it would be better to use a layer instead, since
> > > that what we use elsewhere.
> >
> > I haven't thought about this yet, but I will.
>
> Now that this works, and seems to work well in the situations I'm
> using it (I am also using it in a dependent branch for checkwatches),
> I'm reluctant to spend more time on figuring out how to do it in a
> layer.

Ok.

> > > How well does this play with profiling? Do you still get all profiling
> > > information if you specify --profile=cProfile?
> >
> > Not tried this yet. Not actually sure how to test this; it's a very
> > long time since I did any profiling.
>
> It looks like no profiling information is collected from the
> sub-process. I'm not sure how to fix this. How important is this to
> you? Can this branch be landed without that support, with a bug filed
> to record that it needs to be added?

As long as it's not used too much, it's not crucial to have it.

However, I don't see any tests for this. Oh, now I found them, they just
don't look like tests. Hmm. Some more comments would help, to understand
why, and what's being tested. I would have expected a test that manually
calls the run() method. Would that be hard? It would mean that you would
get much better failure messages, in case of something went wrong.
Having tests in setUp and tearDown isn't ideal.

Sorry, but you'll have to ask someone else to carry on reviewing this
branch, since I'm unavailable next week.

    vote abstain

--
Björn Tillenius | https://launchpad.net/~bjornt

review: Abstain
Revision history for this message
Gavin Panella (allenap) wrote :

Björn Tillenius wrote:
> > > > How well does this play with profiling? Do you still get all profiling
> > > > information if you specify --profile=cProfile?
> > >
> > > Not tried this yet. Not actually sure how to test this; it's a very
> > > long time since I did any profiling.
> >
> > It looks like no profiling information is collected from the
> > sub-process. I'm not sure how to fix this. How important is this to
> > you? Can this branch be landed without that support, with a bug filed
> > to record that it needs to be added?
>
> As long as it's not used too much, it's not crucial to have it.
>
> However, I don't see any tests for this. Oh, now I found them, they just
> don't look like tests. Hmm. Some more comments would help, to understand
> why, and what's being tested. I would have expected a test that manually
> calls the run() method. Would that be hard? It would mean that you would
> get much better failure messages, in case of something went wrong.

I've looked into calling the run() method manually, but it means that we
would only be able to test layer.testSetUp(), test.setUp(), the test
method itself, test.tearDown() and layer.testTearDown(). I guess that's
not bad, but it misses out on layer.setUp() and layer.tearDown(), which
are quite important. The way it's structured at the moment means that
everything gets tested, and tested with the real test objects (i.e. the
runner and result) that zope.testing provides.

I have tried to replicate the zope.testing set up - by using Runner from
zope.testing.testresult.runner - but it's fiddly and adds a ton of noise
to the tests and the output. And I haven't yet figured out how to get it
to work properly.

> Having tests in setUp and tearDown isn't ideal.

It's not ideal, but failures here get reported as test failures, rather
than causing a complete test run failure. Unfortunately, failures in
layer methods do cause the run to fail.

To summarise, I'd like to leave the tests as they are. They demonstrate
the operation of ZopeTestInSubProcess right from layer set-up to
tear-down, which I think is valuable. I'll add some more comments to
make it clearer to future developers.

> Sorry, but you'll have to ask someone else to carry on reviewing this
> branch, since I'm unavailable next week.

Thank you for asking so many good questions :) Have a good break.

Revision history for this message
Gavin Panella (allenap) wrote :

Latest incremental diff, r10440..

=== modified file 'lib/lp/testing/__init__.py'
--- lib/lp/testing/__init__.py 2010-03-11 16:37:22 +0000
+++ lib/lp/testing/__init__.py 2010-03-12 10:58:17 +0000
@@ -593,22 +593,12 @@
593class ZopeTestInSubProcess:593class ZopeTestInSubProcess:
594 """Run tests in a sub-process, respecting Zope idiosyncrasies.594 """Run tests in a sub-process, respecting Zope idiosyncrasies.
595595
596 Use this as a mixin with an interesting `TestCase` or `TestSuite`596 Use this as a mixin with an interesting `TestCase` to isolate
597 subclass to quickly isolate tests with side-effects.597 tests with side-effects. Each and every test *method* in the test
598598 case is run in a new, forked, sub-process. This will slow down
599 When mixed-in with a `TestCase`:599 your tests, so use it sparingly. However, when you need to, for
600600 example, start the Twisted reactor (which cannot currently be
601 Each and every test *method* in the test case is run in a new,601 safely stopped and restarted in process) it is invaluable.
602 forked, sub-process. This will slow down your tests, so use it
603 sparingly. However, when you need to, for example, start the
604 Twisted reactor (which cannot currently be safely stopped and
605 restarted in process) it is invaluable.
606
607 When mixed-in with a `TestSuite`:
608
609 All tests in this suite are run in a single, newly forked,
610 sub-process. This isolates the parent process from side-effects,
611 but the tests in this suite are not isolated from one another.
612602
613 This is basically a reimplementation of subunit's603 This is basically a reimplementation of subunit's
614 `IsolatedTestCase` or `IsolatedTestSuite`, but adjusted to work604 `IsolatedTestCase` or `IsolatedTestSuite`, but adjusted to work
615605
=== modified file 'lib/lp/testing/tests/test_zope_test_in_subprocess.py'
--- lib/lp/testing/tests/test_zope_test_in_subprocess.py 2010-03-11 21:37:41 +0000
+++ lib/lp/testing/tests/test_zope_test_in_subprocess.py 2010-03-15 11:46:49 +0000
@@ -1,7 +1,18 @@
1# Copyright 2010 Canonical Ltd. This software is licensed under the1# Copyright 2010 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4"""Test `lp.testing.ZopeTestInSubProcess`."""4"""Test `lp.testing.ZopeTestInSubProcess`.
5
6How does it do this?
7
8A `TestCase`, mixed-in with `ZopeTestInSubProcess`, is run by the Zope
9test runner. This test case sets its own layer, to keep track of the
10PIDs when certain methods are called. It also records pids for its own
11methods. Assertions are made as these methods are called to ensure that
12they are running in the correct process - the parent or the child.
13
14Recording of the PIDs is handled using the `record_pid` decorator.
15"""
516
6__metaclass__ = type17__metaclass__ = type
718
@@ -37,11 +48,13 @@
37 because the test case runs before these methods are called. In the48 because the test case runs before these methods are called. In the
38 interests of symmetry and clarity, the assertions for setUp() and49 interests of symmetry and clarity, the assertions for setUp() and
39 testSetUp() are done here too.50 testSetUp() are done here too.
51
52 This layer expects to be *instantiated*, which is not the norm for
53 Zope layers. See `TestZopeTestInSubProcess` for its use.
40 """54 """
4155
42 @record_pid56 @record_pid
43 def __init__(self, test):57 def __init__(self):
44 self.test = test
45 # These are needed to satisfy the requirements of the58 # These are needed to satisfy the requirements of the
46 # byzantine Zope layer machinery.59 # byzantine Zope layer machinery.
47 self.__name__ = self.__class__.__name__60 self.__name__ = self.__class__.__name__
@@ -54,12 +67,6 @@
54 "layer.setUp() not called in parent process.")67 "layer.setUp() not called in parent process.")
5568
56 @record_pid69 @record_pid
57 def tearDown(self):
58 # Runs in the parent process.
59 assert self.pid_in___init__ == self.pid_in_tearDown, (
60 "layer.tearDown() not called in parent process.")
61
62 @record_pid
63 def testSetUp(self):70 def testSetUp(self):
64 # Runs in the child process.71 # Runs in the child process.
65 assert self.pid_in___init__ != self.pid_in_testSetUp, (72 assert self.pid_in___init__ != self.pid_in_testSetUp, (
@@ -71,19 +78,30 @@
71 assert self.pid_in_testSetUp == self.pid_in_testTearDown, (78 assert self.pid_in_testSetUp == self.pid_in_testTearDown, (
72 "layer.testTearDown() not called in same process as testSetUp().")79 "layer.testTearDown() not called in same process as testSetUp().")
7380
81 @record_pid
82 def tearDown(self):
83 # Runs in the parent process.
84 assert self.pid_in___init__ == self.pid_in_tearDown, (
85 "layer.tearDown() not called in parent process.")
86
7487
75class TestZopeTestInSubProcess(ZopeTestInSubProcess, unittest.TestCase):88class TestZopeTestInSubProcess(ZopeTestInSubProcess, unittest.TestCase):
76 """Test `ZopeTestInSubProcess`.89 """Test `ZopeTestInSubProcess`.
7790
78 Assert that setUp(), test() and tearDown() are called in the child91 Assert that setUp(), test() and tearDown() are called in the child
79 process.92 process.
93
94 Sets its own layer attribute. This layer is then responsible for
95 recording the PID at interesting moments. Specifically,
96 layer.testSetUp() must be called in the same process as
97 test.setUp().
80 """98 """
8199
82 @record_pid100 @record_pid
83 def __init__(self, method_name='runTest'):101 def __init__(self, method_name='runTest'):
84 # Runs in the parent process.102 # Runs in the parent process.
85 super(TestZopeTestInSubProcess, self).__init__(method_name)103 super(TestZopeTestInSubProcess, self).__init__(method_name)
86 self.layer = TestZopeTestInSubProcessLayer(self)104 self.layer = TestZopeTestInSubProcessLayer()
87105
88 @record_pid106 @record_pid
89 def setUp(self):107 def setUp(self):
@@ -110,9 +128,4 @@
110128
111129
112def test_suite():130def test_suite():
113 suite = unittest.TestSuite()131 return unittest.TestLoader().loadTestsFromName(__name__)
114 loader = unittest.TestLoader()
115 suite.addTests(
116 loader.loadTestsFromTestCase(
117 TestZopeTestInSubProcess))
118 return suite
Revision history for this message
Jonathan Lange (jml) wrote :
Download full text (7.1 KiB)

On Mon, Mar 15, 2010 at 1:30 PM, Gavin Panella
<email address hidden> wrote:
> Latest incremental diff, r10440..
>
> === modified file 'lib/lp/services/job/runner.py'
> --- lib/lp/services/job/runner.py 2010-02-25 12:07:15 +0000
> +++ lib/lp/services/job/runner.py 2010-03-15 13:26:40 +0000
> @@ -401,9 +401,10 @@
> class JobCronScript(LaunchpadCronScript):
> """Base class for scripts that run jobs."""
>
> - def __init__(self, runner_class=JobRunner):
> + def __init__(self, runner_class=JobRunner, test_args=None):
> self.dbuser = getattr(config, self.config_name).dbuser
> - super(JobCronScript, self).__init__(self.config_name, self.dbuser)
> + super(JobCronScript, self).__init__(
> + self.config_name, self.dbuser, test_args)
> self.runner_class = runner_class
>
> def main(self):
>

What's the motivation for this change?

> === modified file 'lib/lp/services/job/tests/test_runner.py'
> --- lib/lp/services/job/tests/test_runner.py 2010-02-24 19:52:01 +0000
> +++ lib/lp/services/job/tests/test_runner.py 2010-03-15 13:26:40 +0000
> @@ -11,23 +11,24 @@
> from unittest import TestLoader
>
> import transaction
> -from canonical.testing import LaunchpadZopelessLayer
> +
> from zope.component import getUtility
> from zope.error.interfaces import IErrorReportingUtility
> from zope.interface import implements
>
> +from canonical.launchpad.webapp import errorlog
> +from canonical.launchpad.webapp.interfaces import (
> + DEFAULT_FLAVOR, IStoreSelector, MAIN_STORE)
> +from canonical.testing import LaunchpadZopelessLayer
> +
> from lp.code.interfaces.branchmergeproposal import (
> - IUpdatePreviewDiffJobSource,)
> -from lp.testing.mail_helpers import pop_notifications
> -from lp.services.job.runner import (
> - JobCronScript, JobRunner, BaseRunnableJob, TwistedJobRunner
> -)
> + IUpdatePreviewDiffJobSource)
> from lp.services.job.interfaces.job import JobStatus, IRunnableJob
> from lp.services.job.model.job import Job
> -from lp.testing import TestCaseWithFactory
> -from canonical.launchpad.webapp import errorlog
> -from canonical.launchpad.webapp.interfaces import (
> - IStoreSelector, MAIN_STORE, DEFAULT_FLAVOR)
> +from lp.services.job.runner import (
> + BaseRunnableJob, JobCronScript, JobRunner, TwistedJobRunner)
> +from lp.testing import TestCaseWithFactory, ZopeTestInSubProcess
> +from lp.testing.mail_helpers import pop_notifications
>
>
> class NullJob(BaseRunnableJob):
> @@ -300,7 +301,7 @@
> self.entries.append(input)
>
>
> -class TestTwistedJobRunner(TestCaseWithFactory):
> +class TestTwistedJobRunner(ZopeTestInSubProcess, TestCaseWithFactory):
>
> layer = LaunchpadZopelessLayer
>
> @@ -325,7 +326,7 @@
> self.assertIn('Job ran too long.', oops.value)
>
>
> -class TestJobCronScript(TestCaseWithFactory):
> +class TestJobCronScript(ZopeTestInSubProcess, TestCaseWithFactory):
>
> layer = LaunchpadZopelessLayer
>
> @@ -351,7 +352,8 @@
> source_interface = IUpdatePreviewDiffJobSource
>
> def __init__(self):
> - super(JobCronScriptSubclass, self).__init__(DummyRunner)
> + super(Job...

Read more...

Revision history for this message
Jonathan Lange (jml) wrote :

Just a few changes required to the comments.

review: Needs Fixing
Revision history for this message
Gavin Panella (allenap) wrote :
Download full text (9.5 KiB)

> On Mon, Mar 15, 2010 at 1:30 PM, Gavin Panella
> <email address hidden> wrote:
> > Latest incremental diff, r10440..
> >
> > === modified file 'lib/lp/services/job/runner.py'
> > --- lib/lp/services/job/runner.py 2010-02-25 12:07:15 +0000
> > +++ lib/lp/services/job/runner.py 2010-03-15 13:26:40 +0000
> > @@ -401,9 +401,10 @@
> > class JobCronScript(LaunchpadCronScript):
> > """Base class for scripts that run jobs."""
> >
> > - def __init__(self, runner_class=JobRunner):
> > + def __init__(self, runner_class=JobRunner, test_args=None):
> > self.dbuser = getattr(config, self.config_name).dbuser
> > - super(JobCronScript, self).__init__(self.config_name, self.dbuser)
> > + super(JobCronScript, self).__init__(
> > + self.config_name, self.dbuser, test_args)
> > self.runner_class = runner_class
> >
> > def main(self):
> >
>
> What's the motivation for this change?

It makes testing easier. I'll explain more below.

>
> > === modified file 'lib/lp/services/job/tests/test_runner.py'
> > --- lib/lp/services/job/tests/test_runner.py 2010-02-24 19:52:01 +0000
> > +++ lib/lp/services/job/tests/test_runner.py 2010-03-15 13:26:40 +0000
> > @@ -11,23 +11,24 @@
> > from unittest import TestLoader
> >
> > import transaction
> > -from canonical.testing import LaunchpadZopelessLayer
> > +
> > from zope.component import getUtility
> > from zope.error.interfaces import IErrorReportingUtility
> > from zope.interface import implements
> >
> > +from canonical.launchpad.webapp import errorlog
> > +from canonical.launchpad.webapp.interfaces import (
> > + DEFAULT_FLAVOR, IStoreSelector, MAIN_STORE)
> > +from canonical.testing import LaunchpadZopelessLayer
> > +
> > from lp.code.interfaces.branchmergeproposal import (
> > - IUpdatePreviewDiffJobSource,)
> > -from lp.testing.mail_helpers import pop_notifications
> > -from lp.services.job.runner import (
> > - JobCronScript, JobRunner, BaseRunnableJob, TwistedJobRunner
> > -)
> > + IUpdatePreviewDiffJobSource)
> > from lp.services.job.interfaces.job import JobStatus, IRunnableJob
> > from lp.services.job.model.job import Job
> > -from lp.testing import TestCaseWithFactory
> > -from canonical.launchpad.webapp import errorlog
> > -from canonical.launchpad.webapp.interfaces import (
> > - IStoreSelector, MAIN_STORE, DEFAULT_FLAVOR)
> > +from lp.services.job.runner import (
> > + BaseRunnableJob, JobCronScript, JobRunner, TwistedJobRunner)
> > +from lp.testing import TestCaseWithFactory, ZopeTestInSubProcess
> > +from lp.testing.mail_helpers import pop_notifications
> >
> >
> > class NullJob(BaseRunnableJob):
> > @@ -300,7 +301,7 @@
> > self.entries.append(input)
> >
> >
> > -class TestTwistedJobRunner(TestCaseWithFactory):
> > +class TestTwistedJobRunner(ZopeTestInSubProcess, TestCaseWithFactory):
> >
> > layer = LaunchpadZopelessLayer
> >
> > @@ -325,7 +326,7 @@
> > self.assertIn('Job ran too long.', oops.value)
> >
> >
> > -class TestJobCronScript(TestCaseWithFactory):
> > +class TestJobCronScript(ZopeTestInSubProcess, TestCaseWithFactory):
> >
> > layer = LaunchpadZopelessLayer
> >
> >...

Read more...

Revision history for this message
Jonathan Lange (jml) wrote :
Download full text (9.7 KiB)

On Tue, Mar 16, 2010 at 4:58 PM, Gavin Panella
<email address hidden> wrote:
>> On Mon, Mar 15, 2010 at 1:30 PM, Gavin Panella
>> <email address hidden> wrote:
>> > Latest incremental diff, r10440..
>> >
>> > === modified file 'lib/lp/services/job/runner.py'
>> > --- lib/lp/services/job/runner.py     2010-02-25 12:07:15 +0000
>> > +++ lib/lp/services/job/runner.py     2010-03-15 13:26:40 +0000
>> > @@ -401,9 +401,10 @@
>> >  class JobCronScript(LaunchpadCronScript):
>> >      """Base class for scripts that run jobs."""
>> >
>> > -    def __init__(self, runner_class=JobRunner):
>> > +    def __init__(self, runner_class=JobRunner, test_args=None):
>> >          self.dbuser = getattr(config, self.config_name).dbuser
>> > -        super(JobCronScript, self).__init__(self.config_name, self.dbuser)
>> > +        super(JobCronScript, self).__init__(
>> > +            self.config_name, self.dbuser, test_args)
>> >          self.runner_class = runner_class
>> >
>> >      def main(self):
>> >
>>
>> What's the motivation for this change?
>
> It makes testing easier. I'll explain more below.
>

>>
>> > === modified file 'lib/lp/services/job/tests/test_runner.py'
>> > --- lib/lp/services/job/tests/test_runner.py  2010-02-24 19:52:01 +0000
>> > +++ lib/lp/services/job/tests/test_runner.py  2010-03-15 13:26:40 +0000
>> > @@ -11,23 +11,24 @@
>> >  from unittest import TestLoader
>> >
>> >  import transaction
>> > -from canonical.testing import LaunchpadZopelessLayer
>> > +
>> >  from zope.component import getUtility
>> >  from zope.error.interfaces import IErrorReportingUtility
>> >  from zope.interface import implements
>> >
>> > +from canonical.launchpad.webapp import errorlog
>> > +from canonical.launchpad.webapp.interfaces import (
>> > +    DEFAULT_FLAVOR, IStoreSelector, MAIN_STORE)
>> > +from canonical.testing import LaunchpadZopelessLayer
>> > +
>> >  from lp.code.interfaces.branchmergeproposal import (
>> > -    IUpdatePreviewDiffJobSource,)
>> > -from lp.testing.mail_helpers import pop_notifications
>> > -from lp.services.job.runner import (
>> > -    JobCronScript, JobRunner, BaseRunnableJob, TwistedJobRunner
>> > -)
>> > +    IUpdatePreviewDiffJobSource)
>> >  from lp.services.job.interfaces.job import JobStatus, IRunnableJob
>> >  from lp.services.job.model.job import Job
>> > -from lp.testing import TestCaseWithFactory
>> > -from canonical.launchpad.webapp import errorlog
>> > -from canonical.launchpad.webapp.interfaces import (
>> > -    IStoreSelector, MAIN_STORE, DEFAULT_FLAVOR)
>> > +from lp.services.job.runner import (
>> > +    BaseRunnableJob, JobCronScript, JobRunner, TwistedJobRunner)
>> > +from lp.testing import TestCaseWithFactory, ZopeTestInSubProcess
>> > +from lp.testing.mail_helpers import pop_notifications
>> >
>> >
>> >  class NullJob(BaseRunnableJob):
>> > @@ -300,7 +301,7 @@
>> >          self.entries.append(input)
>> >
>> >
>> > -class TestTwistedJobRunner(TestCaseWithFactory):
>> > +class TestTwistedJobRunner(ZopeTestInSubProcess, TestCaseWithFactory):
>> >
>> >      layer = LaunchpadZopelessLayer
>> >
>> > @@ -325,7 +326,7 @@
>> >          self.assertIn('Job ran too long.', oops.value)
>> >
>> >
>> > -class...

Read more...

Revision history for this message
Jonathan Lange (jml) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/lp/services/job/runner.py'
--- lib/lp/services/job/runner.py 2010-02-25 12:07:15 +0000
+++ lib/lp/services/job/runner.py 2010-03-16 18:02:40 +0000
@@ -401,9 +401,10 @@
401class JobCronScript(LaunchpadCronScript):401class JobCronScript(LaunchpadCronScript):
402 """Base class for scripts that run jobs."""402 """Base class for scripts that run jobs."""
403403
404 def __init__(self, runner_class=JobRunner):404 def __init__(self, runner_class=JobRunner, test_args=None):
405 self.dbuser = getattr(config, self.config_name).dbuser405 self.dbuser = getattr(config, self.config_name).dbuser
406 super(JobCronScript, self).__init__(self.config_name, self.dbuser)406 super(JobCronScript, self).__init__(
407 self.config_name, self.dbuser, test_args)
407 self.runner_class = runner_class408 self.runner_class = runner_class
408409
409 def main(self):410 def main(self):
410411
=== modified file 'lib/lp/services/job/tests/test_runner.py'
--- lib/lp/services/job/tests/test_runner.py 2010-02-24 19:52:01 +0000
+++ lib/lp/services/job/tests/test_runner.py 2010-03-16 18:02:40 +0000
@@ -11,23 +11,24 @@
11from unittest import TestLoader11from unittest import TestLoader
1212
13import transaction13import transaction
14from canonical.testing import LaunchpadZopelessLayer14
15from zope.component import getUtility15from zope.component import getUtility
16from zope.error.interfaces import IErrorReportingUtility16from zope.error.interfaces import IErrorReportingUtility
17from zope.interface import implements17from zope.interface import implements
1818
19from canonical.launchpad.webapp import errorlog
20from canonical.launchpad.webapp.interfaces import (
21 DEFAULT_FLAVOR, IStoreSelector, MAIN_STORE)
22from canonical.testing import LaunchpadZopelessLayer
23
19from lp.code.interfaces.branchmergeproposal import (24from lp.code.interfaces.branchmergeproposal import (
20 IUpdatePreviewDiffJobSource,)25 IUpdatePreviewDiffJobSource)
21from lp.testing.mail_helpers import pop_notifications
22from lp.services.job.runner import (
23 JobCronScript, JobRunner, BaseRunnableJob, TwistedJobRunner
24)
25from lp.services.job.interfaces.job import JobStatus, IRunnableJob26from lp.services.job.interfaces.job import JobStatus, IRunnableJob
26from lp.services.job.model.job import Job27from lp.services.job.model.job import Job
27from lp.testing import TestCaseWithFactory28from lp.services.job.runner import (
28from canonical.launchpad.webapp import errorlog29 BaseRunnableJob, JobCronScript, JobRunner, TwistedJobRunner)
29from canonical.launchpad.webapp.interfaces import (30from lp.testing import TestCaseWithFactory, ZopeTestInSubProcess
30 IStoreSelector, MAIN_STORE, DEFAULT_FLAVOR)31from lp.testing.mail_helpers import pop_notifications
3132
3233
33class NullJob(BaseRunnableJob):34class NullJob(BaseRunnableJob):
@@ -300,7 +301,7 @@
300 self.entries.append(input)301 self.entries.append(input)
301302
302303
303class TestTwistedJobRunner(TestCaseWithFactory):304class TestTwistedJobRunner(ZopeTestInSubProcess, TestCaseWithFactory):
304305
305 layer = LaunchpadZopelessLayer306 layer = LaunchpadZopelessLayer
306307
@@ -325,7 +326,7 @@
325 self.assertIn('Job ran too long.', oops.value)326 self.assertIn('Job ran too long.', oops.value)
326327
327328
328class TestJobCronScript(TestCaseWithFactory):329class TestJobCronScript(ZopeTestInSubProcess, TestCaseWithFactory):
329330
330 layer = LaunchpadZopelessLayer331 layer = LaunchpadZopelessLayer
331332
@@ -351,7 +352,8 @@
351 source_interface = IUpdatePreviewDiffJobSource352 source_interface = IUpdatePreviewDiffJobSource
352353
353 def __init__(self):354 def __init__(self):
354 super(JobCronScriptSubclass, self).__init__(DummyRunner)355 super(JobCronScriptSubclass, self).__init__(
356 DummyRunner, test_args=[])
355 self.logger = ListLogger()357 self.logger = ListLogger()
356358
357 old_errorlog = errorlog.globalErrorUtility359 old_errorlog = errorlog.globalErrorUtility
358360
=== modified file 'lib/lp/testing/__init__.py'
--- lib/lp/testing/__init__.py 2010-03-01 03:06:02 +0000
+++ lib/lp/testing/__init__.py 2010-03-16 18:02:40 +0000
@@ -29,6 +29,7 @@
29 'validate_mock_class',29 'validate_mock_class',
30 'WindmillTestCase',30 'WindmillTestCase',
31 'with_anonymous_login',31 'with_anonymous_login',
32 'ZopeTestInSubProcess',
32 ]33 ]
3334
34import copy35import copy
@@ -38,6 +39,8 @@
38from pprint import pformat39from pprint import pformat
39import shutil40import shutil
40import subprocess41import subprocess
42import subunit
43import sys
41import tempfile44import tempfile
42import time45import time
4346
@@ -62,6 +65,7 @@
62from zope.interface.verify import verifyClass, verifyObject65from zope.interface.verify import verifyClass, verifyObject
63from zope.security.proxy import (66from zope.security.proxy import (
64 isinstance as zope_isinstance, removeSecurityProxy)67 isinstance as zope_isinstance, removeSecurityProxy)
68from zope.testing.testrunner.runner import TestResult as ZopeTestResult
6569
66from canonical.launchpad.webapp import errorlog70from canonical.launchpad.webapp import errorlog
67from canonical.config import config71from canonical.config import config
@@ -586,6 +590,70 @@
586 self.client.open(url=u'http://launchpad.dev:8085')590 self.client.open(url=u'http://launchpad.dev:8085')
587591
588592
593class ZopeTestInSubProcess:
594 """Run tests in a sub-process, respecting Zope idiosyncrasies.
595
596 Use this as a mixin with an interesting `TestCase` to isolate
597 tests with side-effects. Each and every test *method* in the test
598 case is run in a new, forked, sub-process. This will slow down
599 your tests, so use it sparingly. However, when you need to, for
600 example, start the Twisted reactor (which cannot currently be
601 safely stopped and restarted in process) it is invaluable.
602
603 This is basically a reimplementation of subunit's
604 `IsolatedTestCase` or `IsolatedTestSuite`, but adjusted to work
605 with Zope. In particular, Zope's TestResult object is responsible
606 for calling testSetUp() and testTearDown() on the selected layer.
607 """
608
609 def run(self, result):
610 # The result must be an instance of Zope's TestResult because
611 # we construct a super() of it later on. Other result classes
612 # could be supported with a more general approach, but it's
613 # unlikely that any one approach is going to work for every
614 # class. It's better to fail early and draw attention here.
615 assert isinstance(result, ZopeTestResult), (
616 "result must be a Zope result object, not %r." % (result,))
617 pread, pwrite = os.pipe()
618 pid = os.fork()
619 if pid == 0:
620 # Child.
621 os.close(pread)
622 fdwrite = os.fdopen(pwrite, 'w', 1)
623 # Send results to both the Zope result object (so that
624 # layer setup and teardown are done properly, etc.) and to
625 # the subunit stream client so that the parent process can
626 # obtain the result.
627 result = testtools.MultiTestResult(
628 result, subunit.TestProtocolClient(fdwrite))
629 super(ZopeTestInSubProcess, self).run(result)
630 fdwrite.flush()
631 sys.stdout.flush()
632 sys.stderr.flush()
633 # Exit hard to avoid running onexit handlers and to avoid
634 # anything that could suppress SystemExit; this exit must
635 # not be prevented.
636 os._exit(0)
637 else:
638 # Parent.
639 os.close(pwrite)
640 fdread = os.fdopen(pread, 'rU')
641 # Skip all the Zope-specific result stuff by using a
642 # super() of the result. This is because the Zope result
643 # object calls testSetUp() and testTearDown() on the
644 # layer, and handles post-mortem debugging. These things
645 # do not make sense in the parent process. More
646 # immediately, it also means that the results are not
647 # reported twice; they are reported on stdout by the child
648 # process, so they need to be suppressed here.
649 result = super(ZopeTestResult, result)
650 # Accept the result from the child process.
651 protocol = subunit.TestProtocolServer(result)
652 protocol.readFrom(fdread)
653 fdread.close()
654 os.waitpid(pid, 0)
655
656
589def capture_events(callable_obj, *args, **kwargs):657def capture_events(callable_obj, *args, **kwargs):
590 """Capture the events emitted by a callable.658 """Capture the events emitted by a callable.
591659
592660
=== added file 'lib/lp/testing/tests/test_zope_test_in_subprocess.py'
--- lib/lp/testing/tests/test_zope_test_in_subprocess.py 1970-01-01 00:00:00 +0000
+++ lib/lp/testing/tests/test_zope_test_in_subprocess.py 2010-03-16 18:02:40 +0000
@@ -0,0 +1,131 @@
1# Copyright 2010 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""Test `lp.testing.ZopeTestInSubProcess`.
5
6How does it do this?
7
8A `TestCase`, mixed-in with `ZopeTestInSubProcess`, is run by the Zope
9test runner. This test case sets its own layer, to keep track of the
10PIDs when certain methods are called. It also records pids for its own
11methods. Assertions are made as these methods are called to ensure that
12they are running in the correct process - the parent or the child.
13
14Recording of the PIDs is handled using the `record_pid` decorator.
15"""
16
17__metaclass__ = type
18
19import functools
20import os
21import unittest
22
23from lp.testing import ZopeTestInSubProcess
24
25
26def record_pid(method):
27 """Decorator that records the pid at method invocation.
28
29 Will probably only DTRT with class methods or bound instance
30 methods.
31 """
32 @functools.wraps(method)
33 def wrapper(self, *args, **kwargs):
34 setattr(self, 'pid_in_%s' % method.__name__, os.getpid())
35 return method(self, *args, **kwargs)
36 return wrapper
37
38
39class TestZopeTestInSubProcessLayer:
40 """Helper to test `ZopeTestInSubProcess`.
41
42 Asserts that layers are set up and torn down in the expected way,
43 namely that setUp() and tearDown() are called in the parent
44 process, and testSetUp() and testTearDown() are called in the
45 child process.
46
47 The assertions for tearDown() and testTearDown() must be done here
48 because the test case runs before these methods are called. In the
49 interests of symmetry and clarity, the assertions for setUp() and
50 testSetUp() are done here too.
51
52 This layer expects to be *instantiated*, which is not the norm for
53 Zope layers. See `TestZopeTestInSubProcess` for its use.
54 """
55
56 @record_pid
57 def __init__(self):
58 # These are needed to satisfy the requirements of the
59 # byzantine Zope layer machinery.
60 self.__name__ = self.__class__.__name__
61 self.__bases__ = self.__class__.__bases__
62
63 @record_pid
64 def setUp(self):
65 # Runs in the parent process.
66 assert self.pid_in___init__ == self.pid_in_setUp, (
67 "layer.setUp() not called in parent process.")
68
69 @record_pid
70 def testSetUp(self):
71 # Runs in the child process.
72 assert self.pid_in___init__ != self.pid_in_testSetUp, (
73 "layer.testSetUp() called in parent process.")
74
75 @record_pid
76 def testTearDown(self):
77 # Runs in the child process.
78 assert self.pid_in_testSetUp == self.pid_in_testTearDown, (
79 "layer.testTearDown() not called in same process as testSetUp().")
80
81 @record_pid
82 def tearDown(self):
83 # Runs in the parent process.
84 assert self.pid_in___init__ == self.pid_in_tearDown, (
85 "layer.tearDown() not called in parent process.")
86
87
88class TestZopeTestInSubProcess(ZopeTestInSubProcess, unittest.TestCase):
89 """Test `ZopeTestInSubProcess`.
90
91 Assert that setUp(), test() and tearDown() are called in the child
92 process.
93
94 Sets its own layer attribute. This layer is then responsible for
95 recording the PID at interesting moments. Specifically,
96 layer.testSetUp() must be called in the same process as
97 test.setUp().
98 """
99
100 @record_pid
101 def __init__(self, method_name='runTest'):
102 # Runs in the parent process.
103 super(TestZopeTestInSubProcess, self).__init__(method_name)
104 self.layer = TestZopeTestInSubProcessLayer()
105
106 @record_pid
107 def setUp(self):
108 # Runs in the child process.
109 super(TestZopeTestInSubProcess, self).setUp()
110 self.failUnlessEqual(
111 self.layer.pid_in_testSetUp, self.pid_in_setUp,
112 "setUp() not called in same process as layer.testSetUp().")
113
114 @record_pid
115 def test(self):
116 # Runs in the child process.
117 self.failUnlessEqual(
118 self.pid_in_setUp, self.pid_in_test,
119 "test method not run in same process as setUp().")
120
121 @record_pid
122 def tearDown(self):
123 # Runs in the child process.
124 super(TestZopeTestInSubProcess, self).tearDown()
125 self.failUnlessEqual(
126 self.pid_in_setUp, self.pid_in_tearDown,
127 "tearDown() not run in same process as setUp().")
128
129
130def test_suite():
131 return unittest.TestLoader().loadTestsFromName(__name__)