Merge lp:~mwhudson/launchpad/permit_timeout_from_features-on-participation-bug-861510 into lp:launchpad

Proposed by Michael Hudson-Doyle
Status: Merged
Approved by: Michael Hudson-Doyle
Approved revision: no longer in the source branch.
Merged at revision: 14143
Proposed branch: lp:~mwhudson/launchpad/permit_timeout_from_features-on-participation-bug-861510
Merge into: lp:launchpad
Diff against target: 344 lines (+84/-38)
11 files modified
lib/canonical/launchpad/scripts/__init__.py (+5/-5)
lib/canonical/launchpad/webapp/adapter.py (+5/-3)
lib/canonical/launchpad/webapp/authorization.py (+16/-1)
lib/canonical/launchpad/webapp/ftests/test_adapter.py (+0/-16)
lib/canonical/launchpad/webapp/interaction.py (+22/-1)
lib/canonical/launchpad/webapp/interfaces.py (+19/-0)
lib/canonical/launchpad/webapp/servers.py (+4/-2)
lib/canonical/launchpad/webapp/tests/test_servers.py (+3/-1)
lib/canonical/testing/layers.py (+9/-6)
lib/lp/code/tests/test_doc.py (+1/-1)
lib/lp/testing/__init__.py (+0/-2)
To merge this branch: bzr merge lp:~mwhudson/launchpad/permit_timeout_from_features-on-participation-bug-861510
Reviewer Review Type Date Requested Status
Benji York (community) code Approve
Martin Pool (community) Approve
Review via email: mp+78355@code.launchpad.net

Commit message

[r=benji,mbp][bug=623199,861510] Move permit_timeout_from_features flag from old-fashioned, incompletely-reset thread local storage to a new shiny method of attaching data to the interaction

Description of the change

Hi,

This branch does two things:

1) It creates a pattern for storing stuff in the participation (aka request in an appserver context) rather than thread local variables, and
2) Uses this new pattern to store the flag indicating whether to read the hard_timeout feature flag in this way rather on a thread local.

Part 2) fixes bug 861510, which could have been fixed in other ways, but I've been thinking about a way to do 1) and start fixing bug 623199. I have two and a half failed attempts to store the feature controller itself (only) on the participation, but they all got a bit confusing and I decided to pick an easier target (I wanted to fix _some_ problem in this way, to validate the technique). I quite like the fix -- because the flag defaults to False, I can kill off the half baked attempts we have to clear this flag when it needs to be cleared and can be confident that it can't escape the test isolation (this is one of the reasons I just landed a branch that unconditionally tears down the interaction between tests <wink>).

The core understanding I reached is that in Launchpad, the participation is only ever:

(a) an instance of canonical.launchpad.webapp.interaction.Participation (in scripts, mostly)
(b) an instance of canonical.launchpad.webapp.servers.LaunchpadTestRequest (in tests, clearly -- this is the participation type the lp.testing login helpers create by default)
(c) some subclass of canonical.launchpad.webapp.servers.BasicLaunchpadRequest (in the appserver)

So we only need to add an attribute to instances of these three classes and then we can confidently access it whenever there is a zope interaction set up (whenever we're "logged in" in some sense, although this includes the anonymous "user").

In the thread linked to from bug 623199, I proposed using the annotations dictionary that Launchpad's browser requests already have, but when push came to shove I didn't really see the point and just stuck another attribute on -- if we were to use annotations, we'd have to come up with a constant containing the key and import that around and it just felt unnecessary (also some page templates read the features attribute off the request already, so supporting them was easier this way in my previous branches).

I've successfully tested the branch in EC2.

Cheers,
mwh

To post a comment you must log in.
Revision history for this message
Martin Pool (mbp) wrote :

I'm no kind of authority on what is tasteful in Zope or Launchpad, but this does sound good to me.

I think you should fix your 'XXX explain more' ;-) by adding something similar to the text in <https://bugs.launchpad.net/launchpad/+bug/861510/comments/3>, otherwise people will ask, as I did, why you need special handling of the hard_timeout feature flag, rather than just looking it up in the usual way.

review: Needs Fixing
Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Ah, I'd actually fixed that, but my ISP is having such a bad evening that my push had failed. It's pushed now (over 3G, sigh).

Revision history for this message
Martin Pool (mbp) wrote :

+1, but you definitely need review from someone else too

review: Approve
Revision history for this message
Benji York (benji) wrote :

The permit_timeout_from_features flag may have been conceived to help in
checking the timeout feature flag, but it seems to me that it applies to
all feature flags. Perhaps its name should be
"permit_feature_flag_lookup" and the feature flag code should raise an
exception if asked for a flag before the time is right.

Shouldn't the set_permit_timeout_from_features method and
_permit_feature_timeout thread local be removed since the new flag is
the "right" way to look up and mutate the flag?

Having get_participation_extras return None if no appropriate
participation is found feels like a trap to me (i.e., when someone calls
it they are implicitly asserting that the extras are available). It
should raise an exception instead and there should be a
query_participation_extras function so we can be explicit about
tolerating missing extras.

If it survives, the XXX in get_participation_extras is required to have
a bug and person's name associated with it. For details see
https://dev.launchpad.net/PolicyandProcess/XXXPolicy.

It feels just a little weird that get_participation_extras returns a
participation. I can't think of a design that feels better that doesn't
add an inappropriate level of complexity. Here's the best I could do in
case it inspires something better:

    Add an IHasParticpationExtras interface that only has a get_extras
    method. Participations that support extras would also provide
    IHasParticpationExtras and get_extras would return an
    IParticipationExtras.

In fact, I think the data should really be attached to the interaction
instead of the participation. That would remove the need to search
through the participations to find one with extras and the need to
assert that no more than one have extras. Since the participation is
entirely about associating a principle with the interaction and the
extras aren't principal-dependent, the interaction seems like the right
place to put the extras.

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Can we leave details of how this particular flag works to another merge proposal? This one is already confusing enough :-)

_permit_feature_timeout is not the thread local instance -- that instance is used for other things (that I'd also like to kill off, of course, but baby steps).

I agree in principle that stashing things on the interaction does seem cleaner in some ways. Maybe (sketching) something like this:

class IInteractionExtras(Interface):
    permit_timeout_from_features = Attribute(...)

class InteractionExtras(object):
    permit_timeout_from_features = False

class LaunchpadPermissiveSecurityPolicy(PermissiveSecurityPolicy):
    def __init__(self, *participations):
        PermissiveSecurityPolicy.__init__(self, *participations)
        self.extras = InteractionExtras()

(similar change to LaunchpadSecurityPolicy)

Then

def get_interaction_extras():
    # With some error checking, should fail if no interaction though
    return queryInteraction().extras

One part of this that feels, well, different to what I did is that it doesn't give a separate implementation for tests (LaunchpadTestRequest vs LaunchpadBrowserRequest). Maybe that's a good thing though.

What do you think?

Revision history for this message
Robert Collins (lifeless) wrote :

Re "The permit_timeout_from_features flag may have been conceived to help in
checking the timeout feature flag, but it seems to me that it applies to
all feature flags. Perhaps its name should be
"permit_feature_flag_lookup" and the feature flag code should raise an
exception if asked for a flag before the time is right."

No - its truely just the bootstrap mechanism for the db timeout
feature - there is -no- intent that other feature lookups, if they
happen before the db timeout is found, should be blocked (and it would
be a bad, surprising, bug to block them).

-Rob

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

OK, I've made the changes I proposed and am kicking off a test in ec2. It's worked out to be a fairly small change, which is nice.

I probably should add a couple of docstrings but otherwise, please let me know what you think!

Revision history for this message
Benji York (benji) wrote :

Robert: thanks for explaining that the flag really is just about the
timeout flag and not more general. I see now why that is.

Revision history for this message
Benji York (benji) wrote :

> OK, I've made the changes I proposed and am kicking off a test in ec2.
> It's worked out to be a fairly small change, which is nice.
>
> I probably should add a couple of docstrings but otherwise, please let
> me know what you think!

I like it.

I'm still of the opinion that both get_ and query_ versions would be a
small win (the win being a more specific exception if you should have
handled the no-extras possibility but didn't), but you should feel free
to land as-is.

review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/canonical/launchpad/scripts/__init__.py'
2--- lib/canonical/launchpad/scripts/__init__.py 2011-09-06 07:07:13 +0000
3+++ lib/canonical/launchpad/scripts/__init__.py 2011-10-12 20:43:24 +0000
4@@ -23,7 +23,6 @@
5
6 from zope.configuration.config import ConfigurationMachine
7 from zope.security.management import setSecurityPolicy
8-from zope.security.simplepolicies import PermissiveSecurityPolicy
9 import zope.sendmail.delivery
10 import zope.site.hooks
11
12@@ -38,7 +37,10 @@
13 from canonical.database.postgresql import ConnectionString
14 # Intentional re-export, following along the lines of the logger module.
15 from canonical.launchpad.scripts.loghandlers import WatchedFileHandler
16-from canonical.launchpad.webapp.authorization import LaunchpadSecurityPolicy
17+from canonical.launchpad.webapp.authorization import (
18+ LaunchpadPermissiveSecurityPolicy,
19+ LaunchpadSecurityPolicy,
20+ )
21 from canonical.launchpad.webapp.interaction import (
22 ANONYMOUS,
23 setupInteractionByEmail,
24@@ -88,7 +90,7 @@
25 if use_web_security:
26 setSecurityPolicy(LaunchpadSecurityPolicy)
27 else:
28- setSecurityPolicy(PermissiveSecurityPolicy)
29+ setSecurityPolicy(LaunchpadPermissiveSecurityPolicy)
30
31 # Register atexit handler to kill off mail delivery daemon threads, and
32 # thus avoid spew at exit. See:
33@@ -109,8 +111,6 @@
34
35 # This is a convenient hack to set up a zope interaction, before we get
36 # the proper API for having a principal / user running in scripts.
37- # The script will have full permissions because of the
38- # PermissiveSecurityPolicy set up in script.zcml.
39 setupInteractionByEmail(ANONYMOUS)
40
41
42
43=== modified file 'lib/canonical/launchpad/webapp/adapter.py'
44--- lib/canonical/launchpad/webapp/adapter.py 2011-09-22 02:29:01 +0000
45+++ lib/canonical/launchpad/webapp/adapter.py 2011-10-12 20:43:24 +0000
46@@ -71,6 +71,7 @@
47 ReadOnlyModeViolation,
48 SLAVE_FLAVOR,
49 )
50+from canonical.launchpad.webapp.interaction import get_interaction_extras
51 from canonical.launchpad.webapp.opstats import OpStats
52 from canonical.lazr.timeout import set_default_timeout_function
53 from lp.services import features
54@@ -195,7 +196,6 @@
55 _local.enable_timeout = enable_timeout
56 _local.commit_logger = CommitLogger(transaction)
57 transaction.manager.registerSynch(_local.commit_logger)
58- set_permit_timeout_from_features(False)
59
60
61 def clear_request_started():
62@@ -289,7 +289,7 @@
63 :param enabled: If True permit looking up request timeouts in
64 feature flags.
65 """
66- _local._permit_feature_timeout = enabled
67+ get_interaction_extras().permit_timeout_from_features = enabled
68
69
70 def _get_request_timeout(timeout=None):
71@@ -302,7 +302,9 @@
72 return None
73 if timeout is None:
74 timeout = config.database.db_statement_timeout
75- if getattr(_local, '_permit_feature_timeout', False):
76+ interaction_extras = get_interaction_extras()
77+ if (interaction_extras is not None
78+ and interaction_extras.permit_timeout_from_features):
79 set_permit_timeout_from_features(False)
80 try:
81 timeout_str = features.getFeatureFlag('hard_timeout')
82
83=== modified file 'lib/canonical/launchpad/webapp/authorization.py'
84--- lib/canonical/launchpad/webapp/authorization.py 2011-10-12 01:08:48 +0000
85+++ lib/canonical/launchpad/webapp/authorization.py 2011-10-12 20:43:24 +0000
86@@ -26,10 +26,14 @@
87 checkPermission as check_permission_is_registered,
88 )
89 from zope.security.proxy import removeSecurityProxy
90-from zope.security.simplepolicies import ParanoidSecurityPolicy
91+from zope.security.simplepolicies import (
92+ ParanoidSecurityPolicy,
93+ PermissiveSecurityPolicy,
94+ )
95
96 from canonical.database.sqlbase import block_implicit_flushes
97 from canonical.launchpad.readonly import is_read_only
98+from canonical.launchpad.webapp.interaction import InteractionExtras
99 from canonical.launchpad.webapp.interfaces import (
100 AccessLevel,
101 ILaunchpadContainer,
102@@ -47,6 +51,10 @@
103 class LaunchpadSecurityPolicy(ParanoidSecurityPolicy):
104 classProvides(ISecurityPolicy)
105
106+ def __init__(self, *participations):
107+ ParanoidSecurityPolicy.__init__(self, *participations)
108+ self.extras = InteractionExtras()
109+
110 def _checkRequiredAccessLevel(self, access_level, permission, object):
111 """Check that the principal has the level of access required.
112
113@@ -252,3 +260,10 @@
114 # method, but it is not in an interface, and not implemented by
115 # all classes that implement IApplicationRequest.
116 del p.annotations[LAUNCHPAD_SECURITY_POLICY_CACHE_KEY]
117+
118+
119+class LaunchpadPermissiveSecurityPolicy(PermissiveSecurityPolicy):
120+
121+ def __init__(self, *participations):
122+ PermissiveSecurityPolicy.__init__(self, *participations)
123+ self.extras = InteractionExtras()
124
125=== modified file 'lib/canonical/launchpad/webapp/ftests/test_adapter.py'
126--- lib/canonical/launchpad/webapp/ftests/test_adapter.py 2010-10-04 20:46:55 +0000
127+++ lib/canonical/launchpad/webapp/ftests/test_adapter.py 2011-10-12 20:43:24 +0000
128@@ -7,23 +7,7 @@
129 import unittest
130
131 from canonical.launchpad.testing.systemdocs import LayeredDocFileSuite
132-from canonical.launchpad.webapp import adapter
133 from canonical.testing.layers import LaunchpadFunctionalLayer
134-from lp.testing import TestCase
135-
136-
137-class TestTimeout(TestCase):
138-
139- def test_set_permit_timeout_from_features(self):
140- adapter.set_permit_timeout_from_features(True)
141- self.assertTrue(adapter._local._permit_feature_timeout)
142- adapter.set_permit_timeout_from_features(False)
143- self.assertFalse(adapter._local._permit_feature_timeout)
144-
145- def test_set_request_started_disables_flag_timeout(self):
146- adapter.set_request_started()
147- self.addCleanup(adapter.clear_request_started)
148- self.assertFalse(adapter._local._permit_feature_timeout)
149
150
151 def test_suite():
152
153=== modified file 'lib/canonical/launchpad/webapp/interaction.py'
154--- lib/canonical/launchpad/webapp/interaction.py 2010-08-20 20:31:18 +0000
155+++ lib/canonical/launchpad/webapp/interaction.py 2011-10-12 20:43:24 +0000
156@@ -47,6 +47,7 @@
157
158 from canonical.launchpad.webapp.interfaces import (
159 IOpenLaunchBag,
160+ IInteractionExtras,
161 IPlacelessAuthUtility,
162 )
163
164@@ -54,10 +55,11 @@
165 __all__ = [
166 'ANONYMOUS',
167 'get_current_principal',
168+ 'get_interaction_extras',
169 'setupInteraction',
170 'setupInteractionByEmail',
171 'setupInteractionForPerson',
172- 'Participation',
173+ 'InteractionExtras',
174 ]
175
176
177@@ -171,3 +173,22 @@
178
179 interaction = None
180 principal = None
181+
182+
183+class InteractionExtras:
184+ """Extra data attached to all interactions. See `IInteractionExtras`."""
185+
186+ implements(IInteractionExtras)
187+ permit_timeout_from_features = False
188+
189+
190+def get_interaction_extras():
191+ """Return the active provider of `IInteractionExtras`.
192+
193+ This is looked up from the interaction. If there is no interaction then
194+ return None.
195+ """
196+ interaction = queryInteraction()
197+ if interaction is None:
198+ return None
199+ return interaction.extras
200
201=== modified file 'lib/canonical/launchpad/webapp/interfaces.py'
202--- lib/canonical/launchpad/webapp/interfaces.py 2011-10-03 11:36:55 +0000
203+++ lib/canonical/launchpad/webapp/interfaces.py 2011-10-12 20:43:24 +0000
204@@ -310,6 +310,25 @@
205 '''
206
207
208+class IInteractionExtras(Interface):
209+ """We attach a provider of this interface to all interactions.
210+
211+ Because a fresh provider is constructed for every request and between
212+ every test, it is less error-prone to add things to this interface than to
213+ stash state on a thread local.
214+
215+ If you add something here, you should go and edit
216+ `canonical.launchpad.webapp.interaction.InteractionExtras`,
217+ """
218+
219+ permit_timeout_from_features = Attribute(
220+ """A boolean indicating whether to read the 'hard_timeout' feature
221+ flag. We can't check the feature flags early on in request processing
222+ because this can trigger nested db lookups. See the docstring of
223+ `canonical.launchpad.webapp.servers.set_permit_timeout_from_features`
224+ for more.""")
225+
226+
227 #
228 # Request
229 #
230
231=== modified file 'lib/canonical/launchpad/webapp/servers.py'
232--- lib/canonical/launchpad/webapp/servers.py 2011-10-03 11:36:55 +0000
233+++ lib/canonical/launchpad/webapp/servers.py 2011-10-12 20:43:24 +0000
234@@ -866,8 +866,10 @@
235 False
236
237 """
238- implements(INotificationRequest, IBasicLaunchpadRequest, IParticipation,
239- canonical.launchpad.layers.LaunchpadLayer)
240+ implements(
241+ INotificationRequest, IBasicLaunchpadRequest, IParticipation,
242+ canonical.launchpad.layers.LaunchpadLayer)
243+
244 # These two attributes satisfy IParticipation.
245 principal = None
246 interaction = None
247
248=== modified file 'lib/canonical/launchpad/webapp/tests/test_servers.py'
249--- lib/canonical/launchpad/webapp/tests/test_servers.py 2011-10-03 11:36:55 +0000
250+++ lib/canonical/launchpad/webapp/tests/test_servers.py 2011-10-12 20:43:24 +0000
251@@ -32,7 +32,9 @@
252 Interface,
253 )
254
255-from canonical.launchpad.webapp.interfaces import IFinishReadOnlyRequestEvent
256+from canonical.launchpad.webapp.interfaces import (
257+ IFinishReadOnlyRequestEvent,
258+ )
259 from canonical.launchpad.webapp.publication import LaunchpadBrowserPublication
260 from canonical.launchpad.webapp.servers import (
261 ApplicationServerSettingRequestFactory,
262
263=== modified file 'lib/canonical/testing/layers.py'
264--- lib/canonical/testing/layers.py 2011-10-05 21:33:43 +0000
265+++ lib/canonical/testing/layers.py 2011-10-12 20:43:24 +0000
266@@ -99,7 +99,6 @@
267 endInteraction,
268 getSecurityPolicy,
269 )
270-from zope.security.simplepolicies import PermissiveSecurityPolicy
271 from zope.server.logger.pythonlogger import PythonLogger
272
273 from canonical.config import (
274@@ -113,6 +112,9 @@
275 )
276 from canonical.database.sqlbase import session_store
277 from canonical.launchpad.scripts import execute_zcml_for_scripts
278+from canonical.launchpad.webapp.authorization import (
279+ LaunchpadPermissiveSecurityPolicy,
280+ )
281 from canonical.launchpad.webapp.interfaces import (
282 DEFAULT_FLAVOR,
283 IOpenLaunchBag,
284@@ -1209,9 +1211,10 @@
285 # This should not happen here, it should be caught by the
286 # testTearDown() method. If it does, something very nasty
287 # happened.
288- if getSecurityPolicy() != PermissiveSecurityPolicy:
289+ if getSecurityPolicy() != LaunchpadPermissiveSecurityPolicy:
290 raise LayerInvariantError(
291- "Previous test removed the PermissiveSecurityPolicy.")
292+ "Previous test removed the LaunchpadPermissiveSecurityPolicy."
293+ )
294
295 # execute_zcml_for_scripts() sets up an interaction for the
296 # anonymous user. A previous script may have changed or removed
297@@ -1228,10 +1231,10 @@
298 "Component architecture not loaded or totally screwed")
299 # Make sure that a test that changed the security policy, reset it
300 # back to its default value.
301- if getSecurityPolicy() != PermissiveSecurityPolicy:
302+ if getSecurityPolicy() != LaunchpadPermissiveSecurityPolicy:
303 raise LayerInvariantError(
304- "This test removed the PermissiveSecurityPolicy and didn't "
305- "restore it.")
306+ "This test removed the LaunchpadPermissiveSecurityPolicy and "
307+ "didn't restore it.")
308 logout()
309
310
311
312=== renamed file 'lib/lp/bugs/doc/bugtracker-tokens.txt.disabled' => 'lib/lp/bugs/doc/bugtracker-tokens.txt'
313=== modified file 'lib/lp/code/tests/test_doc.py'
314--- lib/lp/code/tests/test_doc.py 2011-08-12 19:15:43 +0000
315+++ lib/lp/code/tests/test_doc.py 2011-10-12 20:43:24 +0000
316@@ -38,7 +38,7 @@
317
318 To be able to use LaunchpadZopelessLayer.switchDbUser in a test, we need
319 to run in the Zopeless environment. The Zopeless environment normally runs
320- using the PermissiveSecurityPolicy. If we want the test to cover
321+ using the LaunchpadPermissiveSecurityPolicy. If we want the test to cover
322 functionality used in the webapp, it needs to use the
323 LaunchpadSecurityPolicy.
324 """
325
326=== modified file 'lib/lp/testing/__init__.py'
327--- lib/lp/testing/__init__.py 2011-10-05 21:46:17 +0000
328+++ lib/lp/testing/__init__.py 2011-10-12 20:43:24 +0000
329@@ -115,7 +115,6 @@
330 )
331 from canonical.launchpad.webapp.adapter import (
332 print_queries,
333- set_permit_timeout_from_features,
334 start_sql_logging,
335 stop_sql_logging,
336 )
337@@ -579,7 +578,6 @@
338 self.addCleanup(
339 self.attachLibrarianLog,
340 LibrarianLayer.librarian_fixture)
341- set_permit_timeout_from_features(False)
342
343 @adapter(ErrorReportEvent)
344 def _recordOops(self, event):