Merge lp:~adeuring/launchpad/bug-507642 into lp:launchpad

Proposed by Abel Deuring
Status: Merged
Approved by: Jeroen T. Vermeulen
Approved revision: no longer in the source branch.
Merged at revision: not available
Proposed branch: lp:~adeuring/launchpad/bug-507642
Merge into: lp:launchpad
Diff against target: 162 lines (+67/-12)
3 files modified
lib/canonical/launchpad/webapp/snapshot.py (+4/-1)
lib/lp/bugs/interfaces/bug.py (+11/-10)
lib/lp/bugs/tests/test_bugs_webservice.py (+52/-1)
To merge this branch: bzr merge lp:~adeuring/launchpad/bug-507642
Reviewer Review Type Date Requested Status
Jeroen T. Vermeulen (community) code Approve
Review via email: mp+20978@code.launchpad.net

Description of the change

This branch removes the properties IBug.subscriptions,
IBug.users_affected IBug.users_unaffected,
IBug.users_affected_with_dupes and IBug.messages from snapshots
of IBug objects.

It is at present impossible to (un)subscribe from bug 1 via AJAX.
(other manipulations of bug 1 via the webservice API like marking
oneself as being affected by this bug don't work either.) Attempts
to do that lead to OOPSes as reported in bug 507642, bug 505999,
bug 534339. They are caused because a snapshot of the bug involves
making shortlist instances of the above mentioned properties in
snapshot_sql_result(), and the hard limit for the shortlist is set
to 1000, while bug 1 has meanwhile more than 1000 comments.

It would have been enough to mark IBug.messages with doNotSnapshot(),
but bug 1 has ca 300 subscribers and ca 200 people feel affected by
it, and I think that these numbers are close enough to the hard limit
to remove the related properties from the snapshot too.

In order to trigger the failure, we need a bug with "too many" comments,
subscribers or affected people. Instead of creating 1000 bug comments
or people just for the tests, I changed the hard limit for the
shortlist() calls for the tests.

main test:

./bin/test -t test_bugs_webservice

possibly affected tests where snapshots of IBug objects are involved:

./bin/test -t test_bugtask -t test_bugnotification -t test_bugchanges

= Launchpad lint =

Checking for conflicts. and issues in doctests and templates.
Running jslint, xmllint, pyflakes, and pylint.
Using normal rules.

Linting changed files:
  lib/canonical/launchpad/webapp/snapshot.py
  lib/lp/bugs/interfaces/bug.py
  lib/lp/bugs/tests/test_bugs_webservice.py

== Pylint notices ==

lib/canonical/launchpad/webapp/snapshot.py
    11: [F0401] Unable to import 'lazr.lifecycle.interfaces' (No module named lifecycle)

lib/lp/bugs/interfaces/bug.py
    36: [F0401] Unable to import 'lazr.lifecycle.snapshot' (No module named lifecycle)
    52: [F0401] Unable to import 'lazr.restful.declarations' (No module named restful)
    58: [F0401] Unable to import 'lazr.restful.fields' (No module named restful)
    59: [F0401] Unable to import 'lazr.restful.interface' (No module named restful)
    506: [C0322, IBug.addAttachment] Operator not preceded by a space
    comment=Text(), filename=TextLine(), is_patch=Bool(),
    ^
    content_type=TextLine(), description=Text())
    @export_factory_operation(IBugAttachment, [])
    def addAttachment(owner, data, comment, filename, is_patch=False,
    content_type=None, description=None):
    653: [C0322, IBug.getNominations] Operator not preceded by a space
    nominations=List(
    ^
    title=_("Nominations to search through."),
    value_type=Reference(schema=Interface), # IBugNomination
    required=False))
    @operation_returns_collection_of(Interface) # IBugNomination
    @export_read_operation()
    def getNominations(target=None, nominations=None):
    733: [C0322, IBug.markUserAffected] Operator not preceded by a space
    required=False, default=True))
    ^
    @call_with(user=REQUEST_USER)
    @export_write_operation()
    def markUserAffected(user, affected=True):
    748: [C0322, IBug.setCommentVisibility] Operator not preceded by a space
    required=True),
    ^
    visible=Bool(title=_('Show this comment?'), required=True))
    @call_with(user=REQUEST_USER)
    @export_write_operation()
    def setCommentVisibility(user, comment_number, visible):
    760: [C0322, IBug.linkHWSubmission] Operator not preceded by a space
    Interface, title=_('A HWDB submission'), required=True))
    ^
    @export_write_operation()
    def linkHWSubmission(submission):
    767: [C0322, IBug.unlinkHWSubmission] Operator not preceded by a space
    Interface, title=_('A HWDB submission'), required=True))
    ^
    @export_write_operation()
    def unlinkHWSubmission(submission):

To post a comment you must log in.
Revision history for this message
Abel Deuring (adeuring) wrote :

Hi Jeroen,

I added an explicit test that the collections are labeled as "do not snapshot". But I did not run the "full test" of too many subscritpions with the real hard limit, as you suggested: Doing this takes ca 80 second on my machine,even when factory.makePersonNoCimmit() is used. BTW, according to top(1), most time is spent in Python, not in Postgres during this test.

Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

Went through this on IRC. Changes:

 * Don't test the entire integration chain for each affected attribute (exceeding the hard limit on snapshots, then simulating a post). Instead, test once that exceeding the hardlimit on one attribute no longer breaks things; then for each of the attributes test that snapshotting is disabled.

 * Move things out of setUp where appropriate; generate all those bug subscribers or whatever using factory.makePersonNoCommit and there's no need for database commits (after the login).

 * The test docstring was basically one huge sentence, obscuring the essentials.

 * The range(5) should be range(<current limit> + 1)

Go land that baby!

review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/canonical/launchpad/webapp/snapshot.py'
--- lib/canonical/launchpad/webapp/snapshot.py 2009-12-03 21:30:02 +0000
+++ lib/canonical/launchpad/webapp/snapshot.py 2010-03-10 11:35:40 +0000
@@ -12,6 +12,8 @@
1212
13from canonical.launchpad.helpers import shortlist13from canonical.launchpad.helpers import shortlist
1414
15HARD_LIMIT_FOR_SNAPSHOT = 1000
16
15@implementer(ISnapshotValueFactory)17@implementer(ISnapshotValueFactory)
16@adapter(IResultSet) # And ISQLObjectResultSet.18@adapter(IResultSet) # And ISQLObjectResultSet.
17def snapshot_sql_result(value):19def snapshot_sql_result(value):
@@ -19,4 +21,5 @@
19 # SelectResults, which doesn't really help the Snapshot21 # SelectResults, which doesn't really help the Snapshot
20 # object. We therefore list()ify the values; this isn't22 # object. We therefore list()ify the values; this isn't
21 # perfect but allows deltas to be generated reliably.23 # perfect but allows deltas to be generated reliably.
22 return shortlist(value, longest_expected=100, hardlimit=1000)24 return shortlist(
25 value, longest_expected=100, hardlimit=HARD_LIMIT_FOR_SNAPSHOT)
2326
=== modified file 'lib/lp/bugs/interfaces/bug.py'
--- lib/lp/bugs/interfaces/bug.py 2010-03-08 09:02:25 +0000
+++ lib/lp/bugs/interfaces/bug.py 2010-03-10 11:35:40 +0000
@@ -33,6 +33,7 @@
33from canonical.launchpad import _33from canonical.launchpad import _
34from canonical.launchpad.fields import (34from canonical.launchpad.fields import (
35 BugField, ContentNameField, DuplicateBug, PublicPersonChoice, Tag, Title)35 BugField, ContentNameField, DuplicateBug, PublicPersonChoice, Tag, Title)
36from lazr.lifecycle.snapshot import doNotSnapshot
36from lp.bugs.interfaces.bugattachment import IBugAttachment37from lp.bugs.interfaces.bugattachment import IBugAttachment
37from lp.bugs.interfaces.bugtask import (38from lp.bugs.interfaces.bugtask import (
38 BugTaskImportance, BugTaskStatus, IBugTask)39 BugTaskImportance, BugTaskStatus, IBugTask)
@@ -237,10 +238,10 @@
237 readonly=True))238 readonly=True))
238 cve_links = Attribute('Links between this bug and CVE entries.')239 cve_links = Attribute('Links between this bug and CVE entries.')
239 subscriptions = exported(240 subscriptions = exported(
240 CollectionField(241 doNotSnapshot(CollectionField(
241 title=_('Subscriptions.'),242 title=_('Subscriptions.'),
242 value_type=Reference(schema=Interface),243 value_type=Reference(schema=Interface),
243 readonly=True))244 readonly=True)))
244 duplicates = exported(245 duplicates = exported(
245 CollectionField(246 CollectionField(
246 title=_('MultiJoin of the bugs which are dups of this one'),247 title=_('MultiJoin of the bugs which are dups of this one'),
@@ -303,23 +304,23 @@
303 # general master bug.304 # general master bug.
304 Int(title=_('The number of users unaffected by this bug'),305 Int(title=_('The number of users unaffected by this bug'),
305 required=True, readonly=True))306 required=True, readonly=True))
306 users_affected = exported(CollectionField(307 users_affected = exported(doNotSnapshot(CollectionField(
307 title=_('Users affected (not including duplicates)'),308 title=_('Users affected (not including duplicates)'),
308 value_type=Reference(schema=IPerson),309 value_type=Reference(schema=IPerson),
309 readonly=True))310 readonly=True)))
310 users_unaffected = exported(CollectionField(311 users_unaffected = exported(doNotSnapshot(CollectionField(
311 title=_('Users explicitly marked as unaffected '312 title=_('Users explicitly marked as unaffected '
312 '(not including duplicates)'),313 '(not including duplicates)'),
313 value_type=Reference(schema=IPerson),314 value_type=Reference(schema=IPerson),
314 readonly=True))315 readonly=True)))
315 users_affected_count_with_dupes = exported(316 users_affected_count_with_dupes = exported(
316 Int(title=_('The number of users affected by this bug '317 Int(title=_('The number of users affected by this bug '
317 '(including duplicates)'),318 '(including duplicates)'),
318 required=True, readonly=True))319 required=True, readonly=True))
319 users_affected_with_dupes = exported(CollectionField(320 users_affected_with_dupes = exported(doNotSnapshot(CollectionField(
320 title=_('Users affected (including duplicates)'),321 title=_('Users affected (including duplicates)'),
321 value_type=Reference(schema=IPerson),322 value_type=Reference(schema=IPerson),
322 readonly=True))323 readonly=True)))
323324
324 heat = exported(325 heat = exported(
325 Int(title=_("The 'heat' of the bug"),326 Int(title=_("The 'heat' of the bug"),
@@ -332,11 +333,11 @@
332 "The number of comments on this bug, not including the initial "333 "The number of comments on this bug, not including the initial "
333 "comment.")334 "comment.")
334335
335 messages = CollectionField(336 messages = doNotSnapshot(CollectionField(
336 title=_("The messages related to this object, in reverse "337 title=_("The messages related to this object, in reverse "
337 "order of creation (so newest first)."),338 "order of creation (so newest first)."),
338 readonly=True,339 readonly=True,
339 value_type=Reference(schema=IMessage))340 value_type=Reference(schema=IMessage)))
340341
341 indexed_messages = exported(342 indexed_messages = exported(
342 CollectionField(343 CollectionField(
343344
=== modified file 'lib/lp/bugs/tests/test_bugs_webservice.py'
--- lib/lp/bugs/tests/test_bugs_webservice.py 2009-12-08 19:08:15 +0000
+++ lib/lp/bugs/tests/test_bugs_webservice.py 2010-03-10 11:35:40 +0000
@@ -12,11 +12,14 @@
12from simplejson import dumps12from simplejson import dumps
1313
14from zope.component import getMultiAdapter14from zope.component import getMultiAdapter
15from lazr.lifecycle.interfaces import IDoNotSnapshot
1516
16from lp.bugs.browser.bugtask import get_comments_for_bugtask17from lp.bugs.browser.bugtask import get_comments_for_bugtask
17from canonical.launchpad.ftests import login18from lp.bugs.interfaces.bug import IBug
19from canonical.launchpad.ftests import login, logout
18from lp.testing import TestCaseWithFactory20from lp.testing import TestCaseWithFactory
19from canonical.launchpad.testing.pages import LaunchpadWebServiceCaller21from canonical.launchpad.testing.pages import LaunchpadWebServiceCaller
22from canonical.launchpad.webapp import snapshot
20from canonical.launchpad.webapp.servers import LaunchpadTestRequest23from canonical.launchpad.webapp.servers import LaunchpadTestRequest
21from canonical.testing import DatabaseFunctionalLayer24from canonical.testing import DatabaseFunctionalLayer
2225
@@ -152,5 +155,53 @@
152 self.failUnlessEqual(None, latest_message['parent_link'])155 self.failUnlessEqual(None, latest_message['parent_link'])
153156
154157
158class TestPostBugWithLargeCollections(TestCaseWithFactory):
159 """Ensure that large IBug collections cause OOPSes on POSTs for IBug.
160
161 When a POST request on a bug is processed, a snapshot of the bug
162 is created. This can lead to OOPSes as described in bugs 507642,
163 505999, 534339: A snapshot of a database collection field is a
164 shortlist() copy of the data and the creation of the snapshot fails
165 if a collection contains more elements than the hard limit of the
166 sortlist().
167
168 Hence we do not include properties in the snapshot that may have
169 a large number of elements: messages, bug subscriptions and
170 (un)affected users.
171 """
172 layer = DatabaseFunctionalLayer
173
174 def test_no_snapshots_for_large_collections(self):
175 # Ensure that no snapshots are made of the properties comments,
176 # bug subscriptions and (un)affected users.
177 for field_name in (
178 'subscriptions', 'users_affected', 'users_unaffected',
179 'users_affected_with_dupes', 'messages'):
180 self.failUnless(
181 IDoNotSnapshot.providedBy(IBug[field_name]),
182 'IBug.%s should not be included in snapshots, see bug 507642.'
183 % field_name)
184
185 def test_many_subscribers(self):
186 # Many subscriptions do not cause an OOPS for IBug POSTs.
187 bug = self.factory.makeBug()
188 webservice = LaunchpadWebServiceCaller(
189 'launchpad-library', 'salgado-change-anything')
190 real_hard_limit_for_snapshot = snapshot.HARD_LIMIT_FOR_SNAPSHOT
191 snapshot.HARD_LIMIT_FOR_SNAPSHOT = 3
192 try:
193 login('foo.bar@canonical.com')
194 for count in range(snapshot.HARD_LIMIT_FOR_SNAPSHOT + 1):
195 person = self.factory.makePersonNoCommit()
196 bug.subscribe(person, person)
197 logout()
198 response = webservice.named_post(
199 '/bugs/%d' % bug.id, 'subscribe',
200 person='http://api.launchpad.dev/beta/~name12')
201 self.failUnlessEqual(200, response.status)
202 finally:
203 snapshot.HARD_LIMIT_FOR_SNAPSHOT = real_hard_limit_for_snapshot
204
205
155def test_suite():206def test_suite():
156 return unittest.TestLoader().loadTestsFromName(__name__)207 return unittest.TestLoader().loadTestsFromName(__name__)