Launchpad itself

Merge lp:~allenap/launchpad/dynamic-batch-size-bug-546085-devel into lp:launchpad

  1. dynamic-batch-size-bug-546085-devel
  2. Merge into devel
Proposed by Gavin Panella
Status: Merged
Approved by: Gavin Panella
Approved revision: no longer in the source branch.
Merged at revision: not available
Proposed branch: lp:~allenap/launchpad/dynamic-batch-size-bug-546085-devel
Merge into: lp:launchpad
Diff against target: 600 lines (+203/-124)
11 files modified
lib/lp/bugs/doc/checkwatches.txt (+6/-3)
lib/lp/bugs/doc/externalbugtracker-debbugs.txt (+3/-3)
lib/lp/bugs/doc/externalbugtracker.txt (+5/-2)
lib/lp/bugs/externalbugtracker/__init__.py (+4/-1)
lib/lp/bugs/externalbugtracker/base.py (+5/-1)
lib/lp/bugs/externalbugtracker/debbugs.py (+8/-7)
lib/lp/bugs/scripts/checkwatches/tests/test_scheduler.py (+110/-0)
lib/lp/bugs/scripts/checkwatches/tests/test_updater.py (+15/-0)
lib/lp/bugs/scripts/checkwatches/updater.py (+40/-16)
lib/lp/bugs/tests/externalbugtracker.py (+4/-2)
lib/lp/bugs/tests/test_bugwatch.py (+3/-89)
To merge this branch: bzr merge lp:~allenap/launchpad/dynamic-batch-size-bug-546085-devel
Reviewer Review Type Date Requested Status
Abel Deuring (community) code Approve
Gavin Panella (community) code Abstain
Review via email: mp+23105@code.launchpad.net

Commit message

In checkwatches, calculate a sensible batch size at run-time based on the number of watches there are for a bug tracker. Originally landed in db-devel r9213.

Description of the change

This is a cherry-pick of db-devel r9213 into devel. I really should have landed it here in the first place. It's got a good chance of causing conflicts so I don't want it to remain only in db-devel for the whole cycle. It has already been reviewed - https://code.edge.launchpad.net/~allenap/launchpad/dynamic-batch-size-bug-546085/+merge/22538 - so this is more of a rubber-stamping exercise.

To post a comment you must log in.
Revision history for this message
Abel Deuring (adeuring) :
review: Approve (rs)
Revision history for this message
Gavin Panella (allenap) :
review: Abstain (code)
Revision history for this message
Abel Deuring (adeuring) :
review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/lp/bugs/doc/checkwatches.txt'
--- lib/lp/bugs/doc/checkwatches.txt 2010-03-30 17:25:52 +0000
+++ lib/lp/bugs/doc/checkwatches.txt 2010-04-09 14:24:25 +0000
@@ -45,7 +45,7 @@
4545
46 >>> print err46 >>> print err
47 INFO creating lockfile47 INFO creating lockfile
48 DEBUG Using a global batch size of None48 DEBUG No global batch size specified.
49 DEBUG Skipping updating Ubuntu Bugzilla watches.49 DEBUG Skipping updating Ubuntu Bugzilla watches.
50 DEBUG No watches to update on http://bugs.debian.org50 DEBUG No watches to update on http://bugs.debian.org
51 DEBUG No watches to update on mailto:bugs@example.com51 DEBUG No watches to update on mailto:bugs@example.com
@@ -274,7 +274,7 @@
274 ... updater.log = original_log274 ... updater.log = original_log
275 ... externalbugtracker.Roundup.batch_size = batch_size275 ... externalbugtracker.Roundup.batch_size = batch_size
276 ... urllib2.urlopen = urlopen276 ... urllib2.urlopen = urlopen
277 DEBUG Using a global batch size of None277 DEBUG No global batch size specified.
278 INFO Updating 5 watches for 5 bugs on http://bugs.example.com278 INFO Updating 5 watches for 5 bugs on http://bugs.example.com
279 ERROR Connection timed out when updating ... (OOPS-...)279 ERROR Connection timed out when updating ... (OOPS-...)
280280
@@ -412,7 +412,8 @@
412 >>> from lp.bugs.interfaces.externalbugtracker import (412 >>> from lp.bugs.interfaces.externalbugtracker import (
413 ... ISupportsCommentImport, ISupportsCommentPushing,413 ... ISupportsCommentImport, ISupportsCommentPushing,
414 ... ISupportsBackLinking)414 ... ISupportsBackLinking)
415 >>> from lp.bugs.externalbugtracker.base import ExternalBugTracker415 >>> from lp.bugs.externalbugtracker.base import (
416 ... BATCH_SIZE_UNLIMITED, ExternalBugTracker)
416417
417 >>> nowish = datetime.now(utc)418 >>> nowish = datetime.now(utc)
418 >>> class UselessExternalBugTracker(ExternalBugTracker):419 >>> class UselessExternalBugTracker(ExternalBugTracker):
@@ -421,6 +422,8 @@
421 ... ISupportsBackLinking, ISupportsCommentImport,422 ... ISupportsBackLinking, ISupportsCommentImport,
422 ... ISupportsCommentPushing)423 ... ISupportsCommentPushing)
423 ...424 ...
425 ... batch_size = BATCH_SIZE_UNLIMITED
426 ...
424 ... def initializeRemoteBugDB(self, bug_ids):427 ... def initializeRemoteBugDB(self, bug_ids):
425 ... # This just exists to stop errors from being raised.428 ... # This just exists to stop errors from being raised.
426 ... pass429 ... pass
427430
=== modified file 'lib/lp/bugs/doc/externalbugtracker-debbugs.txt'
--- lib/lp/bugs/doc/externalbugtracker-debbugs.txt 2010-03-26 13:48:53 +0000
+++ lib/lp/bugs/doc/externalbugtracker-debbugs.txt 2010-04-09 14:24:25 +0000
@@ -14,7 +14,7 @@
14You can specify the db_location explicitly:14You can specify the db_location explicitly:
1515
16 >>> from lp.bugs.externalbugtracker import (16 >>> from lp.bugs.externalbugtracker import (
17 ... DebBugs)17 ... BATCH_SIZE_UNLIMITED, DebBugs)
18 >>> from canonical.testing import LaunchpadZopelessLayer18 >>> from canonical.testing import LaunchpadZopelessLayer
19 >>> txn = LaunchpadZopelessLayer.txn19 >>> txn = LaunchpadZopelessLayer.txn
20 >>> external_debbugs = DebBugs(20 >>> external_debbugs = DebBugs(
@@ -42,8 +42,8 @@
42worry about batching bug watch updates for performance reasons, so42worry about batching bug watch updates for performance reasons, so
43DebBugs instances don't have a batch_size limit.43DebBugs instances don't have a batch_size limit.
4444
45 >>> print external_debbugs.batch_size45 >>> external_debbugs.batch_size == BATCH_SIZE_UNLIMITED
46 None46 True
4747
4848
49== Retrieving bug status from the debbugs database ==49== Retrieving bug status from the debbugs database ==
5050
=== modified file 'lib/lp/bugs/doc/externalbugtracker.txt'
--- lib/lp/bugs/doc/externalbugtracker.txt 2010-03-26 15:12:59 +0000
+++ lib/lp/bugs/doc/externalbugtracker.txt 2010-04-09 14:24:25 +0000
@@ -1042,13 +1042,16 @@
10421042
1043_getRemoteIdsToCheck() will interpret a batch_size parameter of 0 as an1043_getRemoteIdsToCheck() will interpret a batch_size parameter of 0 as an
1044instruction to ignore the batch size limitation altogether and just return all1044instruction to ignore the batch size limitation altogether and just return all
1045the IDs that need checking.1045the IDs that need checking. The constant BATCH_SIZE_UNLIMITED should
1046be used in place of using 0 verbatim.
1047
1048 >>> from lp.bugs.externalbugtracker import BATCH_SIZE_UNLIMITED
10461049
1047 >>> ids = bug_watch_updater._getRemoteIdsToCheck(1050 >>> ids = bug_watch_updater._getRemoteIdsToCheck(
1048 ... external_bugtracker,1051 ... external_bugtracker,
1049 ... unchecked_watches + watches_with_comments + old_watches,1052 ... unchecked_watches + watches_with_comments + old_watches,
1050 ... external_bugtracker.getCurrentDBTime(), utc_now,1053 ... external_bugtracker.getCurrentDBTime(), utc_now,
1051 ... batch_size=0)1054 ... batch_size=BATCH_SIZE_UNLIMITED)
1052 >>> print sorted(ids['remote_ids_to_check'])1055 >>> print sorted(ids['remote_ids_to_check'])
1053 [u'0', u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9']1056 [u'0', u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9']
10541057
10551058
=== modified file 'lib/lp/bugs/externalbugtracker/__init__.py'
--- lib/lp/bugs/externalbugtracker/__init__.py 2009-06-25 00:40:31 +0000
+++ lib/lp/bugs/externalbugtracker/__init__.py 2010-04-09 14:24:25 +0000
@@ -7,7 +7,7 @@
77
8__metaclass__ = type8__metaclass__ = type
9__all__ = [9__all__ = [
10 'get_external_bugtracker',10 'BATCH_SIZE_UNLIMITED',
11 'BugNotFound',11 'BugNotFound',
12 'BugTrackerConnectError',12 'BugTrackerConnectError',
13 'BugWatchUpdateError',13 'BugWatchUpdateError',
@@ -30,6 +30,7 @@
30 'UnparseableBugData',30 'UnparseableBugData',
31 'UnparseableBugTrackerVersion',31 'UnparseableBugTrackerVersion',
32 'UnsupportedBugTrackerVersion',32 'UnsupportedBugTrackerVersion',
33 'get_external_bugtracker',
33 ]34 ]
3435
35from lp.bugs.externalbugtracker.base import *36from lp.bugs.externalbugtracker.base import *
@@ -41,6 +42,8 @@
41from lp.bugs.externalbugtracker.rt import *42from lp.bugs.externalbugtracker.rt import *
42from lp.bugs.externalbugtracker.trac import *43from lp.bugs.externalbugtracker.trac import *
43from lp.bugs.interfaces.bugtracker import BugTrackerType44from lp.bugs.interfaces.bugtracker import BugTrackerType
45
46
44BUG_TRACKER_CLASSES = {47BUG_TRACKER_CLASSES = {
45 BugTrackerType.BUGZILLA: Bugzilla,48 BugTrackerType.BUGZILLA: Bugzilla,
46 BugTrackerType.DEBBUGS: DebBugs,49 BugTrackerType.DEBBUGS: DebBugs,
4750
=== modified file 'lib/lp/bugs/externalbugtracker/base.py'
--- lib/lp/bugs/externalbugtracker/base.py 2010-03-26 15:12:59 +0000
+++ lib/lp/bugs/externalbugtracker/base.py 2010-04-09 14:24:25 +0000
@@ -5,6 +5,7 @@
55
6__metaclass__ = type6__metaclass__ = type
7__all__ = [7__all__ = [
8 'BATCH_SIZE_UNLIMITED',
8 'BugNotFound',9 'BugNotFound',
9 'BugTrackerAuthenticationError',10 'BugTrackerAuthenticationError',
10 'BugTrackerConnectError',11 'BugTrackerConnectError',
@@ -39,6 +40,9 @@
39# The user agent we send in our requests40# The user agent we send in our requests
40LP_USER_AGENT = "Launchpad Bugscraper/0.2 (https://bugs.launchpad.net/)"41LP_USER_AGENT = "Launchpad Bugscraper/0.2 (https://bugs.launchpad.net/)"
4142
43# To signify that all bug watches should be checked in a single run.
44BATCH_SIZE_UNLIMITED = 0
45
4246
43#47#
44# Errors.48# Errors.
@@ -133,7 +137,7 @@
133137
134 implements(IExternalBugTracker)138 implements(IExternalBugTracker)
135139
136 batch_size = 100140 batch_size = None
137 batch_query_threshold = config.checkwatches.batch_query_threshold141 batch_query_threshold = config.checkwatches.batch_query_threshold
138 comment_template = 'default_remotecomment_template.txt'142 comment_template = 'default_remotecomment_template.txt'
139143
140144
=== modified file 'lib/lp/bugs/externalbugtracker/debbugs.py'
--- lib/lp/bugs/externalbugtracker/debbugs.py 2010-03-31 20:09:55 +0000
+++ lib/lp/bugs/externalbugtracker/debbugs.py 2010-04-09 14:24:25 +0000
@@ -22,18 +22,19 @@
2222
23from canonical.config import config23from canonical.config import config
24from canonical.database.sqlbase import commit24from canonical.database.sqlbase import commit
25from canonical.launchpad.interfaces.message import IMessageSet
26from canonical.launchpad.mail import simple_sendmail
27from canonical.launchpad.webapp import urlsplit
28
25from lp.bugs.externalbugtracker import (29from lp.bugs.externalbugtracker import (
26 BugNotFound, BugTrackerConnectError, ExternalBugTracker,30 BATCH_SIZE_UNLIMITED, BugNotFound, BugTrackerConnectError,
27 InvalidBugId, UnknownRemoteStatusError)31 ExternalBugTracker, InvalidBugId, UnknownRemoteStatusError)
28from canonical.launchpad.interfaces.message import IMessageSet32from lp.bugs.externalbugtracker.isolation import ensure_no_transaction
29from lp.bugs.interfaces.bugtask import BugTaskImportance, BugTaskStatus33from lp.bugs.interfaces.bugtask import BugTaskImportance, BugTaskStatus
30from lp.bugs.interfaces.externalbugtracker import (34from lp.bugs.interfaces.externalbugtracker import (
31 ISupportsBugImport, ISupportsCommentImport, ISupportsCommentPushing,35 ISupportsBugImport, ISupportsCommentImport, ISupportsCommentPushing,
32 UNKNOWN_REMOTE_IMPORTANCE)36 UNKNOWN_REMOTE_IMPORTANCE)
33from canonical.launchpad.mail import simple_sendmail
34from lp.bugs.scripts import debbugs37from lp.bugs.scripts import debbugs
35from canonical.launchpad.webapp import urlsplit
36from lp.bugs.externalbugtracker.isolation import ensure_no_transaction
3738
3839
39debbugsstatusmap = {'open': BugTaskStatus.NEW,40debbugsstatusmap = {'open': BugTaskStatus.NEW,
@@ -59,7 +60,7 @@
59 # Because we keep a local copy of debbugs, we remove the batch_size60 # Because we keep a local copy of debbugs, we remove the batch_size
60 # limit so that all debbugs watches that need checking will be61 # limit so that all debbugs watches that need checking will be
61 # checked each time checkwatches runs.62 # checked each time checkwatches runs.
62 batch_size = None63 batch_size = BATCH_SIZE_UNLIMITED
6364
64 def __init__(self, baseurl, db_location=None):65 def __init__(self, baseurl, db_location=None):
65 super(DebBugs, self).__init__(baseurl)66 super(DebBugs, self).__init__(baseurl)
6667
=== added directory 'lib/lp/bugs/scripts/checkwatches/tests'
=== added file 'lib/lp/bugs/scripts/checkwatches/tests/__init__.py'
=== added file 'lib/lp/bugs/scripts/checkwatches/tests/test_scheduler.py'
--- lib/lp/bugs/scripts/checkwatches/tests/test_scheduler.py 1970-01-01 00:00:00 +0000
+++ lib/lp/bugs/scripts/checkwatches/tests/test_scheduler.py 2010-04-09 14:24:25 +0000
@@ -0,0 +1,110 @@
1# Copyright 2010 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""XXX: Module docstring goes here."""
5
6__metaclass__ = type
7
8import transaction
9import unittest
10
11from datetime import datetime, timedelta
12from pytz import utc
13
14from zope.component import getUtility
15
16from canonical.launchpad.scripts.logger import QuietFakeLogger
17from canonical.testing import DatabaseFunctionalLayer
18
19from lp.bugs.interfaces.bugwatch import (
20 BugWatchActivityStatus, IBugWatchSet)
21from lp.bugs.scripts.checkwatches.scheduler import BugWatchScheduler
22
23from lp.testing import TestCaseWithFactory
24
25
26class TestBugWatchScheduler(TestCaseWithFactory):
27 """Tests for the BugWatchScheduler, which runs as part of garbo."""
28
29 layer = DatabaseFunctionalLayer
30
31 def setUp(self):
32 super(TestBugWatchScheduler, self).setUp('foo.bar@canonical.com')
33 # We'll make sure that all the other bug watches look like
34 # they've been scheduled so that only our watch gets scheduled.
35 for watch in getUtility(IBugWatchSet).search():
36 watch.next_check = datetime.now(utc)
37 self.bug_watch = self.factory.makeBugWatch()
38 self.scheduler = BugWatchScheduler(QuietFakeLogger())
39 transaction.commit()
40
41 def test_scheduler_schedules_unchecked_watches(self):
42 # The BugWatchScheduler will schedule a BugWatch that has never
43 # been checked to be checked immediately.
44 self.bug_watch.next_check = None
45 self.scheduler(1)
46
47 self.assertNotEqual(None, self.bug_watch.next_check)
48 self.assertTrue(
49 self.bug_watch.next_check <= datetime.now(utc))
50
51 def test_scheduler_schedules_working_watches(self):
52 # If a watch has been checked and has never failed its next
53 # check will be scheduled for 24 hours after its last check.
54 now = datetime.now(utc)
55 # Add some succesful activity to ensure that successful activity
56 # is handled correctly.
57 self.bug_watch.addActivity()
58 self.bug_watch.lastchecked = now
59 self.bug_watch.next_check = None
60 transaction.commit()
61 self.scheduler(1)
62
63 self.assertEqual(
64 now + timedelta(hours=24), self.bug_watch.next_check)
65
66 def test_scheduler_schedules_failing_watches(self):
67 # If a watch has failed once, it will be scheduled more than 24
68 # hours after its last check.
69 now = datetime.now(utc)
70 self.bug_watch.lastchecked = now
71
72 # The delay depends on the number of failures that the watch has
73 # had.
74 for failure_count in range(1, 6):
75 self.bug_watch.next_check = None
76 self.bug_watch.addActivity(
77 result=BugWatchActivityStatus.BUG_NOT_FOUND)
78 transaction.commit()
79 self.scheduler(1)
80
81 coefficient = self.scheduler.delay_coefficient * failure_count
82 self.assertEqual(
83 now + timedelta(days=1 + coefficient),
84 self.bug_watch.next_check)
85
86 # The scheduler only looks at the last 5 activity items, so even
87 # if there have been more failures the maximum delay will be 7
88 # days.
89 for count in range(10):
90 self.bug_watch.addActivity(
91 result=BugWatchActivityStatus.BUG_NOT_FOUND)
92 self.bug_watch.next_check = None
93 transaction.commit()
94 self.scheduler(1)
95 self.assertEqual(
96 now + timedelta(days=7), self.bug_watch.next_check)
97
98 def test_scheduler_doesnt_schedule_scheduled_watches(self):
99 # The scheduler will ignore watches whose next_check has been
100 # set.
101 next_check_date = datetime.now(utc) + timedelta(days=1)
102 self.bug_watch.next_check = next_check_date
103 transaction.commit()
104 self.scheduler(1)
105
106 self.assertEqual(next_check_date, self.bug_watch.next_check)
107
108
109def test_suite():
110 return unittest.TestLoader().loadTestsFromName(__name__)
0111
=== renamed file 'lib/lp/bugs/scripts/tests/test_checkwatches.py' => 'lib/lp/bugs/scripts/checkwatches/tests/test_updater.py'
--- lib/lp/bugs/scripts/tests/test_checkwatches.py 2010-03-25 21:59:48 +0000
+++ lib/lp/bugs/scripts/checkwatches/tests/test_updater.py 2010-04-09 14:24:25 +0000
@@ -167,6 +167,21 @@
167 # stop SQL statemnet logging.167 # stop SQL statemnet logging.
168 clear_request_started()168 clear_request_started()
169169
170 def test_suggest_batch_size(self):
171 class RemoteSystem: pass
172 remote_system = RemoteSystem()
173 # When the batch_size is None, suggest_batch_size() will set
174 # it accordingly.
175 remote_system.batch_size = None
176 checkwatches.updater.suggest_batch_size(remote_system, 1)
177 self.failUnlessEqual(100, remote_system.batch_size)
178 remote_system.batch_size = None
179 checkwatches.updater.suggest_batch_size(remote_system, 12350)
180 self.failUnlessEqual(247, remote_system.batch_size)
181 # If the batch_size is already set, it will not be changed.
182 checkwatches.updater.suggest_batch_size(remote_system, 99999)
183 self.failUnlessEqual(247, remote_system.batch_size)
184
170185
171class TestUpdateBugsWithLinkedQuestions(unittest.TestCase):186class TestUpdateBugsWithLinkedQuestions(unittest.TestCase):
172 """Tests for updating bugs with linked questions."""187 """Tests for updating bugs with linked questions."""
173188
=== modified file 'lib/lp/bugs/scripts/checkwatches/updater.py'
--- lib/lp/bugs/scripts/checkwatches/updater.py 2010-03-26 21:03:30 +0000
+++ lib/lp/bugs/scripts/checkwatches/updater.py 2010-04-09 14:24:25 +0000
@@ -61,16 +61,15 @@
6161
62from lp.bugs import externalbugtracker62from lp.bugs import externalbugtracker
63from lp.bugs.externalbugtracker import (63from lp.bugs.externalbugtracker import (
64 BugNotFound, BugTrackerConnectError, BugWatchUpdateError,64 BATCH_SIZE_UNLIMITED, BugNotFound, BugTrackerConnectError,
65 BugWatchUpdateWarning, InvalidBugId, PrivateRemoteBug,65 BugWatchUpdateError, BugWatchUpdateWarning, InvalidBugId,
66 UnknownBugTrackerTypeError, UnknownRemoteStatusError, UnparseableBugData,66 PrivateRemoteBug, UnknownBugTrackerTypeError, UnknownRemoteStatusError,
67 UnparseableBugTrackerVersion, UnsupportedBugTrackerVersion)67 UnparseableBugData, UnparseableBugTrackerVersion,
68from lp.bugs.externalbugtracker.bugzilla import (68 UnsupportedBugTrackerVersion)
69 BugzillaAPI)69from lp.bugs.externalbugtracker.bugzilla import BugzillaAPI
70from lp.bugs.externalbugtracker.isolation import check_no_transaction70from lp.bugs.externalbugtracker.isolation import check_no_transaction
71from lp.bugs.interfaces.bug import IBugSet71from lp.bugs.interfaces.bug import IBugSet
72from lp.bugs.interfaces.externalbugtracker import (72from lp.bugs.interfaces.externalbugtracker import ISupportsBackLinking
73 ISupportsBackLinking)
74from lp.services.limitedlist import LimitedList73from lp.services.limitedlist import LimitedList
75from lp.services.scripts.base import LaunchpadCronScript74from lp.services.scripts.base import LaunchpadCronScript
7675
@@ -78,6 +77,11 @@
78SYNCABLE_GNOME_PRODUCTS = []77SYNCABLE_GNOME_PRODUCTS = []
79MAX_SQL_STATEMENTS_LOGGED = 1000078MAX_SQL_STATEMENTS_LOGGED = 10000
8079
80# The minimum batch size to suggest to an IExternalBugTracker.
81SUGGESTED_BATCH_SIZE_MIN = 100
82# The proportion of all watches to suggest as a batch size.
83SUGGESTED_BATCH_SIZE_PROPORTION = 0.02
84
8185
82class TooMuchTimeSkew(BugWatchUpdateError):86class TooMuchTimeSkew(BugWatchUpdateError):
83 """Time difference between ourselves and the remote server is too much."""87 """Time difference between ourselves and the remote server is too much."""
@@ -242,6 +246,22 @@
242 yield item246 yield item
243247
244248
249def suggest_batch_size(remote_system, num_watches):
250 """Suggest a value for batch_size if it's not set.
251
252 Given the number of bug watches for a `remote_system`, this sets a
253 suggested batch size on it. If `remote_system` already has a batch
254 size set, this does not override it.
255
256 :param remote_system: An `ExternalBugTracker`.
257 :param num_watches: The number of watches for `remote_system`.
258 """
259 if remote_system.batch_size is None:
260 remote_system.batch_size = max(
261 SUGGESTED_BATCH_SIZE_MIN,
262 int(SUGGESTED_BATCH_SIZE_PROPORTION * num_watches))
263
264
245class BugWatchUpdater(object):265class BugWatchUpdater(object):
246 """Takes responsibility for updating remote bug watches."""266 """Takes responsibility for updating remote bug watches."""
247267
@@ -381,7 +401,12 @@
381 `BaseScheduler`. If no scheduler is given, `SerialScheduler`401 `BaseScheduler`. If no scheduler is given, `SerialScheduler`
382 will be used, which simply runs the jobs in order.402 will be used, which simply runs the jobs in order.
383 """403 """
384 self.log.debug("Using a global batch size of %s" % batch_size)404 if batch_size is None:
405 self.log.debug("No global batch size specified.")
406 elif batch_size == BATCH_SIZE_UNLIMITED:
407 self.log.debug("Using an unlimited global batch size.")
408 else:
409 self.log.debug("Using a global batch size of %s" % batch_size)
385410
386 # Default to using the very simple SerialScheduler.411 # Default to using the very simple SerialScheduler.
387 if scheduler is None:412 if scheduler is None:
@@ -497,6 +522,7 @@
497 def _getExternalBugTrackersAndWatches(self, bug_tracker, bug_watches):522 def _getExternalBugTrackersAndWatches(self, bug_tracker, bug_watches):
498 """Return an `ExternalBugTracker` instance for `bug_tracker`."""523 """Return an `ExternalBugTracker` instance for `bug_tracker`."""
499 with self.transaction:524 with self.transaction:
525 num_watches = bug_tracker.watches.count()
500 remotesystem = (526 remotesystem = (
501 externalbugtracker.get_external_bugtracker(bug_tracker))527 externalbugtracker.get_external_bugtracker(bug_tracker))
502 # We special-case the Gnome Bugzilla.528 # We special-case the Gnome Bugzilla.
@@ -506,6 +532,9 @@
506 # Probe the remote system for additional capabilities.532 # Probe the remote system for additional capabilities.
507 remotesystem_to_use = remotesystem.getExternalBugTrackerToUse()533 remotesystem_to_use = remotesystem.getExternalBugTrackerToUse()
508534
535 # Try to hint at how many bug watches to check each time.
536 suggest_batch_size(remotesystem_to_use, num_watches)
537
509 if (is_gnome_bugzilla and538 if (is_gnome_bugzilla and
510 isinstance(remotesystem_to_use, BugzillaAPI) and539 isinstance(remotesystem_to_use, BugzillaAPI) and
511 len(self._syncable_gnome_products) > 0):540 len(self._syncable_gnome_products) > 0):
@@ -656,11 +685,6 @@
656 # by the ExternalBugTracker.685 # by the ExternalBugTracker.
657 batch_size = remotesystem.batch_size686 batch_size = remotesystem.batch_size
658687
659 if batch_size == 0:
660 # A batch_size of 0 means that there's no batch size limit
661 # for this bug tracker.
662 batch_size = None
663
664 with self.transaction:688 with self.transaction:
665 old_bug_watches = set(689 old_bug_watches = set(
666 bug_watch for bug_watch in bug_watches690 bug_watch for bug_watch in bug_watches
@@ -693,7 +717,7 @@
693 # are actually some bugs that we're interested in so as to717 # are actually some bugs that we're interested in so as to
694 # avoid unnecessary network traffic.718 # avoid unnecessary network traffic.
695 if server_time is not None and len(remote_old_ids) > 0:719 if server_time is not None and len(remote_old_ids) > 0:
696 if batch_size is None:720 if batch_size == BATCH_SIZE_UNLIMITED:
697 remote_old_ids_to_check = (721 remote_old_ids_to_check = (
698 remotesystem.getModifiedRemoteBugs(722 remotesystem.getModifiedRemoteBugs(
699 remote_old_ids, oldest_lastchecked))723 remote_old_ids, oldest_lastchecked))
@@ -721,7 +745,7 @@
721 remote_ids_to_check = chain(745 remote_ids_to_check = chain(
722 remote_ids_with_comments, remote_new_ids, remote_old_ids_to_check)746 remote_ids_with_comments, remote_new_ids, remote_old_ids_to_check)
723747
724 if batch_size is not None:748 if batch_size != BATCH_SIZE_UNLIMITED:
725 # Some remote bug IDs may appear in more than one list so749 # Some remote bug IDs may appear in more than one list so
726 # we must filter the list before slicing.750 # we must filter the list before slicing.
727 remote_ids_to_check = islice(751 remote_ids_to_check = islice(
728752
=== modified file 'lib/lp/bugs/tests/externalbugtracker.py'
--- lib/lp/bugs/tests/externalbugtracker.py 2009-12-23 12:14:59 +0000
+++ lib/lp/bugs/tests/externalbugtracker.py 2010-04-09 14:24:25 +0000
@@ -24,8 +24,8 @@
24from canonical.config import config24from canonical.config import config
25from canonical.database.sqlbase import commit, ZopelessTransactionManager25from canonical.database.sqlbase import commit, ZopelessTransactionManager
26from lp.bugs.externalbugtracker import (26from lp.bugs.externalbugtracker import (
27 BugNotFound, BugTrackerConnectError, Bugzilla, DebBugs,27 BATCH_SIZE_UNLIMITED, BugNotFound, BugTrackerConnectError, Bugzilla,
28 ExternalBugTracker, Mantis, RequestTracker, Roundup, SourceForge,28 DebBugs, ExternalBugTracker, Mantis, RequestTracker, Roundup, SourceForge,
29 Trac)29 Trac)
30from lp.bugs.externalbugtracker.trac import (30from lp.bugs.externalbugtracker.trac import (
31 FAULT_TICKET_NOT_FOUND, LP_PLUGIN_BUG_IDS_ONLY, LP_PLUGIN_FULL,31 FAULT_TICKET_NOT_FOUND, LP_PLUGIN_BUG_IDS_ONLY, LP_PLUGIN_FULL,
@@ -152,6 +152,8 @@
152 implementation, though it doesn't actually do anything.152 implementation, though it doesn't actually do anything.
153 """153 """
154154
155 batch_size = BATCH_SIZE_UNLIMITED
156
155 def __init__(self, baseurl='http://example.com/'):157 def __init__(self, baseurl='http://example.com/'):
156 super(TestExternalBugTracker, self).__init__(baseurl)158 super(TestExternalBugTracker, self).__init__(baseurl)
157159
158160
=== modified file 'lib/lp/bugs/tests/test_bugwatch.py'
--- lib/lp/bugs/tests/test_bugwatch.py 2010-03-26 14:49:24 +0000
+++ lib/lp/bugs/tests/test_bugwatch.py 2010-04-09 14:24:25 +0000
@@ -8,27 +8,21 @@
8import transaction8import transaction
9import unittest9import unittest
1010
11from datetime import datetime, timedelta
12from pytz import utc
13
14from urlparse import urlunsplit11from urlparse import urlunsplit
1512
16from zope.component import getUtility13from zope.component import getUtility
1714
18from canonical.launchpad.ftests import login, ANONYMOUS15from canonical.launchpad.ftests import login, ANONYMOUS
16from canonical.launchpad.scripts.garbo import BugWatchActivityPruner
19from canonical.launchpad.scripts.logger import QuietFakeLogger17from canonical.launchpad.scripts.logger import QuietFakeLogger
20from canonical.launchpad.webapp import urlsplit18from canonical.launchpad.webapp import urlsplit
21from canonical.launchpad.scripts.garbo import BugWatchActivityPruner
22from canonical.launchpad.scripts.logger import QuietFakeLogger
23from canonical.testing import (19from canonical.testing import (
24 DatabaseFunctionalLayer, LaunchpadFunctionalLayer, LaunchpadZopelessLayer)20 DatabaseFunctionalLayer, LaunchpadFunctionalLayer, LaunchpadZopelessLayer)
2521
26from lp.bugs.interfaces.bugtracker import BugTrackerType, IBugTrackerSet22from lp.bugs.interfaces.bugtracker import BugTrackerType, IBugTrackerSet
27from lp.bugs.interfaces.bugwatch import (23from lp.bugs.interfaces.bugwatch import (
28 BugWatchActivityStatus, IBugWatchSet, NoBugTrackerFound,24 IBugWatchSet, NoBugTrackerFound, UnrecognizedBugTrackerURL)
29 UnrecognizedBugTrackerURL)25from lp.bugs.scripts.checkwatches.scheduler import MAX_SAMPLE_SIZE
30from lp.bugs.scripts.checkwatches.scheduler import (
31 BugWatchScheduler, MAX_SAMPLE_SIZE)
32from lp.registry.interfaces.person import IPersonSet26from lp.registry.interfaces.person import IPersonSet
3327
34from lp.testing import TestCaseWithFactory28from lp.testing import TestCaseWithFactory
@@ -467,85 +461,5 @@
467 self.failUnless("Activity %s" % i in messages)461 self.failUnless("Activity %s" % i in messages)
468462
469463
470class TestBugWatchScheduler(TestCaseWithFactory):
471 """Tests for the BugWatchScheduler, which runs as part of garbo."""
472
473 layer = DatabaseFunctionalLayer
474
475 def setUp(self):
476 super(TestBugWatchScheduler, self).setUp('foo.bar@canonical.com')
477 # We'll make sure that all the other bug watches look like
478 # they've been scheduled so that only our watch gets scheduled.
479 for watch in getUtility(IBugWatchSet).search():
480 watch.next_check = datetime.now(utc)
481 self.bug_watch = self.factory.makeBugWatch()
482 self.scheduler = BugWatchScheduler(QuietFakeLogger())
483 transaction.commit()
484
485 def test_scheduler_schedules_unchecked_watches(self):
486 # The BugWatchScheduler will schedule a BugWatch that has never
487 # been checked to be checked immediately.
488 self.bug_watch.next_check = None
489 self.scheduler(1)
490
491 self.assertNotEqual(None, self.bug_watch.next_check)
492 self.assertTrue(
493 self.bug_watch.next_check <= datetime.now(utc))
494
495 def test_scheduler_schedules_working_watches(self):
496 # If a watch has been checked and has never failed its next
497 # check will be scheduled for 24 hours after its last check.
498 now = datetime.now(utc)
499 self.bug_watch.lastchecked = now
500 self.bug_watch.next_check = None
501 transaction.commit()
502 self.scheduler(1)
503
504 self.assertEqual(
505 now + timedelta(hours=24), self.bug_watch.next_check)
506
507 def test_scheduler_schedules_failing_watches(self):
508 # If a watch has failed once, it will be scheduled more than 24
509 # hours after its last check.
510 now = datetime.now(utc)
511 self.bug_watch.lastchecked = now
512
513 # The delay depends on the number of failures that the watch has
514 # had.
515 for failure_count in range(1, 6):
516 self.bug_watch.next_check = None
517 self.bug_watch.addActivity(
518 result=BugWatchActivityStatus.BUG_NOT_FOUND)
519 transaction.commit()
520 self.scheduler(1)
521
522 coefficient = self.scheduler.delay_coefficient * failure_count
523 self.assertEqual(
524 now + timedelta(days=1 + coefficient),
525 self.bug_watch.next_check)
526
527 # The scheduler only looks at the last 5 activity items, so even
528 # if there have been more failures the maximum delay will be 7
529 # days.
530 for count in range(10):
531 self.bug_watch.addActivity(
532 result=BugWatchActivityStatus.BUG_NOT_FOUND)
533 self.bug_watch.next_check = None
534 transaction.commit()
535 self.scheduler(1)
536 self.assertEqual(
537 now + timedelta(days=7), self.bug_watch.next_check)
538
539 def test_scheduler_doesnt_schedule_scheduled_watches(self):
540 # The scheduler will ignore watches whose next_check has been
541 # set.
542 next_check_date = datetime.now(utc) + timedelta(days=1)
543 self.bug_watch.next_check = next_check_date
544 transaction.commit()
545 self.scheduler(1)
546
547 self.assertEqual(next_check_date, self.bug_watch.next_check)
548
549
550def test_suite():464def test_suite():
551 return unittest.TestLoader().loadTestsFromName(__name__)465 return unittest.TestLoader().loadTestsFromName(__name__)