Launchpad itself

Merge lp:~allenap/launchpad/dynamic-batch-size-bug-546085 into lp:launchpad/db-devel

  1. dynamic-batch-size-bug-546085
  2. Merge into db-devel
Proposed by Gavin Panella
Status: Merged
Approved by: Gavin Panella
Approved revision: no longer in the source branch.
Merged at revision: not available
Proposed branch: lp:~allenap/launchpad/dynamic-batch-size-bug-546085
Merge into: lp:launchpad/db-devel
Diff against target: 595 lines (+202/-125)
11 files modified
lib/lp/bugs/doc/checkwatches.txt (+6/-3)
lib/lp/bugs/doc/externalbugtracker-debbugs.txt (+3/-3)
lib/lp/bugs/doc/externalbugtracker.txt (+5/-2)
lib/lp/bugs/externalbugtracker/__init__.py (+4/-1)
lib/lp/bugs/externalbugtracker/base.py (+5/-1)
lib/lp/bugs/externalbugtracker/debbugs.py (+8/-7)
lib/lp/bugs/scripts/checkwatches/tests/test_scheduler.py (+110/-0)
lib/lp/bugs/scripts/checkwatches/tests/test_updater.py (+15/-0)
lib/lp/bugs/scripts/checkwatches/updater.py (+40/-16)
lib/lp/bugs/tests/externalbugtracker.py (+4/-2)
lib/lp/bugs/tests/test_bugwatch.py (+2/-90)
To merge this branch: bzr merge lp:~allenap/launchpad/dynamic-batch-size-bug-546085
Reviewer Review Type Date Requested Status
Edwin Grubbs (community) code Approve
Review via email: mp+22538@code.launchpad.net

Commit message

In checkwatches, calculate a sensible batch size at run-time based on the number of watches there are for a bug tracker.

Description of the change

When checkwatches runs it observes a batch_size attribute on the object representing the remote system. This is fine, but for bug trackers with an especially large number of bug watches the batch size is a bit too low. This branch makes batch_size equal to 2% of the number of bug watches recorded for a bug tracker, or 100, whichever is greater. For GNOME Bugs this works out at about 300 (GNOME Bugs is the largest active bug tracker registered in Launchpad, with >15000 bug watches).

I've also moved the test for BugWatchScheduler into the lp.bugs.scripts.checkwatches.tests package, and the tests for the updater too.

To post a comment you must log in.
Revision history for this message
Edwin Grubbs (edwin-grubbs) wrote :

Hi Gavin,

This is a nice branch. I just have two comments besides the lint errors.

merge-conditional

-Edwin

== Pyflakes notices ==

lib/lp/bugs/externalbugtracker/debbugs.py
    124: local variable 'severity' is assigned to but never used

lib/lp/bugs/scripts/checkwatches/tests/test_updater.py
    157: local variable 'bug' is assigned to but never used

>=== modified file 'lib/lp/bugs/externalbugtracker/debbugs.py'
>--- lib/lp/bugs/externalbugtracker/debbugs.py 2010-03-16 16:52:42 +0000
>+++ lib/lp/bugs/externalbugtracker/debbugs.py 2010-03-31 15:33:31 +0000
>@@ -61,7 +61,7 @@
> # Because we keep a local copy of debbugs, we remove the batch_size
> # limit so that all debbugs watches that need checking will be
> # checked each time checkwatches runs.
>- batch_size = None
>+ batch_size = 0

Zero is a magic number. It would be good to define it as a constant
as opposed to relying on comments to explain what it means. You might
want to define the constant as a unique object such as:
    UNLIMITED = object()
so that there is never the possiblity of someone using zero and getting
the opposite results than they expect. That would allow you to get
rid of this if-statement in _getRemoteIdsToCheck() where you change
the meaning of the magic values.
    if batch_size == 0:
        batch_size = None

>
> def __init__(self, baseurl, db_location=None):
> super(DebBugs, self).__init__(baseurl)
>
>=== modified file 'lib/lp/bugs/scripts/checkwatches/updater.py'
>--- lib/lp/bugs/scripts/checkwatches/updater.py 2010-03-26 21:03:30 +0000
>+++ lib/lp/bugs/scripts/checkwatches/updater.py 2010-03-31 15:33:31 +0000
>@@ -242,6 +247,22 @@
> yield item
>
>
>+def suggest_batch_size(remote_system, num_watches):
>+ """Suggest a value for batch_size if it's not set.
>+
>+ Givend the number of bug watches for an `remote_system`, this sets
>+ a suggested batch size on it. If `remote_system` already has a
>+ batch size set, this does not override it.
>+
>+ :param remote_system: An `ExternalBugTracker`.
>+ :param num_watches: The number of watches for `remote_system`.
>+ """
>+ if remote_system.batch_size is None:
>+ remote_system.batch_size = max(
>+ SUGGESTED_BATCH_SIZE_MIN, int(
>+ SUGGESTED_BATCH_SIZE_PROPORTION * num_watches))

I think it would be easier to read with this wrapping.
        remote_system.batch_size = max(
            SUGGESTED_BATCH_SIZE_MIN,
            int(SUGGESTED_BATCH_SIZE_PROPORTION * num_watches))

>+
> class BugWatchUpdater(object):
> """Takes responsibility for updating remote bug watches."""
>

review: Approve (code)
Revision history for this message
Gavin Panella (allenap) wrote :

Post-review diff, constructed from revisions 9177, 9178 and 9180.

post-review-combined.diff
0-- lib/lp/bugs/externalbugtracker/__init__.py 2009-06-25 00:40:31 +00000++ lib/lp/bugs/externalbugtracker/__init__.py 2010-04-07 14:25:18 +0000
@@ -7,7 +7,7 @@
77
8__metaclass__ = type8__metaclass__ = type
9__all__ = [9__all__ = [
10 'get_external_bugtracker',10 'BATCH_SIZE_UNLIMITED',
11 'BugNotFound',11 'BugNotFound',
12 'BugTrackerConnectError',12 'BugTrackerConnectError',
13 'BugWatchUpdateError',13 'BugWatchUpdateError',
@@ -30,6 +30,7 @@
30 'UnparseableBugData',30 'UnparseableBugData',
31 'UnparseableBugTrackerVersion',31 'UnparseableBugTrackerVersion',
32 'UnsupportedBugTrackerVersion',32 'UnsupportedBugTrackerVersion',
33 'get_external_bugtracker',
33 ]34 ]
3435
35from lp.bugs.externalbugtracker.base import *36from lp.bugs.externalbugtracker.base import *
@@ -41,6 +42,8 @@
41from lp.bugs.externalbugtracker.rt import *42from lp.bugs.externalbugtracker.rt import *
42from lp.bugs.externalbugtracker.trac import *43from lp.bugs.externalbugtracker.trac import *
43from lp.bugs.interfaces.bugtracker import BugTrackerType44from lp.bugs.interfaces.bugtracker import BugTrackerType
45
46
44BUG_TRACKER_CLASSES = {47BUG_TRACKER_CLASSES = {
45 BugTrackerType.BUGZILLA: Bugzilla,48 BugTrackerType.BUGZILLA: Bugzilla,
46 BugTrackerType.DEBBUGS: DebBugs,49 BugTrackerType.DEBBUGS: DebBugs,
47-- lib/lp/bugs/externalbugtracker/base.py 2010-03-30 12:19:11 +000050++ lib/lp/bugs/externalbugtracker/base.py 2010-04-07 14:25:18 +0000
@@ -5,6 +5,7 @@
55
6__metaclass__ = type6__metaclass__ = type
7__all__ = [7__all__ = [
8 'BATCH_SIZE_UNLIMITED',
8 'BugNotFound',9 'BugNotFound',
9 'BugTrackerAuthenticationError',10 'BugTrackerAuthenticationError',
10 'BugTrackerConnectError',11 'BugTrackerConnectError',
@@ -39,6 +40,9 @@
39# The user agent we send in our requests40# The user agent we send in our requests
40LP_USER_AGENT = "Launchpad Bugscraper/0.2 (https://bugs.launchpad.net/)"41LP_USER_AGENT = "Launchpad Bugscraper/0.2 (https://bugs.launchpad.net/)"
4142
43# To signify that all bug watches should be checked in a single run.
44BATCH_SIZE_UNLIMITED = 0
45
4246
43#47#
44# Errors.48# Errors.
45-- lib/lp/bugs/externalbugtracker/debbugs.py 2010-03-26 17:32:38 +000049++ lib/lp/bugs/externalbugtracker/debbugs.py 2010-04-07 14:25:18 +0000
@@ -22,18 +22,19 @@
2222
23from canonical.config import config23from canonical.config import config
24from canonical.database.sqlbase import commit24from canonical.database.sqlbase import commit
25from canonical.launchpad.interfaces.message import IMessageSet
26from canonical.launchpad.mail import simple_sendmail
27from canonical.launchpad.webapp import urlsplit
28
25from lp.bugs.externalbugtracker import (29from lp.bugs.externalbugtracker import (
26 BugNotFound, BugTrackerConnectError, ExternalBugTracker,30 BATCH_SIZE_UNLIMITED, BugNotFound, BugTrackerConnectError,
27 InvalidBugId, UnknownRemoteStatusError)31 ExternalBugTracker, InvalidBugId, UnknownRemoteStatusError)
28from canonical.launchpad.interfaces.message import IMessageSet32from lp.bugs.externalbugtracker.isolation import ensure_no_transaction
29from lp.bugs.interfaces.bugtask import BugTaskImportance, BugTaskStatus33from lp.bugs.interfaces.bugtask import BugTaskImportance, BugTaskStatus
30from lp.bugs.interfaces.externalbugtracker import (34from lp.bugs.interfaces.externalbugtracker import (
31 ISupportsBugImport, ISupportsCommentImport, ISupportsCommentPushing,35 ISupportsBugImport, ISupportsCommentImport, ISupportsCommentPushing,
32 UNKNOWN_REMOTE_IMPORTANCE)36 UNKNOWN_REMOTE_IMPORTANCE)
33from canonical.launchpad.mail import simple_sendmail
34from lp.bugs.scripts import debbugs37from lp.bugs.scripts import debbugs
35from canonical.launchpad.webapp import urlsplit
36from lp.bugs.externalbugtracker.isolation import ensure_no_transaction
3738
3839
39debbugsstatusmap = {'open': BugTaskStatus.NEW,40debbugsstatusmap = {'open': BugTaskStatus.NEW,
@@ -61,7 +62,7 @@
61 # Because we keep a local copy of debbugs, we remove the batch_size62 # Because we keep a local copy of debbugs, we remove the batch_size
62 # limit so that all debbugs watches that need checking will be63 # limit so that all debbugs watches that need checking will be
63 # checked each time checkwatches runs.64 # checked each time checkwatches runs.
64 batch_size = 065 batch_size = BATCH_SIZE_UNLIMITED
6566
66 def __init__(self, baseurl, db_location=None):67 def __init__(self, baseurl, db_location=None):
67 super(DebBugs, self).__init__(baseurl)68 super(DebBugs, self).__init__(baseurl)
68-- lib/lp/bugs/scripts/checkwatches/updater.py 2010-03-31 10:51:17 +000069++ lib/lp/bugs/scripts/checkwatches/updater.py 2010-04-07 14:25:18 +0000
@@ -61,16 +61,15 @@
6161
62from lp.bugs import externalbugtracker62from lp.bugs import externalbugtracker
63from lp.bugs.externalbugtracker import (63from lp.bugs.externalbugtracker import (
64 BugNotFound, BugTrackerConnectError, BugWatchUpdateError,64 BATCH_SIZE_UNLIMITED, BugNotFound, BugTrackerConnectError,
65 BugWatchUpdateWarning, InvalidBugId, PrivateRemoteBug,65 BugWatchUpdateError, BugWatchUpdateWarning, InvalidBugId,
66 UnknownBugTrackerTypeError, UnknownRemoteStatusError, UnparseableBugData,66 PrivateRemoteBug, UnknownBugTrackerTypeError, UnknownRemoteStatusError,
67 UnparseableBugTrackerVersion, UnsupportedBugTrackerVersion)67 UnparseableBugData, UnparseableBugTrackerVersion,
68from lp.bugs.externalbugtracker.bugzilla import (68 UnsupportedBugTrackerVersion)
69 BugzillaAPI)69from lp.bugs.externalbugtracker.bugzilla import BugzillaAPI
70from lp.bugs.externalbugtracker.isolation import check_no_transaction70from lp.bugs.externalbugtracker.isolation import check_no_transaction
71from lp.bugs.interfaces.bug import IBugSet71from lp.bugs.interfaces.bug import IBugSet
72from lp.bugs.interfaces.externalbugtracker import (72from lp.bugs.interfaces.externalbugtracker import ISupportsBackLinking
73 ISupportsBackLinking)
74from lp.services.limitedlist import LimitedList73from lp.services.limitedlist import LimitedList
75from lp.services.scripts.base import LaunchpadCronScript74from lp.services.scripts.base import LaunchpadCronScript
7675
@@ -250,17 +249,17 @@
250def suggest_batch_size(remote_system, num_watches):249def suggest_batch_size(remote_system, num_watches):
251 """Suggest a value for batch_size if it's not set.250 """Suggest a value for batch_size if it's not set.
252251
253 Givend the number of bug watches for an `remote_system`, this sets252 Given the number of bug watches for a `remote_system`, this sets a
254 a suggested batch size on it. If `remote_system` already has a253 suggested batch size on it. If `remote_system` already has a batch
255 batch size set, this does not override it.254 size set, this does not override it.
256255
257 :param remote_system: An `ExternalBugTracker`.256 :param remote_system: An `ExternalBugTracker`.
258 :param num_watches: The number of watches for `remote_system`.257 :param num_watches: The number of watches for `remote_system`.
259 """258 """
260 if remote_system.batch_size is None:259 if remote_system.batch_size is None:
261 remote_system.batch_size = max(260 remote_system.batch_size = max(
262 SUGGESTED_BATCH_SIZE_MIN, int(261 SUGGESTED_BATCH_SIZE_MIN,
263 SUGGESTED_BATCH_SIZE_PROPORTION * num_watches))262 int(SUGGESTED_BATCH_SIZE_PROPORTION * num_watches))
264263
265264
266class BugWatchUpdater(object):265class BugWatchUpdater(object):
@@ -402,7 +401,12 @@
402 `BaseScheduler`. If no scheduler is given, `SerialScheduler`401 `BaseScheduler`. If no scheduler is given, `SerialScheduler`
403 will be used, which simply runs the jobs in order.402 will be used, which simply runs the jobs in order.
404 """403 """
405 self.log.debug("Using a global batch size of %s" % batch_size)404 if batch_size is None:
405 self.log.debug("No global batch size specified.")
406 elif batch_size == BATCH_SIZE_UNLIMITED:
407 self.log.debug("Using an unlimited global batch size.")
408 else:
409 self.log.debug("Using a global batch size of %s" % batch_size)
406410
407 # Default to using the very simple SerialScheduler.411 # Default to using the very simple SerialScheduler.
408 if scheduler is None:412 if scheduler is None:
@@ -681,11 +685,6 @@
681 # by the ExternalBugTracker.685 # by the ExternalBugTracker.
682 batch_size = remotesystem.batch_size686 batch_size = remotesystem.batch_size
683687
684 if batch_size == 0:
685 # A batch_size of 0 means that there's no batch size limit
686 # for this bug tracker.
687 batch_size = None
688
689 with self.transaction:688 with self.transaction:
690 old_bug_watches = set(689 old_bug_watches = set(
691 bug_watch for bug_watch in bug_watches690 bug_watch for bug_watch in bug_watches
@@ -718,7 +717,7 @@
718 # are actually some bugs that we're interested in so as to717 # are actually some bugs that we're interested in so as to
719 # avoid unnecessary network traffic.718 # avoid unnecessary network traffic.
720 if server_time is not None and len(remote_old_ids) > 0:719 if server_time is not None and len(remote_old_ids) > 0:
721 if batch_size is None:720 if batch_size == BATCH_SIZE_UNLIMITED:
722 remote_old_ids_to_check = (721 remote_old_ids_to_check = (
723 remotesystem.getModifiedRemoteBugs(722 remotesystem.getModifiedRemoteBugs(
724 remote_old_ids, oldest_lastchecked))723 remote_old_ids, oldest_lastchecked))
@@ -746,7 +745,7 @@
746 remote_ids_to_check = chain(745 remote_ids_to_check = chain(
747 remote_ids_with_comments, remote_new_ids, remote_old_ids_to_check)746 remote_ids_with_comments, remote_new_ids, remote_old_ids_to_check)
748747
749 if batch_size is not None:748 if batch_size != BATCH_SIZE_UNLIMITED:
750 # Some remote bug IDs may appear in more than one list so749 # Some remote bug IDs may appear in more than one list so
751 # we must filter the list before slicing.750 # we must filter the list before slicing.
752 remote_ids_to_check = islice(751 remote_ids_to_check = islice(
753-- lib/lp/bugs/doc/checkwatches.txt 2010-03-30 17:25:52 +0000752++ lib/lp/bugs/doc/checkwatches.txt 2010-04-07 16:08:16 +0000
@@ -45,7 +45,7 @@
4545
46 >>> print err46 >>> print err
47 INFO creating lockfile47 INFO creating lockfile
48 DEBUG Using a global batch size of None48 DEBUG No global batch size specified.
49 DEBUG Skipping updating Ubuntu Bugzilla watches.49 DEBUG Skipping updating Ubuntu Bugzilla watches.
50 DEBUG No watches to update on http://bugs.debian.org50 DEBUG No watches to update on http://bugs.debian.org
51 DEBUG No watches to update on mailto:bugs@example.com51 DEBUG No watches to update on mailto:bugs@example.com
@@ -274,7 +274,7 @@
274 ... updater.log = original_log274 ... updater.log = original_log
275 ... externalbugtracker.Roundup.batch_size = batch_size275 ... externalbugtracker.Roundup.batch_size = batch_size
276 ... urllib2.urlopen = urlopen276 ... urllib2.urlopen = urlopen
277 DEBUG Using a global batch size of None277 DEBUG No global batch size specified.
278 INFO Updating 5 watches for 5 bugs on http://bugs.example.com278 INFO Updating 5 watches for 5 bugs on http://bugs.example.com
279 ERROR Connection timed out when updating ... (OOPS-...)279 ERROR Connection timed out when updating ... (OOPS-...)
280280
@@ -412,7 +412,8 @@
412 >>> from lp.bugs.interfaces.externalbugtracker import (412 >>> from lp.bugs.interfaces.externalbugtracker import (
413 ... ISupportsCommentImport, ISupportsCommentPushing,413 ... ISupportsCommentImport, ISupportsCommentPushing,
414 ... ISupportsBackLinking)414 ... ISupportsBackLinking)
415 >>> from lp.bugs.externalbugtracker.base import ExternalBugTracker415 >>> from lp.bugs.externalbugtracker.base import (
416 ... BATCH_SIZE_UNLIMITED, ExternalBugTracker)
416417
417 >>> nowish = datetime.now(utc)418 >>> nowish = datetime.now(utc)
418 >>> class UselessExternalBugTracker(ExternalBugTracker):419 >>> class UselessExternalBugTracker(ExternalBugTracker):
@@ -421,6 +422,8 @@
421 ... ISupportsBackLinking, ISupportsCommentImport,422 ... ISupportsBackLinking, ISupportsCommentImport,
422 ... ISupportsCommentPushing)423 ... ISupportsCommentPushing)
423 ...424 ...
425 ... batch_size = BATCH_SIZE_UNLIMITED
426 ...
424 ... def initializeRemoteBugDB(self, bug_ids):427 ... def initializeRemoteBugDB(self, bug_ids):
425 ... # This just exists to stop errors from being raised.428 ... # This just exists to stop errors from being raised.
426 ... pass429 ... pass
427-- lib/lp/bugs/doc/externalbugtracker-debbugs.txt 2010-03-26 13:48:53 +0000430++ lib/lp/bugs/doc/externalbugtracker-debbugs.txt 2010-04-07 16:08:16 +0000
@@ -14,7 +14,7 @@
14You can specify the db_location explicitly:14You can specify the db_location explicitly:
1515
16 >>> from lp.bugs.externalbugtracker import (16 >>> from lp.bugs.externalbugtracker import (
17 ... DebBugs)17 ... BATCH_SIZE_UNLIMITED, DebBugs)
18 >>> from canonical.testing import LaunchpadZopelessLayer18 >>> from canonical.testing import LaunchpadZopelessLayer
19 >>> txn = LaunchpadZopelessLayer.txn19 >>> txn = LaunchpadZopelessLayer.txn
20 >>> external_debbugs = DebBugs(20 >>> external_debbugs = DebBugs(
@@ -42,8 +42,8 @@
42worry about batching bug watch updates for performance reasons, so42worry about batching bug watch updates for performance reasons, so
43DebBugs instances don't have a batch_size limit.43DebBugs instances don't have a batch_size limit.
4444
45 >>> print external_debbugs.batch_size45 >>> external_debbugs.batch_size == BATCH_SIZE_UNLIMITED
46 None46 True
4747
4848
49== Retrieving bug status from the debbugs database ==49== Retrieving bug status from the debbugs database ==
50-- lib/lp/bugs/tests/externalbugtracker.py 2009-12-23 12:14:59 +000050++ lib/lp/bugs/tests/externalbugtracker.py 2010-04-07 16:08:16 +0000
@@ -24,8 +24,8 @@
24from canonical.config import config24from canonical.config import config
25from canonical.database.sqlbase import commit, ZopelessTransactionManager25from canonical.database.sqlbase import commit, ZopelessTransactionManager
26from lp.bugs.externalbugtracker import (26from lp.bugs.externalbugtracker import (
27 BugNotFound, BugTrackerConnectError, Bugzilla, DebBugs,27 BATCH_SIZE_UNLIMITED, BugNotFound, BugTrackerConnectError, Bugzilla,
28 ExternalBugTracker, Mantis, RequestTracker, Roundup, SourceForge,28 DebBugs, ExternalBugTracker, Mantis, RequestTracker, Roundup, SourceForge,
29 Trac)29 Trac)
30from lp.bugs.externalbugtracker.trac import (30from lp.bugs.externalbugtracker.trac import (
31 FAULT_TICKET_NOT_FOUND, LP_PLUGIN_BUG_IDS_ONLY, LP_PLUGIN_FULL,31 FAULT_TICKET_NOT_FOUND, LP_PLUGIN_BUG_IDS_ONLY, LP_PLUGIN_FULL,
@@ -152,6 +152,8 @@
152 implementation, though it doesn't actually do anything.152 implementation, though it doesn't actually do anything.
153 """153 """
154154
155 batch_size = BATCH_SIZE_UNLIMITED
156
155 def __init__(self, baseurl='http://example.com/'):157 def __init__(self, baseurl='http://example.com/'):
156 super(TestExternalBugTracker, self).__init__(baseurl)158 super(TestExternalBugTracker, self).__init__(baseurl)
157159
Revision history for this message
Gavin Panella (allenap) wrote :

Conversation re. additional changes from IRC:

<EdwinGrubbs> allenap: I'm surprised that you added a line to tests/externalbugtracker.py to set the batch_size to BATCH_SIZE_UNLIMITED, but it didn't change the results of any of the tests.
<allenap> EdwinGrubbs: It kind of does the same as lines 153 to 157 used to do.
<allenap> EdwinGrubbs: ... because most calls into updateBugWatches() and _getRemoteIdsToCheck() didn't specify batch_size.
<EdwinGrubbs> allenap: ok, that makes sense. Looks good.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/lp/bugs/doc/checkwatches.txt'
--- lib/lp/bugs/doc/checkwatches.txt 2010-04-06 21:36:16 +0000
+++ lib/lp/bugs/doc/checkwatches.txt 2010-04-08 15:38:28 +0000
@@ -45,7 +45,7 @@
4545
46 >>> print err46 >>> print err
47 INFO Creating lockfile: /var/lock/launchpad-checkwatches.lock47 INFO Creating lockfile: /var/lock/launchpad-checkwatches.lock
48 DEBUG Using a global batch size of None48 DEBUG No global batch size specified.
49 DEBUG Skipping updating Ubuntu Bugzilla watches.49 DEBUG Skipping updating Ubuntu Bugzilla watches.
50 DEBUG No watches to update on http://bugs.debian.org50 DEBUG No watches to update on http://bugs.debian.org
51 DEBUG No watches to update on mailto:bugs@example.com51 DEBUG No watches to update on mailto:bugs@example.com
@@ -274,7 +274,7 @@
274 ... updater.log = original_log274 ... updater.log = original_log
275 ... externalbugtracker.Roundup.batch_size = batch_size275 ... externalbugtracker.Roundup.batch_size = batch_size
276 ... urllib2.urlopen = urlopen276 ... urllib2.urlopen = urlopen
277 DEBUG Using a global batch size of None277 DEBUG No global batch size specified.
278 INFO Updating 5 watches for 5 bugs on http://bugs.example.com278 INFO Updating 5 watches for 5 bugs on http://bugs.example.com
279 ERROR Connection timed out when updating ... (OOPS-...)279 ERROR Connection timed out when updating ... (OOPS-...)
280280
@@ -412,7 +412,8 @@
412 >>> from lp.bugs.interfaces.externalbugtracker import (412 >>> from lp.bugs.interfaces.externalbugtracker import (
413 ... ISupportsCommentImport, ISupportsCommentPushing,413 ... ISupportsCommentImport, ISupportsCommentPushing,
414 ... ISupportsBackLinking)414 ... ISupportsBackLinking)
415 >>> from lp.bugs.externalbugtracker.base import ExternalBugTracker415 >>> from lp.bugs.externalbugtracker.base import (
416 ... BATCH_SIZE_UNLIMITED, ExternalBugTracker)
416417
417 >>> nowish = datetime.now(utc)418 >>> nowish = datetime.now(utc)
418 >>> class UselessExternalBugTracker(ExternalBugTracker):419 >>> class UselessExternalBugTracker(ExternalBugTracker):
@@ -421,6 +422,8 @@
421 ... ISupportsBackLinking, ISupportsCommentImport,422 ... ISupportsBackLinking, ISupportsCommentImport,
422 ... ISupportsCommentPushing)423 ... ISupportsCommentPushing)
423 ...424 ...
425 ... batch_size = BATCH_SIZE_UNLIMITED
426 ...
424 ... def initializeRemoteBugDB(self, bug_ids):427 ... def initializeRemoteBugDB(self, bug_ids):
425 ... # This just exists to stop errors from being raised.428 ... # This just exists to stop errors from being raised.
426 ... pass429 ... pass
427430
=== modified file 'lib/lp/bugs/doc/externalbugtracker-debbugs.txt'
--- lib/lp/bugs/doc/externalbugtracker-debbugs.txt 2010-03-26 13:48:53 +0000
+++ lib/lp/bugs/doc/externalbugtracker-debbugs.txt 2010-04-08 15:38:28 +0000
@@ -14,7 +14,7 @@
14You can specify the db_location explicitly:14You can specify the db_location explicitly:
1515
16 >>> from lp.bugs.externalbugtracker import (16 >>> from lp.bugs.externalbugtracker import (
17 ... DebBugs)17 ... BATCH_SIZE_UNLIMITED, DebBugs)
18 >>> from canonical.testing import LaunchpadZopelessLayer18 >>> from canonical.testing import LaunchpadZopelessLayer
19 >>> txn = LaunchpadZopelessLayer.txn19 >>> txn = LaunchpadZopelessLayer.txn
20 >>> external_debbugs = DebBugs(20 >>> external_debbugs = DebBugs(
@@ -42,8 +42,8 @@
42worry about batching bug watch updates for performance reasons, so42worry about batching bug watch updates for performance reasons, so
43DebBugs instances don't have a batch_size limit.43DebBugs instances don't have a batch_size limit.
4444
45 >>> print external_debbugs.batch_size45 >>> external_debbugs.batch_size == BATCH_SIZE_UNLIMITED
46 None46 True
4747
4848
49== Retrieving bug status from the debbugs database ==49== Retrieving bug status from the debbugs database ==
5050
=== modified file 'lib/lp/bugs/doc/externalbugtracker.txt'
--- lib/lp/bugs/doc/externalbugtracker.txt 2010-03-26 15:12:59 +0000
+++ lib/lp/bugs/doc/externalbugtracker.txt 2010-04-08 15:38:28 +0000
@@ -1042,13 +1042,16 @@
10421042
1043_getRemoteIdsToCheck() will interpret a batch_size parameter of 0 as an1043_getRemoteIdsToCheck() will interpret a batch_size parameter of 0 as an
1044instruction to ignore the batch size limitation altogether and just return all1044instruction to ignore the batch size limitation altogether and just return all
1045the IDs that need checking.1045the IDs that need checking. The constant BATCH_SIZE_UNLIMITED should
1046be used in place of using 0 verbatim.
1047
1048 >>> from lp.bugs.externalbugtracker import BATCH_SIZE_UNLIMITED
10461049
1047 >>> ids = bug_watch_updater._getRemoteIdsToCheck(1050 >>> ids = bug_watch_updater._getRemoteIdsToCheck(
1048 ... external_bugtracker,1051 ... external_bugtracker,
1049 ... unchecked_watches + watches_with_comments + old_watches,1052 ... unchecked_watches + watches_with_comments + old_watches,
1050 ... external_bugtracker.getCurrentDBTime(), utc_now,1053 ... external_bugtracker.getCurrentDBTime(), utc_now,
1051 ... batch_size=0)1054 ... batch_size=BATCH_SIZE_UNLIMITED)
1052 >>> print sorted(ids['remote_ids_to_check'])1055 >>> print sorted(ids['remote_ids_to_check'])
1053 [u'0', u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9']1056 [u'0', u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9']
10541057
10551058
=== modified file 'lib/lp/bugs/externalbugtracker/__init__.py'
--- lib/lp/bugs/externalbugtracker/__init__.py 2009-06-25 00:40:31 +0000
+++ lib/lp/bugs/externalbugtracker/__init__.py 2010-04-08 15:38:28 +0000
@@ -7,7 +7,7 @@
77
8__metaclass__ = type8__metaclass__ = type
9__all__ = [9__all__ = [
10 'get_external_bugtracker',10 'BATCH_SIZE_UNLIMITED',
11 'BugNotFound',11 'BugNotFound',
12 'BugTrackerConnectError',12 'BugTrackerConnectError',
13 'BugWatchUpdateError',13 'BugWatchUpdateError',
@@ -30,6 +30,7 @@
30 'UnparseableBugData',30 'UnparseableBugData',
31 'UnparseableBugTrackerVersion',31 'UnparseableBugTrackerVersion',
32 'UnsupportedBugTrackerVersion',32 'UnsupportedBugTrackerVersion',
33 'get_external_bugtracker',
33 ]34 ]
3435
35from lp.bugs.externalbugtracker.base import *36from lp.bugs.externalbugtracker.base import *
@@ -41,6 +42,8 @@
41from lp.bugs.externalbugtracker.rt import *42from lp.bugs.externalbugtracker.rt import *
42from lp.bugs.externalbugtracker.trac import *43from lp.bugs.externalbugtracker.trac import *
43from lp.bugs.interfaces.bugtracker import BugTrackerType44from lp.bugs.interfaces.bugtracker import BugTrackerType
45
46
44BUG_TRACKER_CLASSES = {47BUG_TRACKER_CLASSES = {
45 BugTrackerType.BUGZILLA: Bugzilla,48 BugTrackerType.BUGZILLA: Bugzilla,
46 BugTrackerType.DEBBUGS: DebBugs,49 BugTrackerType.DEBBUGS: DebBugs,
4750
=== modified file 'lib/lp/bugs/externalbugtracker/base.py'
--- lib/lp/bugs/externalbugtracker/base.py 2010-03-26 15:12:59 +0000
+++ lib/lp/bugs/externalbugtracker/base.py 2010-04-08 15:38:28 +0000
@@ -5,6 +5,7 @@
55
6__metaclass__ = type6__metaclass__ = type
7__all__ = [7__all__ = [
8 'BATCH_SIZE_UNLIMITED',
8 'BugNotFound',9 'BugNotFound',
9 'BugTrackerAuthenticationError',10 'BugTrackerAuthenticationError',
10 'BugTrackerConnectError',11 'BugTrackerConnectError',
@@ -39,6 +40,9 @@
39# The user agent we send in our requests40# The user agent we send in our requests
40LP_USER_AGENT = "Launchpad Bugscraper/0.2 (https://bugs.launchpad.net/)"41LP_USER_AGENT = "Launchpad Bugscraper/0.2 (https://bugs.launchpad.net/)"
4142
43# To signify that all bug watches should be checked in a single run.
44BATCH_SIZE_UNLIMITED = 0
45
4246
43#47#
44# Errors.48# Errors.
@@ -133,7 +137,7 @@
133137
134 implements(IExternalBugTracker)138 implements(IExternalBugTracker)
135139
136 batch_size = 100140 batch_size = None
137 batch_query_threshold = config.checkwatches.batch_query_threshold141 batch_query_threshold = config.checkwatches.batch_query_threshold
138 comment_template = 'default_remotecomment_template.txt'142 comment_template = 'default_remotecomment_template.txt'
139143
140144
=== modified file 'lib/lp/bugs/externalbugtracker/debbugs.py'
--- lib/lp/bugs/externalbugtracker/debbugs.py 2010-03-31 20:09:55 +0000
+++ lib/lp/bugs/externalbugtracker/debbugs.py 2010-04-08 15:38:28 +0000
@@ -22,18 +22,19 @@
2222
23from canonical.config import config23from canonical.config import config
24from canonical.database.sqlbase import commit24from canonical.database.sqlbase import commit
25from canonical.launchpad.interfaces.message import IMessageSet
26from canonical.launchpad.mail import simple_sendmail
27from canonical.launchpad.webapp import urlsplit
28
25from lp.bugs.externalbugtracker import (29from lp.bugs.externalbugtracker import (
26 BugNotFound, BugTrackerConnectError, ExternalBugTracker,30 BATCH_SIZE_UNLIMITED, BugNotFound, BugTrackerConnectError,
27 InvalidBugId, UnknownRemoteStatusError)31 ExternalBugTracker, InvalidBugId, UnknownRemoteStatusError)
28from canonical.launchpad.interfaces.message import IMessageSet32from lp.bugs.externalbugtracker.isolation import ensure_no_transaction
29from lp.bugs.interfaces.bugtask import BugTaskImportance, BugTaskStatus33from lp.bugs.interfaces.bugtask import BugTaskImportance, BugTaskStatus
30from lp.bugs.interfaces.externalbugtracker import (34from lp.bugs.interfaces.externalbugtracker import (
31 ISupportsBugImport, ISupportsCommentImport, ISupportsCommentPushing,35 ISupportsBugImport, ISupportsCommentImport, ISupportsCommentPushing,
32 UNKNOWN_REMOTE_IMPORTANCE)36 UNKNOWN_REMOTE_IMPORTANCE)
33from canonical.launchpad.mail import simple_sendmail
34from lp.bugs.scripts import debbugs37from lp.bugs.scripts import debbugs
35from canonical.launchpad.webapp import urlsplit
36from lp.bugs.externalbugtracker.isolation import ensure_no_transaction
3738
3839
39debbugsstatusmap = {'open': BugTaskStatus.NEW,40debbugsstatusmap = {'open': BugTaskStatus.NEW,
@@ -59,7 +60,7 @@
59 # Because we keep a local copy of debbugs, we remove the batch_size60 # Because we keep a local copy of debbugs, we remove the batch_size
60 # limit so that all debbugs watches that need checking will be61 # limit so that all debbugs watches that need checking will be
61 # checked each time checkwatches runs.62 # checked each time checkwatches runs.
62 batch_size = None63 batch_size = BATCH_SIZE_UNLIMITED
6364
64 def __init__(self, baseurl, db_location=None):65 def __init__(self, baseurl, db_location=None):
65 super(DebBugs, self).__init__(baseurl)66 super(DebBugs, self).__init__(baseurl)
6667
=== added directory 'lib/lp/bugs/scripts/checkwatches/tests'
=== added file 'lib/lp/bugs/scripts/checkwatches/tests/__init__.py'
=== added file 'lib/lp/bugs/scripts/checkwatches/tests/test_scheduler.py'
--- lib/lp/bugs/scripts/checkwatches/tests/test_scheduler.py 1970-01-01 00:00:00 +0000
+++ lib/lp/bugs/scripts/checkwatches/tests/test_scheduler.py 2010-04-08 15:38:28 +0000
@@ -0,0 +1,110 @@
1# Copyright 2010 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""XXX: Module docstring goes here."""
5
6__metaclass__ = type
7
8import transaction
9import unittest
10
11from datetime import datetime, timedelta
12from pytz import utc
13
14from zope.component import getUtility
15
16from canonical.launchpad.scripts.logger import QuietFakeLogger
17from canonical.testing import DatabaseFunctionalLayer
18
19from lp.bugs.interfaces.bugwatch import (
20 BugWatchActivityStatus, IBugWatchSet)
21from lp.bugs.scripts.checkwatches.scheduler import BugWatchScheduler
22
23from lp.testing import TestCaseWithFactory
24
25
26class TestBugWatchScheduler(TestCaseWithFactory):
27 """Tests for the BugWatchScheduler, which runs as part of garbo."""
28
29 layer = DatabaseFunctionalLayer
30
31 def setUp(self):
32 super(TestBugWatchScheduler, self).setUp('foo.bar@canonical.com')
33 # We'll make sure that all the other bug watches look like
34 # they've been scheduled so that only our watch gets scheduled.
35 for watch in getUtility(IBugWatchSet).search():
36 watch.next_check = datetime.now(utc)
37 self.bug_watch = self.factory.makeBugWatch()
38 self.scheduler = BugWatchScheduler(QuietFakeLogger())
39 transaction.commit()
40
41 def test_scheduler_schedules_unchecked_watches(self):
42 # The BugWatchScheduler will schedule a BugWatch that has never
43 # been checked to be checked immediately.
44 self.bug_watch.next_check = None
45 self.scheduler(1)
46
47 self.assertNotEqual(None, self.bug_watch.next_check)
48 self.assertTrue(
49 self.bug_watch.next_check <= datetime.now(utc))
50
51 def test_scheduler_schedules_working_watches(self):
52 # If a watch has been checked and has never failed its next
53 # check will be scheduled for 24 hours after its last check.
54 now = datetime.now(utc)
55 # Add some succesful activity to ensure that successful activity
56 # is handled correctly.
57 self.bug_watch.addActivity()
58 self.bug_watch.lastchecked = now
59 self.bug_watch.next_check = None
60 transaction.commit()
61 self.scheduler(1)
62
63 self.assertEqual(
64 now + timedelta(hours=24), self.bug_watch.next_check)
65
66 def test_scheduler_schedules_failing_watches(self):
67 # If a watch has failed once, it will be scheduled more than 24
68 # hours after its last check.
69 now = datetime.now(utc)
70 self.bug_watch.lastchecked = now
71
72 # The delay depends on the number of failures that the watch has
73 # had.
74 for failure_count in range(1, 6):
75 self.bug_watch.next_check = None
76 self.bug_watch.addActivity(
77 result=BugWatchActivityStatus.BUG_NOT_FOUND)
78 transaction.commit()
79 self.scheduler(1)
80
81 coefficient = self.scheduler.delay_coefficient * failure_count
82 self.assertEqual(
83 now + timedelta(days=1 + coefficient),
84 self.bug_watch.next_check)
85
86 # The scheduler only looks at the last 5 activity items, so even
87 # if there have been more failures the maximum delay will be 7
88 # days.
89 for count in range(10):
90 self.bug_watch.addActivity(
91 result=BugWatchActivityStatus.BUG_NOT_FOUND)
92 self.bug_watch.next_check = None
93 transaction.commit()
94 self.scheduler(1)
95 self.assertEqual(
96 now + timedelta(days=7), self.bug_watch.next_check)
97
98 def test_scheduler_doesnt_schedule_scheduled_watches(self):
99 # The scheduler will ignore watches whose next_check has been
100 # set.
101 next_check_date = datetime.now(utc) + timedelta(days=1)
102 self.bug_watch.next_check = next_check_date
103 transaction.commit()
104 self.scheduler(1)
105
106 self.assertEqual(next_check_date, self.bug_watch.next_check)
107
108
109def test_suite():
110 return unittest.TestLoader().loadTestsFromName(__name__)
0111
=== renamed file 'lib/lp/bugs/scripts/tests/test_checkwatches.py' => 'lib/lp/bugs/scripts/checkwatches/tests/test_updater.py'
--- lib/lp/bugs/scripts/tests/test_checkwatches.py 2010-03-25 21:59:48 +0000
+++ lib/lp/bugs/scripts/checkwatches/tests/test_updater.py 2010-04-08 15:38:28 +0000
@@ -167,6 +167,21 @@
167 # stop SQL statemnet logging.167 # stop SQL statemnet logging.
168 clear_request_started()168 clear_request_started()
169169
170 def test_suggest_batch_size(self):
171 class RemoteSystem: pass
172 remote_system = RemoteSystem()
173 # When the batch_size is None, suggest_batch_size() will set
174 # it accordingly.
175 remote_system.batch_size = None
176 checkwatches.updater.suggest_batch_size(remote_system, 1)
177 self.failUnlessEqual(100, remote_system.batch_size)
178 remote_system.batch_size = None
179 checkwatches.updater.suggest_batch_size(remote_system, 12350)
180 self.failUnlessEqual(247, remote_system.batch_size)
181 # If the batch_size is already set, it will not be changed.
182 checkwatches.updater.suggest_batch_size(remote_system, 99999)
183 self.failUnlessEqual(247, remote_system.batch_size)
184
170185
171class TestUpdateBugsWithLinkedQuestions(unittest.TestCase):186class TestUpdateBugsWithLinkedQuestions(unittest.TestCase):
172 """Tests for updating bugs with linked questions."""187 """Tests for updating bugs with linked questions."""
173188
=== modified file 'lib/lp/bugs/scripts/checkwatches/updater.py'
--- lib/lp/bugs/scripts/checkwatches/updater.py 2010-03-26 21:03:30 +0000
+++ lib/lp/bugs/scripts/checkwatches/updater.py 2010-04-08 15:38:28 +0000
@@ -61,16 +61,15 @@
6161
62from lp.bugs import externalbugtracker62from lp.bugs import externalbugtracker
63from lp.bugs.externalbugtracker import (63from lp.bugs.externalbugtracker import (
64 BugNotFound, BugTrackerConnectError, BugWatchUpdateError,64 BATCH_SIZE_UNLIMITED, BugNotFound, BugTrackerConnectError,
65 BugWatchUpdateWarning, InvalidBugId, PrivateRemoteBug,65 BugWatchUpdateError, BugWatchUpdateWarning, InvalidBugId,
66 UnknownBugTrackerTypeError, UnknownRemoteStatusError, UnparseableBugData,66 PrivateRemoteBug, UnknownBugTrackerTypeError, UnknownRemoteStatusError,
67 UnparseableBugTrackerVersion, UnsupportedBugTrackerVersion)67 UnparseableBugData, UnparseableBugTrackerVersion,
68from lp.bugs.externalbugtracker.bugzilla import (68 UnsupportedBugTrackerVersion)
69 BugzillaAPI)69from lp.bugs.externalbugtracker.bugzilla import BugzillaAPI
70from lp.bugs.externalbugtracker.isolation import check_no_transaction70from lp.bugs.externalbugtracker.isolation import check_no_transaction
71from lp.bugs.interfaces.bug import IBugSet71from lp.bugs.interfaces.bug import IBugSet
72from lp.bugs.interfaces.externalbugtracker import (72from lp.bugs.interfaces.externalbugtracker import ISupportsBackLinking
73 ISupportsBackLinking)
74from lp.services.limitedlist import LimitedList73from lp.services.limitedlist import LimitedList
75from lp.services.scripts.base import LaunchpadCronScript74from lp.services.scripts.base import LaunchpadCronScript
7675
@@ -78,6 +77,11 @@
78SYNCABLE_GNOME_PRODUCTS = []77SYNCABLE_GNOME_PRODUCTS = []
79MAX_SQL_STATEMENTS_LOGGED = 1000078MAX_SQL_STATEMENTS_LOGGED = 10000
8079
80# The minimum batch size to suggest to an IExternalBugTracker.
81SUGGESTED_BATCH_SIZE_MIN = 100
82# The proportion of all watches to suggest as a batch size.
83SUGGESTED_BATCH_SIZE_PROPORTION = 0.02
84
8185
82class TooMuchTimeSkew(BugWatchUpdateError):86class TooMuchTimeSkew(BugWatchUpdateError):
83 """Time difference between ourselves and the remote server is too much."""87 """Time difference between ourselves and the remote server is too much."""
@@ -242,6 +246,22 @@
242 yield item246 yield item
243247
244248
249def suggest_batch_size(remote_system, num_watches):
250 """Suggest a value for batch_size if it's not set.
251
252 Given the number of bug watches for a `remote_system`, this sets a
253 suggested batch size on it. If `remote_system` already has a batch
254 size set, this does not override it.
255
256 :param remote_system: An `ExternalBugTracker`.
257 :param num_watches: The number of watches for `remote_system`.
258 """
259 if remote_system.batch_size is None:
260 remote_system.batch_size = max(
261 SUGGESTED_BATCH_SIZE_MIN,
262 int(SUGGESTED_BATCH_SIZE_PROPORTION * num_watches))
263
264
245class BugWatchUpdater(object):265class BugWatchUpdater(object):
246 """Takes responsibility for updating remote bug watches."""266 """Takes responsibility for updating remote bug watches."""
247267
@@ -381,7 +401,12 @@
381 `BaseScheduler`. If no scheduler is given, `SerialScheduler`401 `BaseScheduler`. If no scheduler is given, `SerialScheduler`
382 will be used, which simply runs the jobs in order.402 will be used, which simply runs the jobs in order.
383 """403 """
384 self.log.debug("Using a global batch size of %s" % batch_size)404 if batch_size is None:
405 self.log.debug("No global batch size specified.")
406 elif batch_size == BATCH_SIZE_UNLIMITED:
407 self.log.debug("Using an unlimited global batch size.")
408 else:
409 self.log.debug("Using a global batch size of %s" % batch_size)
385410
386 # Default to using the very simple SerialScheduler.411 # Default to using the very simple SerialScheduler.
387 if scheduler is None:412 if scheduler is None:
@@ -497,6 +522,7 @@
497 def _getExternalBugTrackersAndWatches(self, bug_tracker, bug_watches):522 def _getExternalBugTrackersAndWatches(self, bug_tracker, bug_watches):
498 """Return an `ExternalBugTracker` instance for `bug_tracker`."""523 """Return an `ExternalBugTracker` instance for `bug_tracker`."""
499 with self.transaction:524 with self.transaction:
525 num_watches = bug_tracker.watches.count()
500 remotesystem = (526 remotesystem = (
501 externalbugtracker.get_external_bugtracker(bug_tracker))527 externalbugtracker.get_external_bugtracker(bug_tracker))
502 # We special-case the Gnome Bugzilla.528 # We special-case the Gnome Bugzilla.
@@ -506,6 +532,9 @@
506 # Probe the remote system for additional capabilities.532 # Probe the remote system for additional capabilities.
507 remotesystem_to_use = remotesystem.getExternalBugTrackerToUse()533 remotesystem_to_use = remotesystem.getExternalBugTrackerToUse()
508534
535 # Try to hint at how many bug watches to check each time.
536 suggest_batch_size(remotesystem_to_use, num_watches)
537
509 if (is_gnome_bugzilla and538 if (is_gnome_bugzilla and
510 isinstance(remotesystem_to_use, BugzillaAPI) and539 isinstance(remotesystem_to_use, BugzillaAPI) and
511 len(self._syncable_gnome_products) > 0):540 len(self._syncable_gnome_products) > 0):
@@ -656,11 +685,6 @@
656 # by the ExternalBugTracker.685 # by the ExternalBugTracker.
657 batch_size = remotesystem.batch_size686 batch_size = remotesystem.batch_size
658687
659 if batch_size == 0:
660 # A batch_size of 0 means that there's no batch size limit
661 # for this bug tracker.
662 batch_size = None
663
664 with self.transaction:688 with self.transaction:
665 old_bug_watches = set(689 old_bug_watches = set(
666 bug_watch for bug_watch in bug_watches690 bug_watch for bug_watch in bug_watches
@@ -693,7 +717,7 @@
693 # are actually some bugs that we're interested in so as to717 # are actually some bugs that we're interested in so as to
694 # avoid unnecessary network traffic.718 # avoid unnecessary network traffic.
695 if server_time is not None and len(remote_old_ids) > 0:719 if server_time is not None and len(remote_old_ids) > 0:
696 if batch_size is None:720 if batch_size == BATCH_SIZE_UNLIMITED:
697 remote_old_ids_to_check = (721 remote_old_ids_to_check = (
698 remotesystem.getModifiedRemoteBugs(722 remotesystem.getModifiedRemoteBugs(
699 remote_old_ids, oldest_lastchecked))723 remote_old_ids, oldest_lastchecked))
@@ -721,7 +745,7 @@
721 remote_ids_to_check = chain(745 remote_ids_to_check = chain(
722 remote_ids_with_comments, remote_new_ids, remote_old_ids_to_check)746 remote_ids_with_comments, remote_new_ids, remote_old_ids_to_check)
723747
724 if batch_size is not None:748 if batch_size != BATCH_SIZE_UNLIMITED:
725 # Some remote bug IDs may appear in more than one list so749 # Some remote bug IDs may appear in more than one list so
726 # we must filter the list before slicing.750 # we must filter the list before slicing.
727 remote_ids_to_check = islice(751 remote_ids_to_check = islice(
728752
=== modified file 'lib/lp/bugs/tests/externalbugtracker.py'
--- lib/lp/bugs/tests/externalbugtracker.py 2009-12-23 12:14:59 +0000
+++ lib/lp/bugs/tests/externalbugtracker.py 2010-04-08 15:38:28 +0000
@@ -24,8 +24,8 @@
24from canonical.config import config24from canonical.config import config
25from canonical.database.sqlbase import commit, ZopelessTransactionManager25from canonical.database.sqlbase import commit, ZopelessTransactionManager
26from lp.bugs.externalbugtracker import (26from lp.bugs.externalbugtracker import (
27 BugNotFound, BugTrackerConnectError, Bugzilla, DebBugs,27 BATCH_SIZE_UNLIMITED, BugNotFound, BugTrackerConnectError, Bugzilla,
28 ExternalBugTracker, Mantis, RequestTracker, Roundup, SourceForge,28 DebBugs, ExternalBugTracker, Mantis, RequestTracker, Roundup, SourceForge,
29 Trac)29 Trac)
30from lp.bugs.externalbugtracker.trac import (30from lp.bugs.externalbugtracker.trac import (
31 FAULT_TICKET_NOT_FOUND, LP_PLUGIN_BUG_IDS_ONLY, LP_PLUGIN_FULL,31 FAULT_TICKET_NOT_FOUND, LP_PLUGIN_BUG_IDS_ONLY, LP_PLUGIN_FULL,
@@ -152,6 +152,8 @@
152 implementation, though it doesn't actually do anything.152 implementation, though it doesn't actually do anything.
153 """153 """
154154
155 batch_size = BATCH_SIZE_UNLIMITED
156
155 def __init__(self, baseurl='http://example.com/'):157 def __init__(self, baseurl='http://example.com/'):
156 super(TestExternalBugTracker, self).__init__(baseurl)158 super(TestExternalBugTracker, self).__init__(baseurl)
157159
158160
=== modified file 'lib/lp/bugs/tests/test_bugwatch.py'
--- lib/lp/bugs/tests/test_bugwatch.py 2010-04-07 12:46:15 +0000
+++ lib/lp/bugs/tests/test_bugwatch.py 2010-04-08 15:38:28 +0000
@@ -8,9 +8,6 @@
8import transaction8import transaction
9import unittest9import unittest
1010
11from datetime import datetime, timedelta
12from pytz import utc
13
14from urlparse import urlunsplit11from urlparse import urlunsplit
1512
16from zope.component import getUtility13from zope.component import getUtility
@@ -24,10 +21,8 @@
2421
25from lp.bugs.interfaces.bugtracker import BugTrackerType, IBugTrackerSet22from lp.bugs.interfaces.bugtracker import BugTrackerType, IBugTrackerSet
26from lp.bugs.interfaces.bugwatch import (23from lp.bugs.interfaces.bugwatch import (
27 BugWatchActivityStatus, IBugWatchSet, NoBugTrackerFound,24 IBugWatchSet, NoBugTrackerFound, UnrecognizedBugTrackerURL)
28 UnrecognizedBugTrackerURL)25from lp.bugs.scripts.checkwatches.scheduler import MAX_SAMPLE_SIZE
29from lp.bugs.scripts.checkwatches.scheduler import (
30 BugWatchScheduler, MAX_SAMPLE_SIZE)
31from lp.registry.interfaces.person import IPersonSet26from lp.registry.interfaces.person import IPersonSet
3227
33from lp.testing import TestCaseWithFactory28from lp.testing import TestCaseWithFactory
@@ -466,88 +461,5 @@
466 self.failUnless("Activity %s" % i in messages)461 self.failUnless("Activity %s" % i in messages)
467462
468463
469class TestBugWatchScheduler(TestCaseWithFactory):
470 """Tests for the BugWatchScheduler, which runs as part of garbo."""
471
472 layer = DatabaseFunctionalLayer
473
474 def setUp(self):
475 super(TestBugWatchScheduler, self).setUp('foo.bar@canonical.com')
476 # We'll make sure that all the other bug watches look like
477 # they've been scheduled so that only our watch gets scheduled.
478 for watch in getUtility(IBugWatchSet).search():
479 watch.next_check = datetime.now(utc)
480 self.bug_watch = self.factory.makeBugWatch()
481 self.scheduler = BugWatchScheduler(QuietFakeLogger())
482 transaction.commit()
483
484 def test_scheduler_schedules_unchecked_watches(self):
485 # The BugWatchScheduler will schedule a BugWatch that has never
486 # been checked to be checked immediately.
487 self.bug_watch.next_check = None
488 self.scheduler(1)
489
490 self.assertNotEqual(None, self.bug_watch.next_check)
491 self.assertTrue(
492 self.bug_watch.next_check <= datetime.now(utc))
493
494 def test_scheduler_schedules_working_watches(self):
495 # If a watch has been checked and has never failed its next
496 # check will be scheduled for 24 hours after its last check.
497 now = datetime.now(utc)
498 # Add some succesful activity to ensure that successful activity
499 # is handled correctly.
500 self.bug_watch.addActivity()
501 self.bug_watch.lastchecked = now
502 self.bug_watch.next_check = None
503 transaction.commit()
504 self.scheduler(1)
505
506 self.assertEqual(
507 now + timedelta(hours=24), self.bug_watch.next_check)
508
509 def test_scheduler_schedules_failing_watches(self):
510 # If a watch has failed once, it will be scheduled more than 24
511 # hours after its last check.
512 now = datetime.now(utc)
513 self.bug_watch.lastchecked = now
514
515 # The delay depends on the number of failures that the watch has
516 # had.
517 for failure_count in range(1, 6):
518 self.bug_watch.next_check = None
519 self.bug_watch.addActivity(
520 result=BugWatchActivityStatus.BUG_NOT_FOUND)
521 transaction.commit()
522 self.scheduler(1)
523
524 coefficient = self.scheduler.delay_coefficient * failure_count
525 self.assertEqual(
526 now + timedelta(days=1 + coefficient),
527 self.bug_watch.next_check)
528
529 # The scheduler only looks at the last 5 activity items, so even
530 # if there have been more failures the maximum delay will be 7
531 # days.
532 for count in range(10):
533 self.bug_watch.addActivity(
534 result=BugWatchActivityStatus.BUG_NOT_FOUND)
535 self.bug_watch.next_check = None
536 transaction.commit()
537 self.scheduler(1)
538 self.assertEqual(
539 now + timedelta(days=7), self.bug_watch.next_check)
540
541 def test_scheduler_doesnt_schedule_scheduled_watches(self):
542 # The scheduler will ignore watches whose next_check has been
543 # set.
544 next_check_date = datetime.now(utc) + timedelta(days=1)
545 self.bug_watch.next_check = next_check_date
546 transaction.commit()
547 self.scheduler(1)
548
549 self.assertEqual(next_check_date, self.bug_watch.next_check)
550
551
552def test_suite():464def test_suite():
553 return unittest.TestLoader().loadTestsFromName(__name__)465 return unittest.TestLoader().loadTestsFromName(__name__)

Subscribers

People subscribed via source and target branches

to status/vote changes: