Launchpad itself

Merge lp:~gmb/launchpad/heat-garbo-hourly-bug-509195 into lp:launchpad/db-devel

  1. heat-garbo-hourly-bug-509195
  2. Merge into db-devel
Proposed by Graham Binns
Status: Merged
Merged at revision: not available
Proposed branch: lp:~gmb/launchpad/heat-garbo-hourly-bug-509195
Merge into: lp:launchpad/db-devel
Diff against target: 410 lines (+172/-52)
7 files modified
database/schema/security.cfg (+2/-1)
lib/canonical/config/schema-lazr.conf (+1/-0)
lib/canonical/launchpad/scripts/garbo.py (+19/-16)
lib/lp/bugs/configure.zcml (+1/-0)
lib/lp/bugs/doc/bug-heat.txt (+117/-32)
lib/lp/bugs/interfaces/bug.py (+11/-1)
lib/lp/bugs/model/bug.py (+21/-2)
To merge this branch: bzr merge lp:~gmb/launchpad/heat-garbo-hourly-bug-509195
Reviewer Review Type Date Requested Status
Abel Deuring (community) code Approve
Review via email: mp+20729@code.launchpad.net

Commit message

Bugs with out-of-date heat will now have CalculateBugHeatJobs created for them by garbo-hourly.

Description of the change

This branch adds a job to the hourly garbage collector to mop up bugs with out-of-date heat.

In this branch I've added Bug.heat_last_updated (which was in the DB already but unused), added a getBugsWithOutDatedHeat() method to IBugSet, and updated the existing (unused) BugHeatUpdater TunableLoop in garbo.py to now create CalculateBugHeatJobs for each out-of-date bug rather than updating their heat itself.

I've also added permissions to the garbo db user so that it can add new BugJobs.

To post a comment you must log in.
Revision history for this message
Abel Deuring (adeuring) wrote :

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Graham,

nice banch! I have just two nitpicks, see below.

Abel

> === modified file 'lib/lp/bugs/doc/bug-heat.txt'
> --- lib/lp/bugs/doc/bug-heat.txt 2010-03-03 13:10:17 +0000
> +++ lib/lp/bugs/doc/bug-heat.txt 2010-03-05 10:13:32 +0000

[...]

> +it as a method will add jobs to calculate the heat of for all the bugs
> +whose heat is more than seven days old.

I think one of ('of', 'for') is sufficient ;)

> +
> +Before update_bug_heat is called, we'll ensure that there are no waiting
> +jobs in the bug heat calculation queue.
> +
> + >>> from lp.bugs.interfaces.bugjob import ICalculateBugHeatJobSource
> + >>> for calc_job in
getUtility(ICalculateBugHeatJobSource).iterReady():
> + ... calc_job.job.start()
> + ... calc_job.job.complete()
> +
> + >>> ready_jobs =
list(getUtility(ICalculateBugHeatJobSource).iterReady())
> + >>> len(ready_jobs)
> + 0
> +
> +We need to commit here to ensure that the bugs we've created are
> +available to the update_bug_heat script.
> +
> + >>> import transaction
> + >>> transaction.commit()
> +
> + >>> getUtility(IBugSet).getBugsWithOutdatedHeat(1).count()
> + 2
> +
> +We need to run update_bug_heat() twice to ensure that both the bugs are
> +updated.

I think you call update_bug_heat() in this doc test only once. But it
seems it should be called with chunksize=2 to ensure both bugs are
processed.

> +
> + >>> update_bug_heat(chunk_size=2)
> + DEBUG Adding CalculateBugHeatJobs for 2 Bugs (starting id: ...)
> + DEBUG Adding CalculateBugHeatJob for bug ...
> + DEBUG Adding CalculateBugHeatJob for bug ...

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFLkODxekBPhm8NrtARApukAJoCrVwFFnMkn4bR+JHA6lbUGQaZigCgkHXF
ARR5bSLClYT7Elxqmb/8X7o=
=+2Ba
-----END PGP SIGNATURE-----

Revision history for this message
Abel Deuring (adeuring) :
review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'database/schema/security.cfg'
--- database/schema/security.cfg 2010-03-01 21:59:32 +0000
+++ database/schema/security.cfg 2010-03-05 11:49:30 +0000
@@ -1842,8 +1842,9 @@
1842public.mailinglistsubscription = SELECT, DELETE1842public.mailinglistsubscription = SELECT, DELETE
1843public.teamparticipation = SELECT, DELETE1843public.teamparticipation = SELECT, DELETE
1844public.emailaddress = SELECT, UPDATE1844public.emailaddress = SELECT, UPDATE
1845public.job = SELECT, DELETE1845public.job = SELECT, INSERT, DELETE
1846public.branchjob = SELECT, DELETE1846public.branchjob = SELECT, DELETE
1847public.bugjob = SELECT, INSERT
18471848
1848[garbo_daily]1849[garbo_daily]
1849type=user1850type=user
18501851
=== modified file 'lib/canonical/config/schema-lazr.conf'
--- lib/canonical/config/schema-lazr.conf 2010-02-24 10:18:16 +0000
+++ lib/canonical/config/schema-lazr.conf 2010-03-05 11:49:30 +0000
@@ -36,6 +36,7 @@
36oops_prefix: none36oops_prefix: none
37error_dir: none37error_dir: none
38copy_to_zlog: false38copy_to_zlog: false
39max_heat_age: 7
3940
40[process_apport_blobs]41[process_apport_blobs]
41# The database user which will be used by this process.42# The database user which will be used by this process.
4243
=== modified file 'lib/canonical/launchpad/scripts/garbo.py'
--- lib/canonical/launchpad/scripts/garbo.py 2010-02-02 17:44:24 +0000
+++ lib/canonical/launchpad/scripts/garbo.py 2010-03-05 11:49:30 +0000
@@ -31,8 +31,8 @@
31from canonical.launchpad.webapp.interfaces import (31from canonical.launchpad.webapp.interfaces import (
32 IStoreSelector, AUTH_STORE, MAIN_STORE, MASTER_FLAVOR)32 IStoreSelector, AUTH_STORE, MAIN_STORE, MASTER_FLAVOR)
33from lp.bugs.interfaces.bug import IBugSet33from lp.bugs.interfaces.bug import IBugSet
34from lp.bugs.interfaces.bugjob import ICalculateBugHeatJobSource
34from lp.bugs.model.bugnotification import BugNotification35from lp.bugs.model.bugnotification import BugNotification
35from lp.bugs.scripts.bugheat import BugHeatCalculator
36from lp.code.interfaces.revision import IRevisionSet36from lp.code.interfaces.revision import IRevisionSet
37from lp.code.model.branchjob import BranchJob37from lp.code.model.branchjob import BranchJob
38from lp.code.model.codeimportresult import CodeImportResult38from lp.code.model.codeimportresult import CodeImportResult
@@ -698,18 +698,22 @@
698698
699 maximum_chunk_size = 1000699 maximum_chunk_size = 1000
700700
701 def __init__(self, log, abort_time=None):701 def __init__(self, log, abort_time=None, max_heat_age=None):
702 super(BugHeatUpdater, self).__init__(log, abort_time)702 super(BugHeatUpdater, self).__init__(log, abort_time)
703 self.transaction = transaction703 self.transaction = transaction
704 self.total_processed = 0
705 self.is_done = False
704 self.offset = 0706 self.offset = 0
705 self.total_updated = 0707 if max_heat_age is None:
708 max_heat_age = config.calculate_bug_heat.max_heat_age
709 self.max_heat_age = max_heat_age
706710
707 def isDone(self):711 def isDone(self):
708 """See `ITunableLoop`."""712 """See `ITunableLoop`."""
709 # When the main loop has no more Bugs to process it sets713 # When the main loop has no more Bugs to process it sets
710 # offset to None. Until then, it always has a numerical714 # offset to None. Until then, it always has a numerical
711 # value.715 # value.
712 return self.offset is None716 return self.is_done
713717
714 def __call__(self, chunk_size):718 def __call__(self, chunk_size):
715 """Retrieve a batch of Bugs and update their heat.719 """Retrieve a batch of Bugs and update their heat.
@@ -721,23 +725,23 @@
721 # trying to slice using floats or anything similarly725 # trying to slice using floats or anything similarly
722 # foolish. We shouldn't have to do this.726 # foolish. We shouldn't have to do this.
723 chunk_size = int(chunk_size)727 chunk_size = int(chunk_size)
724
725 start = self.offset728 start = self.offset
726 end = self.offset + chunk_size729 end = self.offset + chunk_size
727730
728 transaction.begin()731 transaction.begin()
729 # XXX 2010-01-08 gmb bug=505850:732 bugs = getUtility(IBugSet).getBugsWithOutdatedHeat(
730 # This method call should be taken out and shot as soon as733 self.max_heat_age)[start:end]
731 # we have a proper permissions system for scripts.
732 bugs = getUtility(IBugSet).dangerousGetAllBugs()[start:end]
733734
734 self.offset = None
735 bug_count = bugs.count()735 bug_count = bugs.count()
736 if bug_count > 0:736 if bug_count > 0:
737 starting_id = bugs.first().id737 starting_id = bugs.first().id
738 self.log.debug("Updating %i Bugs (starting id: %i)" %738 self.log.debug(
739 "Adding CalculateBugHeatJobs for %i Bugs (starting id: %i)" %
739 (bug_count, starting_id))740 (bug_count, starting_id))
741 else:
742 self.is_done = True
740743
744 self.offset = None
741 for bug in bugs:745 for bug in bugs:
742 # We set the starting point of the next batch to the Bug746 # We set the starting point of the next batch to the Bug
743 # id after the one we're looking at now. If there aren't any747 # id after the one we're looking at now. If there aren't any
@@ -745,11 +749,9 @@
745 # will remain set to None.749 # will remain set to None.
746 start += 1750 start += 1
747 self.offset = start751 self.offset = start
748 self.log.debug("Updating heat for bug %s" % bug.id)752 self.log.debug("Adding CalculateBugHeatJob for bug %s" % bug.id)
749 bug_heat_calculator = BugHeatCalculator(bug)753 getUtility(ICalculateBugHeatJobSource).create(bug)
750 heat = bug_heat_calculator.getBugHeat()754 self.total_processed += 1
751 bug.setHeat(heat)
752 self.total_updated += 1
753 transaction.commit()755 transaction.commit()
754756
755757
@@ -836,6 +838,7 @@
836 OpenIDAssociationPruner,838 OpenIDAssociationPruner,
837 OpenIDConsumerAssociationPruner,839 OpenIDConsumerAssociationPruner,
838 RevisionCachePruner,840 RevisionCachePruner,
841 BugHeatUpdater,
839 ]842 ]
840 experimental_tunable_loops = []843 experimental_tunable_loops = []
841844
842845
=== modified file 'lib/lp/bugs/configure.zcml'
--- lib/lp/bugs/configure.zcml 2010-02-19 12:31:43 +0000
+++ lib/lp/bugs/configure.zcml 2010-03-05 11:49:30 +0000
@@ -575,6 +575,7 @@
575 personIsAlsoNotifiedSubscriber575 personIsAlsoNotifiedSubscriber
576 personIsSubscribedToDuplicate576 personIsSubscribedToDuplicate
577 heat577 heat
578 heat_last_updated
578 has_patches579 has_patches
579 latest_patch580 latest_patch
580 latest_patch_uploaded"/>581 latest_patch_uploaded"/>
581582
=== modified file 'lib/lp/bugs/doc/bug-heat.txt'
--- lib/lp/bugs/doc/bug-heat.txt 2010-03-03 13:10:17 +0000
+++ lib/lp/bugs/doc/bug-heat.txt 2010-03-05 11:49:30 +0000
@@ -12,54 +12,139 @@
12 >>> bug.heat12 >>> bug.heat
13 013 0
1414
15It will also have a heat_last_updated of None.
16
17 >>> print bug.heat_last_updated
18 None
19
15The bug's heat can be set by calling its setHeat() method.20The bug's heat can be set by calling its setHeat() method.
1621
17 >>> bug.setHeat(42)22 >>> bug.setHeat(42)
18 >>> bug.heat23 >>> bug.heat
19 4224 42
2025
26Its heat_last_updated will also have been set.
27
28 >>> bug.heat_last_updated
29 datetime.datetime(..., tzinfo=<UTC>)
30
31
32Getting bugs whose heat is outdated
33-----------------------------------
34
35It's possible to get the set of bugs whose heat hasn't been updated for
36a given amount of time by calling IBugSet's getBugsWithOutdatedHeat()
37method.
38
39First, we'll set the heat of all bugs so that none of them are out of
40date.
41
42 >>> from lp.bugs.interfaces.bug import IBugSet
43 >>> for bug in getUtility(IBugSet).dangerousGetAllBugs():
44 ... bug.setHeat(0)
45
46If we call getBugsWithOutdatedHeat() now, the set that is returned will
47be empty because all the bugs have been recently updated.
48getBugsWithOutdatedHeat() takes a single parameter, max_heat_age, which
49is the maximum age, in days, that a bug's heat can be before it gets
50included in the returned set.
51
52 >>> getUtility(IBugSet).getBugsWithOutdatedHeat(1).count()
53 0
54
55IBug.setHeat() takes a timestamp parameter so that we can set the
56heat_last_updated date manually for the purposes of testing. If we make
57a bug's heat older than the max_heat_age that we pass to
58getBugsWithOutdatedHeat() it will appear in the set returned by
59getBugsWithOutdatedHeat().
60
61 >>> from datetime import datetime, timedelta
62 >>> from pytz import timezone
63 >>> old_heat_bug = factory.makeBug()
64 >>> old_heat_bug.setHeat(
65 ... 0, datetime.now(timezone('UTC')) - timedelta(days=2))
66
67 >>> outdated_bugs = getUtility(IBugSet).getBugsWithOutdatedHeat(1)
68 >>> outdated_bugs.count()
69 1
70
71 >>> outdated_bugs[0] == old_heat_bug
72 True
73
74getBugsWithOutdatedHeat() also returns bugs whose heat has never been
75updated.
76
77 >>> new_bug = factory.makeBug()
78 >>> outdated_bugs = getUtility(IBugSet).getBugsWithOutdatedHeat(1)
79 >>> outdated_bugs.count()
80 2
81
82 >>> new_bug in outdated_bugs
83 True
84
2185
22The BugHeatUpdater class86The BugHeatUpdater class
23---------------------------87---------------------------
2488
25In order to calculate bug heat we need to use the BugHeatUpdater89The BugHeatUpdater class is used to create bug heat calculation jobs for
26class, which is designed precisely for that task. It's part of the garbo90bugs with out-of-date heat.
27module and runs as part of the garbo-daily cronjob.
2891
29 >>> from canonical.launchpad.scripts.garbo import BugHeatUpdater92 >>> from canonical.launchpad.scripts.garbo import BugHeatUpdater
30 >>> from canonical.launchpad.scripts import FakeLogger93 >>> from canonical.launchpad.scripts import FakeLogger
3194
32 >>> update_bug_heat = BugHeatUpdater(FakeLogger())95 >>> update_bug_heat = BugHeatUpdater(FakeLogger(), max_heat_age=1)
3396
34BugHeatUpdater implements ITunableLoop and as such is callable. Calling97BugHeatUpdater implements ITunableLoop and as such is callable. Calling
35it as a method will update the heat for all the bugs currently held in98it as a method will add jobs to calculate the heat of for all the bugs
36Launchpad.99whose heat is more than seven days old.
37100
38Before update_bug_heat is called, bug 1 will have no heat.101Before update_bug_heat is called, we'll ensure that there are no waiting
39102jobs in the bug heat calculation queue.
40 >>> from zope.component import getUtility103
41 >>> from lp.bugs.interfaces.bug import IBugSet104 >>> from lp.bugs.interfaces.bugjob import ICalculateBugHeatJobSource
42 >>> bug_1 = getUtility(IBugSet).get(1)105 >>> for calc_job in getUtility(ICalculateBugHeatJobSource).iterReady():
43106 ... calc_job.job.start()
44 >>> bug_1.heat107 ... calc_job.job.complete()
45 0108
46109 >>> ready_jobs = list(getUtility(ICalculateBugHeatJobSource).iterReady())
47We touch bug 1 to make sure its date_last_updated is recent enough (bug heat110 >>> len(ready_jobs)
48decays over time).111 0
49112
50 >>> new_comment = bug_1.newMessage(113We need to commit here to ensure that the bugs we've created are
51 ... owner=bug_1.owner, subject="...", content="...")114available to the update_bug_heat script.
52 >>> import transaction ; transaction.commit()115
53 >>> bug_1 = getUtility(IBugSet).get(1)116 >>> import transaction
54117 >>> transaction.commit()
55 >>> update_bug_heat(chunk_size=1)118
56 DEBUG Updating 1 Bugs (starting id: ...)119 >>> getUtility(IBugSet).getBugsWithOutdatedHeat(1).count()
57 ...120 2
58121
59Bug 1's heat will now be greater than 0.122We need to run update_bug_heat() twice to ensure that both the bugs are
60123updated.
61 >>> bug_1.heat > 0124
62 True125 >>> update_bug_heat(chunk_size=2)
126 DEBUG Adding CalculateBugHeatJobs for 2 Bugs (starting id: ...)
127 DEBUG Adding CalculateBugHeatJob for bug ...
128 DEBUG Adding CalculateBugHeatJob for bug ...
129
130There will now be two CalculateBugHeatJobs in the queue.
131
132 >>> ready_jobs = list(getUtility(ICalculateBugHeatJobSource).iterReady())
133 >>> len(ready_jobs)
134 2
135
136Running them will update the bugs' heat.
137
138 >>> for calc_job in getUtility(ICalculateBugHeatJobSource).iterReady():
139 ... calc_job.job.start()
140 ... calc_job.run()
141 ... calc_job.job.complete()
142
143IBugSet.getBugsWithOutdatedHeat() will now return an empty set since all
144the bugs have been updated.
145
146 >>> getUtility(IBugSet).getBugsWithOutdatedHeat(1).count()
147 0
63148
64149
65Caculating the maximum heat for a target150Caculating the maximum heat for a target
66151
=== modified file 'lib/lp/bugs/interfaces/bug.py'
--- lib/lp/bugs/interfaces/bug.py 2010-02-27 20:20:03 +0000
+++ lib/lp/bugs/interfaces/bug.py 2010-03-05 11:49:30 +0000
@@ -320,6 +320,8 @@
320 heat = exported(320 heat = exported(
321 Int(title=_("The 'heat' of the bug"),321 Int(title=_("The 'heat' of the bug"),
322 required=False, readonly=True))322 required=False, readonly=True))
323 heat_last_updated = Datetime(
324 title=_('Heat Last Updated'), required=False, readonly=True)
323325
324 # Adding related BugMessages provides a hook for getting at326 # Adding related BugMessages provides a hook for getting at
325 # BugMessage.visible when building bug comments.327 # BugMessage.visible when building bug comments.
@@ -778,7 +780,7 @@
778 if the user is the owner or an admin.780 if the user is the owner or an admin.
779 """781 """
780782
781 def setHeat(heat):783 def setHeat(heat, timestamp=None):
782 """Set the heat for the bug."""784 """Set the heat for the bug."""
783785
784class InvalidDuplicateValue(Exception):786class InvalidDuplicateValue(Exception):
@@ -1032,6 +1034,14 @@
1032 # Note, this method should go away when we have a proper1034 # Note, this method should go away when we have a proper
1033 # permissions system for scripts.1035 # permissions system for scripts.
10341036
1037 def getBugsWithOutdatedHeat(max_heat_age):
1038 """Return the set of bugs whose heat is out of date.
1039
1040 :param max_heat_age: The maximum age, in days, that a bug's heat
1041 can be before it is included in the
1042 returned set.
1043 """
1044
10351045
1036class IFileBugData(Interface):1046class IFileBugData(Interface):
1037 """A class containing extra data to be used when filing a bug."""1047 """A class containing extra data to be used when filing a bug."""
10381048
=== modified file 'lib/lp/bugs/model/bug.py'
--- lib/lp/bugs/model/bug.py 2010-02-28 13:41:18 +0000
+++ lib/lp/bugs/model/bug.py 2010-03-05 11:49:30 +0000
@@ -22,7 +22,9 @@
22import operator22import operator
23import re23import re
24from cStringIO import StringIO24from cStringIO import StringIO
25from datetime import datetime, timedelta
25from email.Utils import make_msgid26from email.Utils import make_msgid
27from pytz import timezone
2628
27from zope.contenttype import guess_content_type29from zope.contenttype import guess_content_type
28from zope.component import getUtility30from zope.component import getUtility
@@ -33,7 +35,7 @@
33from sqlobject import SQLMultipleJoin, SQLRelatedJoin35from sqlobject import SQLMultipleJoin, SQLRelatedJoin
34from sqlobject import SQLObjectNotFound36from sqlobject import SQLObjectNotFound
35from storm.expr import (37from storm.expr import (
36 And, Count, Func, In, LeftJoin, Max, Not, Select, SQLRaw, Union)38 And, Count, Func, In, LeftJoin, Max, Not, Or, Select, SQLRaw, Union)
37from storm.store import EmptyResultSet, Store39from storm.store import EmptyResultSet, Store
3840
39from lazr.lifecycle.event import (41from lazr.lifecycle.event import (
@@ -260,6 +262,7 @@
260 users_affected_count = IntCol(notNull=True, default=0)262 users_affected_count = IntCol(notNull=True, default=0)
261 users_unaffected_count = IntCol(notNull=True, default=0)263 users_unaffected_count = IntCol(notNull=True, default=0)
262 heat = IntCol(notNull=True, default=0)264 heat = IntCol(notNull=True, default=0)
265 heat_last_updated = UtcDateTimeCol(default=None)
263 latest_patch_uploaded = UtcDateTimeCol(default=None)266 latest_patch_uploaded = UtcDateTimeCol(default=None)
264267
265 @property268 @property
@@ -1531,9 +1534,13 @@
15311534
1532 return not subscriptions_from_dupes.is_empty()1535 return not subscriptions_from_dupes.is_empty()
15331536
1534 def setHeat(self, heat):1537 def setHeat(self, heat, timestamp=None):
1535 """See `IBug`."""1538 """See `IBug`."""
1539 if timestamp is None:
1540 timestamp = UTC_NOW
1541
1536 self.heat = heat1542 self.heat = heat
1543 self.heat_last_updated = timestamp
1537 for task in self.bugtasks:1544 for task in self.bugtasks:
1538 task.target.recalculateMaxBugHeat()1545 task.target.recalculateMaxBugHeat()
15391546
@@ -1790,6 +1797,18 @@
1790 result_set = store.find(Bug)1797 result_set = store.find(Bug)
1791 return result_set.order_by('id')1798 return result_set.order_by('id')
17921799
1800 def getBugsWithOutdatedHeat(self, max_heat_age):
1801 """See `IBugSet`."""
1802 store = IStore(Bug)
1803 last_updated_cutoff = (
1804 datetime.now(timezone('UTC')) -
1805 timedelta(days=max_heat_age))
1806 last_updated_clause = Or(
1807 Bug.heat_last_updated < last_updated_cutoff,
1808 Bug.heat_last_updated == None)
1809
1810 return store.find(Bug, last_updated_clause).order_by('id')
1811
17931812
1794class BugAffectsPerson(SQLBase):1813class BugAffectsPerson(SQLBase):
1795 """A bug is marked as affecting a user."""1814 """A bug is marked as affecting a user."""

Subscribers

People subscribed via source and target branches

to status/vote changes: