Launchpad itself

Merge lp:~abentley/launchpad/build-duration into lp:launchpad/db-devel

  1. build-duration
  2. Merge into db-devel
Proposed by Aaron Bentley
Status: Merged
Merged at revision: 9627
Proposed branch: lp:~abentley/launchpad/build-duration
Merge into: lp:launchpad/db-devel
Diff against target: 231 lines (+43/-20)
9 files modified
database/sampledata/current-dev.sql (+9/-1)
database/sampledata/current.sql (+9/-1)
database/schema/patch-2207-75-0.sql (+9/-0)
lib/lp/buildmaster/tests/test_buildfarmjob.py (+1/-0)
lib/lp/code/browser/tests/test_sourcepackagerecipe.py (+4/-2)
lib/lp/code/model/tests/test_sourcepackagerecipebuild.py (+3/-1)
lib/lp/soyuz/doc/binarypackagebuild.txt (+3/-1)
lib/lp/soyuz/stories/soyuz/xx-builds-pages.txt (+4/-4)
lib/lp/soyuz/tests/test_binarypackagebuild.py (+1/-10)
To merge this branch: bzr merge lp:~abentley/launchpad/build-duration
Reviewer Review Type Date Requested Status
Edwin Grubbs (community) code Approve
Stuart Bishop (community) db Approve
Robert Collins db Pending
Review via email: mp+31655@code.launchpad.net

Description of the change

= Summary =
Fix bug #612899: Some recipe builds have NULL date_started but non-NULL date_finished

== Proposed fix ==
Fix the broken production data before migration by using the
date_first_dispatched (which we have determined is not broken) to synthesize a
build_duration.

Add a constraint to ensure this bug doesn't re-surface.

== Pre-implementation notes ==
Discussed with bigjools

== Implementation details ==
Some of the sampledata was broken, so I fixed it.

== Tests ==
None

== Demo and Q/A ==
None

= Launchpad lint =

Checking for conflicts and issues in changed files.

Linting changed files:
  database/sampledata/current.sql
  database/sampledata/current-dev.sql
  database/schema/patch-2207-75-0.sql

./database/schema/patch-2207-75-0.sql
       7: Line exceeds 78 characters.
      10: Line exceeds 78 characters.
      12: Line exceeds 78 characters.
      15: Line exceeds 78 characters.
      18: Line exceeds 78 characters.

To post a comment you must log in.
Revision history for this message
Stuart Bishop (stub) wrote :

The change looks fine. Note that updating the existing db patch means it will not have effect on staging until the next full database restore (hopefully this weekend).

review: Approve (db)
Revision history for this message
Aaron Bentley (abentley) wrote :

Addendum: I have had to fix several tests that were assuming the presence of bad data (which is now impossible). I deleted test_estimateDuration_with_bad_history because its only purpose was to test behaviour with bad data.

Revision history for this message
Edwin Grubbs (edwin-grubbs) wrote :

Hi Aaron,

This branch looks good.

-Edwin

review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'database/sampledata/current-dev.sql'
2--- database/sampledata/current-dev.sql 2010-07-30 23:01:04 +0000
3+++ database/sampledata/current-dev.sql 2010-08-04 15:21:28 +0000
4@@ -1,5 +1,6 @@
5 -- Copyright 2010 Canonical Ltd. This software is licensed under the
6 -- GNU Affero General Public License version 3 (see the file LICENSE).
7+-- Created using pg_dump (PostgreSQL) 8.4.4
8
9 SET check_function_bodies = false;
10 SET client_encoding = 'UTF8';
11@@ -2313,7 +2314,7 @@
12 INSERT INTO buildfarmjob (id, processor, virtualized, date_created, date_started, date_finished, date_first_dispatched, builder, status, log, job_type) VALUES (26, 1, true, '2007-07-08 00:00:00', '2007-07-07 23:58:41', '2007-07-08 00:00:01', NULL, 1, 2, 1, 1);
13 INSERT INTO buildfarmjob (id, processor, virtualized, date_created, date_started, date_finished, date_first_dispatched, builder, status, log, job_type) VALUES (27, 1, true, '2007-07-24 00:00:00', '2007-07-23 23:58:41', '2007-07-24 00:00:01', NULL, 1, 1, 1, 1);
14 INSERT INTO buildfarmjob (id, processor, virtualized, date_created, date_started, date_finished, date_first_dispatched, builder, status, log, job_type) VALUES (28, 3, true, '2007-08-10 00:00:00', '2007-08-10 00:00:00', '2007-08-10 00:00:13', NULL, 1, 1, 1, 1);
15-INSERT INTO buildfarmjob (id, processor, virtualized, date_created, date_started, date_finished, date_first_dispatched, builder, status, log, job_type) VALUES (29, 1, false, '2007-08-09 21:54:18.553132', NULL, '2007-08-09 23:59:59', NULL, NULL, 1, NULL, 1);
16+INSERT INTO buildfarmjob (id, processor, virtualized, date_created, date_started, date_finished, date_first_dispatched, builder, status, log, job_type) VALUES (29, 1, false, '2007-08-09 21:54:18.553132', '2007-08-09 23:49:59', '2007-08-09 23:59:59', NULL, NULL, 1, NULL, 1);
17 INSERT INTO buildfarmjob (id, processor, virtualized, date_created, date_started, date_finished, date_first_dispatched, builder, status, log, job_type) VALUES (30, 3, false, '2007-08-10 00:00:01', '2007-08-10 00:00:01', '2007-08-10 00:00:14', NULL, 1, 1, 1, 1);
18
19
20@@ -10543,6 +10544,13 @@
21 ALTER TABLE structuralsubscription ENABLE TRIGGER ALL;
22
23
24+ALTER TABLE suggestivepotemplate DISABLE TRIGGER ALL;
25+
26+
27+
28+ALTER TABLE suggestivepotemplate ENABLE TRIGGER ALL;
29+
30+
31 ALTER TABLE teammembership DISABLE TRIGGER ALL;
32
33 INSERT INTO teammembership (id, person, team, status, date_joined, date_expires, last_changed_by, last_change_comment, proposed_by, acknowledged_by, reviewed_by, date_proposed, date_last_changed, date_acknowledged, date_reviewed, proponent_comment, acknowledger_comment, reviewer_comment, date_created) VALUES (1, 1, 17, 3, '2005-03-03 10:02:53.830191', '2050-03-03 10:02:53.830191', NULL, NULL, 1, NULL, 1, '2005-03-03 10:02:53.830191', NULL, NULL, '2005-03-03 10:02:53.830191', NULL, NULL, NULL, '2008-02-15 13:59:16.98918');
34
35=== modified file 'database/sampledata/current.sql'
36--- database/sampledata/current.sql 2010-07-30 23:01:04 +0000
37+++ database/sampledata/current.sql 2010-08-04 15:21:28 +0000
38@@ -1,5 +1,6 @@
39 -- Copyright 2010 Canonical Ltd. This software is licensed under the
40 -- GNU Affero General Public License version 3 (see the file LICENSE).
41+-- Created using pg_dump (PostgreSQL) 8.4.4
42
43 SET check_function_bodies = false;
44 SET client_encoding = 'UTF8';
45@@ -2270,7 +2271,7 @@
46 INSERT INTO buildfarmjob (id, processor, virtualized, date_created, date_started, date_finished, date_first_dispatched, builder, status, log, job_type) VALUES (26, 1, true, '2007-07-08 00:00:00', '2007-07-07 23:58:41', '2007-07-08 00:00:01', NULL, 1, 2, 1, 1);
47 INSERT INTO buildfarmjob (id, processor, virtualized, date_created, date_started, date_finished, date_first_dispatched, builder, status, log, job_type) VALUES (27, 1, true, '2007-07-24 00:00:00', '2007-07-23 23:58:41', '2007-07-24 00:00:01', NULL, 1, 1, 1, 1);
48 INSERT INTO buildfarmjob (id, processor, virtualized, date_created, date_started, date_finished, date_first_dispatched, builder, status, log, job_type) VALUES (28, 3, true, '2007-08-10 00:00:00', '2007-08-10 00:00:00', '2007-08-10 00:00:13', NULL, 1, 1, 1, 1);
49-INSERT INTO buildfarmjob (id, processor, virtualized, date_created, date_started, date_finished, date_first_dispatched, builder, status, log, job_type) VALUES (29, 1, false, '2007-08-09 21:54:18.553132', NULL, '2007-08-09 23:59:59', NULL, NULL, 1, NULL, 1);
50+INSERT INTO buildfarmjob (id, processor, virtualized, date_created, date_started, date_finished, date_first_dispatched, builder, status, log, job_type) VALUES (29, 1, false, '2007-08-09 21:54:18.553132', '2007-08-09 23:49:59', '2007-08-09 23:59:59', NULL, NULL, 1, NULL, 1);
51 INSERT INTO buildfarmjob (id, processor, virtualized, date_created, date_started, date_finished, date_first_dispatched, builder, status, log, job_type) VALUES (30, 3, false, '2007-08-10 00:00:01', '2007-08-10 00:00:01', '2007-08-10 00:00:14', NULL, 1, 1, 1, 1);
52
53
54@@ -10413,6 +10414,13 @@
55 ALTER TABLE structuralsubscription ENABLE TRIGGER ALL;
56
57
58+ALTER TABLE suggestivepotemplate DISABLE TRIGGER ALL;
59+
60+
61+
62+ALTER TABLE suggestivepotemplate ENABLE TRIGGER ALL;
63+
64+
65 ALTER TABLE teammembership DISABLE TRIGGER ALL;
66
67 INSERT INTO teammembership (id, person, team, status, date_joined, date_expires, last_changed_by, last_change_comment, proposed_by, acknowledged_by, reviewed_by, date_proposed, date_last_changed, date_acknowledged, date_reviewed, proponent_comment, acknowledger_comment, reviewer_comment, date_created) VALUES (1, 1, 17, 3, '2005-03-03 10:02:53.830191', '2050-03-03 10:02:53.830191', NULL, NULL, 1, NULL, 1, '2005-03-03 10:02:53.830191', NULL, NULL, '2005-03-03 10:02:53.830191', NULL, NULL, NULL, '2008-02-15 13:59:16.98918');
68
69=== modified file 'database/schema/patch-2207-75-0.sql'
70--- database/schema/patch-2207-75-0.sql 2010-07-30 10:08:40 +0000
71+++ database/schema/patch-2207-75-0.sql 2010-08-04 15:21:28 +0000
72@@ -3,6 +3,9 @@
73
74 SET client_min_messages=ERROR;
75
76+-- Fix broken production data
77+UPDATE SourcePackageRecipeBuild SET build_duration = date_built - date_first_dispatched WHERE build_duration is NULL;
78+
79 CREATE TEMPORARY TABLE NewBuildFarmJob AS SELECT
80 nextval('buildfarmjob_id_seq') AS id, 1 AS processor, True AS virtualized, date_created, date_built - build_duration AS date_started, date_built AS date_finished, date_first_dispatched, builder, build_state, build_log, 3 AS job_type, id AS sprb_id FROM SourcePackageRecipeBuild;
81
82@@ -29,4 +32,10 @@
83 DROP COLUMN build_state, DROP COLUMN build_log, DROP COLUMN archive,
84 DROP COLUMN pocket, DROP COLUMN upload_log, DROP COLUMN dependencies;
85
86+
87+ALTER TABLE BuildFarmJob
88+ ADD CONSTRAINT started_if_finished CHECK (
89+ date_finished IS NULL OR date_started IS NOT NULL);
90+
91+
92 INSERT INTO LaunchpadDatabaseRevision VALUES (2207, 75, 0);
93
94=== modified file 'lib/lp/buildmaster/tests/test_buildfarmjob.py'
95--- lib/lp/buildmaster/tests/test_buildfarmjob.py 2010-07-18 17:43:23 +0000
96+++ lib/lp/buildmaster/tests/test_buildfarmjob.py 2010-08-04 15:21:28 +0000
97@@ -49,6 +49,7 @@
98 build_farm_job = getUtility(IBuildFarmJobSource).new(
99 job_type=job_type, status=status)
100 removeSecurityProxy(build_farm_job).builder = builder
101+ removeSecurityProxy(build_farm_job).date_started = date_finished
102 removeSecurityProxy(build_farm_job).date_finished = date_finished
103 return build_farm_job
104
105
106=== modified file 'lib/lp/code/browser/tests/test_sourcepackagerecipe.py'
107--- lib/lp/code/browser/tests/test_sourcepackagerecipe.py 2010-07-31 20:55:46 +0000
108+++ lib/lp/code/browser/tests/test_sourcepackagerecipe.py 2010-08-04 15:21:28 +0000
109@@ -515,6 +515,7 @@
110 build = removeSecurityProxy(self.factory.makeSourcePackageRecipeBuild(
111 recipe=recipe, distroseries=self.squirrel, archive=self.ppa))
112 build.status = BuildStatus.FULLYBUILT
113+ build.date_started = datetime(2010, 03, 16, tzinfo=utc)
114 build.date_finished = datetime(2010, 03, 16, tzinfo=utc)
115
116 self.assertTextMatchesExpressionIgnoreWhitespace("""\
117@@ -596,8 +597,9 @@
118 set(view.builds))
119
120 def set_day(build, day):
121- removeSecurityProxy(build).date_finished = datetime(
122- 2010, 03, day, tzinfo=utc)
123+ naked_build = removeSecurityProxy(build)
124+ naked_build.date_started = datetime(2010, 03, day, tzinfo=utc)
125+ naked_build.date_finished = datetime(2010, 03, day, tzinfo=utc)
126 set_day(build1, 16)
127 set_day(build2, 15)
128 # When there are 4+ pending builds, only the the most
129
130=== modified file 'lib/lp/code/model/tests/test_sourcepackagerecipebuild.py'
131--- lib/lp/code/model/tests/test_sourcepackagerecipebuild.py 2010-08-03 08:49:19 +0000
132+++ lib/lp/code/model/tests/test_sourcepackagerecipebuild.py 2010-08-04 15:21:28 +0000
133@@ -326,7 +326,9 @@
134 def prepare_build():
135 queue_record = self.factory.makeSourcePackageRecipeBuildJob()
136 build = queue_record.specific_job.build
137- removeSecurityProxy(build).status = BuildStatus.FULLYBUILT
138+ naked_build = removeSecurityProxy(build)
139+ naked_build.status = BuildStatus.FULLYBUILT
140+ naked_build.date_started = self.factory.getUniqueDate()
141 queue_record.builder = self.factory.makeBuilder()
142 slave = WaitingSlave('BuildStatus.OK')
143 queue_record.builder.setSlaveForTesting(slave)
144
145=== modified file 'lib/lp/soyuz/doc/binarypackagebuild.txt'
146--- lib/lp/soyuz/doc/binarypackagebuild.txt 2010-07-23 09:04:18 +0000
147+++ lib/lp/soyuz/doc/binarypackagebuild.txt 2010-08-04 15:21:28 +0000
148@@ -1200,7 +1200,7 @@
149 >>> properties = {
150 ... 'log': 1,
151 ... 'date_finished': UTC_NOW,
152- ... 'date_started': None,
153+ ... 'date_started': UTC_NOW,
154 ... 'builder': bob,
155 ... 'status': BuildStatus.FAILEDTOUPLOAD,
156 ... 'dependencies': u'whatever',
157@@ -1417,10 +1417,12 @@
158 >>> earlier_builds = src_pkg_earlier.createMissingBuilds()
159 >>> eg_build_date = earlier_builds[0].date_created
160 >>> for build in earlier_builds:
161+ ... build.date_started = eg_build_date - timedelta(1)
162 ... build.date_finished = eg_build_date - timedelta(1)
163
164 >>> later_builds = src_pkg_later.createMissingBuilds()
165 >>> for build in later_builds:
166+ ... build.date_started = eg_build_date
167 ... build.date_finished = eg_build_date
168
169 Ensure that the i386 builds are created by the 'frog' builder,
170
171=== modified file 'lib/lp/soyuz/stories/soyuz/xx-builds-pages.txt'
172--- lib/lp/soyuz/stories/soyuz/xx-builds-pages.txt 2010-04-14 17:34:35 +0000
173+++ lib/lp/soyuz/stories/soyuz/xx-builds-pages.txt 2010-08-04 15:21:28 +0000
174@@ -143,8 +143,8 @@
175 ------------------------------
176 Successfully built
177 i386 build of commercialpackage 1.0-1 in ubuntu breezy-autotest RELEASE
178- Build started at an unknown time on an unknown build machine and finished
179- on 2007-08-09 taking
180+ Build started on 2007-08-09 on an unknown build machine and finished
181+ on 2007-08-09 taking ten minutes
182 ------------------------------
183 Failed to upload
184 i386 build of cdrkit 1.0 in ubuntu breezy-autotest RELEASE
185@@ -200,8 +200,8 @@
186 ------------------------------
187 Successfully built
188 i386 build of commercialpackage 1.0-1 in ubuntu breezy-autotest RELEASE
189- Build started at an unknown time on an unknown build machine and finished
190- on 2007-08-09 taking
191+ Build started on 2007-08-09 on an unknown build machine and finished
192+ on 2007-08-09 taking ten minutes
193 ------------------------------
194 Successfully built
195 i386 build of pmount 0.1-1 in ubuntu hoary RELEASE
196
197=== modified file 'lib/lp/soyuz/tests/test_binarypackagebuild.py'
198--- lib/lp/soyuz/tests/test_binarypackagebuild.py 2010-07-21 07:45:50 +0000
199+++ lib/lp/soyuz/tests/test_binarypackagebuild.py 2010-08-04 15:21:28 +0000
200@@ -10,7 +10,6 @@
201 from zope.component import getUtility
202 from zope.security.proxy import removeSecurityProxy
203
204-from canonical.database.constants import UTC_NOW
205 from canonical.testing import LaunchpadZopelessLayer
206 from lp.services.job.model.job import Job
207 from lp.buildmaster.interfaces.buildbase import BuildStatus
208@@ -83,13 +82,6 @@
209 self.create_previous_build(335)
210 self.assertEqual(335, self.build.estimateDuration().seconds)
211
212- def test_estimateDuration_with_bad_history(self):
213- # If the latest matching build has bad data, ignore it.
214- # See bug 589068.
215- previous_build = self.create_previous_build(335)
216- previous_build.date_started = None
217- self.assertEqual(60, self.build.estimateDuration().seconds)
218-
219 def addFakeBuildLog(self):
220 lfa = self.factory.makeLibraryFileAlias('mybuildlog.txt')
221 removeSecurityProxy(self.build).log = lfa
222@@ -418,8 +410,7 @@
223
224 self.builder = self.factory.makeBuilder()
225 self.builder.setSlaveForTesting(WaitingSlave('BuildStatus.OK'))
226- self.build.buildqueue_record.builder = self.builder
227- self.build.buildqueue_record.setDateStarted(UTC_NOW)
228+ self.build.buildqueue_record.markAsBuilding(self.builder)
229
230 def testDependencies(self):
231 """Verify that storeBuildInfo sets any dependencies."""

Subscribers

People subscribed via source and target branches

to status/vote changes: