Launchpad itself

Merge lp:~michael.nelson/launchpad/567922-binarypackagebuild-packagebuild-3 into lp:launchpad/db-devel

  1. 567922-binarypackagebuild-packagebuild-3
  2. Merge into db-devel
Proposed by Michael Nelson
Status: Merged
Merged at revision: 9405
Proposed branch: lp:~michael.nelson/launchpad/567922-binarypackagebuild-packagebuild-3
Merge into: lp:launchpad/db-devel
Prerequisite: lp:~michael.nelson/launchpad/567922-binarypackagebuild-packagebuild-2
Diff against target: 786 lines (+263/-94) (has conflicts)
13 files modified
BRANCH.TODO (+4/-0)
lib/lp/buildmaster/doc/buildfarmjob.txt (+8/-5)
lib/lp/buildmaster/interfaces/buildbase.py (+1/-1)
lib/lp/buildmaster/interfaces/packagebuild.py (+32/-0)
lib/lp/buildmaster/model/buildbase.py (+103/-57)
lib/lp/buildmaster/model/buildfarmjob.py (+9/-3)
lib/lp/buildmaster/model/buildfarmjobbehavior.py (+3/-2)
lib/lp/buildmaster/model/packagebuild.py (+53/-9)
lib/lp/buildmaster/tests/test_buildbase.py (+3/-3)
lib/lp/buildmaster/tests/test_packagebuild.py (+19/-6)
lib/lp/code/model/sourcepackagerecipebuild.py (+9/-3)
lib/lp/soyuz/model/buildpackagejob.py (+9/-4)
lib/lp/translations/model/translationtemplatesbuildjob.py (+10/-1)
Text conflict in lib/lp/buildmaster/interfaces/buildbase.py
Text conflict in lib/lp/buildmaster/model/buildbase.py
Text conflict in lib/lp/buildmaster/tests/test_buildbase.py
To merge this branch: bzr merge lp:~michael.nelson/launchpad/567922-binarypackagebuild-packagebuild-3
Reviewer Review Type Date Requested Status
Jelmer Vernooij (community) code* Approve
Review via email: mp+24327@code.launchpad.net

Description of the change

Please read the description of the previous MP at:

https://code.edge.launchpad.net/~michael.nelson/launchpad/567922-binarypackagebuild-packagebuild-2/+merge/24211

This branch adds the last three interface methods to IPackageBuild: verifySuccessfulUpload, storeUploadLog and notify, as well as handleStatus() on IPackageBuildDerived (it should not be delegated like the other methods, as it needs to then call sub-handles such as _handleStatus_OK() etc., which need to be called from the top of the chain).

The same tests to run.

The next branch steps back from all this re-shuffling and cleans up the results.

To post a comment you must log in.
Revision history for this message
Jelmer Vernooij (jelmer) wrote :

Sorry, still haven't sorted out my mailing list subscriptions. I promise I'll do that for the next one and reply inline...

Making the BuildBase methods static is ugly, though I can understand why you're doing it and it seems like a reasonable thing to do for the moment.
Some minor issues:

The various versions of _set_build_farm_job have an extra heading space in their docstring.

The extra_info parameter to IPackageBuild.verify is not documented in its interface.

Also, this code will unfortunately conflict once my simplify-uploadprocess branch lands.

review: Approve (code*)
Revision history for this message
Jelmer Vernooij (jelmer) wrote :

It'd be great if this branch can also use Store.flush rather than flush_database_updates as we discussed on IRC.

Revision history for this message
Michael Nelson (michael.nelson) wrote :

On Thu, Apr 29, 2010 at 3:52 PM, Jelmer Vernooij <email address hidden> wrote:
> Review: Approve code*
> Sorry, still haven't sorted out my mailing list subscriptions. I promise I'll do that for the next one and reply inline...
>
> Making the BuildBase methods static is ugly, though I can understand why you're doing it and it seems like a reasonable thing to do for the moment.

Yep - it's temporary just so there's only one implementation while I
keep refactoring (and tests continue to pass). I've updated the
BRANCH.TODO to reflect that (which will ensure the code doesn't land
until it is fixed).

> Some minor issues:
>
> The various versions of _set_build_farm_job have an extra heading space in their docstring.

Fixed.

>
> The extra_info parameter to IPackageBuild.verify is not documented in its interface.

Fixed.

>
> Also, this code will unfortunately conflict once my simplify-uploadprocess branch lands.

Yeah, that's ok. I'm expecting to be resolving conflicts for the next while :)

Incremental diff attached. And I'd already replaced
flush_database_updates as discussed in a previous rev.

Thanks!

changes_jelmer.diff
1=== modified file 'BRANCH.TODO'
2--- BRANCH.TODO 2010-04-13 19:17:03 +0000
3+++ BRANCH.TODO 2010-04-29 15:10:20 +0000
4@@ -2,3 +2,7 @@
5 # landing. There is a test to ensure it is empty in trunk. If there is
6 # stuff still here when you are ready to land, the items should probably
7 # be converted to bugs so they can be scheduled.
8+
9+The static methods on IBuildFarmJob/IPackageBuild should be reverted
10+to normal methods once all *Build classes have transitioned to the new
11+model and IBuildBase is removed.
12
13=== modified file 'lib/lp/buildmaster/interfaces/packagebuild.py'
14--- lib/lp/buildmaster/interfaces/packagebuild.py 2010-04-29 10:56:03 +0000
15+++ lib/lp/buildmaster/interfaces/packagebuild.py 2010-04-29 15:18:45 +0000
16@@ -130,7 +130,13 @@
17 """
18
19 def notify(extra_info=None):
20- """Notify current build state to related people via email."""
21+ """Notify current build state to related people via email.
22+
23+ :param extra_info: Optional extra information that will be included
24+ in the notification email. If the notification is for a
25+ failed-to-upload error then this must be the content of the
26+ upload log.
27+ """
28
29
30 class IPackageBuildSource(Interface):
31
32=== modified file 'lib/lp/code/model/sourcepackagerecipebuild.py'
33--- lib/lp/code/model/sourcepackagerecipebuild.py 2010-04-28 08:32:10 +0000
34+++ lib/lp/code/model/sourcepackagerecipebuild.py 2010-04-29 15:10:58 +0000
35@@ -223,7 +223,7 @@
36 super(SourcePackageRecipeBuildJob, self).__init__()
37
38 def _set_build_farm_job(self):
39- """ Setup the IBuildFarmJob delegate.
40+ """Setup the IBuildFarmJob delegate.
41
42 We override this to provide a delegate specific to package builds."""
43 self.build_farm_job = PackageBuild(self.build)
44
45=== modified file 'lib/lp/soyuz/model/buildpackagejob.py'
46--- lib/lp/soyuz/model/buildpackagejob.py 2010-04-28 08:32:10 +0000
47+++ lib/lp/soyuz/model/buildpackagejob.py 2010-04-29 15:11:32 +0000
48@@ -46,7 +46,7 @@
49 super(BuildPackageJob, self).__init__()
50
51 def _set_build_farm_job(self):
52- """ Setup the IBuildFarmJob delegate.
53+ """Setup the IBuildFarmJob delegate.
54
55 We override this to provide a delegate specific to package builds."""
56 self.build_farm_job = PackageBuild(self.build)
57
58=== modified file 'lib/lp/translations/model/translationtemplatesbuildjob.py'
59--- lib/lp/translations/model/translationtemplatesbuildjob.py 2010-04-28 10:59:12 +0000
60+++ lib/lp/translations/model/translationtemplatesbuildjob.py 2010-04-29 15:11:07 +0000
61@@ -49,7 +49,7 @@
62 super(TranslationTemplatesBuildJob, self).__init__(branch_job)
63
64 def _set_build_farm_job(self):
65- """ Setup the IBuildFarmJob delegate.
66+ """Setup the IBuildFarmJob delegate.
67
68 We override this to provide a non-database delegate that simply
69 provides required functionality to the queue system."""

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'BRANCH.TODO'
2--- BRANCH.TODO 2010-04-13 19:17:03 +0000
3+++ BRANCH.TODO 2010-05-05 15:49:24 +0000
4@@ -2,3 +2,7 @@
5 # landing. There is a test to ensure it is empty in trunk. If there is
6 # stuff still here when you are ready to land, the items should probably
7 # be converted to bugs so they can be scheduled.
8+
9+The static methods on IBuildFarmJob/IPackageBuild should be reverted
10+to normal methods once all *Build classes have transitioned to the new
11+model and IBuildBase is removed.
12
13=== modified file 'lib/lp/buildmaster/doc/buildfarmjob.txt'
14--- lib/lp/buildmaster/doc/buildfarmjob.txt 2010-04-16 14:37:33 +0000
15+++ lib/lp/buildmaster/doc/buildfarmjob.txt 2010-05-05 15:49:24 +0000
16@@ -1,7 +1,8 @@
17 BuildFarmJob
18 ============
19
20- >>> from lp.buildmaster.interfaces.buildfarmjob import IBuildFarmJob
21+ >>> from lp.buildmaster.interfaces.buildfarmjob import (
22+ ... BuildFarmJobType, IBuildFarmJob)
23 >>> from lp.buildmaster.model.buildfarmjob import (
24 ... BuildFarmJob, BuildFarmJobDerived)
25
26@@ -9,12 +10,14 @@
27 specific build farm job classes will automatically delegate to
28 BuildFarmJob by inheriting BuildFarmJobDerived.
29
30- >>> buildfarmjob = BuildFarmJob()
31+ >>> buildfarmjob = BuildFarmJob(BuildFarmJobType.PACKAGEBUILD)
32 >>> verifyObject(IBuildFarmJob, buildfarmjob)
33 True
34
35-The BuildFarmJob class does not (yet) do a lot itself, other than
36-providing default implementations for its methods.
37+The BuildFarmJob class provides many of the common attributes used by
38+different types of build farm jobs, as well as default implementations
39+for the common methods. The BuildFarmJob class provides default
40+implementations for many build-related methods.
41
42 >>> print buildfarmjob.getLogFileName()
43 buildlog.txt
44@@ -24,6 +27,7 @@
45 ...
46 NotImplementedError
47
48+For more details, please refer to lp.buildmaster.tests.test_buildfarmjob.
49
50 BuildFarmJobDerived
51 ===================
52@@ -40,4 +44,3 @@
53 instance of a specific build-farm job implementation associated with a
54 given Job instance which must be called in the context of a concrete
55 derived class. See lp.buildmaster.tests.test_buildqueue for examples.
56-
57
58=== modified file 'lib/lp/buildmaster/interfaces/buildbase.py'
59--- lib/lp/buildmaster/interfaces/buildbase.py 2010-05-05 15:49:22 +0000
60+++ lib/lp/buildmaster/interfaces/buildbase.py 2010-05-05 15:49:24 +0000
61@@ -228,7 +228,7 @@
62 >>>>>>> MERGE-SOURCE
63 """
64
65- def handleStatus(status, librarian, slave_status):
66+ def handleStatus(build, status, librarian, slave_status):
67 """Handle a finished build status from a slave.
68
69 :param status: Slave build status string with 'BuildStatus.' stripped.
70
71=== modified file 'lib/lp/buildmaster/interfaces/packagebuild.py'
72--- lib/lp/buildmaster/interfaces/packagebuild.py 2010-05-05 15:49:22 +0000
73+++ lib/lp/buildmaster/interfaces/packagebuild.py 2010-05-05 15:49:24 +0000
74@@ -6,6 +6,7 @@
75 __all__ = [
76 'IPackageBuild',
77 'IPackageBuildSource',
78+ 'IPackageBuildDerived',
79 ]
80
81
82@@ -118,6 +119,25 @@
83 custom status handlers, but it should not be called externally.
84 """
85
86+ def verifySuccessfulUpload():
87+ """Verify that the upload of this build completed succesfully."""
88+
89+ def storeUploadLog(content):
90+ """Store the given content as the build upload_log.
91+
92+ :param content: string containing the upload-processor log output for
93+ the binaries created in this build.
94+ """
95+
96+ def notify(extra_info=None):
97+ """Notify current build state to related people via email.
98+
99+ :param extra_info: Optional extra information that will be included
100+ in the notification email. If the notification is for a
101+ failed-to-upload error then this must be the content of the
102+ upload log.
103+ """
104+
105
106 class IPackageBuildSource(Interface):
107 """A utility of this interface used to create _things_."""
108@@ -129,3 +149,15 @@
109 :param pocket: An item of `PackagePublishingPocket`.
110 :param dependencies: An optional debian-like dependency line.
111 """
112+
113+
114+class IPackageBuildDerived(Interface):
115+ """Classes deriving from IPackageBuild inherit the default handleStatus.
116+ """
117+ def handleStatus(status, librarian, slave_status):
118+ """Handle a finished build status from a slave.
119+
120+ :param status: Slave build status string with 'BuildStatus.' stripped.
121+ :param slave_status: A dict as returned by IBuilder.slaveStatus
122+ """
123+
124
125=== modified file 'lib/lp/buildmaster/model/buildbase.py'
126--- lib/lp/buildmaster/model/buildbase.py 2010-05-05 15:49:22 +0000
127+++ lib/lp/buildmaster/model/buildbase.py 2010-05-05 15:49:24 +0000
128@@ -125,20 +125,31 @@
129 return None
130 return self._getProxiedFileURL(self.upload_log)
131
132- def handleStatus(self, status, librarian, slave_status):
133+ @staticmethod
134+ def handleStatus(build, status, librarian, slave_status):
135 """See `IBuildBase`."""
136 logger = logging.getLogger(BUILDD_MANAGER_LOG_NAME)
137
138- method = getattr(self, '_handleStatus_' + status, None)
139+ method = getattr(build, '_handleStatus_' + status, None)
140
141 if method is None:
142+<<<<<<< TREE
143 logger.critical("Unknown BuildStatus '%s' for builder '%s'"
144 % (status, self.buildqueue_record.builder.url))
145+=======
146+ if build.buildqueue_record is not None:
147+ logger.critical("Unknown BuildStatus '%s' for builder '%s'"
148+ % (status, build.buildqueue_record.builder.url))
149+ else:
150+ logger.critical("Unknown BuildStatus '%s' for %r"
151+ % (status, build))
152+>>>>>>> MERGE-SOURCE
153 return
154
155- method(librarian, slave_status, logger)
156+ method(build, librarian, slave_status, logger)
157
158 <<<<<<< TREE
159+ def _handleStatus_OK(self, librarian, slave_status, logger):
160 =======
161 def processUpload(self, leaf, root, logger):
162 """Process an upload.
163@@ -168,8 +179,9 @@
164 ztm=ZopelessTransactionManager, log=logger)
165 processor.processUploadQueue(leaf)
166
167+ @staticmethod
168+ def _handleStatus_OK(build, librarian, slave_status, logger):
169 >>>>>>> MERGE-SOURCE
170- def _handleStatus_OK(self, librarian, slave_status, logger):
171 """Handle a package that built successfully.
172
173 Once built successfully, we pull the files, store them in a
174@@ -179,36 +191,41 @@
175 filemap = slave_status['filemap']
176
177 logger.info("Processing successful build %s from builder %s" % (
178- self.buildqueue_record.specific_job.build.title,
179- self.buildqueue_record.builder.name))
180+ build.buildqueue_record.specific_job.build.title,
181+ build.buildqueue_record.builder.name))
182 # Explode before collect a binary that is denied in this
183 # distroseries/pocket
184- if not self.archive.allowUpdatesToReleasePocket():
185- assert self.distroseries.canUploadToPocket(self.pocket), (
186+ if not build.archive.allowUpdatesToReleasePocket():
187+ assert build.distroseries.canUploadToPocket(build.pocket), (
188 "%s (%s) can not be built for pocket %s: illegal status"
189- % (self.title, self.id, self.pocket.name))
190+ % (build.title, build.id, build.pocket.name))
191
192 # ensure we have the correct build root as:
193 # <BUILDMASTER_ROOT>/incoming/<UPLOAD_LEAF>/<TARGET_PATH>/[FILES]
194 root = os.path.abspath(config.builddmaster.root)
195
196 # create a single directory to store build result files
197- upload_leaf = self.getUploadDirLeaf(
198- '%s-%s' % (self.id, self.buildqueue_record.id))
199- upload_dir = self.getUploadDir(upload_leaf)
200+ upload_leaf = build.getUploadDirLeaf(
201+ '%s-%s' % (build.id, build.buildqueue_record.id))
202+ upload_dir = build.getUploadDir(upload_leaf)
203 logger.debug("Storing build result at '%s'" % upload_dir)
204
205 # Build the right UPLOAD_PATH so the distribution and archive
206 # can be correctly found during the upload:
207 # <archive_id>/distribution_name
208 # for all destination archive types.
209+<<<<<<< TREE
210 archive = self.archive
211 distribution_name = self.distribution.name
212 target_path = '%s/%s' % (archive.id, distribution_name)
213 upload_path = os.path.join(upload_dir, target_path)
214+=======
215+ upload_path = os.path.join(upload_dir, str(build.archive.id),
216+ build.distribution.name)
217+>>>>>>> MERGE-SOURCE
218 os.makedirs(upload_path)
219
220- slave = removeSecurityProxy(self.buildqueue_record.builder.slave)
221+ slave = removeSecurityProxy(build.buildqueue_record.builder.slave)
222 successful_copy_from_slave = True
223 for filename in filemap:
224 logger.info("Grabbing file: %s" % filename)
225@@ -220,7 +237,7 @@
226 successful_copy_from_slave = False
227 logger.warning(
228 "A slave tried to upload the file '%s' "
229- "for the build %d." % (filename, self.id))
230+ "for the build %d." % (filename, build.id))
231 break
232 out_file = open(out_file_name, "wb")
233 slave_file = slave.getFile(filemap[filename])
234@@ -229,6 +246,7 @@
235 # We only attempt the upload if we successfully copied all the
236 # files from the slave.
237 if successful_copy_from_slave:
238+<<<<<<< TREE
239 uploader_logfilename = os.path.join(
240 upload_dir, UPLOAD_LOG_FILENAME)
241 uploader_command = self.getUploaderCommand(
242@@ -251,6 +269,14 @@
243 # to not return error, it only happen when the code is broken).
244 uploader_result_code = uploader_process.returncode
245 logger.info("Uploader returned %d" % uploader_result_code)
246+=======
247+ logger.info("Invoking uploader on %s for %s" % (root, upload_leaf))
248+ upload_logger = BufferLogger()
249+ upload_log = build.processUpload(upload_leaf, root, upload_logger)
250+ uploader_log_content = upload_logger.buffer.getvalue()
251+ else:
252+ uploader_log_content = 'Copy from slave was unsuccessful.'
253+>>>>>>> MERGE-SOURCE
254
255 # Quick and dirty hack to carry on on process-upload failures
256 if os.path.exists(upload_dir):
257@@ -290,7 +316,7 @@
258
259 # Store build information, build record was already updated during
260 # the binary upload.
261- self.storeBuildInfo(self, librarian, slave_status)
262+ build.storeBuildInfo(build, librarian, slave_status)
263
264 # Retrive the up-to-date build record and perform consistency
265 # checks. The build record should be updated during the binary
266@@ -305,56 +331,65 @@
267 # uploader about this occurrence. The failure notification will
268 # also contain the information required to manually reprocess the
269 # binary upload when it was the case.
270- if (self.buildstate != BuildStatus.FULLYBUILT or
271+ if (build.buildstate != BuildStatus.FULLYBUILT or
272 not successful_copy_from_slave or
273+<<<<<<< TREE
274 not self.verifySuccessfulUpload()):
275 logger.warning("Build %s upload failed." % self.id)
276 self.buildstate = BuildStatus.FAILEDTOUPLOAD
277 uploader_log_content = self.getUploadLogContent(root,
278 upload_leaf)
279+=======
280+ not build.verifySuccessfulUpload()):
281+ logger.warning("Build %s upload failed." % build.id)
282+ build.buildstate = BuildStatus.FAILEDTOUPLOAD
283+>>>>>>> MERGE-SOURCE
284 # Store the upload_log_contents in librarian so it can be
285 # accessed by anyone with permission to see the build.
286- self.storeUploadLog(uploader_log_content)
287+ build.storeUploadLog(uploader_log_content)
288 # Notify the build failure.
289- self.notify(extra_info=uploader_log_content)
290+ build.notify(extra_info=uploader_log_content)
291 else:
292 logger.info(
293 "Gathered %s %d completely" % (
294- self.__class__.__name__, self.id))
295+ build.__class__.__name__, build.id))
296
297 # Release the builder for another job.
298- self.buildqueue_record.builder.cleanSlave()
299+ build.buildqueue_record.builder.cleanSlave()
300 # Remove BuildQueue record.
301- self.buildqueue_record.destroySelf()
302+ build.buildqueue_record.destroySelf()
303
304- def _handleStatus_PACKAGEFAIL(self, librarian, slave_status, logger):
305+ @staticmethod
306+ def _handleStatus_PACKAGEFAIL(build, librarian, slave_status, logger):
307 """Handle a package that had failed to build.
308
309 Build has failed when trying the work with the target package,
310 set the job status as FAILEDTOBUILD, store available info and
311 remove Buildqueue entry.
312 """
313- self.buildstate = BuildStatus.FAILEDTOBUILD
314- self.storeBuildInfo(self, librarian, slave_status)
315- self.buildqueue_record.builder.cleanSlave()
316- self.notify()
317- self.buildqueue_record.destroySelf()
318+ build.buildstate = BuildStatus.FAILEDTOBUILD
319+ build.storeBuildInfo(build, librarian, slave_status)
320+ build.buildqueue_record.builder.cleanSlave()
321+ build.notify()
322+ build.buildqueue_record.destroySelf()
323
324- def _handleStatus_DEPFAIL(self, librarian, slave_status, logger):
325+ @staticmethod
326+ def _handleStatus_DEPFAIL(build, librarian, slave_status, logger):
327 """Handle a package that had missing dependencies.
328
329 Build has failed by missing dependencies, set the job status as
330 MANUALDEPWAIT, store available information, remove BuildQueue
331 entry and release builder slave for another job.
332 """
333- self.buildstate = BuildStatus.MANUALDEPWAIT
334- self.storeBuildInfo(self, librarian, slave_status)
335+ build.buildstate = BuildStatus.MANUALDEPWAIT
336+ build.storeBuildInfo(build, librarian, slave_status)
337 logger.critical("***** %s is MANUALDEPWAIT *****"
338- % self.buildqueue_record.builder.name)
339- self.buildqueue_record.builder.cleanSlave()
340- self.buildqueue_record.destroySelf()
341+ % build.buildqueue_record.builder.name)
342+ build.buildqueue_record.builder.cleanSlave()
343+ build.buildqueue_record.destroySelf()
344
345- def _handleStatus_CHROOTFAIL(self, librarian, slave_status,
346+ @staticmethod
347+ def _handleStatus_CHROOTFAIL(build, librarian, slave_status,
348 logger):
349 """Handle a package that had failed when unpacking the CHROOT.
350
351@@ -362,15 +397,16 @@
352 job as CHROOTFAIL, store available information, remove BuildQueue
353 and release the builder.
354 """
355- self.buildstate = BuildStatus.CHROOTWAIT
356- self.storeBuildInfo(self, librarian, slave_status)
357+ build.buildstate = BuildStatus.CHROOTWAIT
358+ build.storeBuildInfo(build, librarian, slave_status)
359 logger.critical("***** %s is CHROOTWAIT *****" %
360- self.buildqueue_record.builder.name)
361- self.buildqueue_record.builder.cleanSlave()
362- self.notify()
363- self.buildqueue_record.destroySelf()
364+ build.buildqueue_record.builder.name)
365+ build.buildqueue_record.builder.cleanSlave()
366+ build.notify()
367+ build.buildqueue_record.destroySelf()
368
369- def _handleStatus_BUILDERFAIL(self, librarian, slave_status, logger):
370+ @staticmethod
371+ def _handleStatus_BUILDERFAIL(build, librarian, slave_status, logger):
372 """Handle builder failures.
373
374 Build has been failed when trying to build the target package,
375@@ -378,14 +414,15 @@
376 and 'clean' the builder to do another jobs.
377 """
378 logger.warning("***** %s has failed *****"
379- % self.buildqueue_record.builder.name)
380- self.buildqueue_record.builder.failBuilder(
381+ % build.buildqueue_record.builder.name)
382+ build.buildqueue_record.builder.failBuilder(
383 "Builder returned BUILDERFAIL when asked for its status")
384 # simply reset job
385- self.storeBuildInfo(self, librarian, slave_status)
386- self.buildqueue_record.reset()
387+ build.storeBuildInfo(build, librarian, slave_status)
388+ build.buildqueue_record.reset()
389
390- def _handleStatus_GIVENBACK(self, librarian, slave_status, logger):
391+ @staticmethod
392+ def _handleStatus_GIVENBACK(build, librarian, slave_status, logger):
393 """Handle automatic retry requested by builder.
394
395 GIVENBACK pseudo-state represents a request for automatic retry
396@@ -393,15 +430,15 @@
397 ZERO.
398 """
399 logger.warning("***** %s is GIVENBACK by %s *****"
400- % (self.buildqueue_record.specific_job.build.title,
401- self.buildqueue_record.builder.name))
402- self.storeBuildInfo(self, librarian, slave_status)
403+ % (build.buildqueue_record.specific_job.build.title,
404+ build.buildqueue_record.builder.name))
405+ build.storeBuildInfo(build, librarian, slave_status)
406 # XXX cprov 2006-05-30: Currently this information is not
407 # properly presented in the Web UI. We will discuss it in
408 # the next Paris Summit, infinity has some ideas about how
409 # to use this content. For now we just ensure it's stored.
410- self.buildqueue_record.builder.cleanSlave()
411- self.buildqueue_record.reset()
412+ build.buildqueue_record.builder.cleanSlave()
413+ build.buildqueue_record.reset()
414
415 @staticmethod
416 def getLogFromSlave(build):
417@@ -428,25 +465,34 @@
418 else:
419 build.dependencies = None
420
421- def storeUploadLog(self, content):
422- """See `IBuildBase`."""
423+ @staticmethod
424+ def createUploadLog(build, content, filename=None):
425+ """Creates a file on the librarian for the upload log.
426+
427+ :return: ILibraryFileAlias for the upload log file.
428+ """
429 # The given content is stored in the librarian, restricted as
430 # necessary according to the targeted archive's privacy. The content
431 # object's 'upload_log' attribute will point to the
432 # `LibrarianFileAlias`.
433
434- assert self.upload_log is None, (
435+ assert build.upload_log is None, (
436 "Upload log information already exists and cannot be overridden.")
437
438- filename = 'upload_%s_log.txt' % self.id
439+ if filename is None:
440+ filename = 'upload_%s_log.txt' % build.id
441 contentType = filenameToContentType(filename)
442 file_size = len(content)
443 file_content = StringIO(content)
444- restricted = self.is_private
445+ restricted = build.is_private
446
447- library_file = getUtility(ILibraryFileAliasSet).create(
448+ return getUtility(ILibraryFileAliasSet).create(
449 filename, file_size, file_content, contentType=contentType,
450 restricted=restricted)
451+
452+ def storeUploadLog(self, content):
453+ """See `IBuildBase`."""
454+ library_file = self.createUploadLog(self, content)
455 self.upload_log = library_file
456
457 def queueBuild(self, suspended=False):
458
459=== modified file 'lib/lp/buildmaster/model/buildfarmjob.py'
460--- lib/lp/buildmaster/model/buildfarmjob.py 2010-05-05 15:49:22 +0000
461+++ lib/lp/buildmaster/model/buildfarmjob.py 2010-05-05 15:49:24 +0000
462@@ -201,9 +201,15 @@
463 self._set_build_farm_job()
464
465 def _set_build_farm_job(self):
466- """Set the build farm job to which we will delegate.
467-
468- Sub-classes can override as required.
469+ """Set the default build farm job to which we will delegate.
470+
471+ Sub-classes should override as required.
472+
473+ XXX 2010-04-27 michael.nelson bug=570939
474+ This only exists because certain classes assume that
475+ BuildFarmJob/PackageBuild are in-memory objects that simply
476+ provide methods to update the associated builds.
477+ We can remove it once the above bug is completed.
478 """
479 self.build_farm_job = BuildFarmJob(
480 job_type=BuildFarmJobType.PACKAGEBUILD)
481
482=== modified file 'lib/lp/buildmaster/model/buildfarmjobbehavior.py'
483--- lib/lp/buildmaster/model/buildfarmjobbehavior.py 2010-04-12 16:23:13 +0000
484+++ lib/lp/buildmaster/model/buildfarmjobbehavior.py 2010-05-05 15:49:24 +0000
485@@ -183,8 +183,9 @@
486
487 # XXX: dsilvers 2005-03-02: Confirm the builder has the right build?
488
489- queueItem.specific_job.build.handleStatus(
490- build_status, librarian, slave_status)
491+ build = queueItem.specific_job.build
492+ build.handleStatus(
493+ build, build_status, librarian, slave_status)
494
495
496 class IdleBuildBehavior(BuildFarmJobBehaviorBase):
497
498=== modified file 'lib/lp/buildmaster/model/packagebuild.py'
499--- lib/lp/buildmaster/model/packagebuild.py 2010-05-05 15:49:22 +0000
500+++ lib/lp/buildmaster/model/packagebuild.py 2010-05-05 15:49:24 +0000
501@@ -22,7 +22,7 @@
502 from lp.buildmaster.interfaces.buildbase import BuildStatus
503 from lp.buildmaster.interfaces.buildfarmjob import IBuildFarmJobSource
504 from lp.buildmaster.interfaces.packagebuild import (
505- IPackageBuild, IPackageBuildSource)
506+ IPackageBuild, IPackageBuildDerived, IPackageBuildSource)
507 from lp.buildmaster.model.buildbase import BuildBase
508 from lp.buildmaster.model.buildfarmjob import BuildFarmJobDerived
509 from lp.registry.interfaces.pocket import PackagePublishingPocket
510@@ -62,7 +62,7 @@
511 def __init__(self, build):
512 """Construct a PackageBuild.
513
514- XXX 2010-04-21 michael.nelson bug=536700
515+ XXX 2010-04-21 michael.nelson bug=570939
516 This initialiser is only used by IBuildFarmJobDerived classes
517 that are not yet expecting a concrete BuildFarmJob (and so are
518 expecting to pass in the build to which they refer, such as
519@@ -196,10 +196,54 @@
520 """See `IPackageBuild`."""
521 return BuildBase.storeBuildInfo(self, librarian, slave_status)
522
523-
524-class PackageBuildDerived(BuildFarmJobDerived):
525- """Override the base delegate to use a build farm job specific to
526- packages.
527- """
528- def _set_build_farm_job(self):
529- self.build_farm_job = PackageBuild(self.build)
530+ def verifySuccessfulUpload(self):
531+ """See `IPackageBuild`."""
532+ raise NotImplementedError
533+
534+ def storeUploadLog(self, content):
535+ """See `IPackageBuild`."""
536+ filename = "upload_%s_log.txt" % self.build_farm_job.id
537+ library_file = BuildBase.createUploadLog(
538+ self, content, filename=filename)
539+ self.upload_log = library_file
540+
541+ def notify(self, extra_info=None):
542+ """See `IPackageBuild`."""
543+ raise NotImplementedError
544+
545+
546+class PackageBuildDerived:
547+ """See `IPackageBuildDerived`."""
548+ implements(IPackageBuildDerived)
549+
550+ def handleStatus(self, status, librarian, slave_status):
551+ """See `IPackageBuildDerived`."""
552+ return BuildBase.handleStatus(self, status, librarian, slave_status)
553+
554+ # The following private handlers currently re-use the BuildBase
555+ # implementation until it is no longer in use. If we find in the
556+ # future that it would be useful to delegate these also, they can be
557+ # added to IBuildFarmJob or IPackageBuild as necessary.
558+ def _handleStatus_OK(self, librarian, slave_status, logger):
559+ return BuildBase._handleStatus_OK(
560+ self, librarian, slave_status, logger)
561+
562+ def _handleStatus_PACKAGEFAIL(self, librarian, slave_status, logger):
563+ return BuildBase._handleStatus_PACKAGEFAIL(
564+ self, librarian, slave_status, logger)
565+
566+ def _handleStatus_DEPFAIL(self, librarian, slave_status, logger):
567+ return BuildBase._handleStatus_DEPFAIL(
568+ self, librarian, slave_status, logger)
569+
570+ def _handleStatus_CHROOTFAIL(self, librarian, slave_status, logger):
571+ return BuildBase._handleStatus_CHROOTFAIL(
572+ self, librarian, slave_status, logger)
573+
574+ def _handleStatus_BUILDERFAIL(self, librarian, slave_status, logger):
575+ return BuildBase._handleStatus_BUILDERFAIL(
576+ self, librarian, slave_status, logger)
577+
578+ def _handleStatus_GIVENBACK(self, librarian, slave_status, logger):
579+ return BuildBase._handleStatus_GIVENBACK(
580+ self, librarian, slave_status, logger)
581
582=== modified file 'lib/lp/buildmaster/tests/test_buildbase.py'
583--- lib/lp/buildmaster/tests/test_buildbase.py 2010-05-05 15:49:22 +0000
584+++ lib/lp/buildmaster/tests/test_buildbase.py 2010-05-05 15:49:24 +0000
585@@ -192,7 +192,7 @@
586 # A filemap with plain filenames should not cause a problem.
587 # The call to handleStatus will attempt to get the file from
588 # the slave resulting in a URL error in this test case.
589- self.build.handleStatus('OK', None, {
590+ self.build.handleStatus(self.build, 'OK', None, {
591 'filemap': { 'myfile.py': 'test_file_hash'},
592 })
593
594@@ -202,7 +202,7 @@
595 def test_handleStatus_OK_absolute_filepath(self):
596 # A filemap that tries to write to files outside of
597 # the upload directory will result in a failed upload.
598- self.build.handleStatus('OK', None, {
599+ self.build.handleStatus(self.build, 'OK', None, {
600 'filemap': { '/tmp/myfile.py': 'test_file_hash'},
601 })
602 self.assertEqual(BuildStatus.FAILEDTOUPLOAD, self.build.buildstate)
603@@ -211,7 +211,7 @@
604 def test_handleStatus_OK_relative_filepath(self):
605 # A filemap that tries to write to files outside of
606 # the upload directory will result in a failed upload.
607- self.build.handleStatus('OK', None, {
608+ self.build.handleStatus(self.build, 'OK', None, {
609 'filemap': { '../myfile.py': 'test_file_hash'},
610 })
611 self.assertEqual(BuildStatus.FAILEDTOUPLOAD, self.build.buildstate)
612
613=== modified file 'lib/lp/buildmaster/tests/test_packagebuild.py'
614--- lib/lp/buildmaster/tests/test_packagebuild.py 2010-05-05 15:49:22 +0000
615+++ lib/lp/buildmaster/tests/test_packagebuild.py 2010-05-05 15:49:24 +0000
616@@ -6,12 +6,12 @@
617 __metaclass__ = type
618
619 import unittest
620+import hashlib
621
622 from storm.store import Store
623 from zope.component import getUtility
624 from zope.security.proxy import removeSecurityProxy
625
626-from canonical.database.sqlbase import flush_database_updates
627 from canonical.testing.layers import LaunchpadFunctionalLayer
628
629 from lp.buildmaster.interfaces.buildfarmjob import BuildFarmJobType
630@@ -74,8 +74,8 @@
631
632 def test_saves_record(self):
633 # A package build can be stored in the database.
634- flush_database_updates()
635 store = Store.of(self.package_build)
636+ store.flush()
637 retrieved_build = store.find(
638 PackageBuild,
639 PackageBuild.id == self.package_build.id).one()
640@@ -86,6 +86,9 @@
641 self.assertRaises(NotImplementedError, self.package_build.getTitle)
642 self.assertRaises(
643 NotImplementedError, self.package_build.estimateDuration)
644+ self.assertRaises(
645+ NotImplementedError, self.package_build.verifySuccessfulUpload)
646+ self.assertRaises(NotImplementedError, self.package_build.notify)
647
648 def test_default_values(self):
649 # PackageBuild has a number of default values.
650@@ -105,15 +108,25 @@
651 self.package_build.build_farm_job.id),
652 log_url)
653
654+ def test_storeUploadLog(self):
655+ # The given content is uploaded to the librarian and linked as
656+ # the upload log.
657+ self.package_build.storeUploadLog("Some content")
658+ self.failIfEqual(None, self.package_build.upload_log)
659+ self.failUnlessEqual(
660+ hashlib.sha1("Some content").hexdigest(),
661+ self.package_build.upload_log.content.sha1)
662+
663 def test_upload_log_url(self):
664 # The url of the upload log file is determined by the PackageBuild.
665- lfa = self.factory.makeLibraryFileAlias('myuploadlog.txt')
666- removeSecurityProxy(self.package_build).upload_log = lfa
667+ Store.of(self.package_build).flush()
668+ build_id = self.package_build.build_farm_job.id
669+ self.package_build.storeUploadLog("Some content")
670 log_url = self.package_build.upload_log_url
671 self.failUnlessEqual(
672 'http://launchpad.dev/~joe/'
673- '+archive/ppa/+build/%d/+files/myuploadlog.txt' % (
674- self.package_build.build_farm_job.id),
675+ '+archive/ppa/+build/%d/+files/upload_%d_log.txt' % (
676+ build_id, build_id),
677 log_url)
678
679
680
681=== modified file 'lib/lp/code/model/sourcepackagerecipebuild.py'
682--- lib/lp/code/model/sourcepackagerecipebuild.py 2010-05-05 15:49:22 +0000
683+++ lib/lp/code/model/sourcepackagerecipebuild.py 2010-05-05 15:49:24 +0000
684@@ -27,8 +27,8 @@
685 from lp.buildmaster.interfaces.buildfarmjob import BuildFarmJobType
686 from lp.buildmaster.model.buildbase import BuildBase
687 from lp.buildmaster.model.buildqueue import BuildQueue
688-from lp.buildmaster.model.packagebuild import (
689- PackageBuildDerived)
690+from lp.buildmaster.model.buildfarmjob import BuildFarmJobDerived
691+from lp.buildmaster.model.packagebuild import PackageBuild
692 from lp.code.interfaces.sourcepackagerecipebuild import (
693 ISourcePackageRecipeBuildJob, ISourcePackageRecipeBuildJobSource,
694 ISourcePackageRecipeBuild, ISourcePackageRecipeBuildSource)
695@@ -199,7 +199,7 @@
696 return
697
698
699-class SourcePackageRecipeBuildJob(PackageBuildDerived, Storm):
700+class SourcePackageRecipeBuildJob(BuildFarmJobDerived, Storm):
701 classProvides(ISourcePackageRecipeBuildJobSource)
702 implements(ISourcePackageRecipeBuildJob)
703
704@@ -222,6 +222,12 @@
705 self.job = job
706 super(SourcePackageRecipeBuildJob, self).__init__()
707
708+ def _set_build_farm_job(self):
709+ """Setup the IBuildFarmJob delegate.
710+
711+ We override this to provide a delegate specific to package builds."""
712+ self.build_farm_job = PackageBuild(self.build)
713+
714 @classmethod
715 def new(cls, build, job):
716 """See `ISourcePackageRecipeBuildJobSource`."""
717
718=== modified file 'lib/lp/soyuz/model/buildpackagejob.py'
719--- lib/lp/soyuz/model/buildpackagejob.py 2010-05-05 15:49:22 +0000
720+++ lib/lp/soyuz/model/buildpackagejob.py 2010-05-05 15:49:24 +0000
721@@ -18,8 +18,8 @@
722 from canonical.database.sqlbase import sqlvalues
723
724 from lp.buildmaster.interfaces.buildbase import BuildStatus
725-from lp.buildmaster.model.packagebuild import (
726- PackageBuildDerived)
727+from lp.buildmaster.model.buildfarmjob import BuildFarmJobDerived
728+from lp.buildmaster.model.packagebuild import PackageBuild
729 from lp.registry.interfaces.sourcepackage import SourcePackageUrgency
730 from lp.registry.interfaces.pocket import PackagePublishingPocket
731 from lp.soyuz.interfaces.archive import ArchivePurpose
732@@ -28,7 +28,7 @@
733 from lp.soyuz.interfaces.publishing import PackagePublishingStatus
734
735
736-class BuildPackageJob(PackageBuildDerived, Storm):
737+class BuildPackageJob(BuildFarmJobDerived, Storm):
738 """See `IBuildPackageJob`."""
739 implements(IBuildPackageJob)
740
741@@ -42,10 +42,15 @@
742 build = Reference(build_id, 'BinaryPackageBuild.id')
743
744 def __init__(self, build, job):
745- """ Setup the IBuildFarmJob delegation when new items are created."""
746 self.build, self.job = build, job
747 super(BuildPackageJob, self).__init__()
748
749+ def _set_build_farm_job(self):
750+ """Setup the IBuildFarmJob delegate.
751+
752+ We override this to provide a delegate specific to package builds."""
753+ self.build_farm_job = PackageBuild(self.build)
754+
755 def score(self):
756 """See `IBuildPackageJob`."""
757 score_pocketname = {
758
759=== modified file 'lib/lp/translations/model/translationtemplatesbuildjob.py'
760--- lib/lp/translations/model/translationtemplatesbuildjob.py 2010-04-28 08:24:54 +0000
761+++ lib/lp/translations/model/translationtemplatesbuildjob.py 2010-05-05 15:49:24 +0000
762@@ -22,7 +22,8 @@
763
764 from lp.buildmaster.interfaces.buildfarmjob import BuildFarmJobType
765 from lp.buildmaster.interfaces.buildqueue import IBuildQueueSet
766-from lp.buildmaster.model.buildfarmjob import BuildFarmJobDerived
767+from lp.buildmaster.model.buildfarmjob import (
768+ BuildFarmJob, BuildFarmJobDerived)
769 from lp.buildmaster.model.buildqueue import BuildQueue
770 from lp.code.interfaces.branchjob import IRosettaUploadJobSource
771 from lp.buildmaster.interfaces.buildfarmbranchjob import IBuildFarmBranchJob
772@@ -49,6 +50,14 @@
773 def __init__(self, branch_job):
774 super(TranslationTemplatesBuildJob, self).__init__(branch_job)
775
776+ def _set_build_farm_job(self):
777+ """Setup the IBuildFarmJob delegate.
778+
779+ We override this to provide a non-database delegate that simply
780+ provides required functionality to the queue system."""
781+ self.build_farm_job = BuildFarmJob(
782+ job_type=BuildFarmJobType.TRANSLATIONTEMPLATESBUILD)
783+
784 def score(self):
785 """See `IBuildFarmJob`."""
786 # Hard-code score for now; anything other than 1000 is probably

Subscribers

People subscribed via source and target branches

to status/vote changes: