Launchpad itself

Merge lp:~adeuring/launchpad/bug-494075 into lp:launchpad/db-devel

  1. bug-494075
  2. Merge into db-devel
Proposed by Abel Deuring
Status: Merged
Merged at revision: not available
Proposed branch: lp:~adeuring/launchpad/bug-494075
Merge into: lp:launchpad/db-devel
Diff against target: 1374 lines (+784/-245)
17 files modified
lib/canonical/launchpad/webapp/adapter.py (+13/-1)
lib/canonical/launchpad/webapp/ftests/test_adapter.txt (+8/-2)
lib/lp/buildmaster/buildergroup.py (+8/-3)
lib/lp/buildmaster/interfaces/buildfarmjobbehavior.py (+61/-0)
lib/lp/buildmaster/model/buildfarmjobbehavior.py (+71/-0)
lib/lp/buildmaster/tests/buildfarmjobbehavior.txt (+134/-0)
lib/lp/soyuz/browser/tests/builder-views.txt (+1/-0)
lib/lp/soyuz/configure.zcml (+7/-0)
lib/lp/soyuz/doc/buildd-slavescanner.txt (+48/-12)
lib/lp/soyuz/doc/builder.txt (+12/-0)
lib/lp/soyuz/doc/buildqueue.txt (+10/-0)
lib/lp/soyuz/interfaces/builder.py (+10/-25)
lib/lp/soyuz/interfaces/buildqueue.py (+5/-1)
lib/lp/soyuz/model/binarypackagebuildbehavior.py (+271/-0)
lib/lp/soyuz/model/builder.py (+64/-201)
lib/lp/soyuz/model/buildqueue.py (+7/-0)
lib/lp/soyuz/tests/test_builder.py (+54/-0)
To merge this branch: bzr merge lp:~adeuring/launchpad/bug-494075
Reviewer Review Type Date Requested Status
Brad Crittenden (community) code Approve
Review via email: mp+15823@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Abel Deuring (adeuring) wrote :

When psycopg QueryCancelError occurs, we see only the SQL statement leading to the timeout and its parameters, but not the details of the QueryCancelError. This makes it hard to understand what heppend for example in OOPS-1438B2246.

This branch adds the original QueryCancelError message to the data reported for TimeoutErrors.

test: ./bin/test -vv -t test_adapter.txt

= Launchpad lint =

Checking for conflicts. and issues in doctests and templates.
Running jslint, xmllint, pyflakes, and pylint.
Using normal rules.

Linting changed files:
  lib/canonical/launchpad/webapp/adapter.py
  lib/canonical/launchpad/webapp/ftests/test_adapter.txt
abel@klato4:~/canonical/lp-branches/bug-494075$ make lint
= Launchpad lint =

Checking for conflicts. and issues in doctests and templates.
Running jslint, xmllint, pyflakes, and pylint.
Using normal rules.

Linting changed files:
  lib/canonical/launchpad/webapp/adapter.py
  lib/canonical/launchpad/webapp/ftests/test_adapter.txt

Revision history for this message
Brad Crittenden (bac) wrote :

Hi Abel,

Thanks for this fix. Anything to make chasing down OOPs easier is great.

In your actual diff at http://paste.ubuntu.com/337502/ you need to fix the typos in the docstring:

""A variant o TimeoutError hat reports the original PostgreSQL error."""

Looks like you were thinking faster than your fingers would go.

Other than that it's fine.

review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/canonical/launchpad/webapp/adapter.py'
2--- lib/canonical/launchpad/webapp/adapter.py 2009-08-06 15:45:18 +0000
3+++ lib/canonical/launchpad/webapp/adapter.py 2009-12-08 20:16:13 +0000
4@@ -62,6 +62,18 @@
5 classImplements(TimeoutError, IRequestExpired)
6
7
8+class LaunchpadTimeoutError(TimeoutError):
9+ """A variant of TimeoutError that reports the original PostgreSQL error.
10+ """
11+
12+ def __init__(self, statement, params, original_error):
13+ super(LaunchpadTimeoutError, self).__init__(statement, params)
14+ self.original_error = original_error
15+
16+ def __str__(self):
17+ return ('Statement: %r\nParameters:%r\nOriginal error: %r'
18+ % (self.statement, self.params, self.original_error))
19+
20 def _get_dirty_commit_flags():
21 """Return the current dirty commit status"""
22 from canonical.ftests.pgsql import ConnectionWrapper
23@@ -420,7 +432,7 @@
24 return
25 if isinstance(error, QueryCanceledError):
26 OpStats.stats['timeouts'] += 1
27- raise TimeoutError(statement, params)
28+ raise LaunchpadTimeoutError(statement, params, error)
29
30 def get_remaining_time(self):
31 """See `TimeoutTracer`"""
32
33=== modified file 'lib/canonical/launchpad/webapp/ftests/test_adapter.txt'
34--- lib/canonical/launchpad/webapp/ftests/test_adapter.txt 2009-08-06 15:45:18 +0000
35+++ lib/canonical/launchpad/webapp/ftests/test_adapter.txt 2009-12-08 20:16:13 +0000
36@@ -115,7 +115,10 @@
37 >>> store.execute('SELECT pg_sleep(0.200)', noresult=True)
38 Traceback (most recent call last):
39 ...
40- TimeoutError: ...SELECT pg_sleep(0.200)...
41+ LaunchpadTimeoutError: Statement: 'SELECT pg_sleep(0.200)'
42+ Parameters:()
43+ Original error: QueryCanceledError('canceling statement due to
44+ statement timeout\n',)
45
46 Even though the statement timed out, it is recorded in the statement log:
47
48@@ -175,7 +178,10 @@
49 >>> store.execute('SELECT pg_sleep(0.2)', noresult=True)
50 Traceback (most recent call last):
51 ...
52- TimeoutError: ...SELECT pg_sleep(0.2)...
53+ LaunchpadTimeoutError: Statement: 'SELECT pg_sleep(0.2)'
54+ Parameters:()
55+ Original error: QueryCanceledError('canceling statement due to
56+ statement timeout\n',)
57 >>> clear_request_started()
58
59
60
61=== modified file 'lib/lp/buildmaster/buildergroup.py'
62--- lib/lp/buildmaster/buildergroup.py 2009-11-13 19:34:17 +0000
63+++ lib/lp/buildmaster/buildergroup.py 2009-12-08 20:16:13 +0000
64@@ -171,8 +171,8 @@
65 Perform the required actions for each state.
66 """
67 try:
68- (builder_status, build_id, build_status, logtail, filemap,
69- dependencies) = queueItem.builder.slaveStatus()
70+ slave_status = queueItem.builder.slaveStatus()
71+
72 except (xmlrpclib.Fault, socket.error), info:
73 # XXX cprov 2005-06-29:
74 # Hmm, a problem with the xmlrpc interface,
75@@ -192,6 +192,7 @@
76 'BuilderStatus.WAITING': self.updateBuild_WAITING,
77 }
78
79+ builder_status = slave_status['builder_status']
80 if builder_status not in builder_status_handlers:
81 self.logger.critical(
82 "Builder on %s returned unknown status %s, failing it"
83@@ -209,7 +210,11 @@
84 # from the IBuilder content class, it arrives protected by a Zope
85 # Security Proxy, which is not declared, thus empty. Before passing
86 # it to the status handlers we will simply remove the proxy.
87- logtail = removeSecurityProxy(logtail)
88+ logtail = removeSecurityProxy(slave_status.get('logtail'))
89+ build_id = slave_status.get('build_id')
90+ build_status = slave_status.get('build_status')
91+ filemap = slave_status.get('filemap')
92+ dependencies = slave_status.get('dependencies')
93
94 method = builder_status_handlers[builder_status]
95 try:
96
97=== added file 'lib/lp/buildmaster/interfaces/buildfarmjobbehavior.py'
98--- lib/lp/buildmaster/interfaces/buildfarmjobbehavior.py 1970-01-01 00:00:00 +0000
99+++ lib/lp/buildmaster/interfaces/buildfarmjobbehavior.py 2009-12-08 20:16:13 +0000
100@@ -0,0 +1,61 @@
101+# Copyright 2009 Canonical Ltd. This software is licensed under the
102+# GNU Affero General Public License version 3 (see the file LICENSE).
103+
104+# pylint: disable-msg=E0211,E0213
105+
106+"""Interface for build farm job behaviors."""
107+
108+__metaclass__ = type
109+
110+__all__ = [
111+ 'IBuildFarmJobBehavior',
112+ ]
113+
114+from zope.interface import Attribute, Interface
115+
116+
117+class BuildBehaviorMismatch(Exception):
118+ """
119+ A general exception that can be raised when the builder's current behavior
120+ does not match the expected behavior.
121+ """
122+
123+
124+class IBuildFarmJobBehavior(Interface):
125+
126+ status = Attribute(
127+ "Generated status information for this particular job.")
128+
129+ def setBuilder(builder):
130+ """Sets the associated builder reference for this instance."""
131+
132+ def logStartBuild(build_queue_item, logger):
133+ """Log the start of a specific build queue item.
134+
135+ The form of the log message will vary depending on the type of build.
136+ :param build_queue_item: A BuildQueueItem to build.
137+ :param logger: A logger to be used to log diagnostic information.
138+ """
139+
140+ def dispatchBuildToSlave(build_queue_item, logger):
141+ """Dispatch a specific build to the slave.
142+
143+ :param build_queue_item: The `BuildQueueItem` that will be built.
144+ :logger: A logger to be used to log diagnostic information.
145+ """
146+
147+ def verifyBuildRequest(build_queue_item, logger):
148+ """Carry out any pre-build checks.
149+
150+ :param build_queue_item: The `BuildQueueItem` that is to be built.
151+ :logger: A logger to be used to log diagnostic information.
152+ """
153+
154+ def slaveStatus(self, raw_slave_status):
155+ """Return a dict of custom slave status values for this behavior.
156+
157+ :param raw_slave_status: The value returned by the build slave's
158+ status() method.
159+ :return: a dict of extra key/values to be included in the result
160+ of IBuilder.slaveStatus().
161+ """
162
163=== added file 'lib/lp/buildmaster/model/buildfarmjobbehavior.py'
164--- lib/lp/buildmaster/model/buildfarmjobbehavior.py 1970-01-01 00:00:00 +0000
165+++ lib/lp/buildmaster/model/buildfarmjobbehavior.py 2009-12-08 20:16:13 +0000
166@@ -0,0 +1,71 @@
167+# Copyright 2009 Canonical Ltd. This software is licensed under the
168+# GNU Affero General Public License version 3 (see the file LICENSE).
169+
170+# pylint: disable-msg=E0211,E0213
171+
172+"""Base and idle BuildFarmJobBehavior classes."""
173+
174+__metaclass__ = type
175+
176+__all__ = [
177+ 'BuildFarmJobBehaviorBase',
178+ 'IdleBuildBehavior'
179+ ]
180+
181+from zope.interface import implements
182+
183+from lp.buildmaster.interfaces.buildfarmjobbehavior import (
184+ BuildBehaviorMismatch, IBuildFarmJobBehavior)
185+
186+
187+class BuildFarmJobBehaviorBase:
188+ """Ensures that all behaviors inherit the same initialisation.
189+
190+ All build-farm job behaviors should inherit from this.
191+ """
192+
193+ def __init__(self, buildfarmjob):
194+ """Store a reference to the job_type with which we were created."""
195+ self.buildfarmjob = buildfarmjob
196+ self._builder = None
197+
198+ def setBuilder(self, builder):
199+ """The builder should be set once and not changed."""
200+ self._builder = builder
201+
202+ def verifyBuildRequest(self, build_queue_item, logger):
203+ """The default behavior is a no-op."""
204+ pass
205+
206+ def slaveStatus(self, raw_slave_status):
207+ """See `IBuildFarmJobBehavior`.
208+
209+ The default behavior is that we don't add any extra values."""
210+ return {}
211+
212+
213+class IdleBuildBehavior(BuildFarmJobBehaviorBase):
214+
215+ implements(IBuildFarmJobBehavior)
216+
217+ def __init__(self):
218+ """The idle behavior is special in that a buildfarmjob is not
219+ specified during initialisation as it is not the result of an
220+ adaption.
221+ """
222+ super(IdleBuildBehavior, self).__init__(None)
223+
224+ def logStartBuild(self, build_queue_item, logger):
225+ """See `IBuildFarmJobBehavior`."""
226+ raise BuildBehaviorMismatch(
227+ "Builder was idle when asked to log the start of a build.")
228+
229+ def dispatchBuildToSlave(self, build_queue_item, logger):
230+ """See `IBuildFarmJobBehavior`."""
231+ raise BuildBehaviorMismatch(
232+ "Builder was idle when asked to dispatch a build to the slave.")
233+
234+ @property
235+ def status(self):
236+ """See `IBuildFarmJobBehavior`."""
237+ return "Idle"
238
239=== added file 'lib/lp/buildmaster/tests/buildfarmjobbehavior.txt'
240--- lib/lp/buildmaster/tests/buildfarmjobbehavior.txt 1970-01-01 00:00:00 +0000
241+++ lib/lp/buildmaster/tests/buildfarmjobbehavior.txt 2009-12-08 20:16:13 +0000
242@@ -0,0 +1,134 @@
243+BuildFarmJobBehavior
244+====================
245+
246+The Launchpad build farm was originally designed for building binary
247+packages from source packages, but was subsequently generalised to support
248+other types of build farm jobs.
249+
250+The `BuildFarmJobBehavior` class encapsulates job-type-specific behavior
251+with a standard interface to which our generic IBuilder class delegates.
252+The result is that neither our generic IBuilder class or any call-sites
253+(such as the build master) need any knowledge of different job types or
254+how to handle them.
255+
256+
257+Creating a new behavior
258+-----------------------
259+
260+A new behavior should implement the `IBuildFarmJobBehavior` interface
261+and extend BuildFarmJobBehaviorBase. A new behavior will only be required
262+to define one method - dispatchBuildToSlave() - to correctly implement
263+the interface, but will usually want to customise the other properties and
264+methods as well.
265+
266+ >>> from lp.buildmaster.interfaces.buildfarmjobbehavior import (
267+ ... IBuildFarmJobBehavior)
268+ >>> from lp.buildmaster.model.buildfarmjobbehavior import (
269+ ... BuildFarmJobBehaviorBase)
270+ >>> from zope.interface import implements
271+
272+ >>> class MyNewBuildBehavior(BuildFarmJobBehaviorBase):
273+ ... """A custom build behavior for building blah."""
274+ ... implements(IBuildFarmJobBehavior)
275+ ...
276+ ... def dispatchBuildToSlave(self, build_queue_item, logger):
277+ ... print "Did something special to dispatch MySpecialBuild."
278+ ...
279+ ... @property
280+ ... def status(self):
281+ ... return "Currently building a MyNewBuild object."
282+
283+For this documentation, we'll also need a dummy new build farm job.
284+
285+ >>> from lp.buildmaster.interfaces.buildfarmjob import IBuildFarmJob
286+ >>> class IMyNewBuildFarmJob(IBuildFarmJob):
287+ ... "Normally defines job-type specific database fields."""
288+ >>> class MyNewBuildFarmJob(object):
289+ ... implements(IMyNewBuildFarmJob)
290+
291+Custom behaviors are not normally instantiated directly, instead an adapter is
292+specified for the specific IBuildFarmJob. Normaly we'd add some ZCML to
293+adapt our specific build farm job to its behavior like:
294+
295+ <!-- MyNewBuildBehavior -->
296+ <adapter
297+ for="lp.myapp.interfaces.mynewbuildfarmjob.IMyNewBuildFarmJob"
298+ provides="lp.buildmaster.interfaces.buildfarmjobbehavior.IBuildFarmJobBehavior"
299+ factory="lp.myapp.model.mynewbuildbehavior.MyNewBuildBehavior"
300+ permission="zope.Public" />
301+
302+But for the sake of this documentation we'll add the adapter manually.
303+
304+ >>> from zope.app.testing import ztapi
305+ >>> ztapi.provideAdapter(
306+ ... MyNewBuildFarmJob, IBuildFarmJobBehavior, MyNewBuildBehavior)
307+
308+This will then allow the builder to request and set the required behavior from
309+the current job. Bob the builder currently has a binary package job and so
310+finds itself with a binary package build behavior which defines the status
311+attribute with some binary-build specific information.
312+
313+ >>> from lp.soyuz.model.builder import Builder
314+ >>> from canonical.launchpad.webapp.interfaces import (
315+ ... IStoreSelector, MAIN_STORE, DEFAULT_FLAVOR)
316+ >>> store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
317+ >>> bob = store.find(Builder, Builder.name == 'bob').one()
318+ >>> print bob.status
319+ Building i386 build of mozilla-firefox 0.9 in ubuntu hoary RELEASE
320+
321+XXX Michael Nelson 2009-12-04 bug 484819. At the time of writing the
322+BuildQueue.specific_job method has not been updated to support different
323+job types, so we will set the behavior explicitly here.
324+
325+ >>> bob.current_build_behavior = IBuildFarmJobBehavior(
326+ ... MyNewBuildFarmJob())
327+
328+Once the builder has the relevant behavior, it is able to provide both general
329+builder functionality of its own accord, while delegating any build-type
330+specific functionality to the behavior. For example, if the builder is not
331+disabled, the builder delegates the status property to the behavior.
332+
333+ >>> print bob.status
334+ Currently building a MyNewBuild object.
335+
336+On the other hand, if the builder is disabled, the builder itself determines
337+the status.
338+
339+ >>> bob.builderok = False
340+ >>> print bob.status
341+ Disabled
342+ >>> bob.builderok = True
343+
344+The IBuildFarmJobBehavior interface currently provides customisation points
345+throughout the build life-cycle, from logging the start of a build, verifying
346+that the provided queue item is ready to be built, dispatching the build etc.,
347+and allows further customisation to be added easily.
348+
349+Please refer to the IBuildFarmJobBehavior interface to see the currently
350+provided build-type specific customisation points.
351+
352+
353+The IdleBuildBehavior
354+---------------------
355+
356+If a Builder does not have a currentjob (and therefore an appropriate
357+build behavior) it will automatically get the special IdleBuildBehavior
358+to ensure that it fulfills its contract to implement IBuildFarmJobBehavior.
359+
360+First, we'll reset the current build behavior and destroy the current job.
361+
362+ >>> bob.current_build_behavior = None
363+ >>> bob.currentjob.destroySelf()
364+
365+ >>> print bob.status
366+ Idle
367+
368+Attempting to use any other build-related functionality when a builder is
369+idle, such as making a call to log the start of a build, will raise an
370+appropriate exception.
371+
372+ >>> bob.logStartBuild(None, None)
373+ Traceback (most recent call last):
374+ ...
375+ BuildBehaviorMismatch: Builder was idle when asked to log the start of a
376+ build.
377
378=== modified file 'lib/lp/soyuz/browser/tests/builder-views.txt'
379--- lib/lp/soyuz/browser/tests/builder-views.txt 2009-11-24 07:02:21 +0000
380+++ lib/lp/soyuz/browser/tests/builder-views.txt 2009-12-08 20:16:13 +0000
381@@ -228,6 +228,7 @@
382 >>> login("foo.bar@canonical.com")
383 >>> private_job = store.get(BuildQueue, private_job_id)
384 >>> Store.of(private_job).remove(private_job)
385+ >>> frog.current_build_behavior = None
386
387 >>> login('no-priv@canonical.com')
388 >>> nopriv_view = getMultiAdapter((frog, empty_request), name="+index")
389
390=== modified file 'lib/lp/soyuz/configure.zcml'
391--- lib/lp/soyuz/configure.zcml 2009-11-16 22:06:14 +0000
392+++ lib/lp/soyuz/configure.zcml 2009-12-08 20:16:13 +0000
393@@ -892,4 +892,11 @@
394 interface="lp.soyuz.interfaces.buildpackagejob.IBuildPackageJob"/>
395 </class>
396
397+ <!-- BinaryPackageBuildBehavior -->
398+ <adapter
399+ for="lp.soyuz.interfaces.buildpackagejob.IBuildPackageJob"
400+ provides="lp.buildmaster.interfaces.buildfarmjobbehavior.IBuildFarmJobBehavior"
401+ factory="lp.soyuz.model.binarypackagebuildbehavior.BinaryPackageBuildBehavior"
402+ permission="zope.Public" />
403+
404 </configure>
405
406=== modified file 'lib/lp/soyuz/doc/buildd-slavescanner.txt'
407--- lib/lp/soyuz/doc/buildd-slavescanner.txt 2009-11-13 19:34:17 +0000
408+++ lib/lp/soyuz/doc/buildd-slavescanner.txt 2009-12-08 20:16:13 +0000
409@@ -1494,8 +1494,8 @@
410
411 == Builder Status Handler ==
412
413-IBuilder.slaveStatus should return a size homogeneous tuple despite of
414-the current slave state. This tuple should contain, in this order:
415+IBuilder.slaveStatus should return a dict containing the following
416+items:
417
418 * slave status string: 'BuilderStatus.IDLE'
419 * job identifier string: '1-1'
420@@ -1504,24 +1504,60 @@
421 * result file list: {'foo.deb', 'foo.changes'} or None
422 * dependencies string: 'bar baz zaz' or None
423
424+ # Define a helper to print the slave status dict.
425+ >>> from collections import defaultdict
426+ >>> def printSlaveStatus(status_dict):
427+ ... status_dict = defaultdict(lambda:None, status_dict)
428+ ... print (
429+ ... "builder_status: %(builder_status)s\n"
430+ ... "build_id: %(build_id)s\n"
431+ ... "build_status: %(build_status)s\n"
432+ ... "logtail: %(logtail)r\n"
433+ ... "filemap: %(filemap)s\n"
434+ ... "dependencies: %(dependencies)s\n" % status_dict)
435
436 >>> a_builder.setSlaveForTesting(OkSlave())
437- >>> a_builder.slaveStatus()
438- ('BuilderStatus.IDLE', '', None, None, None, None)
439+ >>> printSlaveStatus(a_builder.slaveStatus())
440+ builder_status: BuilderStatus.IDLE
441+ build_id:
442+ build_status: None
443+ logtail: None
444+ filemap: None
445+ dependencies: None
446
447 >>> a_builder.setSlaveForTesting(BuildingSlave())
448- >>> a_builder.slaveStatus()
449- ('BuilderStatus.BUILDING', '1-1', None, <xmlrpclib.Binary ...>, None, None)
450+ >>> printSlaveStatus(a_builder.slaveStatus())
451+ builder_status: BuilderStatus.BUILDING
452+ build_id: 1-1
453+ build_status: None
454+ logtail: <xmlrpclib.Binary ...>
455+ filemap: None
456+ dependencies: None
457
458 >>> a_builder.setSlaveForTesting(WaitingSlave(state='BuildStatus.OK'))
459- >>> a_builder.slaveStatus()
460- ('BuilderStatus.WAITING', '1-1', 'BuildStatus.OK', None, {}, None)
461+ >>> printSlaveStatus(a_builder.slaveStatus())
462+ builder_status: BuilderStatus.WAITING
463+ build_id: 1-1
464+ build_status: BuildStatus.OK
465+ logtail: None
466+ filemap: {}
467+ dependencies: None
468
469 >>> a_builder.setSlaveForTesting(AbortingSlave())
470- >>> a_builder.slaveStatus()
471- ('BuilderStatus.ABORTING', '1-1', None, None, None, None)
472+ >>> printSlaveStatus(a_builder.slaveStatus())
473+ builder_status: BuilderStatus.ABORTING
474+ build_id: 1-1
475+ build_status: None
476+ logtail: None
477+ filemap: None
478+ dependencies: None
479
480 >>> a_builder.setSlaveForTesting(AbortedSlave())
481- >>> a_builder.slaveStatus()
482- ('BuilderStatus.ABORTED', '1-1', None, None, None, None)
483+ >>> printSlaveStatus(a_builder.slaveStatus())
484+ builder_status: BuilderStatus.ABORTED
485+ build_id: 1-1
486+ build_status: None
487+ logtail: None
488+ filemap: None
489+ dependencies: None
490
491
492=== modified file 'lib/lp/soyuz/doc/builder.txt'
493--- lib/lp/soyuz/doc/builder.txt 2009-11-12 12:52:23 +0000
494+++ lib/lp/soyuz/doc/builder.txt 2009-12-08 20:16:13 +0000
495@@ -40,6 +40,18 @@
496 >>> builder.builderok = True
497 >>> builder.failnotes = None
498
499+A builder provides a current_build_behavior attribute to which all the
500+build-type specific behavior for the current build is delegated. For
501+our sample data, the behavior is specifically a behavior for dealing
502+with binary packages
503+
504+ >>> from zope.security.proxy import isinstance
505+ >>> from lp.soyuz.model.binarypackagebuildbehavior import (
506+ ... BinaryPackageBuildBehavior)
507+ >>> isinstance(
508+ ... builder.current_build_behavior, BinaryPackageBuildBehavior)
509+ True
510+
511 In case of copy archives the status string will show both the copy
512 archive owner as well as the copy archive name.
513
514
515=== modified file 'lib/lp/soyuz/doc/buildqueue.txt'
516--- lib/lp/soyuz/doc/buildqueue.txt 2009-11-13 19:34:17 +0000
517+++ lib/lp/soyuz/doc/buildqueue.txt 2009-12-08 20:16:13 +0000
518@@ -98,6 +98,16 @@
519 (True, 1000)
520
521
522+The BuildQueue item is responsible for providing the required build behavior
523+for the item.
524+
525+ >>> from zope.security.proxy import isinstance
526+ >>> from lp.soyuz.model.binarypackagebuildbehavior import (
527+ ... BinaryPackageBuildBehavior)
528+ >>> isinstance(bq.required_build_behavior, BinaryPackageBuildBehavior)
529+ True
530+
531+
532 == Dispatching and Reseting jobs ==
533
534 The sampledata contains an active job, being built by the 'bob'
535
536=== modified file 'lib/lp/soyuz/interfaces/builder.py'
537--- lib/lp/soyuz/interfaces/builder.py 2009-11-20 18:06:28 +0000
538+++ lib/lp/soyuz/interfaces/builder.py 2009-12-08 20:16:13 +0000
539@@ -19,11 +19,13 @@
540 ]
541
542 from zope.interface import Interface, Attribute
543-from zope.schema import Choice, TextLine, Text, Bool
544+from zope.schema import Bool, Choice, Field, Text, TextLine
545
546 from canonical.launchpad import _
547 from canonical.launchpad.fields import Title, Description
548 from lp.registry.interfaces.role import IHasOwner
549+from lp.buildmaster.interfaces.buildfarmjobbehavior import (
550+ IBuildFarmJobBehavior)
551 from canonical.launchpad.validators.name import name_validator
552 from canonical.launchpad.validators.url import builder_url_validator
553
554@@ -57,7 +59,7 @@
555 """The build slave has suffered an error and cannot be used."""
556
557
558-class IBuilder(IHasOwner):
559+class IBuilder(IHasOwner, IBuildFarmJobBehavior):
560 """Build-slave information and state.
561
562 Builder instance represents a single builder slave machine within the
563@@ -138,6 +140,10 @@
564 "new jobs. "),
565 required=False)
566
567+ current_build_behavior = Field(
568+ title=u"The current behavior of the builder for the current job.",
569+ required=False)
570+
571 def cacheFileOnSlave(logger, libraryfilealias):
572 """Ask the slave to cache a librarian file to its local disk.
573
574@@ -148,19 +154,6 @@
575 file.
576 """
577
578- def cachePrivateSourceOnSlave(logger, build_queue_item):
579- """Ask the slave to download source files for a private build.
580-
581- The slave will cache the files for the source in build_queue_item
582- to its local disk in preparation for a private build. Private builds
583- will always take the source files from the archive rather than the
584- librarian since the archive has more granular access to each
585- archive's files.
586-
587- :param logger: A logger used for providing debug information.
588- :param build_queue_item: The `IBuildQueue` being built.
589- """
590-
591 def checkCanBuildForDistroArchSeries(distro_arch_series):
592 """Check that the slave can compile for the given distro_arch_release.
593
594@@ -215,16 +208,8 @@
595 def slaveStatus():
596 """Get the slave status for this builder.
597
598- * builder_status => string
599- * build_id => string
600- * build_status => string or None
601- * logtail => string or None
602- * filename => dictionary or None
603- * dependencies => string or None
604-
605- :return: a tuple containing (
606- builder_status, build_id, build_status, logtail, filemap,
607- dependencies)
608+ :return: a dict containing at least builder_status, but potentially
609+ other values included by the current build behavior.
610 """
611
612 def slaveStatusSentence():
613
614=== modified file 'lib/lp/soyuz/interfaces/buildqueue.py'
615--- lib/lp/soyuz/interfaces/buildqueue.py 2009-11-20 18:06:28 +0000
616+++ lib/lp/soyuz/interfaces/buildqueue.py 2009-12-08 20:16:13 +0000
617@@ -13,7 +13,7 @@
618 ]
619
620 from zope.interface import Interface, Attribute
621-from zope.schema import Choice, Datetime, Timedelta
622+from zope.schema import Choice, Datetime, Field, Timedelta
623
624 from lazr.restful.fields import Reference
625
626@@ -51,6 +51,10 @@
627 title=_('Job type'), required=True, vocabulary=BuildFarmJobType,
628 description=_("The type of this job."))
629
630+ required_build_behavior = Field(
631+ title=_('The builder behavior required to run this job.'),
632+ required=False, readonly=True)
633+
634 estimated_duration = Timedelta(
635 title=_("Estimated Job Duration"), required=True,
636 description=_("Estimated job duration interval."))
637
638=== added file 'lib/lp/soyuz/model/binarypackagebuildbehavior.py'
639--- lib/lp/soyuz/model/binarypackagebuildbehavior.py 1970-01-01 00:00:00 +0000
640+++ lib/lp/soyuz/model/binarypackagebuildbehavior.py 2009-12-08 20:16:13 +0000
641@@ -0,0 +1,271 @@
642+# Copyright 2009 Canonical Ltd. This software is licensed under the
643+# GNU Affero General Public License version 3 (see the file LICENSE).
644+
645+# pylint: disable-msg=E0211,E0213
646+
647+"""Builder behavior for binary package builds."""
648+
649+__metaclass__ = type
650+
651+__all__ = [
652+ 'BinaryPackageBuildBehavior',
653+ ]
654+
655+import socket
656+import xmlrpclib
657+
658+from canonical.launchpad.webapp import urlappend
659+from lp.buildmaster.interfaces.buildfarmjobbehavior import (
660+ IBuildFarmJobBehavior)
661+from lp.buildmaster.model.buildfarmjobbehavior import (
662+ BuildFarmJobBehaviorBase)
663+from lp.registry.interfaces.pocket import PackagePublishingPocket
664+from lp.soyuz.adapters.archivedependencies import (
665+ get_primary_current_component, get_sources_list_for_building)
666+from lp.soyuz.interfaces.archive import ArchivePurpose
667+from lp.soyuz.interfaces.build import IBuildSet
668+from lp.soyuz.interfaces.builder import BuildSlaveFailure, CannotBuild
669+
670+from zope.component import getUtility
671+from zope.interface import implements
672+
673+
674+class BinaryPackageBuildBehavior(BuildFarmJobBehaviorBase):
675+ """Define the behavior of binary package builds."""
676+
677+ implements(IBuildFarmJobBehavior)
678+
679+ def logStartBuild(self, build_queue_item, logger):
680+ """See `IBuildFarmJobBehavior`."""
681+ build = getUtility(IBuildSet).getByQueueEntry(build_queue_item)
682+ spr = build.sourcepackagerelease
683+
684+ logger.info("startBuild(%s, %s, %s, %s)", self._builder.url,
685+ spr.name, spr.version, build.pocket.title)
686+
687+ @property
688+ def status(self):
689+ """See `IBuildFarmJobBehavior`."""
690+ build = getUtility(IBuildSet).getByQueueEntry(
691+ self._builder.currentjob)
692+ msg = 'Building %s' % build.title
693+ archive = build.archive
694+ if not archive.owner.private and (archive.is_ppa or archive.is_copy):
695+ return '%s [%s/%s]' % (msg, archive.owner.name, archive.name)
696+ else:
697+ return msg
698+
699+ def dispatchBuildToSlave(self, build_queue_item, logger):
700+ """See `IBuildFarmJobBehavior`."""
701+
702+ # Start the binary package build on the slave builder. First
703+ # we send the chroot.
704+ build = getUtility(IBuildSet).getByQueueEntry(build_queue_item)
705+ chroot = build.distroarchseries.getChroot()
706+ self._builder.cacheFileOnSlave(logger, chroot)
707+
708+ # Build filemap structure with the files required in this build
709+ # and send them to the slave.
710+ # If the build is private we tell the slave to get the files from the
711+ # archive instead of the librarian because the slaves cannot
712+ # access the restricted librarian.
713+ private = build.archive.private
714+ if private:
715+ self._cachePrivateSourceOnSlave(build_queue_item, logger)
716+ filemap = {}
717+ for source_file in build.sourcepackagerelease.files:
718+ lfa = source_file.libraryfile
719+ filemap[lfa.filename] = lfa.content.sha1
720+ if not private:
721+ self._builder.cacheFileOnSlave(
722+ logger, source_file.libraryfile)
723+
724+ # Generate a string which can be used to cross-check when obtaining
725+ # results so we know we are referring to the right database object in
726+ # subsequent runs.
727+ buildid = "%s-%s" % (build.id, build_queue_item.id)
728+ chroot_sha1 = chroot.content.sha1
729+ logger.debug(
730+ "Initiating build %s on %s" % (buildid, self._builder.url))
731+
732+ try:
733+ args = self._extraBuildArgs(build)
734+ status, info = self._builder.slave.build(
735+ buildid, "debian", chroot_sha1, filemap, args)
736+ message = """%s (%s):
737+ ***** RESULT *****
738+ %s
739+ %s
740+ %s: %s
741+ ******************
742+ """ % (
743+ self._builder.name,
744+ self._builder.url,
745+ filemap,
746+ args,
747+ status,
748+ info,
749+ )
750+ logger.info(message)
751+ except xmlrpclib.Fault, info:
752+ # Mark builder as 'failed'.
753+ logger.debug(
754+ "Disabling builder: %s" % self._builder.url, exc_info=1)
755+ self._builder.failbuilder(
756+ "Exception (%s) when setting up to new job" % info)
757+ raise BuildSlaveFailure
758+ except socket.error, info:
759+ error_message = "Exception (%s) when setting up new job" % info
760+ self._builder.handleTimeout(logger, error_message)
761+ raise BuildSlaveFailure
762+
763+ def verifyBuildRequest(self, build_queue_item, logger):
764+ """Assert some pre-build checks.
765+
766+ The build request is checked:
767+ * Virtualized builds can't build on a non-virtual builder
768+ * Ensure that we have a chroot
769+ * Ensure that the build pocket allows builds for the current
770+ distroseries state.
771+ """
772+ build = getUtility(IBuildSet).getByQueueEntry(build_queue_item)
773+ assert not (not self._builder.virtualized and build.is_virtualized), (
774+ "Attempt to build non-virtual item on a virtual builder.")
775+
776+ # Assert that we are not silently building SECURITY jobs.
777+ # See findBuildCandidates. Once we start building SECURITY
778+ # correctly from EMBARGOED archive this assertion can be removed.
779+ # XXX Julian 2007-12-18 spec=security-in-soyuz: This is being
780+ # addressed in the work on the blueprint:
781+ # https://blueprints.launchpad.net/soyuz/+spec/security-in-soyuz
782+ target_pocket = build.pocket
783+ assert target_pocket != PackagePublishingPocket.SECURITY, (
784+ "Soyuz is not yet capable of building SECURITY uploads.")
785+
786+ # Ensure build has the needed chroot
787+ chroot = build.distroarchseries.getChroot()
788+ if chroot is None:
789+ raise CannotBuild(
790+ "Missing CHROOT for %s/%s/%s" % (
791+ build.distroseries.distribution.name,
792+ build.distroseries.name,
793+ build.distroarchseries.architecturetag)
794+ )
795+
796+ # The main distribution has policies to prevent uploads to some
797+ # pockets (e.g. security) during different parts of the distribution
798+ # series lifecycle. These do not apply to PPA builds nor any archive
799+ # that allows release pocket updates.
800+ if (build.archive.purpose != ArchivePurpose.PPA and
801+ not build.archive.allowUpdatesToReleasePocket()):
802+ # XXX Robert Collins 2007-05-26: not an explicit CannotBuild
803+ # exception yet because the callers have not been audited
804+ assert build.distroseries.canUploadToPocket(build.pocket), (
805+ "%s (%s) can not be built for pocket %s: invalid pocket due "
806+ "to the series status of %s."
807+ % (build.title, build.id, build.pocket.name,
808+ build.distroseries.name))
809+
810+ def slaveStatus(self, raw_slave_status):
811+ """Parse and return the binary build specific status info.
812+
813+ This includes:
814+ * build_id => string
815+ * build_status => string or None
816+ * logtail => string or None
817+ * filename => dictionary or None
818+ * dependencies => string or None
819+ """
820+ builder_status = raw_slave_status[0]
821+ extra_info = {}
822+ if builder_status == 'BuilderStatus.WAITING':
823+ extra_info['build_status'] = raw_slave_status[1]
824+ extra_info['build_id'] = raw_slave_status[2]
825+ build_status_with_files = [
826+ 'BuildStatus.OK',
827+ 'BuildStatus.PACKAGEFAIL',
828+ 'BuildStatus.DEPFAIL',
829+ ]
830+ if extra_info['build_status'] in build_status_with_files:
831+ extra_info['filemap'] = raw_slave_status[3]
832+ extra_info['dependencies'] = raw_slave_status[4]
833+ else:
834+ extra_info['build_id'] = raw_slave_status[1]
835+ if builder_status == 'BuilderStatus.BUILDING':
836+ extra_info['logtail'] = raw_slave_status[2]
837+
838+ return extra_info
839+
840+ def _cachePrivateSourceOnSlave(self, build_queue_item, logger):
841+ """Ask the slave to download source files for a private build.
842+
843+ The slave will cache the files for the source in build_queue_item
844+ to its local disk in preparation for a private build. Private builds
845+ will always take the source files from the archive rather than the
846+ librarian since the archive has more granular access to each
847+ archive's files.
848+
849+ :param build_queue_item: The `IBuildQueue` being built.
850+ :param logger: A logger used for providing debug information.
851+ """
852+ # The URL to the file in the archive consists of these parts:
853+ # archive_url / makePoolPath() / filename
854+ # Once this is constructed we add the http basic auth info.
855+
856+ # Avoid circular imports.
857+ from lp.soyuz.model.publishing import makePoolPath
858+
859+ build = getUtility(IBuildSet).getByQueueEntry(build_queue_item)
860+ archive = build.archive
861+ archive_url = archive.archive_url
862+ component_name = build.current_component.name
863+ for source_file in build.sourcepackagerelease.files:
864+ file_name = source_file.libraryfile.filename
865+ sha1 = source_file.libraryfile.content.sha1
866+ source_name = build.sourcepackagerelease.sourcepackagename.name
867+ poolpath = makePoolPath(source_name, component_name)
868+ url = urlappend(archive_url, poolpath)
869+ url = urlappend(url, file_name)
870+ logger.debug("Asking builder on %s to ensure it has file %s "
871+ "(%s, %s)" % (
872+ self._builder.url, file_name, url, sha1))
873+ self._builder._sendFileToSlave(
874+ url, sha1, "buildd", archive.buildd_secret)
875+
876+ def _extraBuildArgs(self, build):
877+ """
878+ Return the extra arguments required by the slave for the given build.
879+ """
880+ # Build extra arguments.
881+ args = {}
882+ # turn 'arch_indep' ON only if build is archindep or if
883+ # the specific architecture is the nominatedarchindep for
884+ # this distroseries (in case it requires any archindep source)
885+ args['arch_indep'] = build.distroarchseries.isNominatedArchIndep
886+
887+ suite = build.distroarchseries.distroseries.name
888+ if build.pocket != PackagePublishingPocket.RELEASE:
889+ suite += "-%s" % (build.pocket.name.lower())
890+ args['suite'] = suite
891+
892+ archive_purpose = build.archive.purpose
893+ if (archive_purpose == ArchivePurpose.PPA and
894+ not build.archive.require_virtualized):
895+ # If we're building a non-virtual PPA, override the purpose
896+ # to PRIMARY and use the primary component override.
897+ # This ensures that the package mangling tools will run over
898+ # the built packages.
899+ args['archive_purpose'] = ArchivePurpose.PRIMARY.name
900+ args["ogrecomponent"] = (
901+ get_primary_current_component(build))
902+ else:
903+ args['archive_purpose'] = archive_purpose.name
904+ args["ogrecomponent"] = (
905+ build.current_component.name)
906+
907+ args['archives'] = get_sources_list_for_building(build)
908+
909+ # Let the build slave know whether this is a build in a private
910+ # archive.
911+ args['archive_private'] = build.archive.private
912+ return args
913
914=== modified file 'lib/lp/soyuz/model/builder.py'
915--- lib/lp/soyuz/model/builder.py 2009-11-20 18:06:28 +0000
916+++ lib/lp/soyuz/model/builder.py 2009-12-08 20:16:13 +0000
917@@ -20,6 +20,8 @@
918 import urllib2
919 import xmlrpclib
920
921+from lazr.delegates import delegates
922+
923 from zope.interface import implements
924 from zope.component import getUtility
925
926@@ -31,10 +33,11 @@
927 from canonical.cachedproperty import cachedproperty
928 from canonical.config import config
929 from canonical.buildd.slave import BuilderStatus
930+from lp.buildmaster.interfaces.buildfarmjobbehavior import (
931+ BuildBehaviorMismatch, IBuildFarmJobBehavior)
932 from lp.buildmaster.master import BuilddMaster
933+from lp.buildmaster.model.buildfarmjobbehavior import IdleBuildBehavior
934 from canonical.database.sqlbase import SQLBase, sqlvalues
935-from lp.soyuz.adapters.archivedependencies import (
936- get_primary_current_component, get_sources_list_for_building)
937 from lp.soyuz.model.buildqueue import BuildQueue
938 from lp.registry.interfaces.person import validate_public_person
939 from lp.registry.interfaces.pocket import PackagePublishingPocket
940@@ -54,6 +57,7 @@
941 PackagePublishingStatus)
942 from lp.soyuz.model.buildpackagejob import BuildPackageJob
943 from canonical.launchpad.webapp import urlappend
944+from canonical.lazr.utils import safe_hasattr
945 from canonical.librarian.utils import copy_and_close
946
947
948@@ -111,9 +115,11 @@
949
950 return (stdout, stderr, resume_process.returncode)
951
952+
953 class Builder(SQLBase):
954
955 implements(IBuilder, IHasBuildRecords)
956+ delegates(IBuildFarmJobBehavior, context="current_build_behavior")
957 _table = 'Builder'
958
959 _defaultOrder = ['id']
960@@ -135,6 +141,44 @@
961 vm_host = StringCol(dbName='vm_host')
962 active = BoolCol(dbName='active', notNull=True, default=True)
963
964+ def _getCurrentBuildBehavior(self):
965+ """Return the current build behavior."""
966+ if not safe_hasattr(self, '_current_build_behavior'):
967+ self._current_build_behavior = None
968+
969+ if (self._current_build_behavior is None or
970+ isinstance(self._current_build_behavior, IdleBuildBehavior)):
971+ # If we don't currently have a current build behavior set,
972+ # or we are currently idle, then...
973+ currentjob = self.currentjob
974+ if currentjob is not None:
975+ # ...we'll set it based on our current job.
976+ self._current_build_behavior = (
977+ currentjob.required_build_behavior)
978+ self._current_build_behavior.setBuilder(self)
979+ return self._current_build_behavior
980+ elif self._current_build_behavior is None:
981+ # If we don't have a current job or an idle behavior
982+ # already set, then we just set the idle behavior
983+ # before returning.
984+ self._current_build_behavior = IdleBuildBehavior()
985+ return self._current_build_behavior
986+
987+ else:
988+ # We did have a current non-idle build behavior set, so
989+ # we just return it.
990+ return self._current_build_behavior
991+
992+
993+ def _setCurrentBuildBehavior(self, new_behavior):
994+ """Set the current build behavior."""
995+ self._current_build_behavior = new_behavior
996+ if self._current_build_behavior is not None:
997+ self._current_build_behavior.setBuilder(self)
998+
999+ current_build_behavior = property(
1000+ _getCurrentBuildBehavior, _setCurrentBuildBehavior)
1001+
1002 def cacheFileOnSlave(self, logger, libraryfilealias):
1003 """See `IBuilder`."""
1004 url = libraryfilealias.http_url
1005@@ -143,30 +187,6 @@
1006 url, libraryfilealias.content.sha1))
1007 self._sendFileToSlave(url, libraryfilealias.content.sha1)
1008
1009- def cachePrivateSourceOnSlave(self, logger, build_queue_item):
1010- """See `IBuilder`."""
1011- # The URL to the file in the archive consists of these parts:
1012- # archive_url / makePoolPath() / filename
1013- # Once this is constructed we add the http basic auth info.
1014-
1015- # Avoid circular imports.
1016- from lp.soyuz.model.publishing import makePoolPath
1017-
1018- build = getUtility(IBuildSet).getByQueueEntry(build_queue_item)
1019- archive = build.archive
1020- archive_url = archive.archive_url
1021- component_name = build.current_component.name
1022- for source_file in build.sourcepackagerelease.files:
1023- file_name = source_file.libraryfile.filename
1024- sha1 = source_file.libraryfile.content.sha1
1025- source_name = build.sourcepackagerelease.sourcepackagename.name
1026- poolpath = makePoolPath(source_name, component_name)
1027- url = urlappend(archive_url, poolpath)
1028- url = urlappend(url, file_name)
1029- logger.debug("Asking builder on %s to ensure it has file %s "
1030- "(%s, %s)" % (self.url, file_name, url, sha1))
1031- self._sendFileToSlave(url, sha1, "buildd", archive.buildd_secret)
1032-
1033 def _sendFileToSlave(self, url, sha1, username="", password=""):
1034 """Helper to send the file at 'url' with 'sha1' to this builder."""
1035 if not self.builderok:
1036@@ -256,156 +276,22 @@
1037 """See IBuilder."""
1038 self.slave = proxy
1039
1040- def _verifyBuildRequest(self, build_queue_item, logger):
1041- """Assert some pre-build checks.
1042-
1043- The build request is checked:
1044- * Virtualized builds can't build on a non-virtual builder
1045- * Ensure that we have a chroot
1046- * Ensure that the build pocket allows builds for the current
1047- distroseries state.
1048- """
1049- build = getUtility(IBuildSet).getByQueueEntry(build_queue_item)
1050- assert not (not self.virtualized and build.is_virtualized), (
1051- "Attempt to build non-virtual item on a virtual builder.")
1052-
1053- # Assert that we are not silently building SECURITY jobs.
1054- # See findBuildCandidates. Once we start building SECURITY
1055- # correctly from EMBARGOED archive this assertion can be removed.
1056- # XXX Julian 2007-12-18 spec=security-in-soyuz: This is being
1057- # addressed in the work on the blueprint:
1058- # https://blueprints.launchpad.net/soyuz/+spec/security-in-soyuz
1059- target_pocket = build.pocket
1060- assert target_pocket != PackagePublishingPocket.SECURITY, (
1061- "Soyuz is not yet capable of building SECURITY uploads.")
1062-
1063- # Ensure build has the needed chroot
1064- build = getUtility(IBuildSet).getByQueueEntry(build_queue_item)
1065- chroot = build.distroarchseries.getChroot()
1066- if chroot is None:
1067- raise CannotBuild(
1068- "Missing CHROOT for %s/%s/%s" % (
1069- build.distroseries.distribution.name,
1070- build.distroseries.name,
1071- build.distroarchseries.architecturetag)
1072- )
1073-
1074- # The main distribution has policies to prevent uploads to some
1075- # pockets (e.g. security) during different parts of the distribution
1076- # series lifecycle. These do not apply to PPA builds nor any archive
1077- # that allows release pocket updates.
1078- if (build.archive.purpose != ArchivePurpose.PPA and
1079- not build.archive.allowUpdatesToReleasePocket()):
1080- # XXX Robert Collins 2007-05-26: not an explicit CannotBuild
1081- # exception yet because the callers have not been audited
1082- assert build.distroseries.canUploadToPocket(build.pocket), (
1083- "%s (%s) can not be built for pocket %s: invalid pocket due "
1084- "to the series status of %s."
1085- % (build.title, build.id, build.pocket.name,
1086- build.distroseries.name))
1087-
1088- def _dispatchBuildToSlave(self, build_queue_item, args, buildid, logger):
1089- """Start the build on the slave builder."""
1090- # Send chroot.
1091- build = getUtility(IBuildSet).getByQueueEntry(build_queue_item)
1092- chroot = build.distroarchseries.getChroot()
1093- self.cacheFileOnSlave(logger, chroot)
1094-
1095- # Build filemap structure with the files required in this build
1096- # and send them to the slave.
1097- # If the build is private we tell the slave to get the files from the
1098- # archive instead of the librarian because the slaves cannot
1099- # access the restricted librarian.
1100- private = build.archive.private
1101- if private:
1102- self.cachePrivateSourceOnSlave(logger, build_queue_item)
1103- filemap = {}
1104- for source_file in build.sourcepackagerelease.files:
1105- lfa = source_file.libraryfile
1106- filemap[lfa.filename] = lfa.content.sha1
1107- if not private:
1108- self.cacheFileOnSlave(logger, source_file.libraryfile)
1109-
1110- chroot_sha1 = chroot.content.sha1
1111- try:
1112- status, info = self.slave.build(
1113- buildid, "debian", chroot_sha1, filemap, args)
1114- message = """%s (%s):
1115- ***** RESULT *****
1116- %s
1117- %s
1118- %s: %s
1119- ******************
1120- """ % (self.name, self.url, filemap, args, status, info)
1121- logger.info(message)
1122- except xmlrpclib.Fault, info:
1123- # Mark builder as 'failed'.
1124- logger.debug("Disabling builder: %s" % self.url, exc_info=1)
1125- self.failbuilder(
1126- "Exception (%s) when setting up to new job" % info)
1127- raise BuildSlaveFailure
1128- except socket.error, info:
1129- error_message = "Exception (%s) when setting up new job" % info
1130- self.handleTimeout(logger, error_message)
1131- raise BuildSlaveFailure
1132-
1133 def startBuild(self, build_queue_item, logger):
1134 """See IBuilder."""
1135- build = getUtility(IBuildSet).getByQueueEntry(build_queue_item)
1136- spr = build.sourcepackagerelease
1137- logger.info("startBuild(%s, %s, %s, %s)", self.url,
1138- spr.name, spr.version, build.pocket.title)
1139+ # Set the build behavior depending on the provided build queue item.
1140+ self.current_build_behavior = build_queue_item.required_build_behavior
1141+ self.logStartBuild(build_queue_item, logger)
1142
1143 # Make sure the request is valid; an exception is raised if it's not.
1144- self._verifyBuildRequest(build_queue_item, logger)
1145+ self.verifyBuildRequest(build_queue_item, logger)
1146
1147 # If we are building a virtual build, resume the virtual machine.
1148 if self.virtualized:
1149 self.resumeSlaveHost()
1150
1151- # Build extra arguments.
1152- args = {}
1153- # turn 'arch_indep' ON only if build is archindep or if
1154- # the specific architecture is the nominatedarchindep for
1155- # this distroseries (in case it requires any archindep source)
1156- build = getUtility(IBuildSet).getByQueueEntry(build_queue_item)
1157- args['arch_indep'] = build.distroarchseries.isNominatedArchIndep
1158-
1159- suite = build.distroarchseries.distroseries.name
1160- if build.pocket != PackagePublishingPocket.RELEASE:
1161- suite += "-%s" % (build.pocket.name.lower())
1162- args['suite'] = suite
1163-
1164- archive_purpose = build.archive.purpose
1165- if (archive_purpose == ArchivePurpose.PPA and
1166- not build.archive.require_virtualized):
1167- # If we're building a non-virtual PPA, override the purpose
1168- # to PRIMARY and use the primary component override.
1169- # This ensures that the package mangling tools will run over
1170- # the built packages.
1171- args['archive_purpose'] = ArchivePurpose.PRIMARY.name
1172- args["ogrecomponent"] = (
1173- get_primary_current_component(build))
1174- else:
1175- args['archive_purpose'] = archive_purpose.name
1176- args["ogrecomponent"] = (
1177- build.current_component.name)
1178-
1179- args['archives'] = get_sources_list_for_building(build)
1180-
1181- # Let the build slave know whether this is a build in a private
1182- # archive.
1183- args['archive_private'] = build.archive.private
1184-
1185- # Generate a string which can be used to cross-check when obtaining
1186- # results so we know we are referring to the right database object in
1187- # subsequent runs.
1188- buildid = "%s-%s" % (build.id, build_queue_item.id)
1189- logger.debug("Initiating build %s on %s" % (buildid, self.url))
1190-
1191 # Do it.
1192 build_queue_item.markAsBuilding(self)
1193- self._dispatchBuildToSlave(build_queue_item, args, buildid, logger)
1194+ self.dispatchBuildToSlave(build_queue_item, logger)
1195
1196 # XXX cprov 2009-06-24: This code does not belong to the content
1197 # class domain. Here we cannot make sensible decisions about what
1198@@ -418,25 +304,20 @@
1199 if self.failnotes is not None:
1200 return self.failnotes
1201 return 'Disabled'
1202- # Cache the 'currentjob', so we don't have to hit the database
1203- # more than once.
1204- currentjob = self.currentjob
1205- if currentjob is None:
1206- return 'Idle'
1207
1208- build = getUtility(IBuildSet).getByQueueEntry(currentjob)
1209- msg = 'Building %s' % build.title
1210- archive = build.archive
1211- if not archive.owner.private and (archive.is_ppa or archive.is_copy):
1212- return '%s [%s/%s]' % (msg, archive.owner.name, archive.name)
1213- else:
1214- return msg
1215+ # If the builder is OK then we delegate the status
1216+ # to our current behavior.
1217+ return self.current_build_behavior.status
1218
1219 def failbuilder(self, reason):
1220 """See IBuilder"""
1221 self.builderok = False
1222 self.failnotes = reason
1223
1224+ # XXX Michael Nelson 20091202 bug=491330. The current UI assumes
1225+ # that the builder history will display binary build records, as
1226+ # returned by getBuildRecords() below. See the bug for a discussion
1227+ # of the options.
1228 def getBuildRecords(self, build_state=None, name=None, arch_tag=None,
1229 user=None):
1230 """See IHasBuildRecords."""
1231@@ -447,29 +328,11 @@
1232 """See IBuilder."""
1233 builder_version, builder_arch, mechanisms = self.slave.info()
1234 status_sentence = self.slave.status()
1235- builder_status = status_sentence[0]
1236-
1237- if builder_status == 'BuilderStatus.WAITING':
1238- (build_status, build_id) = status_sentence[1:3]
1239- build_status_with_files = [
1240- 'BuildStatus.OK',
1241- 'BuildStatus.PACKAGEFAIL',
1242- 'BuildStatus.DEPFAIL',
1243- ]
1244- if build_status in build_status_with_files:
1245- (filemap, dependencies) = status_sentence[3:]
1246- else:
1247- filemap = dependencies = None
1248- logtail = None
1249- elif builder_status == 'BuilderStatus.BUILDING':
1250- (build_id, logtail) = status_sentence[1:]
1251- build_status = filemap = dependencies = None
1252- else:
1253- build_id = status_sentence[1]
1254- build_status = logtail = filemap = dependencies = None
1255-
1256- return (builder_status, build_id, build_status, logtail, filemap,
1257- dependencies)
1258+
1259+ status = {'builder_status': status_sentence[0]}
1260+ status.update(
1261+ self.current_build_behavior.slaveStatus(status_sentence))
1262+ return status
1263
1264 def slaveStatusSentence(self):
1265 """See IBuilder."""
1266@@ -671,7 +534,7 @@
1267 logger = self._getSlaveScannerLogger()
1268 try:
1269 self.startBuild(candidate, logger)
1270- except (BuildSlaveFailure, CannotBuild), err:
1271+ except (BuildSlaveFailure, CannotBuild, BuildBehaviorMismatch), err:
1272 logger.warn('Could not build: %s' % err)
1273
1274 def handleTimeout(self, logger, error_message):
1275
1276=== modified file 'lib/lp/soyuz/model/buildqueue.py'
1277--- lib/lp/soyuz/model/buildqueue.py 2009-11-20 18:06:28 +0000
1278+++ lib/lp/soyuz/model/buildqueue.py 2009-12-08 20:16:13 +0000
1279@@ -24,6 +24,8 @@
1280 from canonical.database.sqlbase import SQLBase, sqlvalues
1281 from canonical.launchpad.webapp.interfaces import NotFoundError
1282 from lp.buildmaster.interfaces.buildfarmjob import BuildFarmJobType
1283+from lp.buildmaster.interfaces.buildfarmjobbehavior import (
1284+ IBuildFarmJobBehavior)
1285 from lp.services.job.interfaces.job import JobStatus
1286 from lp.services.job.model.job import Job
1287 from lp.soyuz.interfaces.build import BuildStatus, IBuildSet
1288@@ -49,6 +51,11 @@
1289 estimated_duration = IntervalCol()
1290
1291 @property
1292+ def required_build_behavior(self):
1293+ """See `IBuildQueue`."""
1294+ return IBuildFarmJobBehavior(self.specific_job)
1295+
1296+ @property
1297 def specific_job(self):
1298 """See `IBuildQueue`."""
1299 store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
1300
1301=== modified file 'lib/lp/soyuz/tests/test_builder.py'
1302--- lib/lp/soyuz/tests/test_builder.py 2009-11-13 16:37:05 +0000
1303+++ lib/lp/soyuz/tests/test_builder.py 2009-12-08 20:16:13 +0000
1304@@ -8,10 +8,15 @@
1305 from zope.component import getUtility
1306
1307 from canonical.testing import LaunchpadZopelessLayer
1308+from lp.buildmaster.interfaces.buildfarmjobbehavior import (
1309+ BuildBehaviorMismatch, IBuildFarmJobBehavior)
1310+from lp.buildmaster.model.buildfarmjobbehavior import IdleBuildBehavior
1311 from lp.soyuz.interfaces.archive import ArchivePurpose
1312 from lp.soyuz.interfaces.build import BuildStatus, IBuildSet
1313 from lp.soyuz.interfaces.builder import IBuilderSet
1314 from lp.soyuz.interfaces.publishing import PackagePublishingStatus
1315+from lp.soyuz.model.binarypackagebuildbehavior import (
1316+ BinaryPackageBuildBehavior)
1317 from lp.soyuz.tests.test_publishing import SoyuzTestPublisher
1318 from lp.testing import TestCaseWithFactory
1319
1320@@ -214,5 +219,54 @@
1321 self.failUnlessEqual('primary', build.archive.name)
1322 self.failUnlessEqual('firefox', build.sourcepackagerelease.name)
1323
1324+
1325+class TestCurrentBuildBehavior(TestCaseWithFactory):
1326+ """This test ensures the get/set behavior of IBuilder's
1327+ current_build_behavior property.
1328+ """
1329+
1330+ layer = LaunchpadZopelessLayer
1331+
1332+ def setUp(self):
1333+ """Create a new builder ready for testing."""
1334+ super(TestCurrentBuildBehavior, self).setUp()
1335+ self.builder = self.factory.makeBuilder(name='builder')
1336+
1337+ # Have a publisher and a ppa handy for some of the tests below.
1338+ self.publisher = SoyuzTestPublisher()
1339+ self.publisher.prepareBreezyAutotest()
1340+ self.ppa_joe = self.factory.makeArchive(name="joesppa")
1341+
1342+ self.build = self.publisher.getPubSource(
1343+ sourcename="gedit", status=PackagePublishingStatus.PUBLISHED,
1344+ archive=self.ppa_joe).createMissingBuilds()[0]
1345+
1346+ self.buildfarmjob = self.build.buildqueue_record.specific_job
1347+
1348+ def test_idle_behavior_when_no_current_build(self):
1349+ """We return an idle behavior when there is no behavior specified
1350+ nor a current build.
1351+ """
1352+ self.assertIsInstance(
1353+ self.builder.current_build_behavior, IdleBuildBehavior)
1354+
1355+ def test_set_behavior_sets_builder(self):
1356+ """Setting a builder's behavior also associates the behavior with the
1357+ builder."""
1358+ behavior = IBuildFarmJobBehavior(self.buildfarmjob)
1359+ self.builder.current_build_behavior = behavior
1360+
1361+ self.assertEqual(behavior, self.builder.current_build_behavior)
1362+ self.assertEqual(behavior._builder, self.builder)
1363+
1364+ def test_current_job_behavior(self):
1365+ """The current behavior is set automatically from the current job."""
1366+ # Set the builder attribute on the buildqueue record so that our
1367+ # builder will think it has a current build.
1368+ self.build.buildqueue_record.builder = self.builder
1369+
1370+ self.assertIsInstance(
1371+ self.builder.current_build_behavior, BinaryPackageBuildBehavior)
1372+
1373 def test_suite():
1374 return unittest.TestLoader().loadTestsFromName(__name__)

Subscribers

People subscribed via source and target branches

to status/vote changes: