Launchpad itself

Merge lp:~adeuring/launchpad/bug-494075 into lp:launchpad/db-devel

  1. bug-494075
  2. Merge into db-devel
Proposed by Abel Deuring
Status: Merged
Merged at revision: not available
Proposed branch: lp:~adeuring/launchpad/bug-494075
Merge into: lp:launchpad/db-devel
Diff against target: 1374 lines (+784/-245)
17 files modified
lib/canonical/launchpad/webapp/adapter.py (+13/-1)
lib/canonical/launchpad/webapp/ftests/test_adapter.txt (+8/-2)
lib/lp/buildmaster/buildergroup.py (+8/-3)
lib/lp/buildmaster/interfaces/buildfarmjobbehavior.py (+61/-0)
lib/lp/buildmaster/model/buildfarmjobbehavior.py (+71/-0)
lib/lp/buildmaster/tests/buildfarmjobbehavior.txt (+134/-0)
lib/lp/soyuz/browser/tests/builder-views.txt (+1/-0)
lib/lp/soyuz/configure.zcml (+7/-0)
lib/lp/soyuz/doc/buildd-slavescanner.txt (+48/-12)
lib/lp/soyuz/doc/builder.txt (+12/-0)
lib/lp/soyuz/doc/buildqueue.txt (+10/-0)
lib/lp/soyuz/interfaces/builder.py (+10/-25)
lib/lp/soyuz/interfaces/buildqueue.py (+5/-1)
lib/lp/soyuz/model/binarypackagebuildbehavior.py (+271/-0)
lib/lp/soyuz/model/builder.py (+64/-201)
lib/lp/soyuz/model/buildqueue.py (+7/-0)
lib/lp/soyuz/tests/test_builder.py (+54/-0)
To merge this branch: bzr merge lp:~adeuring/launchpad/bug-494075
Reviewer Review Type Date Requested Status
Brad Crittenden (community) code Approve
Review via email: mp+15823@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Abel Deuring (adeuring) wrote :

When psycopg QueryCancelError occurs, we see only the SQL statement leading to the timeout and its parameters, but not the details of the QueryCancelError. This makes it hard to understand what heppend for example in OOPS-1438B2246.

This branch adds the original QueryCancelError message to the data reported for TimeoutErrors.

test: ./bin/test -vv -t test_adapter.txt

= Launchpad lint =

Checking for conflicts. and issues in doctests and templates.
Running jslint, xmllint, pyflakes, and pylint.
Using normal rules.

Linting changed files:
  lib/canonical/launchpad/webapp/adapter.py
  lib/canonical/launchpad/webapp/ftests/test_adapter.txt
abel@klato4:~/canonical/lp-branches/bug-494075$ make lint
= Launchpad lint =

Checking for conflicts. and issues in doctests and templates.
Running jslint, xmllint, pyflakes, and pylint.
Using normal rules.

Linting changed files:
  lib/canonical/launchpad/webapp/adapter.py
  lib/canonical/launchpad/webapp/ftests/test_adapter.txt

Revision history for this message
Brad Crittenden (bac) wrote :

Hi Abel,

Thanks for this fix. Anything to make chasing down OOPs easier is great.

In your actual diff at http://paste.ubuntu.com/337502/ you need to fix the typos in the docstring:

""A variant o TimeoutError hat reports the original PostgreSQL error."""

Looks like you were thinking faster than your fingers would go.

Other than that it's fine.

review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/canonical/launchpad/webapp/adapter.py'
--- lib/canonical/launchpad/webapp/adapter.py 2009-08-06 15:45:18 +0000
+++ lib/canonical/launchpad/webapp/adapter.py 2009-12-08 20:16:13 +0000
@@ -62,6 +62,18 @@
62classImplements(TimeoutError, IRequestExpired)62classImplements(TimeoutError, IRequestExpired)
6363
6464
65class LaunchpadTimeoutError(TimeoutError):
66 """A variant of TimeoutError that reports the original PostgreSQL error.
67 """
68
69 def __init__(self, statement, params, original_error):
70 super(LaunchpadTimeoutError, self).__init__(statement, params)
71 self.original_error = original_error
72
73 def __str__(self):
74 return ('Statement: %r\nParameters:%r\nOriginal error: %r'
75 % (self.statement, self.params, self.original_error))
76
65def _get_dirty_commit_flags():77def _get_dirty_commit_flags():
66 """Return the current dirty commit status"""78 """Return the current dirty commit status"""
67 from canonical.ftests.pgsql import ConnectionWrapper79 from canonical.ftests.pgsql import ConnectionWrapper
@@ -420,7 +432,7 @@
420 return432 return
421 if isinstance(error, QueryCanceledError):433 if isinstance(error, QueryCanceledError):
422 OpStats.stats['timeouts'] += 1434 OpStats.stats['timeouts'] += 1
423 raise TimeoutError(statement, params)435 raise LaunchpadTimeoutError(statement, params, error)
424436
425 def get_remaining_time(self):437 def get_remaining_time(self):
426 """See `TimeoutTracer`"""438 """See `TimeoutTracer`"""
427439
=== modified file 'lib/canonical/launchpad/webapp/ftests/test_adapter.txt'
--- lib/canonical/launchpad/webapp/ftests/test_adapter.txt 2009-08-06 15:45:18 +0000
+++ lib/canonical/launchpad/webapp/ftests/test_adapter.txt 2009-12-08 20:16:13 +0000
@@ -115,7 +115,10 @@
115 >>> store.execute('SELECT pg_sleep(0.200)', noresult=True)115 >>> store.execute('SELECT pg_sleep(0.200)', noresult=True)
116 Traceback (most recent call last):116 Traceback (most recent call last):
117 ...117 ...
118 TimeoutError: ...SELECT pg_sleep(0.200)...118 LaunchpadTimeoutError: Statement: 'SELECT pg_sleep(0.200)'
119 Parameters:()
120 Original error: QueryCanceledError('canceling statement due to
121 statement timeout\n',)
119122
120Even though the statement timed out, it is recorded in the statement log:123Even though the statement timed out, it is recorded in the statement log:
121124
@@ -175,7 +178,10 @@
175 >>> store.execute('SELECT pg_sleep(0.2)', noresult=True)178 >>> store.execute('SELECT pg_sleep(0.2)', noresult=True)
176 Traceback (most recent call last):179 Traceback (most recent call last):
177 ...180 ...
178 TimeoutError: ...SELECT pg_sleep(0.2)...181 LaunchpadTimeoutError: Statement: 'SELECT pg_sleep(0.2)'
182 Parameters:()
183 Original error: QueryCanceledError('canceling statement due to
184 statement timeout\n',)
179 >>> clear_request_started()185 >>> clear_request_started()
180186
181187
182188
=== modified file 'lib/lp/buildmaster/buildergroup.py'
--- lib/lp/buildmaster/buildergroup.py 2009-11-13 19:34:17 +0000
+++ lib/lp/buildmaster/buildergroup.py 2009-12-08 20:16:13 +0000
@@ -171,8 +171,8 @@
171 Perform the required actions for each state.171 Perform the required actions for each state.
172 """172 """
173 try:173 try:
174 (builder_status, build_id, build_status, logtail, filemap,174 slave_status = queueItem.builder.slaveStatus()
175 dependencies) = queueItem.builder.slaveStatus()175
176 except (xmlrpclib.Fault, socket.error), info:176 except (xmlrpclib.Fault, socket.error), info:
177 # XXX cprov 2005-06-29:177 # XXX cprov 2005-06-29:
178 # Hmm, a problem with the xmlrpc interface,178 # Hmm, a problem with the xmlrpc interface,
@@ -192,6 +192,7 @@
192 'BuilderStatus.WAITING': self.updateBuild_WAITING,192 'BuilderStatus.WAITING': self.updateBuild_WAITING,
193 }193 }
194194
195 builder_status = slave_status['builder_status']
195 if builder_status not in builder_status_handlers:196 if builder_status not in builder_status_handlers:
196 self.logger.critical(197 self.logger.critical(
197 "Builder on %s returned unknown status %s, failing it"198 "Builder on %s returned unknown status %s, failing it"
@@ -209,7 +210,11 @@
209 # from the IBuilder content class, it arrives protected by a Zope210 # from the IBuilder content class, it arrives protected by a Zope
210 # Security Proxy, which is not declared, thus empty. Before passing211 # Security Proxy, which is not declared, thus empty. Before passing
211 # it to the status handlers we will simply remove the proxy.212 # it to the status handlers we will simply remove the proxy.
212 logtail = removeSecurityProxy(logtail)213 logtail = removeSecurityProxy(slave_status.get('logtail'))
214 build_id = slave_status.get('build_id')
215 build_status = slave_status.get('build_status')
216 filemap = slave_status.get('filemap')
217 dependencies = slave_status.get('dependencies')
213218
214 method = builder_status_handlers[builder_status]219 method = builder_status_handlers[builder_status]
215 try:220 try:
216221
=== added file 'lib/lp/buildmaster/interfaces/buildfarmjobbehavior.py'
--- lib/lp/buildmaster/interfaces/buildfarmjobbehavior.py 1970-01-01 00:00:00 +0000
+++ lib/lp/buildmaster/interfaces/buildfarmjobbehavior.py 2009-12-08 20:16:13 +0000
@@ -0,0 +1,61 @@
1# Copyright 2009 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4# pylint: disable-msg=E0211,E0213
5
6"""Interface for build farm job behaviors."""
7
8__metaclass__ = type
9
10__all__ = [
11 'IBuildFarmJobBehavior',
12 ]
13
14from zope.interface import Attribute, Interface
15
16
17class BuildBehaviorMismatch(Exception):
18 """
19 A general exception that can be raised when the builder's current behavior
20 does not match the expected behavior.
21 """
22
23
24class IBuildFarmJobBehavior(Interface):
25
26 status = Attribute(
27 "Generated status information for this particular job.")
28
29 def setBuilder(builder):
30 """Sets the associated builder reference for this instance."""
31
32 def logStartBuild(build_queue_item, logger):
33 """Log the start of a specific build queue item.
34
35 The form of the log message will vary depending on the type of build.
36 :param build_queue_item: A BuildQueueItem to build.
37 :param logger: A logger to be used to log diagnostic information.
38 """
39
40 def dispatchBuildToSlave(build_queue_item, logger):
41 """Dispatch a specific build to the slave.
42
43 :param build_queue_item: The `BuildQueueItem` that will be built.
44 :logger: A logger to be used to log diagnostic information.
45 """
46
47 def verifyBuildRequest(build_queue_item, logger):
48 """Carry out any pre-build checks.
49
50 :param build_queue_item: The `BuildQueueItem` that is to be built.
51 :logger: A logger to be used to log diagnostic information.
52 """
53
54 def slaveStatus(self, raw_slave_status):
55 """Return a dict of custom slave status values for this behavior.
56
57 :param raw_slave_status: The value returned by the build slave's
58 status() method.
59 :return: a dict of extra key/values to be included in the result
60 of IBuilder.slaveStatus().
61 """
062
=== added file 'lib/lp/buildmaster/model/buildfarmjobbehavior.py'
--- lib/lp/buildmaster/model/buildfarmjobbehavior.py 1970-01-01 00:00:00 +0000
+++ lib/lp/buildmaster/model/buildfarmjobbehavior.py 2009-12-08 20:16:13 +0000
@@ -0,0 +1,71 @@
1# Copyright 2009 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4# pylint: disable-msg=E0211,E0213
5
6"""Base and idle BuildFarmJobBehavior classes."""
7
8__metaclass__ = type
9
10__all__ = [
11 'BuildFarmJobBehaviorBase',
12 'IdleBuildBehavior'
13 ]
14
15from zope.interface import implements
16
17from lp.buildmaster.interfaces.buildfarmjobbehavior import (
18 BuildBehaviorMismatch, IBuildFarmJobBehavior)
19
20
21class BuildFarmJobBehaviorBase:
22 """Ensures that all behaviors inherit the same initialisation.
23
24 All build-farm job behaviors should inherit from this.
25 """
26
27 def __init__(self, buildfarmjob):
28 """Store a reference to the job_type with which we were created."""
29 self.buildfarmjob = buildfarmjob
30 self._builder = None
31
32 def setBuilder(self, builder):
33 """The builder should be set once and not changed."""
34 self._builder = builder
35
36 def verifyBuildRequest(self, build_queue_item, logger):
37 """The default behavior is a no-op."""
38 pass
39
40 def slaveStatus(self, raw_slave_status):
41 """See `IBuildFarmJobBehavior`.
42
43 The default behavior is that we don't add any extra values."""
44 return {}
45
46
47class IdleBuildBehavior(BuildFarmJobBehaviorBase):
48
49 implements(IBuildFarmJobBehavior)
50
51 def __init__(self):
52 """The idle behavior is special in that a buildfarmjob is not
53 specified during initialisation as it is not the result of an
54 adaption.
55 """
56 super(IdleBuildBehavior, self).__init__(None)
57
58 def logStartBuild(self, build_queue_item, logger):
59 """See `IBuildFarmJobBehavior`."""
60 raise BuildBehaviorMismatch(
61 "Builder was idle when asked to log the start of a build.")
62
63 def dispatchBuildToSlave(self, build_queue_item, logger):
64 """See `IBuildFarmJobBehavior`."""
65 raise BuildBehaviorMismatch(
66 "Builder was idle when asked to dispatch a build to the slave.")
67
68 @property
69 def status(self):
70 """See `IBuildFarmJobBehavior`."""
71 return "Idle"
072
=== added file 'lib/lp/buildmaster/tests/buildfarmjobbehavior.txt'
--- lib/lp/buildmaster/tests/buildfarmjobbehavior.txt 1970-01-01 00:00:00 +0000
+++ lib/lp/buildmaster/tests/buildfarmjobbehavior.txt 2009-12-08 20:16:13 +0000
@@ -0,0 +1,134 @@
1BuildFarmJobBehavior
2====================
3
4The Launchpad build farm was originally designed for building binary
5packages from source packages, but was subsequently generalised to support
6other types of build farm jobs.
7
8The `BuildFarmJobBehavior` class encapsulates job-type-specific behavior
9with a standard interface to which our generic IBuilder class delegates.
10The result is that neither our generic IBuilder class or any call-sites
11(such as the build master) need any knowledge of different job types or
12how to handle them.
13
14
15Creating a new behavior
16-----------------------
17
18A new behavior should implement the `IBuildFarmJobBehavior` interface
19and extend BuildFarmJobBehaviorBase. A new behavior will only be required
20to define one method - dispatchBuildToSlave() - to correctly implement
21the interface, but will usually want to customise the other properties and
22methods as well.
23
24 >>> from lp.buildmaster.interfaces.buildfarmjobbehavior import (
25 ... IBuildFarmJobBehavior)
26 >>> from lp.buildmaster.model.buildfarmjobbehavior import (
27 ... BuildFarmJobBehaviorBase)
28 >>> from zope.interface import implements
29
30 >>> class MyNewBuildBehavior(BuildFarmJobBehaviorBase):
31 ... """A custom build behavior for building blah."""
32 ... implements(IBuildFarmJobBehavior)
33 ...
34 ... def dispatchBuildToSlave(self, build_queue_item, logger):
35 ... print "Did something special to dispatch MySpecialBuild."
36 ...
37 ... @property
38 ... def status(self):
39 ... return "Currently building a MyNewBuild object."
40
41For this documentation, we'll also need a dummy new build farm job.
42
43 >>> from lp.buildmaster.interfaces.buildfarmjob import IBuildFarmJob
44 >>> class IMyNewBuildFarmJob(IBuildFarmJob):
45 ... "Normally defines job-type specific database fields."""
46 >>> class MyNewBuildFarmJob(object):
47 ... implements(IMyNewBuildFarmJob)
48
49Custom behaviors are not normally instantiated directly, instead an adapter is
50specified for the specific IBuildFarmJob. Normaly we'd add some ZCML to
51adapt our specific build farm job to its behavior like:
52
53 <!-- MyNewBuildBehavior -->
54 <adapter
55 for="lp.myapp.interfaces.mynewbuildfarmjob.IMyNewBuildFarmJob"
56 provides="lp.buildmaster.interfaces.buildfarmjobbehavior.IBuildFarmJobBehavior"
57 factory="lp.myapp.model.mynewbuildbehavior.MyNewBuildBehavior"
58 permission="zope.Public" />
59
60But for the sake of this documentation we'll add the adapter manually.
61
62 >>> from zope.app.testing import ztapi
63 >>> ztapi.provideAdapter(
64 ... MyNewBuildFarmJob, IBuildFarmJobBehavior, MyNewBuildBehavior)
65
66This will then allow the builder to request and set the required behavior from
67the current job. Bob the builder currently has a binary package job and so
68finds itself with a binary package build behavior which defines the status
69attribute with some binary-build specific information.
70
71 >>> from lp.soyuz.model.builder import Builder
72 >>> from canonical.launchpad.webapp.interfaces import (
73 ... IStoreSelector, MAIN_STORE, DEFAULT_FLAVOR)
74 >>> store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
75 >>> bob = store.find(Builder, Builder.name == 'bob').one()
76 >>> print bob.status
77 Building i386 build of mozilla-firefox 0.9 in ubuntu hoary RELEASE
78
79XXX Michael Nelson 2009-12-04 bug 484819. At the time of writing the
80BuildQueue.specific_job method has not been updated to support different
81job types, so we will set the behavior explicitly here.
82
83 >>> bob.current_build_behavior = IBuildFarmJobBehavior(
84 ... MyNewBuildFarmJob())
85
86Once the builder has the relevant behavior, it is able to provide both general
87builder functionality of its own accord, while delegating any build-type
88specific functionality to the behavior. For example, if the builder is not
89disabled, the builder delegates the status property to the behavior.
90
91 >>> print bob.status
92 Currently building a MyNewBuild object.
93
94On the other hand, if the builder is disabled, the builder itself determines
95the status.
96
97 >>> bob.builderok = False
98 >>> print bob.status
99 Disabled
100 >>> bob.builderok = True
101
102The IBuildFarmJobBehavior interface currently provides customisation points
103throughout the build life-cycle, from logging the start of a build, verifying
104that the provided queue item is ready to be built, dispatching the build etc.,
105and allows further customisation to be added easily.
106
107Please refer to the IBuildFarmJobBehavior interface to see the currently
108provided build-type specific customisation points.
109
110
111The IdleBuildBehavior
112---------------------
113
114If a Builder does not have a currentjob (and therefore an appropriate
115build behavior) it will automatically get the special IdleBuildBehavior
116to ensure that it fulfills its contract to implement IBuildFarmJobBehavior.
117
118First, we'll reset the current build behavior and destroy the current job.
119
120 >>> bob.current_build_behavior = None
121 >>> bob.currentjob.destroySelf()
122
123 >>> print bob.status
124 Idle
125
126Attempting to use any other build-related functionality when a builder is
127idle, such as making a call to log the start of a build, will raise an
128appropriate exception.
129
130 >>> bob.logStartBuild(None, None)
131 Traceback (most recent call last):
132 ...
133 BuildBehaviorMismatch: Builder was idle when asked to log the start of a
134 build.
0135
=== modified file 'lib/lp/soyuz/browser/tests/builder-views.txt'
--- lib/lp/soyuz/browser/tests/builder-views.txt 2009-11-24 07:02:21 +0000
+++ lib/lp/soyuz/browser/tests/builder-views.txt 2009-12-08 20:16:13 +0000
@@ -228,6 +228,7 @@
228 >>> login("foo.bar@canonical.com")228 >>> login("foo.bar@canonical.com")
229 >>> private_job = store.get(BuildQueue, private_job_id)229 >>> private_job = store.get(BuildQueue, private_job_id)
230 >>> Store.of(private_job).remove(private_job)230 >>> Store.of(private_job).remove(private_job)
231 >>> frog.current_build_behavior = None
231232
232 >>> login('no-priv@canonical.com')233 >>> login('no-priv@canonical.com')
233 >>> nopriv_view = getMultiAdapter((frog, empty_request), name="+index")234 >>> nopriv_view = getMultiAdapter((frog, empty_request), name="+index")
234235
=== modified file 'lib/lp/soyuz/configure.zcml'
--- lib/lp/soyuz/configure.zcml 2009-11-16 22:06:14 +0000
+++ lib/lp/soyuz/configure.zcml 2009-12-08 20:16:13 +0000
@@ -892,4 +892,11 @@
892 interface="lp.soyuz.interfaces.buildpackagejob.IBuildPackageJob"/>892 interface="lp.soyuz.interfaces.buildpackagejob.IBuildPackageJob"/>
893 </class>893 </class>
894894
895 <!-- BinaryPackageBuildBehavior -->
896 <adapter
897 for="lp.soyuz.interfaces.buildpackagejob.IBuildPackageJob"
898 provides="lp.buildmaster.interfaces.buildfarmjobbehavior.IBuildFarmJobBehavior"
899 factory="lp.soyuz.model.binarypackagebuildbehavior.BinaryPackageBuildBehavior"
900 permission="zope.Public" />
901
895</configure>902</configure>
896903
=== modified file 'lib/lp/soyuz/doc/buildd-slavescanner.txt'
--- lib/lp/soyuz/doc/buildd-slavescanner.txt 2009-11-13 19:34:17 +0000
+++ lib/lp/soyuz/doc/buildd-slavescanner.txt 2009-12-08 20:16:13 +0000
@@ -1494,8 +1494,8 @@
14941494
1495== Builder Status Handler ==1495== Builder Status Handler ==
14961496
1497IBuilder.slaveStatus should return a size homogeneous tuple despite of1497IBuilder.slaveStatus should return a dict containing the following
1498the current slave state. This tuple should contain, in this order:1498items:
14991499
1500 * slave status string: 'BuilderStatus.IDLE'1500 * slave status string: 'BuilderStatus.IDLE'
1501 * job identifier string: '1-1'1501 * job identifier string: '1-1'
@@ -1504,24 +1504,60 @@
1504 * result file list: {'foo.deb', 'foo.changes'} or None1504 * result file list: {'foo.deb', 'foo.changes'} or None
1505 * dependencies string: 'bar baz zaz' or None1505 * dependencies string: 'bar baz zaz' or None
15061506
1507 # Define a helper to print the slave status dict.
1508 >>> from collections import defaultdict
1509 >>> def printSlaveStatus(status_dict):
1510 ... status_dict = defaultdict(lambda:None, status_dict)
1511 ... print (
1512 ... "builder_status: %(builder_status)s\n"
1513 ... "build_id: %(build_id)s\n"
1514 ... "build_status: %(build_status)s\n"
1515 ... "logtail: %(logtail)r\n"
1516 ... "filemap: %(filemap)s\n"
1517 ... "dependencies: %(dependencies)s\n" % status_dict)
15071518
1508 >>> a_builder.setSlaveForTesting(OkSlave())1519 >>> a_builder.setSlaveForTesting(OkSlave())
1509 >>> a_builder.slaveStatus()1520 >>> printSlaveStatus(a_builder.slaveStatus())
1510 ('BuilderStatus.IDLE', '', None, None, None, None)1521 builder_status: BuilderStatus.IDLE
1522 build_id:
1523 build_status: None
1524 logtail: None
1525 filemap: None
1526 dependencies: None
15111527
1512 >>> a_builder.setSlaveForTesting(BuildingSlave())1528 >>> a_builder.setSlaveForTesting(BuildingSlave())
1513 >>> a_builder.slaveStatus()1529 >>> printSlaveStatus(a_builder.slaveStatus())
1514 ('BuilderStatus.BUILDING', '1-1', None, <xmlrpclib.Binary ...>, None, None)1530 builder_status: BuilderStatus.BUILDING
1531 build_id: 1-1
1532 build_status: None
1533 logtail: <xmlrpclib.Binary ...>
1534 filemap: None
1535 dependencies: None
15151536
1516 >>> a_builder.setSlaveForTesting(WaitingSlave(state='BuildStatus.OK'))1537 >>> a_builder.setSlaveForTesting(WaitingSlave(state='BuildStatus.OK'))
1517 >>> a_builder.slaveStatus()1538 >>> printSlaveStatus(a_builder.slaveStatus())
1518 ('BuilderStatus.WAITING', '1-1', 'BuildStatus.OK', None, {}, None)1539 builder_status: BuilderStatus.WAITING
1540 build_id: 1-1
1541 build_status: BuildStatus.OK
1542 logtail: None
1543 filemap: {}
1544 dependencies: None
15191545
1520 >>> a_builder.setSlaveForTesting(AbortingSlave())1546 >>> a_builder.setSlaveForTesting(AbortingSlave())
1521 >>> a_builder.slaveStatus()1547 >>> printSlaveStatus(a_builder.slaveStatus())
1522 ('BuilderStatus.ABORTING', '1-1', None, None, None, None)1548 builder_status: BuilderStatus.ABORTING
1549 build_id: 1-1
1550 build_status: None
1551 logtail: None
1552 filemap: None
1553 dependencies: None
15231554
1524 >>> a_builder.setSlaveForTesting(AbortedSlave())1555 >>> a_builder.setSlaveForTesting(AbortedSlave())
1525 >>> a_builder.slaveStatus()1556 >>> printSlaveStatus(a_builder.slaveStatus())
1526 ('BuilderStatus.ABORTED', '1-1', None, None, None, None)1557 builder_status: BuilderStatus.ABORTED
1558 build_id: 1-1
1559 build_status: None
1560 logtail: None
1561 filemap: None
1562 dependencies: None
15271563
15281564
=== modified file 'lib/lp/soyuz/doc/builder.txt'
--- lib/lp/soyuz/doc/builder.txt 2009-11-12 12:52:23 +0000
+++ lib/lp/soyuz/doc/builder.txt 2009-12-08 20:16:13 +0000
@@ -40,6 +40,18 @@
40 >>> builder.builderok = True40 >>> builder.builderok = True
41 >>> builder.failnotes = None41 >>> builder.failnotes = None
4242
43A builder provides a current_build_behavior attribute to which all the
44build-type specific behavior for the current build is delegated. For
45our sample data, the behavior is specifically a behavior for dealing
46with binary packages
47
48 >>> from zope.security.proxy import isinstance
49 >>> from lp.soyuz.model.binarypackagebuildbehavior import (
50 ... BinaryPackageBuildBehavior)
51 >>> isinstance(
52 ... builder.current_build_behavior, BinaryPackageBuildBehavior)
53 True
54
43In case of copy archives the status string will show both the copy55In case of copy archives the status string will show both the copy
44archive owner as well as the copy archive name.56archive owner as well as the copy archive name.
4557
4658
=== modified file 'lib/lp/soyuz/doc/buildqueue.txt'
--- lib/lp/soyuz/doc/buildqueue.txt 2009-11-13 19:34:17 +0000
+++ lib/lp/soyuz/doc/buildqueue.txt 2009-12-08 20:16:13 +0000
@@ -98,6 +98,16 @@
98 (True, 1000)98 (True, 1000)
9999
100100
101The BuildQueue item is responsible for providing the required build behavior
102for the item.
103
104 >>> from zope.security.proxy import isinstance
105 >>> from lp.soyuz.model.binarypackagebuildbehavior import (
106 ... BinaryPackageBuildBehavior)
107 >>> isinstance(bq.required_build_behavior, BinaryPackageBuildBehavior)
108 True
109
110
101== Dispatching and Reseting jobs ==111== Dispatching and Reseting jobs ==
102112
103The sampledata contains an active job, being built by the 'bob'113The sampledata contains an active job, being built by the 'bob'
104114
=== modified file 'lib/lp/soyuz/interfaces/builder.py'
--- lib/lp/soyuz/interfaces/builder.py 2009-11-20 18:06:28 +0000
+++ lib/lp/soyuz/interfaces/builder.py 2009-12-08 20:16:13 +0000
@@ -19,11 +19,13 @@
19 ]19 ]
2020
21from zope.interface import Interface, Attribute21from zope.interface import Interface, Attribute
22from zope.schema import Choice, TextLine, Text, Bool22from zope.schema import Bool, Choice, Field, Text, TextLine
2323
24from canonical.launchpad import _24from canonical.launchpad import _
25from canonical.launchpad.fields import Title, Description25from canonical.launchpad.fields import Title, Description
26from lp.registry.interfaces.role import IHasOwner26from lp.registry.interfaces.role import IHasOwner
27from lp.buildmaster.interfaces.buildfarmjobbehavior import (
28 IBuildFarmJobBehavior)
27from canonical.launchpad.validators.name import name_validator29from canonical.launchpad.validators.name import name_validator
28from canonical.launchpad.validators.url import builder_url_validator30from canonical.launchpad.validators.url import builder_url_validator
2931
@@ -57,7 +59,7 @@
57 """The build slave has suffered an error and cannot be used."""59 """The build slave has suffered an error and cannot be used."""
5860
5961
60class IBuilder(IHasOwner):62class IBuilder(IHasOwner, IBuildFarmJobBehavior):
61 """Build-slave information and state.63 """Build-slave information and state.
6264
63 Builder instance represents a single builder slave machine within the65 Builder instance represents a single builder slave machine within the
@@ -138,6 +140,10 @@
138 "new jobs. "),140 "new jobs. "),
139 required=False)141 required=False)
140142
143 current_build_behavior = Field(
144 title=u"The current behavior of the builder for the current job.",
145 required=False)
146
141 def cacheFileOnSlave(logger, libraryfilealias):147 def cacheFileOnSlave(logger, libraryfilealias):
142 """Ask the slave to cache a librarian file to its local disk.148 """Ask the slave to cache a librarian file to its local disk.
143149
@@ -148,19 +154,6 @@
148 file.154 file.
149 """155 """
150156
151 def cachePrivateSourceOnSlave(logger, build_queue_item):
152 """Ask the slave to download source files for a private build.
153
154 The slave will cache the files for the source in build_queue_item
155 to its local disk in preparation for a private build. Private builds
156 will always take the source files from the archive rather than the
157 librarian since the archive has more granular access to each
158 archive's files.
159
160 :param logger: A logger used for providing debug information.
161 :param build_queue_item: The `IBuildQueue` being built.
162 """
163
164 def checkCanBuildForDistroArchSeries(distro_arch_series):157 def checkCanBuildForDistroArchSeries(distro_arch_series):
165 """Check that the slave can compile for the given distro_arch_release.158 """Check that the slave can compile for the given distro_arch_release.
166159
@@ -215,16 +208,8 @@
215 def slaveStatus():208 def slaveStatus():
216 """Get the slave status for this builder.209 """Get the slave status for this builder.
217210
218 * builder_status => string211 :return: a dict containing at least builder_status, but potentially
219 * build_id => string212 other values included by the current build behavior.
220 * build_status => string or None
221 * logtail => string or None
222 * filename => dictionary or None
223 * dependencies => string or None
224
225 :return: a tuple containing (
226 builder_status, build_id, build_status, logtail, filemap,
227 dependencies)
228 """213 """
229214
230 def slaveStatusSentence():215 def slaveStatusSentence():
231216
=== modified file 'lib/lp/soyuz/interfaces/buildqueue.py'
--- lib/lp/soyuz/interfaces/buildqueue.py 2009-11-20 18:06:28 +0000
+++ lib/lp/soyuz/interfaces/buildqueue.py 2009-12-08 20:16:13 +0000
@@ -13,7 +13,7 @@
13 ]13 ]
1414
15from zope.interface import Interface, Attribute15from zope.interface import Interface, Attribute
16from zope.schema import Choice, Datetime, Timedelta16from zope.schema import Choice, Datetime, Field, Timedelta
1717
18from lazr.restful.fields import Reference18from lazr.restful.fields import Reference
1919
@@ -51,6 +51,10 @@
51 title=_('Job type'), required=True, vocabulary=BuildFarmJobType,51 title=_('Job type'), required=True, vocabulary=BuildFarmJobType,
52 description=_("The type of this job."))52 description=_("The type of this job."))
5353
54 required_build_behavior = Field(
55 title=_('The builder behavior required to run this job.'),
56 required=False, readonly=True)
57
54 estimated_duration = Timedelta(58 estimated_duration = Timedelta(
55 title=_("Estimated Job Duration"), required=True,59 title=_("Estimated Job Duration"), required=True,
56 description=_("Estimated job duration interval."))60 description=_("Estimated job duration interval."))
5761
=== added file 'lib/lp/soyuz/model/binarypackagebuildbehavior.py'
--- lib/lp/soyuz/model/binarypackagebuildbehavior.py 1970-01-01 00:00:00 +0000
+++ lib/lp/soyuz/model/binarypackagebuildbehavior.py 2009-12-08 20:16:13 +0000
@@ -0,0 +1,271 @@
1# Copyright 2009 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4# pylint: disable-msg=E0211,E0213
5
6"""Builder behavior for binary package builds."""
7
8__metaclass__ = type
9
10__all__ = [
11 'BinaryPackageBuildBehavior',
12 ]
13
14import socket
15import xmlrpclib
16
17from canonical.launchpad.webapp import urlappend
18from lp.buildmaster.interfaces.buildfarmjobbehavior import (
19 IBuildFarmJobBehavior)
20from lp.buildmaster.model.buildfarmjobbehavior import (
21 BuildFarmJobBehaviorBase)
22from lp.registry.interfaces.pocket import PackagePublishingPocket
23from lp.soyuz.adapters.archivedependencies import (
24 get_primary_current_component, get_sources_list_for_building)
25from lp.soyuz.interfaces.archive import ArchivePurpose
26from lp.soyuz.interfaces.build import IBuildSet
27from lp.soyuz.interfaces.builder import BuildSlaveFailure, CannotBuild
28
29from zope.component import getUtility
30from zope.interface import implements
31
32
33class BinaryPackageBuildBehavior(BuildFarmJobBehaviorBase):
34 """Define the behavior of binary package builds."""
35
36 implements(IBuildFarmJobBehavior)
37
38 def logStartBuild(self, build_queue_item, logger):
39 """See `IBuildFarmJobBehavior`."""
40 build = getUtility(IBuildSet).getByQueueEntry(build_queue_item)
41 spr = build.sourcepackagerelease
42
43 logger.info("startBuild(%s, %s, %s, %s)", self._builder.url,
44 spr.name, spr.version, build.pocket.title)
45
46 @property
47 def status(self):
48 """See `IBuildFarmJobBehavior`."""
49 build = getUtility(IBuildSet).getByQueueEntry(
50 self._builder.currentjob)
51 msg = 'Building %s' % build.title
52 archive = build.archive
53 if not archive.owner.private and (archive.is_ppa or archive.is_copy):
54 return '%s [%s/%s]' % (msg, archive.owner.name, archive.name)
55 else:
56 return msg
57
58 def dispatchBuildToSlave(self, build_queue_item, logger):
59 """See `IBuildFarmJobBehavior`."""
60
61 # Start the binary package build on the slave builder. First
62 # we send the chroot.
63 build = getUtility(IBuildSet).getByQueueEntry(build_queue_item)
64 chroot = build.distroarchseries.getChroot()
65 self._builder.cacheFileOnSlave(logger, chroot)
66
67 # Build filemap structure with the files required in this build
68 # and send them to the slave.
69 # If the build is private we tell the slave to get the files from the
70 # archive instead of the librarian because the slaves cannot
71 # access the restricted librarian.
72 private = build.archive.private
73 if private:
74 self._cachePrivateSourceOnSlave(build_queue_item, logger)
75 filemap = {}
76 for source_file in build.sourcepackagerelease.files:
77 lfa = source_file.libraryfile
78 filemap[lfa.filename] = lfa.content.sha1
79 if not private:
80 self._builder.cacheFileOnSlave(
81 logger, source_file.libraryfile)
82
83 # Generate a string which can be used to cross-check when obtaining
84 # results so we know we are referring to the right database object in
85 # subsequent runs.
86 buildid = "%s-%s" % (build.id, build_queue_item.id)
87 chroot_sha1 = chroot.content.sha1
88 logger.debug(
89 "Initiating build %s on %s" % (buildid, self._builder.url))
90
91 try:
92 args = self._extraBuildArgs(build)
93 status, info = self._builder.slave.build(
94 buildid, "debian", chroot_sha1, filemap, args)
95 message = """%s (%s):
96 ***** RESULT *****
97 %s
98 %s
99 %s: %s
100 ******************
101 """ % (
102 self._builder.name,
103 self._builder.url,
104 filemap,
105 args,
106 status,
107 info,
108 )
109 logger.info(message)
110 except xmlrpclib.Fault, info:
111 # Mark builder as 'failed'.
112 logger.debug(
113 "Disabling builder: %s" % self._builder.url, exc_info=1)
114 self._builder.failbuilder(
115 "Exception (%s) when setting up to new job" % info)
116 raise BuildSlaveFailure
117 except socket.error, info:
118 error_message = "Exception (%s) when setting up new job" % info
119 self._builder.handleTimeout(logger, error_message)
120 raise BuildSlaveFailure
121
122 def verifyBuildRequest(self, build_queue_item, logger):
123 """Assert some pre-build checks.
124
125 The build request is checked:
126 * Virtualized builds can't build on a non-virtual builder
127 * Ensure that we have a chroot
128 * Ensure that the build pocket allows builds for the current
129 distroseries state.
130 """
131 build = getUtility(IBuildSet).getByQueueEntry(build_queue_item)
132 assert not (not self._builder.virtualized and build.is_virtualized), (
133 "Attempt to build non-virtual item on a virtual builder.")
134
135 # Assert that we are not silently building SECURITY jobs.
136 # See findBuildCandidates. Once we start building SECURITY
137 # correctly from EMBARGOED archive this assertion can be removed.
138 # XXX Julian 2007-12-18 spec=security-in-soyuz: This is being
139 # addressed in the work on the blueprint:
140 # https://blueprints.launchpad.net/soyuz/+spec/security-in-soyuz
141 target_pocket = build.pocket
142 assert target_pocket != PackagePublishingPocket.SECURITY, (
143 "Soyuz is not yet capable of building SECURITY uploads.")
144
145 # Ensure build has the needed chroot
146 chroot = build.distroarchseries.getChroot()
147 if chroot is None:
148 raise CannotBuild(
149 "Missing CHROOT for %s/%s/%s" % (
150 build.distroseries.distribution.name,
151 build.distroseries.name,
152 build.distroarchseries.architecturetag)
153 )
154
155 # The main distribution has policies to prevent uploads to some
156 # pockets (e.g. security) during different parts of the distribution
157 # series lifecycle. These do not apply to PPA builds nor any archive
158 # that allows release pocket updates.
159 if (build.archive.purpose != ArchivePurpose.PPA and
160 not build.archive.allowUpdatesToReleasePocket()):
161 # XXX Robert Collins 2007-05-26: not an explicit CannotBuild
162 # exception yet because the callers have not been audited
163 assert build.distroseries.canUploadToPocket(build.pocket), (
164 "%s (%s) can not be built for pocket %s: invalid pocket due "
165 "to the series status of %s."
166 % (build.title, build.id, build.pocket.name,
167 build.distroseries.name))
168
169 def slaveStatus(self, raw_slave_status):
170 """Parse and return the binary build specific status info.
171
172 This includes:
173 * build_id => string
174 * build_status => string or None
175 * logtail => string or None
176 * filename => dictionary or None
177 * dependencies => string or None
178 """
179 builder_status = raw_slave_status[0]
180 extra_info = {}
181 if builder_status == 'BuilderStatus.WAITING':
182 extra_info['build_status'] = raw_slave_status[1]
183 extra_info['build_id'] = raw_slave_status[2]
184 build_status_with_files = [
185 'BuildStatus.OK',
186 'BuildStatus.PACKAGEFAIL',
187 'BuildStatus.DEPFAIL',
188 ]
189 if extra_info['build_status'] in build_status_with_files:
190 extra_info['filemap'] = raw_slave_status[3]
191 extra_info['dependencies'] = raw_slave_status[4]
192 else:
193 extra_info['build_id'] = raw_slave_status[1]
194 if builder_status == 'BuilderStatus.BUILDING':
195 extra_info['logtail'] = raw_slave_status[2]
196
197 return extra_info
198
199 def _cachePrivateSourceOnSlave(self, build_queue_item, logger):
200 """Ask the slave to download source files for a private build.
201
202 The slave will cache the files for the source in build_queue_item
203 to its local disk in preparation for a private build. Private builds
204 will always take the source files from the archive rather than the
205 librarian since the archive has more granular access to each
206 archive's files.
207
208 :param build_queue_item: The `IBuildQueue` being built.
209 :param logger: A logger used for providing debug information.
210 """
211 # The URL to the file in the archive consists of these parts:
212 # archive_url / makePoolPath() / filename
213 # Once this is constructed we add the http basic auth info.
214
215 # Avoid circular imports.
216 from lp.soyuz.model.publishing import makePoolPath
217
218 build = getUtility(IBuildSet).getByQueueEntry(build_queue_item)
219 archive = build.archive
220 archive_url = archive.archive_url
221 component_name = build.current_component.name
222 for source_file in build.sourcepackagerelease.files:
223 file_name = source_file.libraryfile.filename
224 sha1 = source_file.libraryfile.content.sha1
225 source_name = build.sourcepackagerelease.sourcepackagename.name
226 poolpath = makePoolPath(source_name, component_name)
227 url = urlappend(archive_url, poolpath)
228 url = urlappend(url, file_name)
229 logger.debug("Asking builder on %s to ensure it has file %s "
230 "(%s, %s)" % (
231 self._builder.url, file_name, url, sha1))
232 self._builder._sendFileToSlave(
233 url, sha1, "buildd", archive.buildd_secret)
234
235 def _extraBuildArgs(self, build):
236 """
237 Return the extra arguments required by the slave for the given build.
238 """
239 # Build extra arguments.
240 args = {}
241 # turn 'arch_indep' ON only if build is archindep or if
242 # the specific architecture is the nominatedarchindep for
243 # this distroseries (in case it requires any archindep source)
244 args['arch_indep'] = build.distroarchseries.isNominatedArchIndep
245
246 suite = build.distroarchseries.distroseries.name
247 if build.pocket != PackagePublishingPocket.RELEASE:
248 suite += "-%s" % (build.pocket.name.lower())
249 args['suite'] = suite
250
251 archive_purpose = build.archive.purpose
252 if (archive_purpose == ArchivePurpose.PPA and
253 not build.archive.require_virtualized):
254 # If we're building a non-virtual PPA, override the purpose
255 # to PRIMARY and use the primary component override.
256 # This ensures that the package mangling tools will run over
257 # the built packages.
258 args['archive_purpose'] = ArchivePurpose.PRIMARY.name
259 args["ogrecomponent"] = (
260 get_primary_current_component(build))
261 else:
262 args['archive_purpose'] = archive_purpose.name
263 args["ogrecomponent"] = (
264 build.current_component.name)
265
266 args['archives'] = get_sources_list_for_building(build)
267
268 # Let the build slave know whether this is a build in a private
269 # archive.
270 args['archive_private'] = build.archive.private
271 return args
0272
=== modified file 'lib/lp/soyuz/model/builder.py'
--- lib/lp/soyuz/model/builder.py 2009-11-20 18:06:28 +0000
+++ lib/lp/soyuz/model/builder.py 2009-12-08 20:16:13 +0000
@@ -20,6 +20,8 @@
20import urllib220import urllib2
21import xmlrpclib21import xmlrpclib
2222
23from lazr.delegates import delegates
24
23from zope.interface import implements25from zope.interface import implements
24from zope.component import getUtility26from zope.component import getUtility
2527
@@ -31,10 +33,11 @@
31from canonical.cachedproperty import cachedproperty33from canonical.cachedproperty import cachedproperty
32from canonical.config import config34from canonical.config import config
33from canonical.buildd.slave import BuilderStatus35from canonical.buildd.slave import BuilderStatus
36from lp.buildmaster.interfaces.buildfarmjobbehavior import (
37 BuildBehaviorMismatch, IBuildFarmJobBehavior)
34from lp.buildmaster.master import BuilddMaster38from lp.buildmaster.master import BuilddMaster
39from lp.buildmaster.model.buildfarmjobbehavior import IdleBuildBehavior
35from canonical.database.sqlbase import SQLBase, sqlvalues40from canonical.database.sqlbase import SQLBase, sqlvalues
36from lp.soyuz.adapters.archivedependencies import (
37 get_primary_current_component, get_sources_list_for_building)
38from lp.soyuz.model.buildqueue import BuildQueue41from lp.soyuz.model.buildqueue import BuildQueue
39from lp.registry.interfaces.person import validate_public_person42from lp.registry.interfaces.person import validate_public_person
40from lp.registry.interfaces.pocket import PackagePublishingPocket43from lp.registry.interfaces.pocket import PackagePublishingPocket
@@ -54,6 +57,7 @@
54 PackagePublishingStatus)57 PackagePublishingStatus)
55from lp.soyuz.model.buildpackagejob import BuildPackageJob58from lp.soyuz.model.buildpackagejob import BuildPackageJob
56from canonical.launchpad.webapp import urlappend59from canonical.launchpad.webapp import urlappend
60from canonical.lazr.utils import safe_hasattr
57from canonical.librarian.utils import copy_and_close61from canonical.librarian.utils import copy_and_close
5862
5963
@@ -111,9 +115,11 @@
111115
112 return (stdout, stderr, resume_process.returncode)116 return (stdout, stderr, resume_process.returncode)
113117
118
114class Builder(SQLBase):119class Builder(SQLBase):
115120
116 implements(IBuilder, IHasBuildRecords)121 implements(IBuilder, IHasBuildRecords)
122 delegates(IBuildFarmJobBehavior, context="current_build_behavior")
117 _table = 'Builder'123 _table = 'Builder'
118124
119 _defaultOrder = ['id']125 _defaultOrder = ['id']
@@ -135,6 +141,44 @@
135 vm_host = StringCol(dbName='vm_host')141 vm_host = StringCol(dbName='vm_host')
136 active = BoolCol(dbName='active', notNull=True, default=True)142 active = BoolCol(dbName='active', notNull=True, default=True)
137143
144 def _getCurrentBuildBehavior(self):
145 """Return the current build behavior."""
146 if not safe_hasattr(self, '_current_build_behavior'):
147 self._current_build_behavior = None
148
149 if (self._current_build_behavior is None or
150 isinstance(self._current_build_behavior, IdleBuildBehavior)):
151 # If we don't currently have a current build behavior set,
152 # or we are currently idle, then...
153 currentjob = self.currentjob
154 if currentjob is not None:
155 # ...we'll set it based on our current job.
156 self._current_build_behavior = (
157 currentjob.required_build_behavior)
158 self._current_build_behavior.setBuilder(self)
159 return self._current_build_behavior
160 elif self._current_build_behavior is None:
161 # If we don't have a current job or an idle behavior
162 # already set, then we just set the idle behavior
163 # before returning.
164 self._current_build_behavior = IdleBuildBehavior()
165 return self._current_build_behavior
166
167 else:
168 # We did have a current non-idle build behavior set, so
169 # we just return it.
170 return self._current_build_behavior
171
172
173 def _setCurrentBuildBehavior(self, new_behavior):
174 """Set the current build behavior."""
175 self._current_build_behavior = new_behavior
176 if self._current_build_behavior is not None:
177 self._current_build_behavior.setBuilder(self)
178
179 current_build_behavior = property(
180 _getCurrentBuildBehavior, _setCurrentBuildBehavior)
181
138 def cacheFileOnSlave(self, logger, libraryfilealias):182 def cacheFileOnSlave(self, logger, libraryfilealias):
139 """See `IBuilder`."""183 """See `IBuilder`."""
140 url = libraryfilealias.http_url184 url = libraryfilealias.http_url
@@ -143,30 +187,6 @@
143 url, libraryfilealias.content.sha1))187 url, libraryfilealias.content.sha1))
144 self._sendFileToSlave(url, libraryfilealias.content.sha1)188 self._sendFileToSlave(url, libraryfilealias.content.sha1)
145189
146 def cachePrivateSourceOnSlave(self, logger, build_queue_item):
147 """See `IBuilder`."""
148 # The URL to the file in the archive consists of these parts:
149 # archive_url / makePoolPath() / filename
150 # Once this is constructed we add the http basic auth info.
151
152 # Avoid circular imports.
153 from lp.soyuz.model.publishing import makePoolPath
154
155 build = getUtility(IBuildSet).getByQueueEntry(build_queue_item)
156 archive = build.archive
157 archive_url = archive.archive_url
158 component_name = build.current_component.name
159 for source_file in build.sourcepackagerelease.files:
160 file_name = source_file.libraryfile.filename
161 sha1 = source_file.libraryfile.content.sha1
162 source_name = build.sourcepackagerelease.sourcepackagename.name
163 poolpath = makePoolPath(source_name, component_name)
164 url = urlappend(archive_url, poolpath)
165 url = urlappend(url, file_name)
166 logger.debug("Asking builder on %s to ensure it has file %s "
167 "(%s, %s)" % (self.url, file_name, url, sha1))
168 self._sendFileToSlave(url, sha1, "buildd", archive.buildd_secret)
169
170 def _sendFileToSlave(self, url, sha1, username="", password=""):190 def _sendFileToSlave(self, url, sha1, username="", password=""):
171 """Helper to send the file at 'url' with 'sha1' to this builder."""191 """Helper to send the file at 'url' with 'sha1' to this builder."""
172 if not self.builderok:192 if not self.builderok:
@@ -256,156 +276,22 @@
256 """See IBuilder."""276 """See IBuilder."""
257 self.slave = proxy277 self.slave = proxy
258278
259 def _verifyBuildRequest(self, build_queue_item, logger):
260 """Assert some pre-build checks.
261
262 The build request is checked:
263 * Virtualized builds can't build on a non-virtual builder
264 * Ensure that we have a chroot
265 * Ensure that the build pocket allows builds for the current
266 distroseries state.
267 """
268 build = getUtility(IBuildSet).getByQueueEntry(build_queue_item)
269 assert not (not self.virtualized and build.is_virtualized), (
270 "Attempt to build non-virtual item on a virtual builder.")
271
272 # Assert that we are not silently building SECURITY jobs.
273 # See findBuildCandidates. Once we start building SECURITY
274 # correctly from EMBARGOED archive this assertion can be removed.
275 # XXX Julian 2007-12-18 spec=security-in-soyuz: This is being
276 # addressed in the work on the blueprint:
277 # https://blueprints.launchpad.net/soyuz/+spec/security-in-soyuz
278 target_pocket = build.pocket
279 assert target_pocket != PackagePublishingPocket.SECURITY, (
280 "Soyuz is not yet capable of building SECURITY uploads.")
281
282 # Ensure build has the needed chroot
283 build = getUtility(IBuildSet).getByQueueEntry(build_queue_item)
284 chroot = build.distroarchseries.getChroot()
285 if chroot is None:
286 raise CannotBuild(
287 "Missing CHROOT for %s/%s/%s" % (
288 build.distroseries.distribution.name,
289 build.distroseries.name,
290 build.distroarchseries.architecturetag)
291 )
292
293 # The main distribution has policies to prevent uploads to some
294 # pockets (e.g. security) during different parts of the distribution
295 # series lifecycle. These do not apply to PPA builds nor any archive
296 # that allows release pocket updates.
297 if (build.archive.purpose != ArchivePurpose.PPA and
298 not build.archive.allowUpdatesToReleasePocket()):
299 # XXX Robert Collins 2007-05-26: not an explicit CannotBuild
300 # exception yet because the callers have not been audited
301 assert build.distroseries.canUploadToPocket(build.pocket), (
302 "%s (%s) can not be built for pocket %s: invalid pocket due "
303 "to the series status of %s."
304 % (build.title, build.id, build.pocket.name,
305 build.distroseries.name))
306
307 def _dispatchBuildToSlave(self, build_queue_item, args, buildid, logger):
308 """Start the build on the slave builder."""
309 # Send chroot.
310 build = getUtility(IBuildSet).getByQueueEntry(build_queue_item)
311 chroot = build.distroarchseries.getChroot()
312 self.cacheFileOnSlave(logger, chroot)
313
314 # Build filemap structure with the files required in this build
315 # and send them to the slave.
316 # If the build is private we tell the slave to get the files from the
317 # archive instead of the librarian because the slaves cannot
318 # access the restricted librarian.
319 private = build.archive.private
320 if private:
321 self.cachePrivateSourceOnSlave(logger, build_queue_item)
322 filemap = {}
323 for source_file in build.sourcepackagerelease.files:
324 lfa = source_file.libraryfile
325 filemap[lfa.filename] = lfa.content.sha1
326 if not private:
327 self.cacheFileOnSlave(logger, source_file.libraryfile)
328
329 chroot_sha1 = chroot.content.sha1
330 try:
331 status, info = self.slave.build(
332 buildid, "debian", chroot_sha1, filemap, args)
333 message = """%s (%s):
334 ***** RESULT *****
335 %s
336 %s
337 %s: %s
338 ******************
339 """ % (self.name, self.url, filemap, args, status, info)
340 logger.info(message)
341 except xmlrpclib.Fault, info:
342 # Mark builder as 'failed'.
343 logger.debug("Disabling builder: %s" % self.url, exc_info=1)
344 self.failbuilder(
345 "Exception (%s) when setting up to new job" % info)
346 raise BuildSlaveFailure
347 except socket.error, info:
348 error_message = "Exception (%s) when setting up new job" % info
349 self.handleTimeout(logger, error_message)
350 raise BuildSlaveFailure
351
352 def startBuild(self, build_queue_item, logger):279 def startBuild(self, build_queue_item, logger):
353 """See IBuilder."""280 """See IBuilder."""
354 build = getUtility(IBuildSet).getByQueueEntry(build_queue_item)281 # Set the build behavior depending on the provided build queue item.
355 spr = build.sourcepackagerelease282 self.current_build_behavior = build_queue_item.required_build_behavior
356 logger.info("startBuild(%s, %s, %s, %s)", self.url,283 self.logStartBuild(build_queue_item, logger)
357 spr.name, spr.version, build.pocket.title)
358284
359 # Make sure the request is valid; an exception is raised if it's not.285 # Make sure the request is valid; an exception is raised if it's not.
360 self._verifyBuildRequest(build_queue_item, logger)286 self.verifyBuildRequest(build_queue_item, logger)
361287
362 # If we are building a virtual build, resume the virtual machine.288 # If we are building a virtual build, resume the virtual machine.
363 if self.virtualized:289 if self.virtualized:
364 self.resumeSlaveHost()290 self.resumeSlaveHost()
365291
366 # Build extra arguments.
367 args = {}
368 # turn 'arch_indep' ON only if build is archindep or if
369 # the specific architecture is the nominatedarchindep for
370 # this distroseries (in case it requires any archindep source)
371 build = getUtility(IBuildSet).getByQueueEntry(build_queue_item)
372 args['arch_indep'] = build.distroarchseries.isNominatedArchIndep
373
374 suite = build.distroarchseries.distroseries.name
375 if build.pocket != PackagePublishingPocket.RELEASE:
376 suite += "-%s" % (build.pocket.name.lower())
377 args['suite'] = suite
378
379 archive_purpose = build.archive.purpose
380 if (archive_purpose == ArchivePurpose.PPA and
381 not build.archive.require_virtualized):
382 # If we're building a non-virtual PPA, override the purpose
383 # to PRIMARY and use the primary component override.
384 # This ensures that the package mangling tools will run over
385 # the built packages.
386 args['archive_purpose'] = ArchivePurpose.PRIMARY.name
387 args["ogrecomponent"] = (
388 get_primary_current_component(build))
389 else:
390 args['archive_purpose'] = archive_purpose.name
391 args["ogrecomponent"] = (
392 build.current_component.name)
393
394 args['archives'] = get_sources_list_for_building(build)
395
396 # Let the build slave know whether this is a build in a private
397 # archive.
398 args['archive_private'] = build.archive.private
399
400 # Generate a string which can be used to cross-check when obtaining
401 # results so we know we are referring to the right database object in
402 # subsequent runs.
403 buildid = "%s-%s" % (build.id, build_queue_item.id)
404 logger.debug("Initiating build %s on %s" % (buildid, self.url))
405
406 # Do it.292 # Do it.
407 build_queue_item.markAsBuilding(self)293 build_queue_item.markAsBuilding(self)
408 self._dispatchBuildToSlave(build_queue_item, args, buildid, logger)294 self.dispatchBuildToSlave(build_queue_item, logger)
409295
410 # XXX cprov 2009-06-24: This code does not belong to the content296 # XXX cprov 2009-06-24: This code does not belong to the content
411 # class domain. Here we cannot make sensible decisions about what297 # class domain. Here we cannot make sensible decisions about what
@@ -418,25 +304,20 @@
418 if self.failnotes is not None:304 if self.failnotes is not None:
419 return self.failnotes305 return self.failnotes
420 return 'Disabled'306 return 'Disabled'
421 # Cache the 'currentjob', so we don't have to hit the database
422 # more than once.
423 currentjob = self.currentjob
424 if currentjob is None:
425 return 'Idle'
426307
427 build = getUtility(IBuildSet).getByQueueEntry(currentjob)308 # If the builder is OK then we delegate the status
428 msg = 'Building %s' % build.title309 # to our current behavior.
429 archive = build.archive310 return self.current_build_behavior.status
430 if not archive.owner.private and (archive.is_ppa or archive.is_copy):
431 return '%s [%s/%s]' % (msg, archive.owner.name, archive.name)
432 else:
433 return msg
434311
435 def failbuilder(self, reason):312 def failbuilder(self, reason):
436 """See IBuilder"""313 """See IBuilder"""
437 self.builderok = False314 self.builderok = False
438 self.failnotes = reason315 self.failnotes = reason
439316
317 # XXX Michael Nelson 20091202 bug=491330. The current UI assumes
318 # that the builder history will display binary build records, as
319 # returned by getBuildRecords() below. See the bug for a discussion
320 # of the options.
440 def getBuildRecords(self, build_state=None, name=None, arch_tag=None,321 def getBuildRecords(self, build_state=None, name=None, arch_tag=None,
441 user=None):322 user=None):
442 """See IHasBuildRecords."""323 """See IHasBuildRecords."""
@@ -447,29 +328,11 @@
447 """See IBuilder."""328 """See IBuilder."""
448 builder_version, builder_arch, mechanisms = self.slave.info()329 builder_version, builder_arch, mechanisms = self.slave.info()
449 status_sentence = self.slave.status()330 status_sentence = self.slave.status()
450 builder_status = status_sentence[0]331
451332 status = {'builder_status': status_sentence[0]}
452 if builder_status == 'BuilderStatus.WAITING':333 status.update(
453 (build_status, build_id) = status_sentence[1:3]334 self.current_build_behavior.slaveStatus(status_sentence))
454 build_status_with_files = [335 return status
455 'BuildStatus.OK',
456 'BuildStatus.PACKAGEFAIL',
457 'BuildStatus.DEPFAIL',
458 ]
459 if build_status in build_status_with_files:
460 (filemap, dependencies) = status_sentence[3:]
461 else:
462 filemap = dependencies = None
463 logtail = None
464 elif builder_status == 'BuilderStatus.BUILDING':
465 (build_id, logtail) = status_sentence[1:]
466 build_status = filemap = dependencies = None
467 else:
468 build_id = status_sentence[1]
469 build_status = logtail = filemap = dependencies = None
470
471 return (builder_status, build_id, build_status, logtail, filemap,
472 dependencies)
473336
474 def slaveStatusSentence(self):337 def slaveStatusSentence(self):
475 """See IBuilder."""338 """See IBuilder."""
@@ -671,7 +534,7 @@
671 logger = self._getSlaveScannerLogger()534 logger = self._getSlaveScannerLogger()
672 try:535 try:
673 self.startBuild(candidate, logger)536 self.startBuild(candidate, logger)
674 except (BuildSlaveFailure, CannotBuild), err:537 except (BuildSlaveFailure, CannotBuild, BuildBehaviorMismatch), err:
675 logger.warn('Could not build: %s' % err)538 logger.warn('Could not build: %s' % err)
676539
677 def handleTimeout(self, logger, error_message):540 def handleTimeout(self, logger, error_message):
678541
=== modified file 'lib/lp/soyuz/model/buildqueue.py'
--- lib/lp/soyuz/model/buildqueue.py 2009-11-20 18:06:28 +0000
+++ lib/lp/soyuz/model/buildqueue.py 2009-12-08 20:16:13 +0000
@@ -24,6 +24,8 @@
24from canonical.database.sqlbase import SQLBase, sqlvalues24from canonical.database.sqlbase import SQLBase, sqlvalues
25from canonical.launchpad.webapp.interfaces import NotFoundError25from canonical.launchpad.webapp.interfaces import NotFoundError
26from lp.buildmaster.interfaces.buildfarmjob import BuildFarmJobType26from lp.buildmaster.interfaces.buildfarmjob import BuildFarmJobType
27from lp.buildmaster.interfaces.buildfarmjobbehavior import (
28 IBuildFarmJobBehavior)
27from lp.services.job.interfaces.job import JobStatus29from lp.services.job.interfaces.job import JobStatus
28from lp.services.job.model.job import Job30from lp.services.job.model.job import Job
29from lp.soyuz.interfaces.build import BuildStatus, IBuildSet31from lp.soyuz.interfaces.build import BuildStatus, IBuildSet
@@ -49,6 +51,11 @@
49 estimated_duration = IntervalCol()51 estimated_duration = IntervalCol()
5052
51 @property53 @property
54 def required_build_behavior(self):
55 """See `IBuildQueue`."""
56 return IBuildFarmJobBehavior(self.specific_job)
57
58 @property
52 def specific_job(self):59 def specific_job(self):
53 """See `IBuildQueue`."""60 """See `IBuildQueue`."""
54 store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)61 store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
5562
=== modified file 'lib/lp/soyuz/tests/test_builder.py'
--- lib/lp/soyuz/tests/test_builder.py 2009-11-13 16:37:05 +0000
+++ lib/lp/soyuz/tests/test_builder.py 2009-12-08 20:16:13 +0000
@@ -8,10 +8,15 @@
8from zope.component import getUtility8from zope.component import getUtility
99
10from canonical.testing import LaunchpadZopelessLayer10from canonical.testing import LaunchpadZopelessLayer
11from lp.buildmaster.interfaces.buildfarmjobbehavior import (
12 BuildBehaviorMismatch, IBuildFarmJobBehavior)
13from lp.buildmaster.model.buildfarmjobbehavior import IdleBuildBehavior
11from lp.soyuz.interfaces.archive import ArchivePurpose14from lp.soyuz.interfaces.archive import ArchivePurpose
12from lp.soyuz.interfaces.build import BuildStatus, IBuildSet15from lp.soyuz.interfaces.build import BuildStatus, IBuildSet
13from lp.soyuz.interfaces.builder import IBuilderSet16from lp.soyuz.interfaces.builder import IBuilderSet
14from lp.soyuz.interfaces.publishing import PackagePublishingStatus17from lp.soyuz.interfaces.publishing import PackagePublishingStatus
18from lp.soyuz.model.binarypackagebuildbehavior import (
19 BinaryPackageBuildBehavior)
15from lp.soyuz.tests.test_publishing import SoyuzTestPublisher20from lp.soyuz.tests.test_publishing import SoyuzTestPublisher
16from lp.testing import TestCaseWithFactory21from lp.testing import TestCaseWithFactory
1722
@@ -214,5 +219,54 @@
214 self.failUnlessEqual('primary', build.archive.name)219 self.failUnlessEqual('primary', build.archive.name)
215 self.failUnlessEqual('firefox', build.sourcepackagerelease.name)220 self.failUnlessEqual('firefox', build.sourcepackagerelease.name)
216221
222
223class TestCurrentBuildBehavior(TestCaseWithFactory):
224 """This test ensures the get/set behavior of IBuilder's
225 current_build_behavior property.
226 """
227
228 layer = LaunchpadZopelessLayer
229
230 def setUp(self):
231 """Create a new builder ready for testing."""
232 super(TestCurrentBuildBehavior, self).setUp()
233 self.builder = self.factory.makeBuilder(name='builder')
234
235 # Have a publisher and a ppa handy for some of the tests below.
236 self.publisher = SoyuzTestPublisher()
237 self.publisher.prepareBreezyAutotest()
238 self.ppa_joe = self.factory.makeArchive(name="joesppa")
239
240 self.build = self.publisher.getPubSource(
241 sourcename="gedit", status=PackagePublishingStatus.PUBLISHED,
242 archive=self.ppa_joe).createMissingBuilds()[0]
243
244 self.buildfarmjob = self.build.buildqueue_record.specific_job
245
246 def test_idle_behavior_when_no_current_build(self):
247 """We return an idle behavior when there is no behavior specified
248 nor a current build.
249 """
250 self.assertIsInstance(
251 self.builder.current_build_behavior, IdleBuildBehavior)
252
253 def test_set_behavior_sets_builder(self):
254 """Setting a builder's behavior also associates the behavior with the
255 builder."""
256 behavior = IBuildFarmJobBehavior(self.buildfarmjob)
257 self.builder.current_build_behavior = behavior
258
259 self.assertEqual(behavior, self.builder.current_build_behavior)
260 self.assertEqual(behavior._builder, self.builder)
261
262 def test_current_job_behavior(self):
263 """The current behavior is set automatically from the current job."""
264 # Set the builder attribute on the buildqueue record so that our
265 # builder will think it has a current build.
266 self.build.buildqueue_record.builder = self.builder
267
268 self.assertIsInstance(
269 self.builder.current_build_behavior, BinaryPackageBuildBehavior)
270
217def test_suite():271def test_suite():
218 return unittest.TestLoader().loadTestsFromName(__name__)272 return unittest.TestLoader().loadTestsFromName(__name__)

Subscribers

People subscribed via source and target branches

to status/vote changes: