Launchpad itself

Merge lp:~wgrant/launchpad/multi-arch-builders into lp:launchpad

  1. multi-arch-builders
  2. Merge into devel
Proposed by William Grant
Status: Merged
Approved by: Jeroen T. Vermeulen
Approved revision: no longer in the source branch.
Merged at revision: 11306
Proposed branch: lp:~wgrant/launchpad/multi-arch-builders
Merge into: lp:launchpad
Diff against target: 415 lines (+40/-108)
17 files modified
lib/canonical/buildd/binarypackage.py (+2/-1)
lib/canonical/buildd/debian.py (+5/-4)
lib/canonical/buildd/debian/changelog (+7/-0)
lib/canonical/buildd/sbuild (+3/-0)
lib/canonical/buildd/sbuild-package (+3/-7)
lib/canonical/buildd/sbuildrc (+1/-1)
lib/canonical/buildd/tests/test_translationtemplatesbuildmanager.py (+3/-0)
lib/canonical/buildd/update-debian-chroot (+2/-6)
lib/lp/buildmaster/doc/builder.txt (+0/-34)
lib/lp/buildmaster/interfaces/builder.py (+0/-14)
lib/lp/buildmaster/model/builder.py (+1/-36)
lib/lp/buildmaster/tests/test_manager.py (+1/-0)
lib/lp/code/model/recipebuilder.py (+1/-0)
lib/lp/code/model/tests/test_recipebuilder.py (+2/-0)
lib/lp/soyuz/model/binarypackagebuildbehavior.py (+2/-0)
lib/lp/soyuz/tests/soyuzbuilddhelpers.py (+0/-3)
lib/lp/translations/model/translationtemplatesbuildbehavior.py (+7/-2)
To merge this branch: bzr merge lp:~wgrant/launchpad/multi-arch-builders
Reviewer Review Type Date Requested Status
Jeroen T. Vermeulen (community) Approve
Review via email: mp+31740@code.launchpad.net

Commit message

The architecture of a build slave can now be set through the web UI, and the slave will respect the setting.

Description of the change

This branch mostly fixes bug #491418, adjusting launchpad-buildd to take an optional arch_tag extra argument, which is then used when dealing with the chroot and performing the actual build. This lets buildd-manager decide which architecture it needs a slave to be for a particular build.

The master support for this is incomplete: in this branch, builders must have their architecture tweaked manually through the UI. The master's slave architecture check has also been removed, which means that an incorrect architecture setting could result in chroot setup failures rather than the current builder failure.

Eventually the master will be able to dynamically assign builds of different architectures to the same builder, but that requires more work.

To post a comment you must log in.
Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

Looks good. Can't judge all the consequences, but that's not what a review is for.

Can't think of anything to fix either. Go clean up some lint or something.

Jeroen

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/canonical/buildd/binarypackage.py'
2--- lib/canonical/buildd/binarypackage.py 2010-06-12 03:48:34 +0000
3+++ lib/canonical/buildd/binarypackage.py 2010-08-05 21:16:05 +0000
4@@ -67,7 +67,7 @@
5
6 def doRunBuild(self):
7 """Run the sbuild process to build the package."""
8- args = ["sbuild-package", self._buildid ]
9+ args = ["sbuild-package", self._buildid, self.arch_tag]
10 if self.suite:
11 args.extend([self.suite])
12 args.extend(self._sbuildargs)
13@@ -82,6 +82,7 @@
14 args.extend(["--purpose=" + self.archive_purpose])
15 if self.build_debug_symbols:
16 args.extend(["--build-debug-symbols"])
17+ args.extend(["--architecture=" + self.arch_tag])
18 args.extend(["--comp=" + self.component])
19 args.extend([self._dscfile])
20 self.runSubProcess( self._sbuildpath, args )
21
22=== modified file 'lib/canonical/buildd/debian.py'
23--- lib/canonical/buildd/debian.py 2010-06-12 03:48:34 +0000
24+++ lib/canonical/buildd/debian.py 2010-08-05 21:16:05 +0000
25@@ -47,6 +47,7 @@
26 def initiate(self, files, chroot, extra_args):
27 """Initiate a build with a given set of files and chroot."""
28
29+ self.arch_tag = extra_args.get('arch_tag', self._slave.getArch())
30 self.sources_list = extra_args.get('archives')
31
32 BuildManager.initiate(self, files, chroot, extra_args)
33@@ -62,8 +63,9 @@
34
35 def doUpdateChroot(self):
36 """Perform the chroot upgrade."""
37- self.runSubProcess(self._updatepath,
38- ["update-debian-chroot", self._buildid])
39+ self.runSubProcess(
40+ self._updatepath,
41+ ["update-debian-chroot", self._buildid, self.arch_tag])
42
43 def doRunBuild(self):
44 """Run the main build process.
45@@ -95,8 +97,7 @@
46 yield filename
47
48 def getChangesFilename(self):
49- changes = (
50- self._dscfile[:-4] + "_" + self._slave.getArch() + ".changes")
51+ changes = self._dscfile[:-4] + "_" + self.arch_tag + ".changes"
52 return get_build_path(self._buildid, changes)
53
54 def gatherResults(self):
55
56=== modified file 'lib/canonical/buildd/debian/changelog'
57--- lib/canonical/buildd/debian/changelog 2010-07-23 20:23:04 +0000
58+++ lib/canonical/buildd/debian/changelog 2010-08-05 21:16:05 +0000
59@@ -1,3 +1,10 @@
60+launchpad-buildd (68) UNRELEASED; urgency=low
61+
62+ * Take an 'arch_tag' argument, so the master can override the slave
63+ architecture.
64+
65+ -- William Grant <wgrant@ubuntu.com> Sun, 01 Aug 2010 22:00:32 +1000
66+
67 launchpad-buildd (67) hardy-cat; urgency=low
68
69 * Force aptitude installation for recipe builds on maverick
70
71=== modified file 'lib/canonical/buildd/sbuild'
72--- lib/canonical/buildd/sbuild 2010-07-19 18:14:54 +0000
73+++ lib/canonical/buildd/sbuild 2010-08-05 21:16:05 +0000
74@@ -564,6 +564,9 @@
75 $main::build_source++;
76 $conf::purge_build_directory = "never";
77 }
78+ elsif (/^--architecture=(.)/) {
79+ $conf::arch=$1.$';
80+ }
81 elsif (/^--archive=(.)/) {
82 $main::archive=$1.$';
83 }
84
85=== modified file 'lib/canonical/buildd/sbuild-package'
86--- lib/canonical/buildd/sbuild-package 2010-01-11 23:03:50 +0000
87+++ lib/canonical/buildd/sbuild-package 2010-08-05 21:16:05 +0000
88@@ -20,8 +20,9 @@
89 SBUILD=/usr/bin/sbuild
90 BUILDID=$1
91 SUITE=$2
92+ARCHITECTURETAG=$3
93
94-shift; shift
95+shift 3
96
97 export NR_PROCESSORS=$(grep ^processor /proc/cpuinfo | wc -l)
98
99@@ -34,11 +35,6 @@
100 cd "$HOME/build-$BUILDID"
101
102 hostarch=$(dpkg --print-architecture)
103-architecturetag=$hostarch
104-
105-if [ -f /etc/launchpad-buildd/default ]; then
106- eval `grep architecturetag /etc/launchpad-buildd/default | sed 's/ //g'`
107-fi
108
109 WARN=""
110 case $hostarch in
111@@ -51,7 +47,7 @@
112 WARN="--warnonly"
113 ;;
114 amd64)
115- if [ "$hostarch" != "$architecturetag" ]; then
116+ if [ "$hostarch" != "$ARCHITECTURETAG" ]; then
117 SBUILD="linux32 $SBUILD"
118 WARN="--warnonly"
119 fi
120
121=== modified file 'lib/canonical/buildd/sbuildrc'
122--- lib/canonical/buildd/sbuildrc 2009-12-15 00:19:56 +0000
123+++ lib/canonical/buildd/sbuildrc 2010-08-05 21:16:05 +0000
124@@ -9,7 +9,7 @@
125 $arch="@ARCHTAG@";
126
127 # Maintainer name to use in .changes files (mandatory, no default!)
128-$maintainer_name="Ubuntu/$arch Build Daemon <buildd\@@FQDN@>";
129+$maintainer_name="Ubuntu Build Daemon <buildd\@@FQDN@>";
130
131 # When to purge the build directory afterwards; possible values are
132 # "never", "successful", and "always"
133
134=== modified file 'lib/canonical/buildd/tests/test_translationtemplatesbuildmanager.py'
135--- lib/canonical/buildd/tests/test_translationtemplatesbuildmanager.py 2010-04-01 09:31:13 +0000
136+++ lib/canonical/buildd/tests/test_translationtemplatesbuildmanager.py 2010-08-05 21:16:05 +0000
137@@ -43,6 +43,9 @@
138 def wasCalled(self, name):
139 return name in self._was_called
140
141+ def getArch(self):
142+ return 'i386'
143+
144 addWaitingFile = FakeMethod()
145
146
147
148=== modified file 'lib/canonical/buildd/update-debian-chroot'
149--- lib/canonical/buildd/update-debian-chroot 2009-12-15 18:42:11 +0000
150+++ lib/canonical/buildd/update-debian-chroot 2010-08-05 21:16:05 +0000
151@@ -15,6 +15,7 @@
152 CHROOT=/usr/sbin/chroot
153 APTGET=/usr/bin/apt-get
154 BUILDID="$1"
155+ARCHITECTURETAG="$2"
156 ROOT=$HOME/build-$BUILDID/chroot-autobuild
157 OPTIONS="-o DPkg::Options::=--force-confold"
158
159@@ -25,18 +26,13 @@
160 echo "Updating debian chroot for build $BUILDID"
161
162 hostarch=$(dpkg --print-architecture)
163-architecturetag=$hostarch
164-
165-if [ -f /etc/launchpad-buildd/default ]; then
166- eval `grep architecturetag /etc/launchpad-buildd/default | sed 's/ //g'`
167-fi
168
169 case $hostarch in
170 hppa|powerpc|sparc)
171 CHROOT="linux32 $CHROOT"
172 ;;
173 amd64)
174- if [ "$hostarch" != "$architecturetag" ]; then
175+ if [ "$hostarch" != "$ARCHITECTURETAG" ]; then
176 CHROOT="linux32 $CHROOT"
177 fi
178 ;;
179
180=== modified file 'lib/lp/buildmaster/doc/builder.txt'
181--- lib/lp/buildmaster/doc/builder.txt 2010-08-03 17:48:22 +0000
182+++ lib/lp/buildmaster/doc/builder.txt 2010-08-05 21:16:05 +0000
183@@ -261,40 +261,6 @@
184 >>> config_data = config.pop('vm_resume_command')
185
186
187-Slave architecture checks
188-=========================
189-
190-Builder.checkSlaveArchitecture() asks the slave for its version and tries
191-to match it against a DistroArchSeries with a ProcessorFamily containing
192-the Builder's Processor. If it fails, it will raise an exception so the
193-builder can be marked as failed.
194-
195-A fictitious i386 variant is rejected, since there are no DASes with that
196-tag.
197-
198- >>> from lp.soyuz.tests.soyuzbuilddhelpers import OkSlave
199- >>> bob.setSlaveForTesting(OkSlave('i387'))
200- >>> bob.checkSlaveArchitecture()
201- Traceback (most recent call last):
202- ...
203- BuildDaemonError: Bad slave architecture tag: i387 (registered family: x86)
204-
205-hppa isn't in the x86 family, so it too is rejected.
206-
207- >>> from lp.soyuz.tests.soyuzbuilddhelpers import OkSlave
208- >>> bob.setSlaveForTesting(OkSlave('hppa'))
209- >>> bob.checkSlaveArchitecture()
210- Traceback (most recent call last):
211- ...
212- BuildDaemonError: Bad slave architecture tag: hppa (registered family: x86)
213-
214-But i386, a real x86 variant, passes without objection.
215-
216- >>> from lp.soyuz.tests.soyuzbuilddhelpers import OkSlave
217- >>> bob.setSlaveForTesting(OkSlave('i386'))
218- >>> bob.checkSlaveArchitecture()
219-
220-
221 Rescuing lost slaves
222 ====================
223
224
225=== modified file 'lib/lp/buildmaster/interfaces/builder.py'
226--- lib/lp/buildmaster/interfaces/builder.py 2010-06-04 14:50:46 +0000
227+++ lib/lp/buildmaster/interfaces/builder.py 2010-08-05 21:16:05 +0000
228@@ -150,20 +150,6 @@
229 title=u"The current behavior of the builder for the current job.",
230 required=False)
231
232- def checkSlaveArchitecture():
233- """Check that the slave can compile for its nominated processor.
234-
235- This will query the builder to determine its actual architecture (as
236- opposed to what we expect it to be). It will then look for a
237- DistroArchSeries with the returned architecture tag, and confirm that
238- the processor type matches.
239-
240- :raises BuildDaemonError: When the builder is down or of the wrong
241- architecture.
242- :raises ProtocolVersionMismatch: When the builder returns an
243- unsupported protocol version.
244- """
245-
246 def checkSlaveAlive():
247 """Check that the buildd slave is alive.
248
249
250=== modified file 'lib/lp/buildmaster/model/builder.py'
251--- lib/lp/buildmaster/model/builder.py 2010-08-03 14:48:22 +0000
252+++ lib/lp/buildmaster/model/builder.py 2010-08-05 21:16:05 +0000
253@@ -25,7 +25,6 @@
254 from sqlobject import (
255 BoolCol, ForeignKey, IntCol, SQLObjectNotFound, StringCol)
256 from storm.expr import Coalesce, Count, Sum
257-from storm.store import Store
258 from zope.component import getUtility
259 from zope.interface import implements
260
261@@ -42,8 +41,7 @@
262 from lp.app.errors import NotFoundError
263 from lp.buildmaster.interfaces.builder import (
264 BuildDaemonError, BuildSlaveFailure, CannotBuild, CannotFetchFile,
265- CannotResumeHost, CorruptBuildCookie, IBuilder, IBuilderSet,
266- ProtocolVersionMismatch)
267+ CannotResumeHost, CorruptBuildCookie, IBuilder, IBuilderSet)
268 from lp.buildmaster.interfaces.buildfarmjob import IBuildFarmJobSet
269 from lp.buildmaster.interfaces.buildfarmjobbehavior import (
270 BuildBehaviorMismatch)
271@@ -204,7 +202,6 @@
272
273 try:
274 builder.checkSlaveAlive()
275- builder.checkSlaveArchitecture()
276 builder.rescueIfLost(logger)
277 # Catch only known exceptions.
278 # XXX cprov 2007-06-15 bug=120571: ValueError & TypeError catching is
279@@ -284,38 +281,6 @@
280 current_build_behavior = property(
281 _getCurrentBuildBehavior, _setCurrentBuildBehavior)
282
283- def checkSlaveArchitecture(self):
284- """See `IBuilder`."""
285- # XXX cprov 2007-06-15 bug=545839:
286- # This function currently depends on the operating system specific
287- # details of the build slave to return a processor-family-name (the
288- # architecturetag) which matches the distro_arch_series. In reality,
289- # we should be checking the processor itself (e.g. amd64) as that is
290- # what the distro policy is set from, the architecture tag is both
291- # distro specific and potentially different for radically different
292- # distributions - its not the right thing to be comparing.
293-
294- from lp.soyuz.model.distroarchseries import DistroArchSeries
295-
296- # query the slave for its active details.
297- # XXX cprov 2007-06-15: Why is 'mechanisms' ignored?
298- builder_vers, builder_arch, mechanisms = self.slave.info()
299- # we can only understand one version of slave today:
300- if builder_vers != '1.0':
301- raise ProtocolVersionMismatch("Protocol version mismatch")
302-
303- # Find a distroarchseries with the returned arch tag.
304- # This is ugly, sick and wrong, but so is the whole concept. See the
305- # XXX above and its bug for details.
306- das = Store.of(self).find(
307- DistroArchSeries, architecturetag=builder_arch,
308- processorfamily=self.processor.family).any()
309-
310- if das is None:
311- raise BuildDaemonError(
312- "Bad slave architecture tag: %s (registered family: %s)" %
313- (builder_arch, self.processor.family.name))
314-
315 def checkSlaveAlive(self):
316 """See IBuilder."""
317 if self.slave.echo("Test")[0] != "Test":
318
319=== modified file 'lib/lp/buildmaster/tests/test_manager.py'
320--- lib/lp/buildmaster/tests/test_manager.py 2010-07-28 09:56:24 +0000
321+++ lib/lp/buildmaster/tests/test_manager.py 2010-08-05 21:16:05 +0000
322@@ -644,6 +644,7 @@
323 {'alsa-utils_1.0.9a-4ubuntu1.dsc':
324 '4e3961baf4f56fdbc95d0dd47f3c5bc275da8a33'},
325 {'arch_indep': True,
326+ 'arch_tag': 'i386',
327 'archive_private': False,
328 'archive_purpose': 'PRIMARY',
329 'archives':
330
331=== modified file 'lib/lp/code/model/recipebuilder.py'
332--- lib/lp/code/model/recipebuilder.py 2010-05-27 17:55:16 +0000
333+++ lib/lp/code/model/recipebuilder.py 2010-08-05 21:16:05 +0000
334@@ -62,6 +62,7 @@
335 if self.build.pocket != PackagePublishingPocket.RELEASE:
336 suite += "-%s" % (self.build.pocket.name.lower())
337 args['suite'] = suite
338+ args['arch_tag'] = distroarchseries.architecturetag
339 args["author_name"] = self.build.requester.displayname
340 args["author_email"] = self.build.requester.preferredemail.email
341 args["recipe_text"] = str(self.build.recipe.builder_recipe)
342
343=== modified file 'lib/lp/code/model/tests/test_recipebuilder.py'
344--- lib/lp/code/model/tests/test_recipebuilder.py 2010-06-11 04:45:18 +0000
345+++ lib/lp/code/model/tests/test_recipebuilder.py 2010-08-05 21:16:05 +0000
346@@ -140,6 +140,7 @@
347 expected_archives.append(
348 "deb http://foo %s main" % job.build.distroseries.name)
349 self.assertEqual({
350+ 'arch_tag': 'i386',
351 'author_email': u'requester@ubuntu.com',
352 'suite': u'mydistro',
353 'author_name': u'Joe User',
354@@ -169,6 +170,7 @@
355 job.build, distroarchseries, None)
356 logger = BufferLogger()
357 self.assertEqual({
358+ 'arch_tag': 'i386',
359 'author_email': u'requester@ubuntu.com',
360 'suite': u'mydistro',
361 'author_name': u'Joe User',
362
363=== modified file 'lib/lp/soyuz/model/binarypackagebuildbehavior.py'
364--- lib/lp/soyuz/model/binarypackagebuildbehavior.py 2010-05-20 13:24:01 +0000
365+++ lib/lp/soyuz/model/binarypackagebuildbehavior.py 2010-08-05 21:16:05 +0000
366@@ -192,6 +192,8 @@
367 suite += "-%s" % (build.pocket.name.lower())
368 args['suite'] = suite
369
370+ args['arch_tag'] = build.distro_arch_series.architecturetag
371+
372 archive_purpose = build.archive.purpose
373 if (archive_purpose == ArchivePurpose.PPA and
374 not build.archive.require_virtualized):
375
376=== modified file 'lib/lp/soyuz/tests/soyuzbuilddhelpers.py'
377--- lib/lp/soyuz/tests/soyuzbuilddhelpers.py 2010-07-31 23:10:08 +0000
378+++ lib/lp/soyuz/tests/soyuzbuilddhelpers.py 2010-08-05 21:16:05 +0000
379@@ -71,9 +71,6 @@
380 def checkSlaveAlive(self):
381 pass
382
383- def checkSlaveArchitecture(self):
384- pass
385-
386 def rescueIfLost(self, logger=None):
387 rescueBuilderIfLost(self, logger)
388
389
390=== modified file 'lib/lp/translations/model/translationtemplatesbuildbehavior.py'
391--- lib/lp/translations/model/translationtemplatesbuildbehavior.py 2010-05-24 10:14:41 +0000
392+++ lib/lp/translations/model/translationtemplatesbuildbehavior.py 2010-08-05 21:16:05 +0000
393@@ -44,15 +44,20 @@
394 self._builder.slave.cacheFile(logger, chroot)
395 cookie = self.buildfarmjob.generateSlaveBuildCookie()
396
397- args = self.buildfarmjob.metadata
398+ args = {'arch_tag': self._getDistroArchSeries().architecturetag}
399+ args.update(self.buildfarmjob.metadata)
400+
401 filemap = {}
402
403 self._builder.slave.build(
404 cookie, self.build_type, chroot_sha1, filemap, args)
405
406 def _getChroot(self):
407+ return self._getDistroArchSeries().getChroot()
408+
409+ def _getDistroArchSeries(self):
410 ubuntu = getUtility(ILaunchpadCelebrities).ubuntu
411- return ubuntu.currentseries.nominatedarchindep.getChroot()
412+ return ubuntu.currentseries.nominatedarchindep
413
414 def logStartBuild(self, logger):
415 """See `IBuildFarmJobBehavior`."""