Launchpad itself

Merge lp:~henninge/launchpad/bug-509557-invoke-pottery into lp:launchpad

  1. bug-509557-invoke-pottery
  2. Merge into devel
Proposed by Henning Eggers
Status: Merged
Approved by: Henning Eggers
Approved revision: no longer in the source branch.
Merged at revision: not available
Proposed branch: lp:~henninge/launchpad/bug-509557-invoke-pottery
Merge into: lp:launchpad
Diff against target: 694 lines (+225/-208)
10 files modified
lib/canonical/buildd/debian/changelog (+6/-0)
lib/canonical/buildd/debian/rules (+8/-15)
lib/canonical/buildd/generate-translation-templates (+54/-0)
lib/canonical/buildd/pottery/generate_translation_templates.py (+17/-10)
lib/canonical/buildd/template-buildd-slave.conf (+3/-0)
lib/canonical/buildd/tests/test_generate_translation_templates.py (+10/-6)
lib/canonical/buildd/tests/test_translationtemplatesbuildmanager.py (+77/-48)
lib/canonical/buildd/translationtemplates.py (+48/-127)
lib/lp/translations/tests/test_pottery_detect_intltool.py (+1/-1)
scripts/rosetta/pottery-generate-intltool.py (+1/-1)
To merge this branch: bzr merge lp:~henninge/launchpad/bug-509557-invoke-pottery
Reviewer Review Type Date Requested Status
Paul Hummer (community) code Approve
Review via email: mp+21172@code.launchpad.net

Commit message

Build slave for translation templates generation now mostly working.

Description of the change

= Bug 509557 =

Connects all dots within the build slave to get translation templates build jobs working. It undoes some things done in previous branches, namely:

 - Moves pottery code from lp.translations.pottery into canonical.buildd.pottery. This way it does not need to be copied from there into the buildd package directory by "debian/rules package" anymore. But don't worry, it will still get copied around a lot.

 - The build manager that drives the slave through a state machine is now based on DebianBuildManager because it shares most of it's requirements and thus code. Made the TranslationTemplatesBuildManager much shorter, too.

Another big change is the introduction of the generate-translation-templates script that copies the pottery code into the build chroot and starts the generate_translation_templates.py script in the chroot.

== Implementation details ==

=== modified file 'lib/canonical/buildd/debian/changelog'

The package received a new version number.

=== modified file 'lib/canonical/buildd/debian/rules'

Removed getting the pottery code from the lp tree. Actually, just one new line is left in the file; it copies the pottery directory into the package.

=== added file 'lib/canonical/buildd/generate-translation-templates'

The new shell script that prepares the chroot to run the python script that generates the templates.

=== added directory 'lib/canonical/buildd/pottery'
=== added file 'lib/canonical/buildd/pottery/__init__.py'

The new home for the slave-side pottery code

=== renamed file 'lib/canonical/buildd/generate_translation_templates.py' => 'lib/canonical/buildd/pottery/generate_translation_templates.py'

The script is now part of the package to be copied into the chroot with it. Also it's use of the HOME dir as temporary build directory does not go down well with tests. It's OK on the build slave, though, so I made that setting overridable.

=== renamed file 'lib/lp/translations/pottery/buildd.py' => 'lib/canonical/buildd/pottery/intltool.py'

The file was named to "intltool.py" because it specifically generates templates using intltool. Other tools will get there own module here.

=== modified file 'lib/canonical/buildd/template-buildd-slave.conf'

The TranslationTemplatesBuildManagaer gets its own section like the others to find the generate script to call.

=== modified file 'lib/canonical/buildd/tests/test_generate_translation_templates.py'

New paths and use of temporary directory.

=== modified file 'lib/canonical/buildd/tests/test_translationtemplatesbuildmanager.py'

Oh yeah, big changes here. This is testing the state machine of the build manager. Since the new build manager only implements two states, this test was reduced to test just those. But to get proper coverage I also added tests to check the fail behaviour.

The FakeSlave had to be pimped to react friendly to methods that it doesn't know about. It now even keeps track of those so that the test can see that build manager properly told the slave about a failure. I think that's cool.

=== modified file 'lib/canonical/buildd/translationtemplates.py'

So, this is where it all happens. As mentioned earlier, most of this is deletion because most of the states are already implemented correctly in DebianBuildManager. This adds states for installation of extra packages (bzr and intltool) and calling the generate script.

DebianBuildManager calls doRunBuild to start the build but ist starts with doInstall here before it goes to doGenrate, so I added an alias to doInstall so that DebianBuildManager can find it.

Also, runInChroot was removed because it was flawed and only one usage remains inthe file (in doInstall) which is better done directly now.

I expect another state to be added soon to retrieve the generated templates.

=== modified file 'lib/lp/translations/tests/test_pottery_detect_intltool.py'
=== modified file 'scripts/rosetta/pottery-generate-intltool.py'

New paths for old files.

= Test =

Oh yeah, that ... I tested this code using the steps I described here:
https://dev.launchpad.net/BuildFarm/TryOutBuildSlave

In addition, run the tests mentioned here:

bin/text -vvt generate_translation_templates -t translationtemplatesbuildmanager -t pottery_detect_intltool

= Launchpad lint =

Checking for conflicts. and issues in doctests and templates.
Running jslint, xmllint, pyflakes, and pylint.
Using normal rules.

Linting changed files:
  lib/canonical/buildd/generate-translation-templates
  lib/canonical/buildd/pottery/
  lib/canonical/buildd/template-buildd-slave.conf
  lib/canonical/buildd/translationtemplates.py
  lib/canonical/buildd/debian/changelog
  lib/canonical/buildd/debian/rules
  lib/canonical/buildd/pottery/__init__.py
  lib/canonical/buildd/pottery/generate_translation_templates.py
  lib/canonical/buildd/tests/test_generate_translation_templates.py
  lib/canonical/buildd/tests/test_translationtemplatesbuildmanager.py
  lib/lp/translations/tests/test_pottery_detect_intltool.py
  scripts/rosetta/pottery-generate-intltool.py

== Pylint notices ==

scripts/rosetta/pottery-generate-intltool.py
    8: [W0403] Relative import '_pythonpath'

To post a comment you must log in.
Revision history for this message
Paul Hummer (rockstar) wrote :

Looks good. I'm a little wary of adding new things to lib/canonical but it looks like it's a necessity to getting things done right now, and is not adding any more technical debt.

review: Approve (code)
Revision history for this message
Henning Eggers (henninge) wrote :

Thanks a lot, Paul. I expect that someday lib/canonical/buildd will move somewhere else and thus take this code with it.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/canonical/buildd/debian/changelog'
2--- lib/canonical/buildd/debian/changelog 2010-01-14 21:05:08 +0000
3+++ lib/canonical/buildd/debian/changelog 2010-03-11 22:47:24 +0000
4@@ -1,3 +1,9 @@
5+launchpad-buildd (59) karmic; urgency=low
6+
7+ * Added translation template generation code (pottery).
8+
9+ -- Henning Eggers <henning@canonical.com> Thu, 11 Mar 2010 12:59:25 +0100
10+
11 launchpad-buildd (58~1) karmic; urgency=low
12
13 * Misc fixes to match APIs.
14
15=== modified file 'lib/canonical/buildd/debian/rules'
16--- lib/canonical/buildd/debian/rules 2010-02-25 16:49:51 +0000
17+++ lib/canonical/buildd/debian/rules 2010-03-11 22:47:24 +0000
18@@ -21,10 +21,11 @@
19 targetshare = $(target)/usr/share/launchpad-buildd
20 pytarget = $(targetshare)/canonical/buildd
21
22-pyfiles = debian.py slave.py binarypackage.py utils.py __init__.py sourcepackagerecipe.py translationtemplates.py
23+pyfiles = debian.py slave.py binarypackage.py utils.py __init__.py \
24+ sourcepackagerecipe.py translationtemplates.py
25 slavebins = unpack-chroot mount-chroot update-debian-chroot sbuild-package \
26 scan-for-processes umount-chroot remove-build apply-ogre-model \
27-override-sources-list buildrecipe generate_translation_templates.py
28+override-sources-list buildrecipe generate-translation-templates
29
30 BUILDDUID=65500
31 BUILDDGID=65500
32@@ -39,26 +40,19 @@
33 var/run/launchpad-buildd var/log/launchpad-buildd \
34 etc/launchpad-buildd \
35 usr/share/launchpad-buildd/canonical/launchpad/daemons \
36- usr/share/doc/launchpad-buildd \
37- usr/share/launchpad-buildd/lp \
38- usr/share/launchpad-buildd/lp/translations \
39- usr/share/launchpad-buildd/lp/translations/pottery
40+ usr/share/doc/launchpad-buildd
41
42 # Do installs here
43 touch $(pytarget)/../launchpad/__init__.py
44 touch $(pytarget)/../launchpad/daemons/__init__.py
45 cp launchpad-files/tachandler.py $(pytarget)/../launchpad/daemons/
46 install -m644 launchpad-files/buildd-slave.tac $(targetshare)/buildd-slave.tac
47- touch $(pytarget)/../../lp/__init__.py
48- touch $(pytarget)/../../lp/translations/__init__.py
49- touch $(pytarget)/../../lp/translations/pottery/__init__.py
50- cp launchpad-files/pottery_buildd.py \
51- $(pytarget)/../../lp/translations/pottery/buildd.py
52+ cp -r pottery $(pytarget)
53 for pyfile in $(pyfiles); do \
54- install -m644 ./$$pyfile $(pytarget)/$$pyfile; \
55- done
56+ install -m644 ./$$pyfile $(pytarget)/$$pyfile; \
57+ done
58 for slavebin in $(slavebins); do \
59- install -m755 ./$$slavebin $(targetshare)/slavebin/$$slavebin; \
60+ install -m755 ./$$slavebin $(targetshare)/slavebin/$$slavebin; \
61 done
62 install -m755 ./sbuild $(target)/usr/bin/sbuild
63 touch $(targetshare)/canonical/__init__.py
64@@ -100,7 +94,6 @@
65 mkdir -p launchpad-files
66 install -m644 $(daemons)/buildd-slave.tac launchpad-files/buildd-slave.tac
67 cp ../launchpad/daemons/tachandler.py launchpad-files/tachandler.py
68- cp ../../lp/translations/pottery/buildd.py launchpad-files/pottery_buildd.py
69 debuild -uc -us -S
70
71 build:
72
73=== added file 'lib/canonical/buildd/generate-translation-templates'
74--- lib/canonical/buildd/generate-translation-templates 1970-01-01 00:00:00 +0000
75+++ lib/canonical/buildd/generate-translation-templates 2010-03-11 22:47:24 +0000
76@@ -0,0 +1,54 @@
77+#!/bin/sh
78+#
79+# Copyright 2010 Canonical Ltd. This software is licensed under the
80+# GNU Affero General Public License version 3 (see the file LICENSE).
81+#
82+# Author: Henning Eggers <henning@canonical.com>
83+
84+# Buildd Slave tool to generate translation templates. Boiler plate code
85+# copied from sbuild-package.
86+
87+# Expects build id as arg 1.
88+# Expects branch url as arg 2.
89+
90+# Must run as user with password-less sudo ability.
91+
92+exec 2>&1
93+
94+export LANG=C LC_ALL=C
95+
96+CHMOD=/bin/chmod
97+CHROOT=/usr/sbin/chroot
98+CP=/bin/cp
99+INSTALL=/usr/bin/install
100+MKDIR=/bin/mkdir
101+SU=/bin/su
102+SUDO=/usr/bin/sudo
103+TOUCH=/usr/bin/touch
104+
105+BUILDID=$1
106+BRANCH_URL=$2
107+
108+BUILDD_HOME=/usr/share/launchpad-buildd
109+SLAVEBIN=$BUILDD_HOME/slavebin
110+BUILD_CHROOT="$HOME/build-$BUILDID/chroot-autobuild"
111+USER=$(whoami)
112+
113+BUILDD_PACKAGE=canonical/buildd
114+POTTERY=$BUILDD_PACKAGE/pottery
115+# The script should be smarter about detecting the python version.
116+PYMODULES=/usr/lib/pymodules/python2.6
117+
118+GENERATE_SCRIPT=$PYMODULES/$POTTERY/generate_translation_templates.py
119+
120+# Copy pottery files to chroot.
121+$SUDO $MKDIR -p $BUILD_CHROOT$PYMODULES/$BUILDD_PACKAGE
122+$SUDO $TOUCH $BUILD_CHROOT$PYMODULES/canonical/__init__.py
123+$SUDO $TOUCH $BUILD_CHROOT$PYMODULES/canonical/buildd/__init__.py
124+$SUDO $CP -r $BUILDD_HOME/$POTTERY $BUILD_CHROOT$PYMODULES/$BUILDD_PACKAGE
125+$SUDO $CHMOD -R go+rX $BUILD_CHROOT$PYMODULES/canonical
126+$SUDO $CHMOD 755 $BUILD_CHROOT$GENERATE_SCRIPT
127+
128+
129+# Enter chroot, switch back to unprivileged user, execute the generate script.
130+$SUDO $CHROOT $BUILD_CHROOT $SU - $USER -c "$GENERATE_SCRIPT $BRANCH_URL"
131
132=== added directory 'lib/canonical/buildd/pottery'
133=== added file 'lib/canonical/buildd/pottery/__init__.py'
134=== renamed file 'lib/canonical/buildd/generate_translation_templates.py' => 'lib/canonical/buildd/pottery/generate_translation_templates.py'
135--- lib/canonical/buildd/generate_translation_templates.py 2010-03-04 08:12:25 +0000
136+++ lib/canonical/buildd/pottery/generate_translation_templates.py 2010-03-11 22:47:24 +0000
137@@ -1,4 +1,4 @@
138-#! /usr/bin/python2.5
139+#! /usr/bin/python
140 # Copyright 2010 Canonical Ltd. This software is licensed under the
141 # GNU Affero General Public License version 3 (see the file LICENSE).
142
143@@ -10,28 +10,29 @@
144 from bzrlib.branch import Branch
145 from bzrlib.export import export
146
147-from lp.translations.pottery.buildd import generate_pots
148+from canonical.buildd.pottery import intltool
149
150
151 class GenerateTranslationTemplates:
152 """Script to generate translation templates from a branch."""
153
154- def __init__(self, branch_spec):
155+ def __init__(self, branch_spec, work_dir):
156 """Prepare to generate templates for a branch.
157
158 :param branch_spec: Either a branch URL or the path of a local
159 branch. URLs are recognized by the occurrence of ':'. In
160 the case of a URL, this will make up a path for the branch
161 and check out the branch to there.
162+ :param work_dir: The directory to work in. Must exist.
163 """
164- self.home = os.environ['HOME']
165+ self.work_dir = work_dir
166 self.branch_spec = branch_spec
167
168 def _getBranch(self):
169 """Set `self.branch_dir`, and check out branch if needed."""
170 if ':' in self.branch_spec:
171 # This is a branch URL. Check out the branch.
172- self.branch_dir = os.path.join(self.home, 'source-tree')
173+ self.branch_dir = os.path.join(self.work_dir, 'source-tree')
174 self._checkout(self.branch_spec)
175 else:
176 # This is a local filesystem path. Use the branch in-place.
177@@ -50,7 +51,7 @@
178 def generate(self):
179 """Do It. Generate templates."""
180 self._getBranch()
181- pots = generate_pots(self.branch_dir)
182+ pots = intltool.generate_pots(self.branch_dir)
183 print "\n".join(
184 [os.path.normpath(os.path.join(self.branch_dir, potpath))
185 for potpath in pots])
186@@ -58,8 +59,14 @@
187
188
189 if __name__ == '__main__':
190- if len(sys.argv) != 2:
191- print "Usage: %s branch" % sys.argv[0]
192- print "Where 'branch' is a branch URL or directory."
193+ if len(sys.argv) < 2:
194+ print "Usage: %s branch [workdir]" % sys.argv[0]
195+ print " 'branch' is a branch URL or directory."
196+ print " 'workdir' is a directory, defaults to HOME."
197 sys.exit(1)
198- sys.exit(GenerateTranslationTemplates(sys.argv[1]).generate())
199+ if len(sys.argv) == 3:
200+ workdir = sys.argv[2]
201+ else:
202+ workdir = os.environ['HOME']
203+ script = GenerateTranslationTemplates(sys.argv[1], workdir)
204+ sys.exit(script.generate())
205
206=== renamed file 'lib/lp/translations/pottery/buildd.py' => 'lib/canonical/buildd/pottery/intltool.py'
207=== modified file 'lib/canonical/buildd/template-buildd-slave.conf'
208--- lib/canonical/buildd/template-buildd-slave.conf 2010-01-13 05:26:33 +0000
209+++ lib/canonical/buildd/template-buildd-slave.conf 2010-03-11 22:47:24 +0000
210@@ -27,3 +27,6 @@
211
212 [sourcepackagerecipemanager]
213 buildrecipepath = /usr/share/launchpad-buildd/slavebin/buildrecipe
214+
215+[translationtemplatesmanager]
216+generatepath = /usr/share/launchpad-buildd/slavebin/generate-translation-templates
217
218=== modified file 'lib/canonical/buildd/tests/test_generate_translation_templates.py'
219--- lib/canonical/buildd/tests/test_generate_translation_templates.py 2010-03-04 08:12:25 +0000
220+++ lib/canonical/buildd/tests/test_generate_translation_templates.py 2010-03-11 22:47:24 +0000
221@@ -6,7 +6,7 @@
222
223 from lp.testing.fakemethod import FakeMethod
224
225-from canonical.buildd.generate_translation_templates import (
226+from canonical.buildd.pottery.generate_translation_templates import (
227 GenerateTranslationTemplates)
228
229 from canonical.launchpad.ftests.script import run_script
230@@ -24,7 +24,8 @@
231 # check out that branch into a directory called "source-tree."
232 branch_url = 'lp://~my/translation/branch'
233
234- generator = GenerateTranslationTemplates(branch_url)
235+ generator = GenerateTranslationTemplates(
236+ branch_url, self.makeTemporaryDirectory())
237 generator._checkout = FakeMethod()
238 generator._getBranch()
239
240@@ -36,7 +37,8 @@
241 # works directly in that directory.
242 branch_dir = '/home/me/branch'
243
244- generator = GenerateTranslationTemplates(branch_dir)
245+ generator = GenerateTranslationTemplates(
246+ branch_dir, self.makeTemporaryDirectory())
247 generator._checkout = FakeMethod()
248 generator._getBranch()
249
250@@ -71,7 +73,8 @@
251 branch = self._createBranch({'marker.txt': marker_text})
252 branch_url = branch.getPullURL()
253
254- generator = GenerateTranslationTemplates(branch_url)
255+ generator = GenerateTranslationTemplates(
256+ branch_url, self.makeTemporaryDirectory())
257 generator.branch_dir = self.makeTemporaryDirectory()
258 generator._getBranch()
259
260@@ -80,9 +83,10 @@
261
262 def test_script(self):
263 tempdir = self.makeTemporaryDirectory()
264+ workdir = self.makeTemporaryDirectory()
265 (retval, out, err) = run_script(
266- 'lib/canonical/buildd/generate_translation_templates.py',
267- args=[tempdir])
268+ 'lib/canonical/buildd/pottery/generate_translation_templates.py',
269+ args=[tempdir, workdir])
270 self.assertEqual(0, retval)
271
272
273
274=== modified file 'lib/canonical/buildd/tests/test_translationtemplatesbuildmanager.py'
275--- lib/canonical/buildd/tests/test_translationtemplatesbuildmanager.py 2010-01-21 12:34:41 +0000
276+++ lib/canonical/buildd/tests/test_translationtemplatesbuildmanager.py 2010-03-11 22:47:24 +0000
277@@ -22,10 +22,26 @@
278 def __init__(self, tempdir):
279 self._cachepath = tempdir
280 self._config = FakeConfig()
281+ self._was_called = set()
282
283 def cachePath(self, file):
284 return os.path.join(self._cachepath, file)
285
286+ def anyMethod(self, *args, **kwargs):
287+ pass
288+
289+ fake_methods = ['emptyLog', 'chrootFail', 'buildFail', 'builderFail',]
290+ def __getattr__(self, name):
291+ """Remember which fake methods were called."""
292+ if name not in self.fake_methods:
293+ raise AttributeError(
294+ "'%s' object has no attribute '%s'" % (self.__class__, name))
295+ self._was_called.add(name)
296+ return self.anyMethod
297+
298+ def wasCalled(self, name):
299+ return name in self._was_called
300+
301
302 class MockBuildManager(TranslationTemplatesBuildManager):
303 def __init__(self, *args, **kwargs):
304@@ -33,7 +49,7 @@
305 self.commands = []
306
307 def runSubProcess(self, path, command):
308- self.commands.append(command)
309+ self.commands.append([path]+command)
310 return 0
311
312
313@@ -46,55 +62,27 @@
314 home_dir = os.path.join(self.working_dir, 'home')
315 for dir in (slave_dir, home_dir):
316 os.mkdir(dir)
317- slave = FakeSlave(slave_dir)
318- buildid = '123'
319- self.buildmanager = MockBuildManager(slave, buildid)
320+ self.slave = FakeSlave(slave_dir)
321+ self.buildid = '123'
322+ self.buildmanager = MockBuildManager(self.slave, self.buildid)
323 self.buildmanager.home = home_dir
324 self.chrootdir = os.path.join(
325- home_dir, 'build-%s' % buildid, 'chroot-autobuild')
326+ home_dir, 'build-%s' % self.buildid, 'chroot-autobuild')
327
328 def getState(self):
329 """Retrieve build manager's state."""
330 return self.buildmanager._state
331
332- def test_initiate(self):
333- # Creating a BuildManager spawns no child processes.
334- self.assertEqual([], self.buildmanager.commands)
335-
336- # Initiating the build executes the first command. It leaves
337- # the build manager in the INIT state.
338- self.buildmanager.initiate({}, 'chroot.tar.gz', {'branch_url': 'foo'})
339- self.assertEqual(1, len(self.buildmanager.commands))
340- self.assertEqual(TranslationTemplatesBuildState.INIT, self.getState())
341-
342 def test_iterate(self):
343+ # Two iteration steps are specific to this build manager.
344 url = 'lp:~my/branch'
345 # The build manager's iterate() kicks off the consecutive states
346 # after INIT.
347 self.buildmanager.initiate({}, 'chroot.tar.gz', {'branch_url': url})
348
349- # UNPACK: execute unpack-chroot.
350- self.buildmanager.iterate(0)
351- self.assertEqual(
352- TranslationTemplatesBuildState.UNPACK, self.getState())
353- self.assertEqual('unpack-chroot', self.buildmanager.commands[-1][0])
354-
355- # MOUNT: Set up realistic chroot environment.
356- self.buildmanager.iterate(0)
357- self.assertEqual(
358- TranslationTemplatesBuildState.MOUNT, self.getState())
359- self.assertEqual('mount-chroot', self.buildmanager.commands[-1][0])
360-
361- # UPDATE: Get the latest versions of installed packages.
362- self.buildmanager.iterate(0)
363- self.assertEqual(
364- TranslationTemplatesBuildState.UPDATE, self.getState())
365- expected_command = [
366- '/usr/bin/sudo',
367- '/usr/sbin/chroot', self.chrootdir,
368- 'update-debian-chroot',
369- ]
370- self.assertEqual(expected_command, self.buildmanager.commands[-1][:4])
371+ # Skip states that are done in DebianBuldManager to the state
372+ # directly before INSTALL.
373+ self.buildmanager._state = TranslationTemplatesBuildState.UPDATE
374
375 # INSTALL: Install additional packages needed for this job into
376 # the chroot.
377@@ -103,10 +91,10 @@
378 TranslationTemplatesBuildState.INSTALL, self.getState())
379 expected_command = [
380 '/usr/bin/sudo',
381- '/usr/sbin/chroot', self.chrootdir,
382+ 'sudo', 'chroot', self.chrootdir,
383 'apt-get',
384 ]
385- self.assertEqual(expected_command, self.buildmanager.commands[-1][:4])
386+ self.assertEqual(expected_command, self.buildmanager.commands[-1][:5])
387
388 # GENERATE: Run the slave's payload, the script that generates
389 # templates.
390@@ -114,19 +102,60 @@
391 self.assertEqual(
392 TranslationTemplatesBuildState.GENERATE, self.getState())
393 expected_command = [
394- '/usr/bin/sudo',
395- '/usr/sbin/chroot', self.chrootdir,
396- '/usr/bin/sudo', '-u', self.buildmanager.username,
397- 'generate-translation-templates.py',
398- url,
399+ 'generatepath', 'generatepath', self.buildid, url,
400 ]
401 self.assertEqual(expected_command, self.buildmanager.commands[-1])
402+ self.assertFalse(self.slave.wasCalled('chrootFail'))
403
404- # CLEANUP.
405+ # The control returns to the DebianBuildManager in the REAP state.
406 self.buildmanager.iterate(0)
407- self.assertEqual(
408- TranslationTemplatesBuildState.CLEANUP, self.getState())
409- self.assertEqual('remove-build', self.buildmanager.commands[-1][0])
410+ expected_command = [
411+ 'processscanpath', 'processscanpath', self.buildid
412+ ]
413+ self.assertEqual(
414+ TranslationTemplatesBuildState.REAP, self.getState())
415+ self.assertEqual(expected_command, self.buildmanager.commands[-1])
416+ self.assertFalse(self.slave.wasCalled('buildFail'))
417+
418+ def test_iterate_fail_INSTALL(self):
419+ # See that a failing INSTALL is handled properly.
420+ url = 'lp:~my/branch'
421+ # The build manager's iterate() kicks off the consecutive states
422+ # after INIT.
423+ self.buildmanager.initiate({}, 'chroot.tar.gz', {'branch_url': url})
424+
425+ # Skip states to the INSTALL state.
426+ self.buildmanager._state = TranslationTemplatesBuildState.INSTALL
427+
428+ # The buildmanager fails and iterates to the UMOUNT state.
429+ self.buildmanager.iterate(-1)
430+ self.assertEqual(
431+ TranslationTemplatesBuildState.UMOUNT, self.getState())
432+ expected_command = [
433+ 'umountpath', 'umount-chroot', self.buildid
434+ ]
435+ self.assertEqual(expected_command, self.buildmanager.commands[-1])
436+ self.assertTrue(self.slave.wasCalled('chrootFail'))
437+
438+ def test_iterate_fail_GENERATE(self):
439+ # See that a failing GENERATE is handled properly.
440+ url = 'lp:~my/branch'
441+ # The build manager's iterate() kicks off the consecutive states
442+ # after INIT.
443+ self.buildmanager.initiate({}, 'chroot.tar.gz', {'branch_url': url})
444+
445+ # Skip states to the INSTALL state.
446+ self.buildmanager._state = TranslationTemplatesBuildState.GENERATE
447+
448+ # The buildmanager fails and iterates to the REAP state.
449+ self.buildmanager.iterate(-1)
450+ expected_command = [
451+ 'processscanpath', 'processscanpath', self.buildid
452+ ]
453+ self.assertEqual(
454+ TranslationTemplatesBuildState.REAP, self.getState())
455+ self.assertEqual(expected_command, self.buildmanager.commands[-1])
456+ self.assertTrue(self.slave.wasCalled('buildFail'))
457
458
459 def test_suite():
460
461=== modified file 'lib/canonical/buildd/translationtemplates.py'
462--- lib/canonical/buildd/translationtemplates.py 2010-02-24 18:09:59 +0000
463+++ lib/canonical/buildd/translationtemplates.py 2010-03-11 22:47:24 +0000
464@@ -4,130 +4,38 @@
465 __metaclass__ = type
466
467 import os
468-import pwd
469-
470-from canonical.buildd.slave import BuildManager
471-
472-
473-class TranslationTemplatesBuildState(object):
474- "States this kind of build goes through."""
475- INIT = "INIT"
476- UNPACK = "UNPACK"
477- MOUNT = "MOUNT"
478- UPDATE = "UPDATE"
479+
480+from canonical.buildd.debian import DebianBuildManager, DebianBuildState
481+
482+class TranslationTemplatesBuildState(DebianBuildState):
483 INSTALL = "INSTALL"
484 GENERATE = "GENERATE"
485- CLEANUP = "CLEANUP"
486-
487-
488-class TranslationTemplatesBuildManager(BuildManager):
489+
490+
491+class TranslationTemplatesBuildManager(DebianBuildManager):
492 """Generate translation templates from branch.
493
494 This is the implementation of `TranslationTemplatesBuildJob`. The
495 latter runs on the master server; TranslationTemplatesBuildManager
496 runs on the build slave.
497 """
498- def __init__(self, *args, **kwargs):
499- super(TranslationTemplatesBuildManager, self).__init__(
500- *args, **kwargs)
501- self._state = TranslationTemplatesBuildState.INIT
502- self.alreadyfailed = False
503+
504+ initial_build_state = TranslationTemplatesBuildState.INSTALL
505+
506+ def __init__(self, slave, buildid):
507+ super(TranslationTemplatesBuildManager, self).__init__(slave, buildid)
508+ self._generatepath = slave._config.get(
509+ "translationtemplatesmanager", "generatepath")
510
511 def initiate(self, files, chroot, extra_args):
512 """See `BuildManager`."""
513- self.branch_url = extra_args['branch_url']
514- self.username = pwd.getpwuid(os.getuid())[0]
515+ self._branch_url = extra_args['branch_url']
516+ self._chroot_path = os.path.join(
517+ self.home, 'build-' + self._buildid, 'chroot-autobuild')
518
519 super(TranslationTemplatesBuildManager, self).initiate(
520 files, chroot, extra_args)
521
522- self.chroot_path = os.path.join(
523- self.home, 'build-' + self._buildid, 'chroot-autobuild')
524-
525- def iterate(self, success):
526- func = getattr(self, 'iterate_' + self._state, None)
527- if func is None:
528- raise ValueError("Unknown %s state: %s" % (
529- self.__class__.__name__, self._state))
530- func(success)
531-
532- def iterate_INIT(self, success):
533- """Next step after initialization."""
534- if success == 0:
535- self._state = TranslationTemplatesBuildState.UNPACK
536- self.doUnpack()
537- else:
538- if not self.alreadyfailed:
539- self._slave.builderFail()
540- self.alreadyfailed = True
541- self._state = TranslationTemplatesBuildState.CLEANUP
542- self.build_implementation.doCleanup()
543-
544- def iterate_UNPACK(self, success):
545- if success == 0:
546- self._state = TranslationTemplatesBuildState.MOUNT
547- self.doMounting()
548- else:
549- if not self.alreadyfailed:
550- self._slave.chrootFail()
551- self.alreadyfailed = True
552- self._state = TranslationTemplatesBuildState.CLEANUP
553- self.doCleanup()
554-
555- def iterate_MOUNT(self, success):
556- if success == 0:
557- self._state = TranslationTemplatesBuildState.UPDATE
558- self.doUpdate()
559- else:
560- if not self.alreadyfailed:
561- self._slave.chrootFail()
562- self.alreadyfailed = True
563- self._state = TranslationTemplatesBuildState.CLEANUP
564- self.doCleanup()
565-
566- def iterate_UPDATE(self, success):
567- if success == 0:
568- self._state = TranslationTemplatesBuildState.INSTALL
569- self.doInstall()
570- else:
571- if not self.alreadyfailed:
572- self._slave.chrootFail()
573- self.alreadyfailed = True
574- self._state = TranslationTemplatesBuildState.CLEANUP
575- self.doCleanup()
576-
577- def iterate_INSTALL(self, success):
578- if success == 0:
579- self._state = TranslationTemplatesBuildState.GENERATE
580- self.doGenerate()
581- else:
582- if not self.alreadyfailed:
583- self._slave.chrootFail()
584- self.alreadyfailed = True
585- self._state = TranslationTemplatesBuildState.CLEANUP
586- self.doCleanup()
587-
588- def iterate_GENERATE(self, success):
589- if success == 0:
590- self._state = TranslationTemplatesBuildState.CLEANUP
591- self.doCleanup()
592- else:
593- if not self.alreadyfailed:
594- self._slave.buildFail()
595- self.alreadyfailed = True
596- self._state = TranslationTemplatesBuildState.CLEANUP
597- self.doCleanup()
598-
599- def iterate_CLEANUP(self, success):
600- if success == 0:
601- if not self.alreadyfailed:
602- self._slave.buildOK()
603- else:
604- if not self.alreadyfailed:
605- self._slave.builderFail()
606- self.alreadyfailed = True
607- self._slave.buildComplete()
608-
609 def doInstall(self):
610 """Install packages required."""
611 required_packages = [
612@@ -135,25 +43,38 @@
613 'intltool',
614 ]
615 command = ['apt-get', 'install', '-y'] + required_packages
616- self.runInChroot(self.home, command, as_root=True)
617+ chroot = ['sudo', 'chroot', self._chroot_path]
618+ self.runSubProcess('/usr/bin/sudo', chroot + command)
619
620- def doUpdate(self):
621- """Update chroot."""
622- command = ['update-debian-chroot', self._buildid]
623- self.runInChroot(self.home, command, as_root=True)
624+ # To satisfy DebianPackageManagers needs without having a misleading
625+ # method name here.
626+ doRunBuild = doInstall
627
628 def doGenerate(self):
629 """Generate templates."""
630- command = ['generate-translation-templates.py', self.branch_url]
631- self.runInChroot(self.home, command)
632-
633- def runInChroot(self, path, command, as_root=False):
634- """Run command in chroot."""
635- chroot = ['/usr/bin/sudo', '/usr/sbin/chroot', self.chroot_path]
636- if as_root:
637- sudo = []
638- else:
639- # We have to sudo to chroot, so if the command should _not_
640- # be run as root, we then need to sudo back to who we were.
641- sudo = ['/usr/bin/sudo', '-u', self.username]
642- return self.runSubProcess(path, chroot + sudo + command)
643+ command = [self._generatepath, self._buildid, self._branch_url]
644+ self.runSubProcess(self._generatepath, command)
645+
646+ def iterate_INSTALL(self, success):
647+ """Installation was done."""
648+ if success == 0:
649+ self._state = TranslationTemplatesBuildState.GENERATE
650+ self.doGenerate()
651+ else:
652+ if not self.alreadyfailed:
653+ self._slave.chrootFail()
654+ self.alreadyfailed = True
655+ self._state = TranslationTemplatesBuildState.UMOUNT
656+ self.doUnmounting()
657+
658+ def iterate_GENERATE(self, success):
659+ if success == 0:
660+ self._state = TranslationTemplatesBuildState.REAP
661+ self.doReapProcesses()
662+ else:
663+ if not self.alreadyfailed:
664+ self._slave.buildFail()
665+ self.alreadyfailed = True
666+ self._state = TranslationTemplatesBuildState.REAP
667+ self.doReapProcesses()
668+
669
670=== modified file 'lib/lp/translations/tests/test_pottery_detect_intltool.py'
671--- lib/lp/translations/tests/test_pottery_detect_intltool.py 2010-02-25 13:25:39 +0000
672+++ lib/lp/translations/tests/test_pottery_detect_intltool.py 2010-03-11 22:47:24 +0000
673@@ -11,7 +11,7 @@
674 from bzrlib.bzrdir import BzrDir
675 from canonical.launchpad.scripts.tests import run_script
676 from lp.translations.pottery.detect_intltool import is_intltool_structure
677-from lp.translations.pottery.buildd import (
678+from canonical.buildd.pottery.intltool import (
679 ConfigFile, check_potfiles_in, find_intltool_dirs, find_potfiles_in,
680 generate_pot, generate_pots, get_translation_domain)
681 from lp.testing import TestCase
682
683=== modified file 'scripts/rosetta/pottery-generate-intltool.py'
684--- scripts/rosetta/pottery-generate-intltool.py 2010-02-25 13:25:39 +0000
685+++ scripts/rosetta/pottery-generate-intltool.py 2010-03-11 22:47:24 +0000
686@@ -8,7 +8,7 @@
687 import _pythonpath
688
689
690-from lp.translations.pottery.buildd import generate_pots
691+from canonical.buildd.pottery.intltool import generate_pots
692
693
694 if __name__ == "__main__":