Launchpad itself

Merge lp:~henninge/launchpad/bug-509557-invoke-pottery into lp:launchpad

  1. bug-509557-invoke-pottery
  2. Merge into devel
Proposed by Henning Eggers
Status: Merged
Approved by: Henning Eggers
Approved revision: no longer in the source branch.
Merged at revision: not available
Proposed branch: lp:~henninge/launchpad/bug-509557-invoke-pottery
Merge into: lp:launchpad
Diff against target: 694 lines (+225/-208)
10 files modified
lib/canonical/buildd/debian/changelog (+6/-0)
lib/canonical/buildd/debian/rules (+8/-15)
lib/canonical/buildd/generate-translation-templates (+54/-0)
lib/canonical/buildd/pottery/generate_translation_templates.py (+17/-10)
lib/canonical/buildd/template-buildd-slave.conf (+3/-0)
lib/canonical/buildd/tests/test_generate_translation_templates.py (+10/-6)
lib/canonical/buildd/tests/test_translationtemplatesbuildmanager.py (+77/-48)
lib/canonical/buildd/translationtemplates.py (+48/-127)
lib/lp/translations/tests/test_pottery_detect_intltool.py (+1/-1)
scripts/rosetta/pottery-generate-intltool.py (+1/-1)
To merge this branch: bzr merge lp:~henninge/launchpad/bug-509557-invoke-pottery
Reviewer Review Type Date Requested Status
Paul Hummer (community) code Approve
Review via email: mp+21172@code.launchpad.net

Commit message

Build slave for translation templates generation now mostly working.

Description of the change

= Bug 509557 =

Connects all dots within the build slave to get translation templates build jobs working. It undoes some things done in previous branches, namely:

 - Moves pottery code from lp.translations.pottery into canonical.buildd.pottery. This way it does not need to be copied from there into the buildd package directory by "debian/rules package" anymore. But don't worry, it will still get copied around a lot.

 - The build manager that drives the slave through a state machine is now based on DebianBuildManager because it shares most of it's requirements and thus code. Made the TranslationTemplatesBuildManager much shorter, too.

Another big change is the introduction of the generate-translation-templates script that copies the pottery code into the build chroot and starts the generate_translation_templates.py script in the chroot.

== Implementation details ==

=== modified file 'lib/canonical/buildd/debian/changelog'

The package received a new version number.

=== modified file 'lib/canonical/buildd/debian/rules'

Removed getting the pottery code from the lp tree. Actually, just one new line is left in the file; it copies the pottery directory into the package.

=== added file 'lib/canonical/buildd/generate-translation-templates'

The new shell script that prepares the chroot to run the python script that generates the templates.

=== added directory 'lib/canonical/buildd/pottery'
=== added file 'lib/canonical/buildd/pottery/__init__.py'

The new home for the slave-side pottery code

=== renamed file 'lib/canonical/buildd/generate_translation_templates.py' => 'lib/canonical/buildd/pottery/generate_translation_templates.py'

The script is now part of the package to be copied into the chroot with it. Also it's use of the HOME dir as temporary build directory does not go down well with tests. It's OK on the build slave, though, so I made that setting overridable.

=== renamed file 'lib/lp/translations/pottery/buildd.py' => 'lib/canonical/buildd/pottery/intltool.py'

The file was named to "intltool.py" because it specifically generates templates using intltool. Other tools will get there own module here.

=== modified file 'lib/canonical/buildd/template-buildd-slave.conf'

The TranslationTemplatesBuildManagaer gets its own section like the others to find the generate script to call.

=== modified file 'lib/canonical/buildd/tests/test_generate_translation_templates.py'

New paths and use of temporary directory.

=== modified file 'lib/canonical/buildd/tests/test_translationtemplatesbuildmanager.py'

Oh yeah, big changes here. This is testing the state machine of the build manager. Since the new build manager only implements two states, this test was reduced to test just those. But to get proper coverage I also added tests to check the fail behaviour.

The FakeSlave had to be pimped to react friendly to methods that it doesn't know about. It now even keeps track of those so that the test can see that build manager properly told the slave about a failure. I think that's cool.

=== modified file 'lib/canonical/buildd/translationtemplates.py'

So, this is where it all happens. As mentioned earlier, most of this is deletion because most of the states are already implemented correctly in DebianBuildManager. This adds states for installation of extra packages (bzr and intltool) and calling the generate script.

DebianBuildManager calls doRunBuild to start the build but ist starts with doInstall here before it goes to doGenrate, so I added an alias to doInstall so that DebianBuildManager can find it.

Also, runInChroot was removed because it was flawed and only one usage remains inthe file (in doInstall) which is better done directly now.

I expect another state to be added soon to retrieve the generated templates.

=== modified file 'lib/lp/translations/tests/test_pottery_detect_intltool.py'
=== modified file 'scripts/rosetta/pottery-generate-intltool.py'

New paths for old files.

= Test =

Oh yeah, that ... I tested this code using the steps I described here:
https://dev.launchpad.net/BuildFarm/TryOutBuildSlave

In addition, run the tests mentioned here:

bin/text -vvt generate_translation_templates -t translationtemplatesbuildmanager -t pottery_detect_intltool

= Launchpad lint =

Checking for conflicts. and issues in doctests and templates.
Running jslint, xmllint, pyflakes, and pylint.
Using normal rules.

Linting changed files:
  lib/canonical/buildd/generate-translation-templates
  lib/canonical/buildd/pottery/
  lib/canonical/buildd/template-buildd-slave.conf
  lib/canonical/buildd/translationtemplates.py
  lib/canonical/buildd/debian/changelog
  lib/canonical/buildd/debian/rules
  lib/canonical/buildd/pottery/__init__.py
  lib/canonical/buildd/pottery/generate_translation_templates.py
  lib/canonical/buildd/tests/test_generate_translation_templates.py
  lib/canonical/buildd/tests/test_translationtemplatesbuildmanager.py
  lib/lp/translations/tests/test_pottery_detect_intltool.py
  scripts/rosetta/pottery-generate-intltool.py

== Pylint notices ==

scripts/rosetta/pottery-generate-intltool.py
    8: [W0403] Relative import '_pythonpath'

To post a comment you must log in.
Revision history for this message
Paul Hummer (rockstar) wrote :

Looks good. I'm a little wary of adding new things to lib/canonical but it looks like it's a necessity to getting things done right now, and is not adding any more technical debt.

review: Approve (code)
Revision history for this message
Henning Eggers (henninge) wrote :

Thanks a lot, Paul. I expect that someday lib/canonical/buildd will move somewhere else and thus take this code with it.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/canonical/buildd/debian/changelog'
--- lib/canonical/buildd/debian/changelog 2010-01-14 21:05:08 +0000
+++ lib/canonical/buildd/debian/changelog 2010-03-11 22:47:24 +0000
@@ -1,3 +1,9 @@
1launchpad-buildd (59) karmic; urgency=low
2
3 * Added translation template generation code (pottery).
4
5 -- Henning Eggers <henning@canonical.com> Thu, 11 Mar 2010 12:59:25 +0100
6
1launchpad-buildd (58~1) karmic; urgency=low7launchpad-buildd (58~1) karmic; urgency=low
28
3 * Misc fixes to match APIs.9 * Misc fixes to match APIs.
410
=== modified file 'lib/canonical/buildd/debian/rules'
--- lib/canonical/buildd/debian/rules 2010-02-25 16:49:51 +0000
+++ lib/canonical/buildd/debian/rules 2010-03-11 22:47:24 +0000
@@ -21,10 +21,11 @@
21targetshare = $(target)/usr/share/launchpad-buildd21targetshare = $(target)/usr/share/launchpad-buildd
22pytarget = $(targetshare)/canonical/buildd22pytarget = $(targetshare)/canonical/buildd
2323
24pyfiles = debian.py slave.py binarypackage.py utils.py __init__.py sourcepackagerecipe.py translationtemplates.py24pyfiles = debian.py slave.py binarypackage.py utils.py __init__.py \
25 sourcepackagerecipe.py translationtemplates.py
25slavebins = unpack-chroot mount-chroot update-debian-chroot sbuild-package \26slavebins = unpack-chroot mount-chroot update-debian-chroot sbuild-package \
26scan-for-processes umount-chroot remove-build apply-ogre-model \27scan-for-processes umount-chroot remove-build apply-ogre-model \
27override-sources-list buildrecipe generate_translation_templates.py28override-sources-list buildrecipe generate-translation-templates
2829
29BUILDDUID=6550030BUILDDUID=65500
30BUILDDGID=6550031BUILDDGID=65500
@@ -39,26 +40,19 @@
39 var/run/launchpad-buildd var/log/launchpad-buildd \40 var/run/launchpad-buildd var/log/launchpad-buildd \
40 etc/launchpad-buildd \41 etc/launchpad-buildd \
41 usr/share/launchpad-buildd/canonical/launchpad/daemons \42 usr/share/launchpad-buildd/canonical/launchpad/daemons \
42 usr/share/doc/launchpad-buildd \43 usr/share/doc/launchpad-buildd
43 usr/share/launchpad-buildd/lp \
44 usr/share/launchpad-buildd/lp/translations \
45 usr/share/launchpad-buildd/lp/translations/pottery
4644
47 # Do installs here45 # Do installs here
48 touch $(pytarget)/../launchpad/__init__.py46 touch $(pytarget)/../launchpad/__init__.py
49 touch $(pytarget)/../launchpad/daemons/__init__.py47 touch $(pytarget)/../launchpad/daemons/__init__.py
50 cp launchpad-files/tachandler.py $(pytarget)/../launchpad/daemons/48 cp launchpad-files/tachandler.py $(pytarget)/../launchpad/daemons/
51 install -m644 launchpad-files/buildd-slave.tac $(targetshare)/buildd-slave.tac49 install -m644 launchpad-files/buildd-slave.tac $(targetshare)/buildd-slave.tac
52 touch $(pytarget)/../../lp/__init__.py50 cp -r pottery $(pytarget)
53 touch $(pytarget)/../../lp/translations/__init__.py
54 touch $(pytarget)/../../lp/translations/pottery/__init__.py
55 cp launchpad-files/pottery_buildd.py \
56 $(pytarget)/../../lp/translations/pottery/buildd.py
57 for pyfile in $(pyfiles); do \51 for pyfile in $(pyfiles); do \
58 install -m644 ./$$pyfile $(pytarget)/$$pyfile; \52 install -m644 ./$$pyfile $(pytarget)/$$pyfile; \
59 done53 done
60 for slavebin in $(slavebins); do \54 for slavebin in $(slavebins); do \
61 install -m755 ./$$slavebin $(targetshare)/slavebin/$$slavebin; \55 install -m755 ./$$slavebin $(targetshare)/slavebin/$$slavebin; \
62 done56 done
63 install -m755 ./sbuild $(target)/usr/bin/sbuild57 install -m755 ./sbuild $(target)/usr/bin/sbuild
64 touch $(targetshare)/canonical/__init__.py58 touch $(targetshare)/canonical/__init__.py
@@ -100,7 +94,6 @@
100 mkdir -p launchpad-files94 mkdir -p launchpad-files
101 install -m644 $(daemons)/buildd-slave.tac launchpad-files/buildd-slave.tac95 install -m644 $(daemons)/buildd-slave.tac launchpad-files/buildd-slave.tac
102 cp ../launchpad/daemons/tachandler.py launchpad-files/tachandler.py96 cp ../launchpad/daemons/tachandler.py launchpad-files/tachandler.py
103 cp ../../lp/translations/pottery/buildd.py launchpad-files/pottery_buildd.py
104 debuild -uc -us -S97 debuild -uc -us -S
10598
106build:99build:
107100
=== added file 'lib/canonical/buildd/generate-translation-templates'
--- lib/canonical/buildd/generate-translation-templates 1970-01-01 00:00:00 +0000
+++ lib/canonical/buildd/generate-translation-templates 2010-03-11 22:47:24 +0000
@@ -0,0 +1,54 @@
1#!/bin/sh
2#
3# Copyright 2010 Canonical Ltd. This software is licensed under the
4# GNU Affero General Public License version 3 (see the file LICENSE).
5#
6# Author: Henning Eggers <henning@canonical.com>
7
8# Buildd Slave tool to generate translation templates. Boiler plate code
9# copied from sbuild-package.
10
11# Expects build id as arg 1.
12# Expects branch url as arg 2.
13
14# Must run as user with password-less sudo ability.
15
16exec 2>&1
17
18export LANG=C LC_ALL=C
19
20CHMOD=/bin/chmod
21CHROOT=/usr/sbin/chroot
22CP=/bin/cp
23INSTALL=/usr/bin/install
24MKDIR=/bin/mkdir
25SU=/bin/su
26SUDO=/usr/bin/sudo
27TOUCH=/usr/bin/touch
28
29BUILDID=$1
30BRANCH_URL=$2
31
32BUILDD_HOME=/usr/share/launchpad-buildd
33SLAVEBIN=$BUILDD_HOME/slavebin
34BUILD_CHROOT="$HOME/build-$BUILDID/chroot-autobuild"
35USER=$(whoami)
36
37BUILDD_PACKAGE=canonical/buildd
38POTTERY=$BUILDD_PACKAGE/pottery
39# The script should be smarter about detecting the python version.
40PYMODULES=/usr/lib/pymodules/python2.6
41
42GENERATE_SCRIPT=$PYMODULES/$POTTERY/generate_translation_templates.py
43
44# Copy pottery files to chroot.
45$SUDO $MKDIR -p $BUILD_CHROOT$PYMODULES/$BUILDD_PACKAGE
46$SUDO $TOUCH $BUILD_CHROOT$PYMODULES/canonical/__init__.py
47$SUDO $TOUCH $BUILD_CHROOT$PYMODULES/canonical/buildd/__init__.py
48$SUDO $CP -r $BUILDD_HOME/$POTTERY $BUILD_CHROOT$PYMODULES/$BUILDD_PACKAGE
49$SUDO $CHMOD -R go+rX $BUILD_CHROOT$PYMODULES/canonical
50$SUDO $CHMOD 755 $BUILD_CHROOT$GENERATE_SCRIPT
51
52
53# Enter chroot, switch back to unprivileged user, execute the generate script.
54$SUDO $CHROOT $BUILD_CHROOT $SU - $USER -c "$GENERATE_SCRIPT $BRANCH_URL"
055
=== added directory 'lib/canonical/buildd/pottery'
=== added file 'lib/canonical/buildd/pottery/__init__.py'
=== renamed file 'lib/canonical/buildd/generate_translation_templates.py' => 'lib/canonical/buildd/pottery/generate_translation_templates.py'
--- lib/canonical/buildd/generate_translation_templates.py 2010-03-04 08:12:25 +0000
+++ lib/canonical/buildd/pottery/generate_translation_templates.py 2010-03-11 22:47:24 +0000
@@ -1,4 +1,4 @@
1#! /usr/bin/python2.51#! /usr/bin/python
2# Copyright 2010 Canonical Ltd. This software is licensed under the2# Copyright 2010 Canonical Ltd. This software is licensed under the
3# GNU Affero General Public License version 3 (see the file LICENSE).3# GNU Affero General Public License version 3 (see the file LICENSE).
44
@@ -10,28 +10,29 @@
10from bzrlib.branch import Branch10from bzrlib.branch import Branch
11from bzrlib.export import export11from bzrlib.export import export
1212
13from lp.translations.pottery.buildd import generate_pots13from canonical.buildd.pottery import intltool
1414
1515
16class GenerateTranslationTemplates:16class GenerateTranslationTemplates:
17 """Script to generate translation templates from a branch."""17 """Script to generate translation templates from a branch."""
1818
19 def __init__(self, branch_spec):19 def __init__(self, branch_spec, work_dir):
20 """Prepare to generate templates for a branch.20 """Prepare to generate templates for a branch.
2121
22 :param branch_spec: Either a branch URL or the path of a local22 :param branch_spec: Either a branch URL or the path of a local
23 branch. URLs are recognized by the occurrence of ':'. In23 branch. URLs are recognized by the occurrence of ':'. In
24 the case of a URL, this will make up a path for the branch24 the case of a URL, this will make up a path for the branch
25 and check out the branch to there.25 and check out the branch to there.
26 :param work_dir: The directory to work in. Must exist.
26 """27 """
27 self.home = os.environ['HOME']28 self.work_dir = work_dir
28 self.branch_spec = branch_spec29 self.branch_spec = branch_spec
2930
30 def _getBranch(self):31 def _getBranch(self):
31 """Set `self.branch_dir`, and check out branch if needed."""32 """Set `self.branch_dir`, and check out branch if needed."""
32 if ':' in self.branch_spec:33 if ':' in self.branch_spec:
33 # This is a branch URL. Check out the branch.34 # This is a branch URL. Check out the branch.
34 self.branch_dir = os.path.join(self.home, 'source-tree')35 self.branch_dir = os.path.join(self.work_dir, 'source-tree')
35 self._checkout(self.branch_spec)36 self._checkout(self.branch_spec)
36 else:37 else:
37 # This is a local filesystem path. Use the branch in-place.38 # This is a local filesystem path. Use the branch in-place.
@@ -50,7 +51,7 @@
50 def generate(self):51 def generate(self):
51 """Do It. Generate templates."""52 """Do It. Generate templates."""
52 self._getBranch()53 self._getBranch()
53 pots = generate_pots(self.branch_dir)54 pots = intltool.generate_pots(self.branch_dir)
54 print "\n".join(55 print "\n".join(
55 [os.path.normpath(os.path.join(self.branch_dir, potpath))56 [os.path.normpath(os.path.join(self.branch_dir, potpath))
56 for potpath in pots])57 for potpath in pots])
@@ -58,8 +59,14 @@
5859
5960
60if __name__ == '__main__':61if __name__ == '__main__':
61 if len(sys.argv) != 2:62 if len(sys.argv) < 2:
62 print "Usage: %s branch" % sys.argv[0]63 print "Usage: %s branch [workdir]" % sys.argv[0]
63 print "Where 'branch' is a branch URL or directory."64 print " 'branch' is a branch URL or directory."
65 print " 'workdir' is a directory, defaults to HOME."
64 sys.exit(1)66 sys.exit(1)
65 sys.exit(GenerateTranslationTemplates(sys.argv[1]).generate())67 if len(sys.argv) == 3:
68 workdir = sys.argv[2]
69 else:
70 workdir = os.environ['HOME']
71 script = GenerateTranslationTemplates(sys.argv[1], workdir)
72 sys.exit(script.generate())
6673
=== renamed file 'lib/lp/translations/pottery/buildd.py' => 'lib/canonical/buildd/pottery/intltool.py'
=== modified file 'lib/canonical/buildd/template-buildd-slave.conf'
--- lib/canonical/buildd/template-buildd-slave.conf 2010-01-13 05:26:33 +0000
+++ lib/canonical/buildd/template-buildd-slave.conf 2010-03-11 22:47:24 +0000
@@ -27,3 +27,6 @@
2727
28[sourcepackagerecipemanager]28[sourcepackagerecipemanager]
29buildrecipepath = /usr/share/launchpad-buildd/slavebin/buildrecipe29buildrecipepath = /usr/share/launchpad-buildd/slavebin/buildrecipe
30
31[translationtemplatesmanager]
32generatepath = /usr/share/launchpad-buildd/slavebin/generate-translation-templates
3033
=== modified file 'lib/canonical/buildd/tests/test_generate_translation_templates.py'
--- lib/canonical/buildd/tests/test_generate_translation_templates.py 2010-03-04 08:12:25 +0000
+++ lib/canonical/buildd/tests/test_generate_translation_templates.py 2010-03-11 22:47:24 +0000
@@ -6,7 +6,7 @@
66
7from lp.testing.fakemethod import FakeMethod7from lp.testing.fakemethod import FakeMethod
88
9from canonical.buildd.generate_translation_templates import (9from canonical.buildd.pottery.generate_translation_templates import (
10 GenerateTranslationTemplates)10 GenerateTranslationTemplates)
1111
12from canonical.launchpad.ftests.script import run_script12from canonical.launchpad.ftests.script import run_script
@@ -24,7 +24,8 @@
24 # check out that branch into a directory called "source-tree."24 # check out that branch into a directory called "source-tree."
25 branch_url = 'lp://~my/translation/branch'25 branch_url = 'lp://~my/translation/branch'
2626
27 generator = GenerateTranslationTemplates(branch_url)27 generator = GenerateTranslationTemplates(
28 branch_url, self.makeTemporaryDirectory())
28 generator._checkout = FakeMethod()29 generator._checkout = FakeMethod()
29 generator._getBranch()30 generator._getBranch()
3031
@@ -36,7 +37,8 @@
36 # works directly in that directory.37 # works directly in that directory.
37 branch_dir = '/home/me/branch'38 branch_dir = '/home/me/branch'
3839
39 generator = GenerateTranslationTemplates(branch_dir)40 generator = GenerateTranslationTemplates(
41 branch_dir, self.makeTemporaryDirectory())
40 generator._checkout = FakeMethod()42 generator._checkout = FakeMethod()
41 generator._getBranch()43 generator._getBranch()
4244
@@ -71,7 +73,8 @@
71 branch = self._createBranch({'marker.txt': marker_text})73 branch = self._createBranch({'marker.txt': marker_text})
72 branch_url = branch.getPullURL()74 branch_url = branch.getPullURL()
7375
74 generator = GenerateTranslationTemplates(branch_url)76 generator = GenerateTranslationTemplates(
77 branch_url, self.makeTemporaryDirectory())
75 generator.branch_dir = self.makeTemporaryDirectory()78 generator.branch_dir = self.makeTemporaryDirectory()
76 generator._getBranch()79 generator._getBranch()
7780
@@ -80,9 +83,10 @@
8083
81 def test_script(self):84 def test_script(self):
82 tempdir = self.makeTemporaryDirectory()85 tempdir = self.makeTemporaryDirectory()
86 workdir = self.makeTemporaryDirectory()
83 (retval, out, err) = run_script(87 (retval, out, err) = run_script(
84 'lib/canonical/buildd/generate_translation_templates.py',88 'lib/canonical/buildd/pottery/generate_translation_templates.py',
85 args=[tempdir])89 args=[tempdir, workdir])
86 self.assertEqual(0, retval)90 self.assertEqual(0, retval)
8791
8892
8993
=== modified file 'lib/canonical/buildd/tests/test_translationtemplatesbuildmanager.py'
--- lib/canonical/buildd/tests/test_translationtemplatesbuildmanager.py 2010-01-21 12:34:41 +0000
+++ lib/canonical/buildd/tests/test_translationtemplatesbuildmanager.py 2010-03-11 22:47:24 +0000
@@ -22,10 +22,26 @@
22 def __init__(self, tempdir):22 def __init__(self, tempdir):
23 self._cachepath = tempdir23 self._cachepath = tempdir
24 self._config = FakeConfig()24 self._config = FakeConfig()
25 self._was_called = set()
2526
26 def cachePath(self, file):27 def cachePath(self, file):
27 return os.path.join(self._cachepath, file)28 return os.path.join(self._cachepath, file)
2829
30 def anyMethod(self, *args, **kwargs):
31 pass
32
33 fake_methods = ['emptyLog', 'chrootFail', 'buildFail', 'builderFail',]
34 def __getattr__(self, name):
35 """Remember which fake methods were called."""
36 if name not in self.fake_methods:
37 raise AttributeError(
38 "'%s' object has no attribute '%s'" % (self.__class__, name))
39 self._was_called.add(name)
40 return self.anyMethod
41
42 def wasCalled(self, name):
43 return name in self._was_called
44
2945
30class MockBuildManager(TranslationTemplatesBuildManager):46class MockBuildManager(TranslationTemplatesBuildManager):
31 def __init__(self, *args, **kwargs):47 def __init__(self, *args, **kwargs):
@@ -33,7 +49,7 @@
33 self.commands = []49 self.commands = []
3450
35 def runSubProcess(self, path, command):51 def runSubProcess(self, path, command):
36 self.commands.append(command)52 self.commands.append([path]+command)
37 return 053 return 0
3854
3955
@@ -46,55 +62,27 @@
46 home_dir = os.path.join(self.working_dir, 'home')62 home_dir = os.path.join(self.working_dir, 'home')
47 for dir in (slave_dir, home_dir):63 for dir in (slave_dir, home_dir):
48 os.mkdir(dir)64 os.mkdir(dir)
49 slave = FakeSlave(slave_dir)65 self.slave = FakeSlave(slave_dir)
50 buildid = '123'66 self.buildid = '123'
51 self.buildmanager = MockBuildManager(slave, buildid)67 self.buildmanager = MockBuildManager(self.slave, self.buildid)
52 self.buildmanager.home = home_dir68 self.buildmanager.home = home_dir
53 self.chrootdir = os.path.join(69 self.chrootdir = os.path.join(
54 home_dir, 'build-%s' % buildid, 'chroot-autobuild')70 home_dir, 'build-%s' % self.buildid, 'chroot-autobuild')
5571
56 def getState(self):72 def getState(self):
57 """Retrieve build manager's state."""73 """Retrieve build manager's state."""
58 return self.buildmanager._state74 return self.buildmanager._state
5975
60 def test_initiate(self):
61 # Creating a BuildManager spawns no child processes.
62 self.assertEqual([], self.buildmanager.commands)
63
64 # Initiating the build executes the first command. It leaves
65 # the build manager in the INIT state.
66 self.buildmanager.initiate({}, 'chroot.tar.gz', {'branch_url': 'foo'})
67 self.assertEqual(1, len(self.buildmanager.commands))
68 self.assertEqual(TranslationTemplatesBuildState.INIT, self.getState())
69
70 def test_iterate(self):76 def test_iterate(self):
77 # Two iteration steps are specific to this build manager.
71 url = 'lp:~my/branch'78 url = 'lp:~my/branch'
72 # The build manager's iterate() kicks off the consecutive states79 # The build manager's iterate() kicks off the consecutive states
73 # after INIT.80 # after INIT.
74 self.buildmanager.initiate({}, 'chroot.tar.gz', {'branch_url': url})81 self.buildmanager.initiate({}, 'chroot.tar.gz', {'branch_url': url})
7582
76 # UNPACK: execute unpack-chroot.83 # Skip states that are done in DebianBuldManager to the state
77 self.buildmanager.iterate(0)84 # directly before INSTALL.
78 self.assertEqual(85 self.buildmanager._state = TranslationTemplatesBuildState.UPDATE
79 TranslationTemplatesBuildState.UNPACK, self.getState())
80 self.assertEqual('unpack-chroot', self.buildmanager.commands[-1][0])
81
82 # MOUNT: Set up realistic chroot environment.
83 self.buildmanager.iterate(0)
84 self.assertEqual(
85 TranslationTemplatesBuildState.MOUNT, self.getState())
86 self.assertEqual('mount-chroot', self.buildmanager.commands[-1][0])
87
88 # UPDATE: Get the latest versions of installed packages.
89 self.buildmanager.iterate(0)
90 self.assertEqual(
91 TranslationTemplatesBuildState.UPDATE, self.getState())
92 expected_command = [
93 '/usr/bin/sudo',
94 '/usr/sbin/chroot', self.chrootdir,
95 'update-debian-chroot',
96 ]
97 self.assertEqual(expected_command, self.buildmanager.commands[-1][:4])
9886
99 # INSTALL: Install additional packages needed for this job into87 # INSTALL: Install additional packages needed for this job into
100 # the chroot.88 # the chroot.
@@ -103,10 +91,10 @@
103 TranslationTemplatesBuildState.INSTALL, self.getState())91 TranslationTemplatesBuildState.INSTALL, self.getState())
104 expected_command = [92 expected_command = [
105 '/usr/bin/sudo',93 '/usr/bin/sudo',
106 '/usr/sbin/chroot', self.chrootdir,94 'sudo', 'chroot', self.chrootdir,
107 'apt-get',95 'apt-get',
108 ]96 ]
109 self.assertEqual(expected_command, self.buildmanager.commands[-1][:4])97 self.assertEqual(expected_command, self.buildmanager.commands[-1][:5])
11098
111 # GENERATE: Run the slave's payload, the script that generates99 # GENERATE: Run the slave's payload, the script that generates
112 # templates.100 # templates.
@@ -114,19 +102,60 @@
114 self.assertEqual(102 self.assertEqual(
115 TranslationTemplatesBuildState.GENERATE, self.getState())103 TranslationTemplatesBuildState.GENERATE, self.getState())
116 expected_command = [104 expected_command = [
117 '/usr/bin/sudo',105 'generatepath', 'generatepath', self.buildid, url,
118 '/usr/sbin/chroot', self.chrootdir,
119 '/usr/bin/sudo', '-u', self.buildmanager.username,
120 'generate-translation-templates.py',
121 url,
122 ]106 ]
123 self.assertEqual(expected_command, self.buildmanager.commands[-1])107 self.assertEqual(expected_command, self.buildmanager.commands[-1])
108 self.assertFalse(self.slave.wasCalled('chrootFail'))
124109
125 # CLEANUP.110 # The control returns to the DebianBuildManager in the REAP state.
126 self.buildmanager.iterate(0)111 self.buildmanager.iterate(0)
127 self.assertEqual(112 expected_command = [
128 TranslationTemplatesBuildState.CLEANUP, self.getState())113 'processscanpath', 'processscanpath', self.buildid
129 self.assertEqual('remove-build', self.buildmanager.commands[-1][0])114 ]
115 self.assertEqual(
116 TranslationTemplatesBuildState.REAP, self.getState())
117 self.assertEqual(expected_command, self.buildmanager.commands[-1])
118 self.assertFalse(self.slave.wasCalled('buildFail'))
119
120 def test_iterate_fail_INSTALL(self):
121 # See that a failing INSTALL is handled properly.
122 url = 'lp:~my/branch'
123 # The build manager's iterate() kicks off the consecutive states
124 # after INIT.
125 self.buildmanager.initiate({}, 'chroot.tar.gz', {'branch_url': url})
126
127 # Skip states to the INSTALL state.
128 self.buildmanager._state = TranslationTemplatesBuildState.INSTALL
129
130 # The buildmanager fails and iterates to the UMOUNT state.
131 self.buildmanager.iterate(-1)
132 self.assertEqual(
133 TranslationTemplatesBuildState.UMOUNT, self.getState())
134 expected_command = [
135 'umountpath', 'umount-chroot', self.buildid
136 ]
137 self.assertEqual(expected_command, self.buildmanager.commands[-1])
138 self.assertTrue(self.slave.wasCalled('chrootFail'))
139
140 def test_iterate_fail_GENERATE(self):
141 # See that a failing GENERATE is handled properly.
142 url = 'lp:~my/branch'
143 # The build manager's iterate() kicks off the consecutive states
144 # after INIT.
145 self.buildmanager.initiate({}, 'chroot.tar.gz', {'branch_url': url})
146
147 # Skip states to the INSTALL state.
148 self.buildmanager._state = TranslationTemplatesBuildState.GENERATE
149
150 # The buildmanager fails and iterates to the REAP state.
151 self.buildmanager.iterate(-1)
152 expected_command = [
153 'processscanpath', 'processscanpath', self.buildid
154 ]
155 self.assertEqual(
156 TranslationTemplatesBuildState.REAP, self.getState())
157 self.assertEqual(expected_command, self.buildmanager.commands[-1])
158 self.assertTrue(self.slave.wasCalled('buildFail'))
130159
131160
132def test_suite():161def test_suite():
133162
=== modified file 'lib/canonical/buildd/translationtemplates.py'
--- lib/canonical/buildd/translationtemplates.py 2010-02-24 18:09:59 +0000
+++ lib/canonical/buildd/translationtemplates.py 2010-03-11 22:47:24 +0000
@@ -4,130 +4,38 @@
4__metaclass__ = type4__metaclass__ = type
55
6import os6import os
7import pwd7
88from canonical.buildd.debian import DebianBuildManager, DebianBuildState
9from canonical.buildd.slave import BuildManager9
1010class TranslationTemplatesBuildState(DebianBuildState):
11
12class TranslationTemplatesBuildState(object):
13 "States this kind of build goes through."""
14 INIT = "INIT"
15 UNPACK = "UNPACK"
16 MOUNT = "MOUNT"
17 UPDATE = "UPDATE"
18 INSTALL = "INSTALL"11 INSTALL = "INSTALL"
19 GENERATE = "GENERATE"12 GENERATE = "GENERATE"
20 CLEANUP = "CLEANUP"13
2114
2215class TranslationTemplatesBuildManager(DebianBuildManager):
23class TranslationTemplatesBuildManager(BuildManager):
24 """Generate translation templates from branch.16 """Generate translation templates from branch.
2517
26 This is the implementation of `TranslationTemplatesBuildJob`. The18 This is the implementation of `TranslationTemplatesBuildJob`. The
27 latter runs on the master server; TranslationTemplatesBuildManager19 latter runs on the master server; TranslationTemplatesBuildManager
28 runs on the build slave.20 runs on the build slave.
29 """21 """
30 def __init__(self, *args, **kwargs):22
31 super(TranslationTemplatesBuildManager, self).__init__(23 initial_build_state = TranslationTemplatesBuildState.INSTALL
32 *args, **kwargs)24
33 self._state = TranslationTemplatesBuildState.INIT25 def __init__(self, slave, buildid):
34 self.alreadyfailed = False26 super(TranslationTemplatesBuildManager, self).__init__(slave, buildid)
27 self._generatepath = slave._config.get(
28 "translationtemplatesmanager", "generatepath")
3529
36 def initiate(self, files, chroot, extra_args):30 def initiate(self, files, chroot, extra_args):
37 """See `BuildManager`."""31 """See `BuildManager`."""
38 self.branch_url = extra_args['branch_url']32 self._branch_url = extra_args['branch_url']
39 self.username = pwd.getpwuid(os.getuid())[0]33 self._chroot_path = os.path.join(
34 self.home, 'build-' + self._buildid, 'chroot-autobuild')
4035
41 super(TranslationTemplatesBuildManager, self).initiate(36 super(TranslationTemplatesBuildManager, self).initiate(
42 files, chroot, extra_args)37 files, chroot, extra_args)
4338
44 self.chroot_path = os.path.join(
45 self.home, 'build-' + self._buildid, 'chroot-autobuild')
46
47 def iterate(self, success):
48 func = getattr(self, 'iterate_' + self._state, None)
49 if func is None:
50 raise ValueError("Unknown %s state: %s" % (
51 self.__class__.__name__, self._state))
52 func(success)
53
54 def iterate_INIT(self, success):
55 """Next step after initialization."""
56 if success == 0:
57 self._state = TranslationTemplatesBuildState.UNPACK
58 self.doUnpack()
59 else:
60 if not self.alreadyfailed:
61 self._slave.builderFail()
62 self.alreadyfailed = True
63 self._state = TranslationTemplatesBuildState.CLEANUP
64 self.build_implementation.doCleanup()
65
66 def iterate_UNPACK(self, success):
67 if success == 0:
68 self._state = TranslationTemplatesBuildState.MOUNT
69 self.doMounting()
70 else:
71 if not self.alreadyfailed:
72 self._slave.chrootFail()
73 self.alreadyfailed = True
74 self._state = TranslationTemplatesBuildState.CLEANUP
75 self.doCleanup()
76
77 def iterate_MOUNT(self, success):
78 if success == 0:
79 self._state = TranslationTemplatesBuildState.UPDATE
80 self.doUpdate()
81 else:
82 if not self.alreadyfailed:
83 self._slave.chrootFail()
84 self.alreadyfailed = True
85 self._state = TranslationTemplatesBuildState.CLEANUP
86 self.doCleanup()
87
88 def iterate_UPDATE(self, success):
89 if success == 0:
90 self._state = TranslationTemplatesBuildState.INSTALL
91 self.doInstall()
92 else:
93 if not self.alreadyfailed:
94 self._slave.chrootFail()
95 self.alreadyfailed = True
96 self._state = TranslationTemplatesBuildState.CLEANUP
97 self.doCleanup()
98
99 def iterate_INSTALL(self, success):
100 if success == 0:
101 self._state = TranslationTemplatesBuildState.GENERATE
102 self.doGenerate()
103 else:
104 if not self.alreadyfailed:
105 self._slave.chrootFail()
106 self.alreadyfailed = True
107 self._state = TranslationTemplatesBuildState.CLEANUP
108 self.doCleanup()
109
110 def iterate_GENERATE(self, success):
111 if success == 0:
112 self._state = TranslationTemplatesBuildState.CLEANUP
113 self.doCleanup()
114 else:
115 if not self.alreadyfailed:
116 self._slave.buildFail()
117 self.alreadyfailed = True
118 self._state = TranslationTemplatesBuildState.CLEANUP
119 self.doCleanup()
120
121 def iterate_CLEANUP(self, success):
122 if success == 0:
123 if not self.alreadyfailed:
124 self._slave.buildOK()
125 else:
126 if not self.alreadyfailed:
127 self._slave.builderFail()
128 self.alreadyfailed = True
129 self._slave.buildComplete()
130
131 def doInstall(self):39 def doInstall(self):
132 """Install packages required."""40 """Install packages required."""
133 required_packages = [41 required_packages = [
@@ -135,25 +43,38 @@
135 'intltool',43 'intltool',
136 ]44 ]
137 command = ['apt-get', 'install', '-y'] + required_packages45 command = ['apt-get', 'install', '-y'] + required_packages
138 self.runInChroot(self.home, command, as_root=True)46 chroot = ['sudo', 'chroot', self._chroot_path]
47 self.runSubProcess('/usr/bin/sudo', chroot + command)
13948
140 def doUpdate(self):49 # To satisfy DebianPackageManagers needs without having a misleading
141 """Update chroot."""50 # method name here.
142 command = ['update-debian-chroot', self._buildid]51 doRunBuild = doInstall
143 self.runInChroot(self.home, command, as_root=True)
14452
145 def doGenerate(self):53 def doGenerate(self):
146 """Generate templates."""54 """Generate templates."""
147 command = ['generate-translation-templates.py', self.branch_url]55 command = [self._generatepath, self._buildid, self._branch_url]
148 self.runInChroot(self.home, command)56 self.runSubProcess(self._generatepath, command)
14957
150 def runInChroot(self, path, command, as_root=False):58 def iterate_INSTALL(self, success):
151 """Run command in chroot."""59 """Installation was done."""
152 chroot = ['/usr/bin/sudo', '/usr/sbin/chroot', self.chroot_path]60 if success == 0:
153 if as_root:61 self._state = TranslationTemplatesBuildState.GENERATE
154 sudo = []62 self.doGenerate()
155 else:63 else:
156 # We have to sudo to chroot, so if the command should _not_64 if not self.alreadyfailed:
157 # be run as root, we then need to sudo back to who we were.65 self._slave.chrootFail()
158 sudo = ['/usr/bin/sudo', '-u', self.username]66 self.alreadyfailed = True
159 return self.runSubProcess(path, chroot + sudo + command)67 self._state = TranslationTemplatesBuildState.UMOUNT
68 self.doUnmounting()
69
70 def iterate_GENERATE(self, success):
71 if success == 0:
72 self._state = TranslationTemplatesBuildState.REAP
73 self.doReapProcesses()
74 else:
75 if not self.alreadyfailed:
76 self._slave.buildFail()
77 self.alreadyfailed = True
78 self._state = TranslationTemplatesBuildState.REAP
79 self.doReapProcesses()
80
16081
=== modified file 'lib/lp/translations/tests/test_pottery_detect_intltool.py'
--- lib/lp/translations/tests/test_pottery_detect_intltool.py 2010-02-25 13:25:39 +0000
+++ lib/lp/translations/tests/test_pottery_detect_intltool.py 2010-03-11 22:47:24 +0000
@@ -11,7 +11,7 @@
11from bzrlib.bzrdir import BzrDir11from bzrlib.bzrdir import BzrDir
12from canonical.launchpad.scripts.tests import run_script12from canonical.launchpad.scripts.tests import run_script
13from lp.translations.pottery.detect_intltool import is_intltool_structure13from lp.translations.pottery.detect_intltool import is_intltool_structure
14from lp.translations.pottery.buildd import (14from canonical.buildd.pottery.intltool import (
15 ConfigFile, check_potfiles_in, find_intltool_dirs, find_potfiles_in,15 ConfigFile, check_potfiles_in, find_intltool_dirs, find_potfiles_in,
16 generate_pot, generate_pots, get_translation_domain)16 generate_pot, generate_pots, get_translation_domain)
17from lp.testing import TestCase17from lp.testing import TestCase
1818
=== modified file 'scripts/rosetta/pottery-generate-intltool.py'
--- scripts/rosetta/pottery-generate-intltool.py 2010-02-25 13:25:39 +0000
+++ scripts/rosetta/pottery-generate-intltool.py 2010-03-11 22:47:24 +0000
@@ -8,7 +8,7 @@
8import _pythonpath8import _pythonpath
99
1010
11from lp.translations.pottery.buildd import generate_pots11from canonical.buildd.pottery.intltool import generate_pots
1212
1313
14if __name__ == "__main__":14if __name__ == "__main__":