launchpad-buildd

Merge lp:~cjwatson/launchpad-buildd/charm into lp:launchpad-buildd

  1. charm
  2. Merge into trunk
Proposed by Colin Watson
Status: Rejected
Rejected by: Colin Watson
Proposed branch: lp:~cjwatson/launchpad-buildd/charm
Merge into: lp:launchpad-buildd
Diff against target: 324 lines (+273/-0)
8 files modified
.bzrignore (+2/-0)
charm/Makefile (+77/-0)
charm/README.md (+59/-0)
charm/config.yaml (+3/-0)
charm/layer.yaml (+13/-0)
charm/metadata.yaml (+21/-0)
charm/reactive/launchpad-buildd.py (+96/-0)
debian/changelog (+2/-0)
To merge this branch: bzr merge lp:~cjwatson/launchpad-buildd/charm
Reviewer Review Type Date Requested Status
Launchpad code reviewers Pending
Review via email: mp+312468@code.launchpad.net

Commit message

Add a Juju charm which can be used to deploy local launchpad-buildd instances.

To post a comment you must log in.
Revision history for this message
Colin Watson (cjwatson) wrote :

Superseded by https://code.launchpad.net/~cjwatson/launchpad-buildd/+git/launchpad-buildd/+merge/383062.

Unmerged revisions

213. By Colin Watson

Split out push and push-with-packages targets, since we don't need to push (dummy) resources every time.

212. By Colin Watson

Improve handling of attaching resources after initial deployment.

211. By Colin Watson

Add "push" target.

210. By Colin Watson

Unhold packages before installing them.

209. By Colin Watson

Set ntphost to the empty string rather than removing it entirely, to avoid confusing the postinst.

208. By Colin Watson

Treat zero-sized resources as missing, since the charm store doesn't support truly-optional resources.

207. By Colin Watson

Avoid formatting glitch.

206. By Colin Watson

Add a Juju charm which can be used to deploy local launchpad-buildd
instances.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file '.bzrignore'
--- .bzrignore 2015-11-04 14:10:28 +0000
+++ .bzrignore 2016-12-05 14:48:05 +0000
@@ -1,5 +1,7 @@
1*.egg-info1*.egg-info
2*.pyc2*.pyc
3charm/dist
4charm/tmp
3dist5dist
4debian/files6debian/files
5debian/launchpad-buildd7debian/launchpad-buildd
68
=== added directory 'charm'
=== added file 'charm/Makefile'
--- charm/Makefile 1970-01-01 00:00:00 +0000
+++ charm/Makefile 2016-12-05 14:48:05 +0000
@@ -0,0 +1,77 @@
1NAME ?= launchpad-buildd
2TOPDIR ?= $(CURDIR)/..
3BUILDDIR ?= $(CURDIR)/dist
4TMPDIR ?= $(CURDIR)/tmp
5
6CHARM_SERIES ?= xenial
7CHARM_SRC ?= $(CURDIR)
8JUJU_REPOSITORY = $(BUILDDIR)
9CHARMDIR = $(BUILDDIR)/$(CHARM_SERIES)/$(NAME)
10CHARMREPODIR = $(BUILDDIR)/build
11RESOURCES = $(BUILDDIR)/resources/$(NAME)
12CHARM = $(CHARMDIR)/.done
13LAYER_PATH = $(TMPDIR)/layer
14INTERFACE_PATH = $(TMPDIR)/interface
15EXTRA_CHARM_BUILD_ARGS ?=
16DEPLOY_ENV ?= devel
17CHARM_STORE_URL ?= cs:~launchpad/launchpad-buildd
18
19export INTERFACE_PATH
20export LAYER_PATH
21export JUJU_REPOSITORY
22
23
24build: $(CHARM)
25
26packages: $(RESOURCES)
27
28build-with-packages: build packages
29
30$(BUILDDIR):
31 mkdir -p $@
32
33$(CHARM): $(CHARM_SRC) | $(BUILDDIR)
34 rm -rf $(CHARMDIR)
35 charm build -o $(BUILDDIR) -s $(CHARM_SERIES) -n $(NAME) $(EXTRA_CHARM_BUILD_ARGS)
36 touch $(CHARMDIR)/dummy-launchpad-buildd.deb
37 touch $(CHARMDIR)/dummy-python-lpbuildd.deb
38 touch $@
39
40clean:
41 rm -rf $(BUILDDIR) $(TMPDIR)
42
43$(RESOURCES):
44 rm -rf $(TMPDIR)
45 mkdir -p $(TMPDIR)
46 rsync -a --exclude /charm $(TOPDIR)/ $(TMPDIR)/$(NAME)/
47 cd $(TMPDIR)/$(NAME)/ && debuild -b -uc -us
48 mkdir -p $(RESOURCES)
49 cp -a $(TMPDIR)/launchpad-buildd_*.deb \
50 $(RESOURCES)/launchpad-buildd.deb
51 cp -a $(TMPDIR)/python-lpbuildd_*.deb $(RESOURCES)/python-lpbuildd.deb
52
53create-privileged-model:
54 juju add-model privileged localhost
55 lxc profile set juju-privileged security.privileged true
56
57deploy:
58 juju deploy $(CHARMDIR)
59
60deploy-with-packages:
61 juju deploy \
62 --resource=launchpad-buildd=$(RESOURCES)/launchpad-buildd.deb \
63 --resource=python-lpbuildd=$(RESOURCES)/python-lpbuildd.deb \
64 $(CHARMDIR)
65
66push:
67 charm push $(CHARMDIR) $(CHARM_STORE_URL)
68
69push-with-packages:
70 charm push \
71 --resource=launchpad-buildd=$(CHARMDIR)/dummy-launchpad-buildd.deb \
72 --resource=python-lpbuildd=$(CHARMDIR)/dummy-python-lpbuildd.deb \
73 $(CHARMDIR) $(CHARM_STORE_URL)
74
75.PHONY: build packages build-with-packages clean
76.PHONY: create-privileged-model deploy deploy-with-packages
77.PHONY: push push-with-packages
078
=== added file 'charm/README.md'
--- charm/README.md 1970-01-01 00:00:00 +0000
+++ charm/README.md 2016-12-05 14:48:05 +0000
@@ -0,0 +1,59 @@
1# Overview
2
3This charm installs a Launchpad builder, which can build packages in
4response to requests from a Launchpad instance. It is mainly intended for
5use by Launchpad developers testing changes to builder handling.
6
7# Setup
8
9Builders need to be able to unpack chroots, which involves being able to
10create device nodes. Unprivileged LXD containers cannot do this. If you
11want to use this with the LXD provider, you should therefore do this first:
12
13```
14make create-privileged-model
15```
16
17... or, if you need more control, some variation on this:
18
19```
20juju add-model privileged localhost
21lxc profile set juju-privileged security.privileged true
22```
23
24# Deployment
25
26You can either deploy the stock launchpad-buildd package from a PPA, or
27build your own.
28
29Installing from a PPA is the default; just deploy this charm.
30
31If you're building your own package, then you already have the
32launchpad-buildd code checked out. In the `charm/` subdirectory, build the
33charm and packages together:
34
35```
36make build-with-packages
37```
38
39Then deploy the charm, attaching the packages as resources:
40
41```
42make deploy-with-packages
43```
44
45Either way, this should eventually give you a running builder. Find out its
46host name (e.g. `juju-XXXXXX-0.lxd`) and [add it to your local Launchpad
47instance](https://launchpad.dev/builders/+new) (e.g.
48`http://juju-XXXXXX-0.lxd:8221/`).
49
50# Notes
51
52This charm gives you a non-virtualized builder, since there is no reset from
53a base image between builds; you'll need to make sure that any archives or
54snaps with builds you intend to dispatch to this builder have the "Require
55virtualized builders" option disabled.
56
57The Launchpad development wiki has [instructions on setting up the rest of
58Launchpad](https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally).
59You can skip the parts about installing the builder.
060
=== added file 'charm/config.yaml'
--- charm/config.yaml 1970-01-01 00:00:00 +0000
+++ charm/config.yaml 2016-12-05 14:48:05 +0000
@@ -0,0 +1,3 @@
1options:
2 install_sources:
3 default: ppa:launchpad/buildd-staging
04
=== added file 'charm/layer.yaml'
--- charm/layer.yaml 1970-01-01 00:00:00 +0000
+++ charm/layer.yaml 2016-12-05 14:48:05 +0000
@@ -0,0 +1,13 @@
1repo: lp:launchpad-buildd
2includes:
3 - layer:basic
4 - layer:apt
5options:
6 apt:
7 packages:
8 - bzr-builder
9 - git-build-recipe
10 - quilt
11ignore:
12 - dist
13 - tmp
014
=== added file 'charm/metadata.yaml'
--- charm/metadata.yaml 1970-01-01 00:00:00 +0000
+++ charm/metadata.yaml 2016-12-05 14:48:05 +0000
@@ -0,0 +1,21 @@
1name: launchpad-buildd
2summary: Launchpad builder
3description: |
4 A system that can build packages in response to requests from a
5 Launchpad instance.
6tags:
7 - application_development
8maintainer: Colin Watson <cjwatson@canonical.com>
9subordinate: false
10series:
11 - xenial
12resources:
13 launchpad-buildd:
14 type: file
15 filename: launchpad-buildd.deb
16 description: The main Launchpad builder package.
17 python-lpbuildd:
18 type: file
19 filename: python-lpbuildd.deb
20 description: Python libraries supporting the Launchpad builder.
21min-juju-version: 2.0.0
022
=== added directory 'charm/reactive'
=== added file 'charm/reactive/launchpad-buildd.py'
--- charm/reactive/launchpad-buildd.py 1970-01-01 00:00:00 +0000
+++ charm/reactive/launchpad-buildd.py 2016-12-05 14:48:05 +0000
@@ -0,0 +1,96 @@
1# Copyright 2016 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4from __future__ import print_function
5
6import os.path
7import re
8import shutil
9import subprocess
10
11from charmhelpers import fetch
12from charmhelpers.core import (
13 hookenv,
14 host,
15 )
16from charms.apt import status_set
17from charms.reactive import (
18 hook,
19 only_once,
20 remove_state,
21 set_state,
22 when,
23 when_not,
24 )
25
26
27@only_once
28def install():
29 with open("/etc/default/launchpad-buildd", "w") as default_file:
30 print("RUN_NETWORK_REQUESTS_AS_ROOT=yes", file=default_file)
31 set_state("launchpad-buildd.needs_install")
32
33
34@hook("upgrade-charm", "config-changed")
35def mark_needs_install():
36 set_state("launchpad-buildd.needs_install")
37
38
39@when_not("apt.needs_update")
40@when("launchpad-buildd.needs_install")
41def install_packages():
42 cache_dir = os.path.join(hookenv.charm_dir(), "cache")
43 host.mkdir(cache_dir, perms=0o755)
44 to_install = []
45 packages = ["launchpad-buildd", "python-lpbuildd"]
46 options = ["--option=Dpkg::Options::=--force-confold"]
47 resource_paths = [hookenv.resource_get(package) for package in packages]
48 if all(path and os.path.getsize(path) for path in resource_paths):
49 # Install from resources.
50 changed = False
51 for package, resource_path in zip(packages, resource_paths):
52 local_path = os.path.join(cache_dir, "{}.deb".format(package))
53 to_install.append((local_path, resource_path))
54 if host.file_hash(local_path) != host.file_hash(resource_path):
55 changed = True
56 if not changed:
57 return
58 options.append("--reinstall")
59 else:
60 # We don't have resource-provided packages, so just install from the
61 # PPA.
62 to_install.extend([(None, package) for package in packages])
63 new_paths = [new_path for _, new_path in to_install]
64 try:
65 status_set(None, "Installing {}".format(",".join(packages)))
66 fetch.apt_unhold(packages)
67 fetch.apt_install(new_paths, options=options)
68 fetch.apt_hold(packages)
69 except subprocess.CalledProcessError:
70 status_set(
71 "blocked",
72 "Unable to install packages {}".format(",".join(packages)))
73 else:
74 for local_path, resource_path in to_install:
75 if local_path is not None:
76 shutil.copy2(resource_path, local_path)
77 # ntp.buildd isn't likely to work outside of the Canonical
78 # datacentre, and LXD containers can't set the system time. Let's
79 # just not worry about NTP.
80 config_path = "/etc/launchpad-buildd/default"
81 with open(config_path) as config_file:
82 config = config_file.read()
83 config = re.sub(r"^ntphost = .*", "ntphost = ", config, flags=re.M)
84 with open(config_path + ".new", "w") as new_config_file:
85 new_config_file.write(config)
86 os.rename(config_path + ".new", config_path)
87 remove_state("launchpad-buildd.needs_install")
88 set_state("launchpad-buildd.installed")
89
90
91@when("apt.installed.bzr-builder")
92@when("apt.installed.git-build-recipe")
93@when("apt.installed.quilt")
94@when("launchpad-buildd.installed")
95def mark_active():
96 status_set("active", "Builder running")
097
=== modified file 'debian/changelog'
--- debian/changelog 2016-11-30 10:20:32 +0000
+++ debian/changelog 2016-12-05 14:48:05 +0000
@@ -2,6 +2,8 @@
22
3 * buildsnap: Grant access to the proxy during the build phase as well as3 * buildsnap: Grant access to the proxy during the build phase as well as
4 during the pull phase (LP: #1642281).4 during the pull phase (LP: #1642281).
5 * Add a Juju charm which can be used to deploy local launchpad-buildd
6 instances.
57
6 -- Colin Watson <cjwatson@ubuntu.com> Fri, 04 Nov 2016 10:47:41 +00008 -- Colin Watson <cjwatson@ubuntu.com> Fri, 04 Nov 2016 10:47:41 +0000
79

Subscribers

People subscribed via source and target branches

to all changes: