Juju Charms Collection
turnip package

Merge lp:~cjwatson/charms/trusty/turnip/build-label into lp:~canonical-launchpad-branches/charms/trusty/turnip/devel

  1. Trusty Tahr (14.04)
  2. build-label
  3. Merge into devel
Proposed by Colin Watson
Status: Merged
Merged at revision: 81
Proposed branch: lp:~cjwatson/charms/trusty/turnip/build-label
Merge into: lp:~canonical-launchpad-branches/charms/trusty/turnip/devel
Diff against target: 491 lines (+209/-72)
12 files modified
.bzrignore (+1/-2)
Makefile.common (+24/-21)
README.md (+0/-2)
config.yaml (+28/-0)
deploy-requirements.txt (+0/-2)
hooks/actions.py (+149/-38)
hooks/services.py (+2/-2)
templates/turnip-httpserver.conf.j2 (+1/-1)
templates/turnip-packbackendserver.conf.j2 (+1/-1)
templates/turnip-packfrontendserver.conf.j2 (+1/-1)
templates/turnip-sshserver.conf.j2 (+1/-1)
templates/turnip-virtserver.conf.j2 (+1/-1)
To merge this branch: bzr merge lp:~cjwatson/charms/trusty/turnip/build-label
Reviewer Review Type Date Requested Status
Kit Randel (community) Approve
Review via email: mp+275468@code.launchpad.net

Commit message

Allow updating the code payload separately from the charm using a build label.

Description of the change

Allow updating the code payload separately from the charm using a build label.

The approach used here is based heavily on the software-center-agent charm. The preferred approach is to store the payload in Swift (Canonical developers can use their Canonistack credentials for this), but you can also use "make deploy" or "make rollout" to do an initial deployment or an updated code rollout respectively and it'll push the payload around manually. The previous payload is kept in place for the sake of quick rollouts, but at the moment we prune all but the previous and current payloads: this has the benefit of not needing to worry about working out which the newest ones are.

The virtualenv moves inside the payload directory so that each payload gets its own. /srv/turnip/code remains as a symlink to the current payload, which is convenient and saves us having to substitute the current build label into the Upstart jobs.

There are no tests directly here, but it'll at least get integration testing by way of the corresponding changes to the Mojo spec.

To post a comment you must log in.
Revision history for this message
Kit Randel (blr) wrote :

Just a minor formatting issue.

I'm working through updating Rutabaga's charm similarly as a somewhat circumlocuitous form of review. No doubt fine, given the mojo spec ran, but thought it might be worth leaving this open in the event that I came across anything in the process.

Given our services will be duplicating a reasonable amount of charm code, I did wonder if we should consider maintaining our own charmhelpers like library.

review: Approve
Revision history for this message
Kit Randel (blr) wrote :

Although flake8 is in turnip-dependencies, I don't see it in any requirements manifest, so `make lint` will fail here afaict.

Revision history for this message
Kit Randel (blr) wrote :

> Although flake8 is in turnip-dependencies, I don't see it in any requirements
> manifest, so `make lint` will fail here afaict.

Apologies, disregard that, comment was intended for the turnipcake MP.

lp:~cjwatson/charms/trusty/turnip/build-label updated
82. By Colin Watson

Fix over-indentation.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file '.bzrignore'
2--- .bzrignore 2015-04-27 01:59:44 +0000
3+++ .bzrignore 2015-11-02 11:16:19 +0000
4@@ -1,5 +1,4 @@
5 *.pyc
6 .coverage
7 .venv
8-files/pip-cache
9-files/turnip.tar.gz
10+files/*
11
12=== modified file 'Makefile.common'
13--- Makefile.common 2015-03-29 02:58:13 +0000
14+++ Makefile.common 2015-11-02 11:16:19 +0000
15@@ -5,13 +5,13 @@
16 PWD := $(shell pwd)
17 HOOKS_DIR := $(PWD)/hooks
18 SOURCE_DIR ?= $(shell dirname $(PWD))/.source/$(APP_NAME)
19-PIP_CACHE := $(PWD)/files/pip-cache
20+FILES_DIR := $(PWD)/files
21
22-ifeq ($(PIP_SOURCE_DIR),)
23-PIP_CACHE_ARGS :=
24-else
25-PIP_CACHE_ARGS := --no-index --find-links=file://$(PIP_SOURCE_DIR)
26-endif
27+BUILD_LABEL = $(shell bzr log -rlast: --show-ids $(SOURCE_DIR) | sed -n 's/^revision-id: //p')
28+TARBALL = $(APP_NAME).tar.gz
29+ASSET = $(FILES_DIR)/$(BUILD_LABEL)/$(TARBALL)
30+UNIT = $(APP_NAME)/0
31+CHARM_UNIT_PATH := /var/lib/juju/agents/unit-$(APP_NAME)-0/charm
32
33 all: setup lint test
34
35@@ -20,9 +20,22 @@
36 @juju upgrade-charm --repository=../.. $(APP_NAME)
37
38
39-deploy: tarball pip-cache
40+deploy: payload
41 @echo "Deploying $(APP_NAME)..."
42 @juju deploy --repository=../.. local:trusty/$(APP_NAME)
43+ @$(MAKE) rollout SKIP_BUILD=true
44+
45+
46+# deploy a new revision/branch
47+rollout: _PATH=$(CHARM_UNIT_PATH)/files/$(BUILD_LABEL)
48+rollout:
49+ifneq ($(SKIP_BUILD),true)
50+ $(MAKE) payload
51+endif
52+ # manually copy our asset to be in the right place, rather than upgrade-charm
53+ juju scp $(ASSET) $(UNIT):$(TARBALL)
54+ juju ssh $(UNIT) 'sudo mkdir -p $(_PATH) && sudo mv $(TARBALL) $(_PATH)/'
55+ juju set $(APP_NAME) build_label=$(BUILD_LABEL)
56
57
58 ifeq ($(NO_FETCH_CODE),)
59@@ -39,23 +52,14 @@
60 endif
61
62
63-pip-cache: fetch-code
64- @echo "Updating python dependency cache..."
65- @mkdir -p $(PIP_CACHE)
66- @pip install $(PIP_CACHE_ARGS) --no-use-wheel --download $(PIP_CACHE) \
67- -r $(SOURCE_DIR)/requirements.txt \
68- -r deploy-requirements.txt
69-
70-
71 check-rev:
72 ifndef REV
73 $(error Revision number required to fetch source: e.g. $ REV=10 make deploy)
74 endif
75
76-tarball: fetch-code
77- @echo "Creating tarball for deploy..."
78- @mkdir -p files/
79- @tar czf files/$(APP_NAME).tar.gz -C $(SOURCE_DIR) .
80+payload: fetch-code
81+ @echo "Building asset for $(BUILD_LABEL)..."
82+ @$(MAKE) -C $(SOURCE_DIR) build-tarball TARBALL_BUILDS_DIR=$(FILES_DIR)
83
84
85 # The following targets are for charm maintenance.
86@@ -65,7 +69,6 @@
87 @find . -depth -name '__pycache__' -exec rm -rf '{}' \;
88 @rm -f .coverage
89 @rm -rf $(SOURCE_DIR)
90- @rm -rf $(PIP_CACHE)
91 @rm -rf .venv
92
93
94@@ -94,4 +97,4 @@
95 @bzr cat lp:charm-helpers/tools/charm_helpers_sync/charm_helpers_sync.py > /tmp/charm_helpers_sync.py
96
97
98-.PHONY: clean lint setup tarball test upgrade
99+.PHONY: clean lint setup payload test upgrade
100
101=== modified file 'README.md'
102--- README.md 2015-02-11 21:48:08 +0000
103+++ README.md 2015-11-02 11:16:19 +0000
104@@ -11,8 +11,6 @@
105
106 $ REV=revno make deploy
107
108-The deploy target will create a fresh python package cache in files/pip-cache and sets the turnip config variable 'revision' to that specified by REV.
109-
110 # Todo
111
112 * Refactor to use charmhelpers.core.services
113
114=== modified file 'config.yaml'
115--- config.yaml 2015-05-22 15:42:11 +0000
116+++ config.yaml 2015-11-02 11:16:19 +0000
117@@ -3,6 +3,10 @@
118 type: string
119 default: 'turnip'
120 description: Name of this application.
121+ build_label:
122+ type: string
123+ default: ""
124+ description: Build label to run.
125 nagios_context:
126 default: "juju"
127 type: string
128@@ -120,6 +124,30 @@
129 description: |
130 Hosts that should be allowed to rsync logs. Note that this relies on
131 basenode.
132+ swift_username:
133+ type: string
134+ default: ""
135+ description: Username to use when accessing Swift.
136+ swift_password:
137+ type: string
138+ default: ""
139+ description: Password to use when accessing Swift.
140+ swift_auth_url:
141+ type: string
142+ default: ""
143+ description: URL for authenticating against Keystone.
144+ swift_region_name:
145+ type: string
146+ default: ""
147+ description: Swift region.
148+ swift_tenant_name:
149+ type: string
150+ default: ""
151+ description: Entity that owns resources.
152+ swift_container_name:
153+ type: string
154+ default: ""
155+ description: Container to put objects in.
156
157 # apt configuration used by charmhelpers.
158 install_sources:
159
160=== removed file 'deploy-requirements.txt'
161--- deploy-requirements.txt 2015-03-27 07:43:45 +0000
162+++ deploy-requirements.txt 1970-01-01 00:00:00 +0000
163@@ -1,2 +0,0 @@
164-envdir==0.7
165-gunicorn==19.3.0
166
167=== modified file 'hooks/actions.py'
168--- hooks/actions.py 2015-05-22 15:42:11 +0000
169+++ hooks/actions.py 2015-11-02 11:16:19 +0000
170@@ -1,4 +1,5 @@
171 import base64
172+import errno
173 import grp
174 import os
175 import pwd
176@@ -26,19 +27,29 @@
177
178 # Globals
179 CHARM_FILES_DIR = os.path.join(hookenv.charm_dir(), 'files')
180+CHARM_SCRIPTS_DIR = os.path.join(hookenv.charm_dir(), 'scripts')
181 REQUIRED_PACKAGES = [
182- 'python-virtualenv', 'python-dev', 'python-pygit2', 'git', 'cgit',
183+ 'python-virtualenv',
184+ 'python-dev',
185+ 'libgit2-dev',
186+ 'libffi-dev',
187+ 'git',
188+ 'cgit',
189 # Unfortunately we need build-essential to compile some extensions,
190 # notably Twisted. Using wheels rather than a pip-cache might let us
191 # avoid this in future.
192 'build-essential',
193+ 'python-swiftclient',
194 ]
195 BASE_DIR = config['base_dir']
196+PAYLOADS_DIR = os.path.join(BASE_DIR, 'payloads')
197 CODE_DIR = os.path.join(BASE_DIR, 'code')
198-VENV_DIR = os.path.join(BASE_DIR, 'venv')
199+VENV_DIR = os.path.join(CODE_DIR, 'env')
200+OLD_VENV_DIR = os.path.join(BASE_DIR, 'venv')
201 LOGS_DIR = os.path.join(BASE_DIR, 'logs')
202 DATA_DIR = os.path.join(BASE_DIR, 'data')
203 KEY_DIR = os.path.join(BASE_DIR, 'keys')
204+CODE_TARBALL = 'turnip.tar.gz'
205
206 CODE_USER = config['code_user']
207 CODE_GROUP = config['code_group']
208@@ -62,7 +73,7 @@
209 def make_srv_location():
210 hookenv.log('Creating directories...')
211
212- for dir in (BASE_DIR, CODE_DIR):
213+ for dir in (BASE_DIR, PAYLOADS_DIR):
214 host.mkdir(dir, owner=CODE_USER, group=CODE_GROUP, perms=0o755)
215 for dir in (LOGS_DIR, DATA_DIR, KEY_DIR):
216 host.mkdir(dir, owner=USER, group=GROUP, perms=0o755)
217@@ -87,23 +98,143 @@
218 host.add_user_to_group(USER, CGIT_GROUP)
219
220
221-def unpack_source(service_name):
222- hookenv.log('Deploying source...')
223-
224+def get_swift_creds(config):
225+ return {
226+ 'user': config['swift_username'],
227+ 'project': config['swift_tenant_name'],
228+ 'password': config['swift_password'],
229+ 'authurl': config['swift_auth_url'],
230+ 'region': config['swift_region_name'],
231+ }
232+
233+
234+def swift_base_cmd(**swift_creds):
235+ return [
236+ 'swift',
237+ '--os-username=' + swift_creds['user'],
238+ '--os-tenant-name=' + swift_creds['project'],
239+ '--os-password=' + swift_creds['password'],
240+ '--os-auth-url=' + swift_creds['authurl'],
241+ '--os-region-name=' + swift_creds['region'],
242+ ]
243+
244+
245+def swift_get_etag(name, container=None, **swift_creds):
246+ cmd = swift_base_cmd(**swift_creds) + ['stat', container, name]
247+ file_stat = subprocess.check_output(cmd).splitlines()
248+ for line in file_stat:
249+ words = line.split()
250+ if words[0] == 'ETag:':
251+ return words[1]
252+
253+
254+def swift_fetch(source, target, container=None, **swift_creds):
255+ cmd = swift_base_cmd(**swift_creds) + [
256+ 'download', '--output=' + target, container, source]
257+ subprocess.check_call(cmd)
258+
259+
260+def unlink_force(path):
261+ """Unlink path, without worrying about whether it exists."""
262+ try:
263+ os.unlink(path)
264+ except OSError as e:
265+ if e.errno != errno.ENOENT:
266+ raise
267+
268+
269+def symlink_force(source, link_name):
270+ """Create symlink link_name -> source, even if link_name exists."""
271+ unlink_force(link_name)
272+ os.symlink(source, link_name)
273+
274+
275+def install_python_packages(target_dir):
276+ hookenv.log('Installing Python dependencies...')
277+ subprocess.check_call(
278+ ['sudo', '-u', CODE_USER, 'make', '-C', target_dir, 'build',
279+ 'PIP_SOURCE_DIR=%s' % os.path.join(target_dir, 'pip-cache')])
280+
281+
282+def prune_payloads(keep):
283+ for entry in os.listdir(PAYLOADS_DIR):
284+ if entry in keep:
285+ continue
286+ entry_path = os.path.join(PAYLOADS_DIR, entry)
287+ if os.path.isdir(entry_path):
288+ hookenv.log('Purging old build in %s...' % entry_path)
289+ shutil.rmtree(entry_path)
290+
291+
292+def deploy_code(service_name):
293 make_srv_location()
294
295+ current_build_label = None
296+ if os.path.islink(CODE_DIR):
297+ current_build_label = os.path.basename(os.path.realpath(CODE_DIR))
298+ elif os.path.isdir(os.path.join(CODE_DIR, '.bzr')):
299+ log_output = subprocess.check_output(
300+ ['bzr', 'log', '-rlast:', '--show-ids', CODE_DIR])
301+ for line in log_output.splitlines():
302+ if line.startswith('revision-id: '):
303+ current_build_label = line[len('revision-id: '):]
304+ desired_build_label = config['build_label']
305+ if not desired_build_label:
306+ if current_build_label is not None:
307+ hookenv.log(
308+ 'No desired build label, but build %s is already deployed' %
309+ current_build_label)
310+ # We're probably upgrading from a charm that used old-style code
311+ # assets, so make sure we at least have a virtualenv available
312+ # from the current preferred location.
313+ if not os.path.isdir(VENV_DIR) and os.path.isdir(OLD_VENV_DIR):
314+ os.symlink(OLD_VENV_DIR, VENV_DIR)
315+ return
316+ else:
317+ raise AssertionError('Build label unset, so cannot deploy code')
318+ if current_build_label == desired_build_label:
319+ hookenv.log('Build %s already deployed' % desired_build_label)
320+ return
321+ hookenv.log('Deploying build %s...' % desired_build_label)
322+
323 # Copy source archive
324- archive_path = os.path.join(BASE_DIR, 'turnip.tar.gz')
325-
326- with open(os.path.join(CHARM_FILES_DIR, 'turnip.tar.gz')) as file:
327- host.write_file(archive_path, file.read(), perms=0o644)
328-
329- # Unpack source
330- archive.extract_tarfile(archive_path, CODE_DIR)
331- os.chown(
332- CODE_DIR,
333- pwd.getpwnam(CODE_USER).pw_uid, grp.getgrnam(CODE_GROUP).gr_gid)
334- host.lchownr(CODE_DIR, CODE_USER, CODE_GROUP)
335+ archive_path = os.path.join(PAYLOADS_DIR, desired_build_label + '.tar.gz')
336+ object_name = os.path.join(desired_build_label, CODE_TARBALL)
337+
338+ try:
339+ if config['swift_container_name']:
340+ swift_creds = get_swift_creds(config)
341+ swift_container = config['swift_container_name']
342+ swift_fetch(
343+ os.path.join('turnip-builds', object_name), archive_path,
344+ container=swift_container, **swift_creds)
345+ else:
346+ with open(os.path.join(CHARM_FILES_DIR, object_name)) as file:
347+ host.write_file(archive_path, file.read(), perms=0o644)
348+
349+ # Unpack source
350+ target_dir = os.path.join(PAYLOADS_DIR, desired_build_label)
351+ if os.path.isdir(target_dir):
352+ shutil.rmtree(target_dir)
353+ archive.extract_tarfile(archive_path, target_dir)
354+ os.chown(
355+ target_dir,
356+ pwd.getpwnam(CODE_USER).pw_uid, grp.getgrnam(CODE_GROUP).gr_gid)
357+ host.lchownr(target_dir, CODE_USER, CODE_GROUP)
358+
359+ install_python_packages(target_dir)
360+
361+ if not os.path.islink(CODE_DIR) and os.path.isdir(CODE_DIR):
362+ old_payload_dir = os.path.join(PAYLOADS_DIR, current_build_label)
363+ if os.path.exists(old_payload_dir):
364+ shutil.rmtree(CODE_DIR)
365+ else:
366+ os.rename(CODE_DIR, old_payload_dir)
367+ symlink_force(
368+ os.path.relpath(target_dir, os.path.dirname(CODE_DIR)), CODE_DIR)
369+ prune_payloads([desired_build_label, current_build_label])
370+ finally:
371+ unlink_force(archive_path)
372
373
374 def install_packages(service_name):
375@@ -112,26 +243,6 @@
376 fetch.apt_install(REQUIRED_PACKAGES, fatal=True)
377
378
379-def install_python_packages(service_name):
380- hookenv.log('Installing Python dependencies...')
381- pip_cache = os.path.join(CHARM_FILES_DIR, 'pip-cache')
382- code_reqs = os.path.join(CODE_DIR, 'requirements.txt')
383- deploy_reqs = os.path.join(hookenv.charm_dir(), 'deploy-requirements.txt')
384-
385- pip_bin = os.path.join(VENV_DIR, 'bin', 'pip')
386-
387- subprocess.call([
388- 'sudo', '-u', CODE_USER, 'virtualenv', '--system-site-packages',
389- VENV_DIR])
390- subprocess.check_call([
391- 'sudo', '-u', CODE_USER, pip_bin, 'install', '--no-index',
392- '--find-links={}'.format(pip_cache), '-r', code_reqs,
393- '-r', deploy_reqs])
394- subprocess.check_call([
395- 'sudo', '-u', CODE_USER, pip_bin, 'install', '--no-deps',
396- '-e', CODE_DIR])
397-
398-
399 def write_ssh_keys(service_name):
400 if PUBLIC_KEY and PRIVATE_KEY:
401 keys = [
402@@ -302,7 +413,7 @@
403
404
405 def install_nrpe_scripts(service_name):
406- src = os.path.join(CHARM_FILES_DIR, "nrpe")
407+ src = os.path.join(CHARM_SCRIPTS_DIR, "nrpe")
408 dst = "/usr/local/lib/nagios/plugins"
409 if not os.path.exists(dst):
410 os.makedirs(dst)
411
412=== modified file 'hooks/services.py'
413--- hooks/services.py 2015-05-22 15:42:11 +0000
414+++ hooks/services.py 2015-11-02 11:16:19 +0000
415@@ -34,8 +34,8 @@
416 actions.execd_preinstall('turnip')
417 actions.install_packages('turnip')
418 actions.create_users('turnip')
419- actions.unpack_source('turnip')
420- actions.install_python_packages('turnip')
421+ if hookenv.hook_name() in ('install', 'upgrade-charm', 'config-changed'):
422+ actions.deploy_code('turnip')
423
424 extra_requirements = []
425
426
427=== renamed directory 'files/nrpe' => 'scripts/nrpe'
428=== modified file 'templates/turnip-httpserver.conf.j2'
429--- templates/turnip-httpserver.conf.j2 2015-04-26 22:33:55 +0000
430+++ templates/turnip-httpserver.conf.j2 2015-11-02 11:16:19 +0000
431@@ -4,7 +4,7 @@
432 setuid {{ user }}
433 setgid {{ group }}
434
435-env PYTHON_HOME={{ base_dir }}/venv
436+env PYTHON_HOME={{ base_dir }}/code/env
437
438 start on runlevel [2345]
439 stop on runlevel [016]
440
441=== modified file 'templates/turnip-packbackendserver.conf.j2'
442--- templates/turnip-packbackendserver.conf.j2 2015-04-26 22:33:55 +0000
443+++ templates/turnip-packbackendserver.conf.j2 2015-11-02 11:16:19 +0000
444@@ -4,7 +4,7 @@
445 setuid {{ user }}
446 setgid {{ group }}
447
448-env PYTHON_HOME={{ base_dir }}/venv
449+env PYTHON_HOME={{ base_dir }}/code/env
450
451 start on runlevel [2345]
452 stop on runlevel [016]
453
454=== modified file 'templates/turnip-packfrontendserver.conf.j2'
455--- templates/turnip-packfrontendserver.conf.j2 2015-04-26 22:33:55 +0000
456+++ templates/turnip-packfrontendserver.conf.j2 2015-11-02 11:16:19 +0000
457@@ -4,7 +4,7 @@
458 setuid {{ user }}
459 setgid {{ group }}
460
461-env PYTHON_HOME={{ base_dir }}/venv
462+env PYTHON_HOME={{ base_dir }}/code/env
463
464 start on runlevel [2345]
465 stop on runlevel [016]
466
467=== modified file 'templates/turnip-sshserver.conf.j2'
468--- templates/turnip-sshserver.conf.j2 2015-04-26 22:33:55 +0000
469+++ templates/turnip-sshserver.conf.j2 2015-11-02 11:16:19 +0000
470@@ -4,7 +4,7 @@
471 setuid {{ user }}
472 setgid {{ group }}
473
474-env PYTHON_HOME={{ base_dir }}/venv
475+env PYTHON_HOME={{ base_dir }}/code/env
476
477 start on runlevel [2345]
478 stop on runlevel [016]
479
480=== modified file 'templates/turnip-virtserver.conf.j2'
481--- templates/turnip-virtserver.conf.j2 2015-04-26 22:33:55 +0000
482+++ templates/turnip-virtserver.conf.j2 2015-11-02 11:16:19 +0000
483@@ -4,7 +4,7 @@
484 setuid {{ user }}
485 setgid {{ group }}
486
487-env PYTHON_HOME={{ base_dir }}/venv
488+env PYTHON_HOME={{ base_dir }}/code/env
489
490 start on runlevel [2345]
491 stop on runlevel [016]

Subscribers

People subscribed via source and target branches

to all changes: