Juju Charms Collection
haproxy package

Merge lp:~free.ekanayaka/charms/trusty/haproxy/ssl-crt-support into lp:charms/trusty/haproxy

  1. Trusty Tahr (14.04)
  2. ssl-crt-support
  3. Merge into trunk
Proposed by Free Ekanayaka
Status: Merged
Merged at revision: 88
Proposed branch: lp:~free.ekanayaka/charms/trusty/haproxy/ssl-crt-support
Merge into: lp:charms/trusty/haproxy
Diff against target: 4000 lines (+2735/-212)
29 files modified
README.md (+28/-0)
config-manager.txt (+1/-1)
config.yaml (+25/-0)
data/openssl.cnf (+21/-0)
hooks/charmhelpers/__init__.py (+38/-0)
hooks/charmhelpers/contrib/__init__.py (+15/-0)
hooks/charmhelpers/contrib/charmsupport/__init__.py (+15/-0)
hooks/charmhelpers/contrib/charmsupport/nrpe.py (+116/-10)
hooks/charmhelpers/contrib/charmsupport/volumes.py (+21/-2)
hooks/charmhelpers/core/__init__.py (+15/-0)
hooks/charmhelpers/core/decorators.py (+57/-0)
hooks/charmhelpers/core/fstab.py (+134/-0)
hooks/charmhelpers/core/hookenv.py (+269/-41)
hooks/charmhelpers/core/host.py (+254/-47)
hooks/charmhelpers/core/services/__init__.py (+18/-0)
hooks/charmhelpers/core/services/base.py (+329/-0)
hooks/charmhelpers/core/services/helpers.py (+259/-0)
hooks/charmhelpers/core/sysctl.py (+56/-0)
hooks/charmhelpers/core/templating.py (+68/-0)
hooks/charmhelpers/fetch/__init__.py (+309/-79)
hooks/charmhelpers/fetch/archiveurl.py (+121/-8)
hooks/charmhelpers/fetch/bzrurl.py (+39/-5)
hooks/charmhelpers/fetch/giturl.py (+71/-0)
hooks/hooks.py (+216/-14)
hooks/tests/test_config_changed_hooks.py (+28/-0)
hooks/tests/test_helpers.py (+87/-2)
hooks/tests/test_install.py (+21/-2)
hooks/tests/test_peer_hooks.py (+59/-1)
tests/10_deploy_test.py (+45/-0)
To merge this branch: bzr merge lp:~free.ekanayaka/charms/trusty/haproxy/ssl-crt-support
Reviewer Review Type Date Requested Status
Chris Glass (community) Approve
Review Queue (community) automated testing Needs Fixing
Review via email: mp+249094@code.launchpad.net

Description of the change

Add SSL termination support.

In particular:

- Update charmhelpers (to use host.lsb_release)

- Add ssl_cert/ssl_key config keys to set custom SSL certificate (or the special "SELFSIGNED" value to generate self-signed one), which match the naming and semantics of the apache2 charm.

- Add a crts service yaml keyword to specify the list of SSL certificates a service should use.

- Add global_default_dh_param config key to tweak the tune.ssl.default-dh-param configuration option of haproxy.cfg.

To post a comment you must log in.
Revision history for this message
Review Queue (review-queue) wrote :

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11001-results

review: Needs Fixing (automated testing)
Revision history for this message
Review Queue (review-queue) wrote :

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11002-results

review: Needs Fixing (automated testing)
lp:~free.ekanayaka/charms/trusty/haproxy/ssl-crt-support updated
98. By Free Ekanayaka

Fix comment

Revision history for this message
Review Queue (review-queue) wrote :

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11025-results

review: Needs Fixing (automated testing)
lp:~free.ekanayaka/charms/trusty/haproxy/ssl-crt-support updated
99. By Free Ekanayaka

Add retry for SSL request

Revision history for this message
Chris Glass (tribaal) wrote :

This looks good, I added a couple of comments inline (mainly docstrings and exlanations). I'll give it another quick pass after they are fixed.

Note: most of the diff is charmhelpers sync.

Revision history for this message
Free Ekanayaka (free.ekanayaka) wrote :

Thanks Chris, should be all fixed.

Revision history for this message
James Troup (elmo) wrote :

I don't see anything about permissions in the diff; is the sensitive SSL material properly protected?

Revision history for this message
Free Ekanayaka (free.ekanayaka) wrote :

You mean file-level permissions? I can surely add a "chmod 600" to all SSL files.

lp:~free.ekanayaka/charms/trusty/haproxy/ssl-crt-support updated
100. By Free Ekanayaka

Address review comments

101. By Free Ekanayaka

Ensure SSL files have the right permissions

Revision history for this message
Free Ekanayaka (free.ekanayaka) wrote :

> I don't see anything about permissions in the diff; is the sensitive SSL
> material properly protected?

I added code that will create all SSL files (including the self-signed one) with permission 600 and user/group haproxy.root.

Revision history for this message
Chris Glass (tribaal) wrote :

Looks good, added a couple of more inline comments. I think it's good to go with those addressed/acknowledged.

Revision history for this message
Free Ekanayaka (free.ekanayaka) :
Revision history for this message
Chris Glass (tribaal) wrote :

OK, I'll merge this in now. Thanks for your contribution! +1

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'README.md'
2--- README.md 2014-09-08 18:21:01 +0000
3+++ README.md 2015-02-19 17:05:21 +0000
4@@ -89,6 +89,34 @@
5 traffic goes to the correct haproxy listener which will in turn forward the
6 traffic to the correct backend server/port
7
8+## SSL Termination
9+
10+You can turn on SSL termination by using the `ssl_cert`/`ssl_key` service configuration
11+options and then using the `crts` key in the services yaml, e.g.:
12+
13+ #!/bin/bash
14+ # hooks/website-relation-changed
15+
16+ host=$(unit-get private-address)
17+ port=80
18+
19+ relation-set "services=
20+ - { service_name: my_web_app,
21+ service_options: [mode http, balance leastconn],
22+ crts: [DEFAULT]
23+ servers: [[my_web_app_1, $host, $port, option httpchk GET / HTTP/1.0],
24+ [... optionally more servers here ...]]}
25+ - { ... optionally more services here ... }
26+ "
27+
28+where the DEFAULT keyword means use the certificate set with `ssl_cert`/`ssl_key` (or
29+alternatively you can inline different base64-encode certificates).
30+
31+Note that in order to use SSL termination you need haproxy 1.5 or later, which
32+is not available in stock trusty, but you can get it from trusty-backports setting
33+the `source` configuration option to `backports` or to whatever PPA/archive you
34+wish to use.
35+
36 ## Development
37
38 The following steps are needed for testing and development of the charm,
39
40=== modified file 'config-manager.txt'
41--- config-manager.txt 2013-08-22 02:09:50 +0000
42+++ config-manager.txt 2015-02-19 17:05:21 +0000
43@@ -3,4 +3,4 @@
44 #
45 # make sourcedeps
46
47-./build/charm-helpers lp:charm-helpers;revno=70
48+./build/charm-helpers lp:charm-helpers;revno=303
49
50=== modified file 'config.yaml'
51--- config.yaml 2015-01-22 09:29:03 +0000
52+++ config.yaml 2015-02-19 17:05:21 +0000
53@@ -39,6 +39,15 @@
54 type: boolean
55 description: |
56 Whether to enable the stats UNIX socket.
57+ global_default_dh_param:
58+ default: 1024
59+ type: int
60+ description: |
61+ Sets the maximum size of the Diffie-Hellman parameters used for generating
62+ the ephemeral/temporary Diffie-Hellman key in case of DHE key exchange.
63+ Default value if 1024, higher values will increase the CPU load, and values
64+ greater than 1024 bits are not supported by Java 7 and earlier clients. This
65+ config key will be ignored if the installed haproxy package has no SSL support.
66 default_log:
67 default: "global"
68 type: string
69@@ -126,6 +135,22 @@
70 with a cookie. Session are sticky by default. To turn off sticky sessions,
71 remove the 'cookie SRVNAME insert' and 'cookie S{i}' stanzas from
72 `service_options` and `server_options`.
73+ ssl_cert:
74+ type: string
75+ description: |
76+ base64 encoded default SSL certificate. If the keyword 'SELFSIGNED'
77+ is used, the certificate and key will be autogenerated as
78+ self-signed. This is the certificate used by services configured
79+ using keyword 'DEFAULT' as SSL certificate. This config key will be
80+ ignored if the installed haproxy package has no SSL support.
81+ default: ""
82+ ssl_key:
83+ type: string
84+ description: |
85+ base64 encoded private key for the default SSL certificate. If ssl_cert
86+ is specified as SELFSIGNED or the installed haproxy package has no SSL
87+ support, this will be ignored.
88+ default: ""
89 sysctl:
90 default: ""
91 type: string
92
93=== added directory 'data'
94=== added file 'data/openssl.cnf'
95--- data/openssl.cnf 1970-01-01 00:00:00 +0000
96+++ data/openssl.cnf 2015-02-19 17:05:21 +0000
97@@ -0,0 +1,21 @@
98+RANDFILE = /dev/urandom
99+
100+[ req ]
101+default_days = 3650
102+default_bits = 1024
103+default_keyfile = privkey.pem
104+distinguished_name = req_distinguished_name
105+prompt = no
106+policy = policy_anything
107+x509_extensions = v3_ca
108+
109+[ req_distinguished_name ]
110+commonName = $ENV::OPENSSL_CN
111+
112+[ v3_ca ]
113+# Extensions to add to a certificate request
114+subjectAltName = @alt_names
115+
116+[alt_names]
117+DNS.1 = $ENV::OPENSSL_PUBLIC
118+DNS.2 = $ENV::OPENSSL_PRIVATE
119
120=== modified file 'hooks/charmhelpers/__init__.py'
121--- hooks/charmhelpers/__init__.py 2013-08-21 19:14:32 +0000
122+++ hooks/charmhelpers/__init__.py 2015-02-19 17:05:21 +0000
123@@ -0,0 +1,38 @@
124+# Copyright 2014-2015 Canonical Limited.
125+#
126+# This file is part of charm-helpers.
127+#
128+# charm-helpers is free software: you can redistribute it and/or modify
129+# it under the terms of the GNU Lesser General Public License version 3 as
130+# published by the Free Software Foundation.
131+#
132+# charm-helpers is distributed in the hope that it will be useful,
133+# but WITHOUT ANY WARRANTY; without even the implied warranty of
134+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
135+# GNU Lesser General Public License for more details.
136+#
137+# You should have received a copy of the GNU Lesser General Public License
138+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
139+
140+# Bootstrap charm-helpers, installing its dependencies if necessary using
141+# only standard libraries.
142+import subprocess
143+import sys
144+
145+try:
146+ import six # flake8: noqa
147+except ImportError:
148+ if sys.version_info.major == 2:
149+ subprocess.check_call(['apt-get', 'install', '-y', 'python-six'])
150+ else:
151+ subprocess.check_call(['apt-get', 'install', '-y', 'python3-six'])
152+ import six # flake8: noqa
153+
154+try:
155+ import yaml # flake8: noqa
156+except ImportError:
157+ if sys.version_info.major == 2:
158+ subprocess.check_call(['apt-get', 'install', '-y', 'python-yaml'])
159+ else:
160+ subprocess.check_call(['apt-get', 'install', '-y', 'python3-yaml'])
161+ import yaml # flake8: noqa
162
163=== modified file 'hooks/charmhelpers/contrib/__init__.py'
164--- hooks/charmhelpers/contrib/__init__.py 2013-08-21 19:14:32 +0000
165+++ hooks/charmhelpers/contrib/__init__.py 2015-02-19 17:05:21 +0000
166@@ -0,0 +1,15 @@
167+# Copyright 2014-2015 Canonical Limited.
168+#
169+# This file is part of charm-helpers.
170+#
171+# charm-helpers is free software: you can redistribute it and/or modify
172+# it under the terms of the GNU Lesser General Public License version 3 as
173+# published by the Free Software Foundation.
174+#
175+# charm-helpers is distributed in the hope that it will be useful,
176+# but WITHOUT ANY WARRANTY; without even the implied warranty of
177+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
178+# GNU Lesser General Public License for more details.
179+#
180+# You should have received a copy of the GNU Lesser General Public License
181+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
182
183=== modified file 'hooks/charmhelpers/contrib/charmsupport/__init__.py'
184--- hooks/charmhelpers/contrib/charmsupport/__init__.py 2013-08-21 19:14:32 +0000
185+++ hooks/charmhelpers/contrib/charmsupport/__init__.py 2015-02-19 17:05:21 +0000
186@@ -0,0 +1,15 @@
187+# Copyright 2014-2015 Canonical Limited.
188+#
189+# This file is part of charm-helpers.
190+#
191+# charm-helpers is free software: you can redistribute it and/or modify
192+# it under the terms of the GNU Lesser General Public License version 3 as
193+# published by the Free Software Foundation.
194+#
195+# charm-helpers is distributed in the hope that it will be useful,
196+# but WITHOUT ANY WARRANTY; without even the implied warranty of
197+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
198+# GNU Lesser General Public License for more details.
199+#
200+# You should have received a copy of the GNU Lesser General Public License
201+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
202
203=== modified file 'hooks/charmhelpers/contrib/charmsupport/nrpe.py'
204--- hooks/charmhelpers/contrib/charmsupport/nrpe.py 2013-08-21 19:14:32 +0000
205+++ hooks/charmhelpers/contrib/charmsupport/nrpe.py 2015-02-19 17:05:21 +0000
206@@ -1,3 +1,19 @@
207+# Copyright 2014-2015 Canonical Limited.
208+#
209+# This file is part of charm-helpers.
210+#
211+# charm-helpers is free software: you can redistribute it and/or modify
212+# it under the terms of the GNU Lesser General Public License version 3 as
213+# published by the Free Software Foundation.
214+#
215+# charm-helpers is distributed in the hope that it will be useful,
216+# but WITHOUT ANY WARRANTY; without even the implied warranty of
217+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
218+# GNU Lesser General Public License for more details.
219+#
220+# You should have received a copy of the GNU Lesser General Public License
221+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
222+
223 """Compatibility with the nrpe-external-master charm"""
224 # Copyright 2012 Canonical Ltd.
225 #
226@@ -18,6 +34,7 @@
227 log,
228 relation_ids,
229 relation_set,
230+ relations_of_type,
231 )
232
233 from charmhelpers.core.host import service
234@@ -54,6 +71,12 @@
235 # juju-myservice-0
236 # If you're running multiple environments with the same services in them
237 # this allows you to differentiate between them.
238+# nagios_servicegroups:
239+# default: ""
240+# type: string
241+# description: |
242+# A comma-separated list of nagios servicegroups.
243+# If left empty, the nagios_context will be used as the servicegroup
244 #
245 # 3. Add custom checks (Nagios plugins) to files/nrpe-external-master
246 #
247@@ -125,10 +148,8 @@
248
249 def _locate_cmd(self, check_cmd):
250 search_path = (
251- '/',
252- os.path.join(os.environ['CHARM_DIR'],
253- 'files/nrpe-external-master'),
254 '/usr/lib/nagios/plugins',
255+ '/usr/local/lib/nagios/plugins',
256 )
257 parts = shlex.split(check_cmd)
258 for path in search_path:
259@@ -140,7 +161,7 @@
260 log('Check command not found: {}'.format(parts[0]))
261 return ''
262
263- def write(self, nagios_context, hostname):
264+ def write(self, nagios_context, hostname, nagios_servicegroups=None):
265 nrpe_check_file = '/etc/nagios/nrpe.d/{}.cfg'.format(
266 self.command)
267 with open(nrpe_check_file, 'w') as nrpe_check_config:
268@@ -152,16 +173,21 @@
269 log('Not writing service config as {} is not accessible'.format(
270 NRPE.nagios_exportdir))
271 else:
272- self.write_service_config(nagios_context, hostname)
273+ self.write_service_config(nagios_context, hostname,
274+ nagios_servicegroups)
275
276- def write_service_config(self, nagios_context, hostname):
277+ def write_service_config(self, nagios_context, hostname,
278+ nagios_servicegroups=None):
279 for f in os.listdir(NRPE.nagios_exportdir):
280 if re.search('.*{}.cfg'.format(self.command), f):
281 os.remove(os.path.join(NRPE.nagios_exportdir, f))
282
283+ if not nagios_servicegroups:
284+ nagios_servicegroups = nagios_context
285+
286 templ_vars = {
287 'nagios_hostname': hostname,
288- 'nagios_servicegroup': nagios_context,
289+ 'nagios_servicegroup': nagios_servicegroups,
290 'description': self.description,
291 'shortname': self.shortname,
292 'command': self.command,
293@@ -181,12 +207,19 @@
294 nagios_exportdir = '/var/lib/nagios/export'
295 nrpe_confdir = '/etc/nagios/nrpe.d'
296
297- def __init__(self):
298+ def __init__(self, hostname=None):
299 super(NRPE, self).__init__()
300 self.config = config()
301 self.nagios_context = self.config['nagios_context']
302+ if 'nagios_servicegroups' in self.config:
303+ self.nagios_servicegroups = self.config['nagios_servicegroups']
304+ else:
305+ self.nagios_servicegroups = 'juju'
306 self.unit_name = local_unit().replace('/', '-')
307- self.hostname = "{}-{}".format(self.nagios_context, self.unit_name)
308+ if hostname:
309+ self.hostname = hostname
310+ else:
311+ self.hostname = "{}-{}".format(self.nagios_context, self.unit_name)
312 self.checks = []
313
314 def add_check(self, *args, **kwargs):
315@@ -207,7 +240,8 @@
316 nrpe_monitors = {}
317 monitors = {"monitors": {"remote": {"nrpe": nrpe_monitors}}}
318 for nrpecheck in self.checks:
319- nrpecheck.write(self.nagios_context, self.hostname)
320+ nrpecheck.write(self.nagios_context, self.hostname,
321+ self.nagios_servicegroups)
322 nrpe_monitors[nrpecheck.shortname] = {
323 "command": nrpecheck.command,
324 }
325@@ -216,3 +250,75 @@
326
327 for rid in relation_ids("local-monitors"):
328 relation_set(relation_id=rid, monitors=yaml.dump(monitors))
329+
330+
331+def get_nagios_hostcontext(relation_name='nrpe-external-master'):
332+ """
333+ Query relation with nrpe subordinate, return the nagios_host_context
334+
335+ :param str relation_name: Name of relation nrpe sub joined to
336+ """
337+ for rel in relations_of_type(relation_name):
338+ if 'nagios_hostname' in rel:
339+ return rel['nagios_host_context']
340+
341+
342+def get_nagios_hostname(relation_name='nrpe-external-master'):
343+ """
344+ Query relation with nrpe subordinate, return the nagios_hostname
345+
346+ :param str relation_name: Name of relation nrpe sub joined to
347+ """
348+ for rel in relations_of_type(relation_name):
349+ if 'nagios_hostname' in rel:
350+ return rel['nagios_hostname']
351+
352+
353+def get_nagios_unit_name(relation_name='nrpe-external-master'):
354+ """
355+ Return the nagios unit name prepended with host_context if needed
356+
357+ :param str relation_name: Name of relation nrpe sub joined to
358+ """
359+ host_context = get_nagios_hostcontext(relation_name)
360+ if host_context:
361+ unit = "%s:%s" % (host_context, local_unit())
362+ else:
363+ unit = local_unit()
364+ return unit
365+
366+
367+def add_init_service_checks(nrpe, services, unit_name):
368+ """
369+ Add checks for each service in list
370+
371+ :param NRPE nrpe: NRPE object to add check to
372+ :param list services: List of services to check
373+ :param str unit_name: Unit name to use in check description
374+ """
375+ for svc in services:
376+ upstart_init = '/etc/init/%s.conf' % svc
377+ sysv_init = '/etc/init.d/%s' % svc
378+ if os.path.exists(upstart_init):
379+ nrpe.add_check(
380+ shortname=svc,
381+ description='process check {%s}' % unit_name,
382+ check_cmd='check_upstart_job %s' % svc
383+ )
384+ elif os.path.exists(sysv_init):
385+ cronpath = '/etc/cron.d/nagios-service-check-%s' % svc
386+ cron_file = ('*/5 * * * * root '
387+ '/usr/local/lib/nagios/plugins/check_exit_status.pl '
388+ '-s /etc/init.d/%s status > '
389+ '/var/lib/nagios/service-check-%s.txt\n' % (svc,
390+ svc)
391+ )
392+ f = open(cronpath, 'w')
393+ f.write(cron_file)
394+ f.close()
395+ nrpe.add_check(
396+ shortname=svc,
397+ description='process check {%s}' % unit_name,
398+ check_cmd='check_status_file.py -f '
399+ '/var/lib/nagios/service-check-%s.txt' % svc,
400+ )
401
402=== modified file 'hooks/charmhelpers/contrib/charmsupport/volumes.py'
403--- hooks/charmhelpers/contrib/charmsupport/volumes.py 2013-08-21 19:14:32 +0000
404+++ hooks/charmhelpers/contrib/charmsupport/volumes.py 2015-02-19 17:05:21 +0000
405@@ -1,8 +1,25 @@
406+# Copyright 2014-2015 Canonical Limited.
407+#
408+# This file is part of charm-helpers.
409+#
410+# charm-helpers is free software: you can redistribute it and/or modify
411+# it under the terms of the GNU Lesser General Public License version 3 as
412+# published by the Free Software Foundation.
413+#
414+# charm-helpers is distributed in the hope that it will be useful,
415+# but WITHOUT ANY WARRANTY; without even the implied warranty of
416+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
417+# GNU Lesser General Public License for more details.
418+#
419+# You should have received a copy of the GNU Lesser General Public License
420+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
421+
422 '''
423 Functions for managing volumes in juju units. One volume is supported per unit.
424 Subordinates may have their own storage, provided it is on its own partition.
425
426-Configuration stanzas:
427+Configuration stanzas::
428+
429 volume-ephemeral:
430 type: boolean
431 default: true
432@@ -20,7 +37,8 @@
433 is 'true' and no volume-map value is set. Use 'juju set' to set a
434 value and 'juju resolved' to complete configuration.
435
436-Usage:
437+Usage::
438+
439 from charmsupport.volumes import configure_volume, VolumeConfigurationError
440 from charmsupport.hookenv import log, ERROR
441 def post_mount_hook():
442@@ -34,6 +52,7 @@
443 after_change=post_mount_hook)
444 except VolumeConfigurationError:
445 log('Storage could not be configured', ERROR)
446+
447 '''
448
449 # XXX: Known limitations
450
451=== modified file 'hooks/charmhelpers/core/__init__.py'
452--- hooks/charmhelpers/core/__init__.py 2013-08-21 19:14:32 +0000
453+++ hooks/charmhelpers/core/__init__.py 2015-02-19 17:05:21 +0000
454@@ -0,0 +1,15 @@
455+# Copyright 2014-2015 Canonical Limited.
456+#
457+# This file is part of charm-helpers.
458+#
459+# charm-helpers is free software: you can redistribute it and/or modify
460+# it under the terms of the GNU Lesser General Public License version 3 as
461+# published by the Free Software Foundation.
462+#
463+# charm-helpers is distributed in the hope that it will be useful,
464+# but WITHOUT ANY WARRANTY; without even the implied warranty of
465+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
466+# GNU Lesser General Public License for more details.
467+#
468+# You should have received a copy of the GNU Lesser General Public License
469+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
470
471=== added file 'hooks/charmhelpers/core/decorators.py'
472--- hooks/charmhelpers/core/decorators.py 1970-01-01 00:00:00 +0000
473+++ hooks/charmhelpers/core/decorators.py 2015-02-19 17:05:21 +0000
474@@ -0,0 +1,57 @@
475+# Copyright 2014-2015 Canonical Limited.
476+#
477+# This file is part of charm-helpers.
478+#
479+# charm-helpers is free software: you can redistribute it and/or modify
480+# it under the terms of the GNU Lesser General Public License version 3 as
481+# published by the Free Software Foundation.
482+#
483+# charm-helpers is distributed in the hope that it will be useful,
484+# but WITHOUT ANY WARRANTY; without even the implied warranty of
485+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
486+# GNU Lesser General Public License for more details.
487+#
488+# You should have received a copy of the GNU Lesser General Public License
489+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
490+
491+#
492+# Copyright 2014 Canonical Ltd.
493+#
494+# Authors:
495+# Edward Hope-Morley <opentastic@gmail.com>
496+#
497+
498+import time
499+
500+from charmhelpers.core.hookenv import (
501+ log,
502+ INFO,
503+)
504+
505+
506+def retry_on_exception(num_retries, base_delay=0, exc_type=Exception):
507+ """If the decorated function raises exception exc_type, allow num_retries
508+ retry attempts before raise the exception.
509+ """
510+ def _retry_on_exception_inner_1(f):
511+ def _retry_on_exception_inner_2(*args, **kwargs):
512+ retries = num_retries
513+ multiplier = 1
514+ while True:
515+ try:
516+ return f(*args, **kwargs)
517+ except exc_type:
518+ if not retries:
519+ raise
520+
521+ delay = base_delay * multiplier
522+ multiplier += 1
523+ log("Retrying '%s' %d more times (delay=%s)" %
524+ (f.__name__, retries, delay), level=INFO)
525+ retries -= 1
526+ if delay:
527+ time.sleep(delay)
528+
529+ return _retry_on_exception_inner_2
530+
531+ return _retry_on_exception_inner_1
532
533=== added file 'hooks/charmhelpers/core/fstab.py'
534--- hooks/charmhelpers/core/fstab.py 1970-01-01 00:00:00 +0000
535+++ hooks/charmhelpers/core/fstab.py 2015-02-19 17:05:21 +0000
536@@ -0,0 +1,134 @@
537+#!/usr/bin/env python
538+# -*- coding: utf-8 -*-
539+
540+# Copyright 2014-2015 Canonical Limited.
541+#
542+# This file is part of charm-helpers.
543+#
544+# charm-helpers is free software: you can redistribute it and/or modify
545+# it under the terms of the GNU Lesser General Public License version 3 as
546+# published by the Free Software Foundation.
547+#
548+# charm-helpers is distributed in the hope that it will be useful,
549+# but WITHOUT ANY WARRANTY; without even the implied warranty of
550+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
551+# GNU Lesser General Public License for more details.
552+#
553+# You should have received a copy of the GNU Lesser General Public License
554+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
555+
556+__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
557+
558+import io
559+import os
560+
561+
562+class Fstab(io.FileIO):
563+ """This class extends file in order to implement a file reader/writer
564+ for file `/etc/fstab`
565+ """
566+
567+ class Entry(object):
568+ """Entry class represents a non-comment line on the `/etc/fstab` file
569+ """
570+ def __init__(self, device, mountpoint, filesystem,
571+ options, d=0, p=0):
572+ self.device = device
573+ self.mountpoint = mountpoint
574+ self.filesystem = filesystem
575+
576+ if not options:
577+ options = "defaults"
578+
579+ self.options = options
580+ self.d = int(d)
581+ self.p = int(p)
582+
583+ def __eq__(self, o):
584+ return str(self) == str(o)
585+
586+ def __str__(self):
587+ return "{} {} {} {} {} {}".format(self.device,
588+ self.mountpoint,
589+ self.filesystem,
590+ self.options,
591+ self.d,
592+ self.p)
593+
594+ DEFAULT_PATH = os.path.join(os.path.sep, 'etc', 'fstab')
595+
596+ def __init__(self, path=None):
597+ if path:
598+ self._path = path
599+ else:
600+ self._path = self.DEFAULT_PATH
601+ super(Fstab, self).__init__(self._path, 'rb+')
602+
603+ def _hydrate_entry(self, line):
604+ # NOTE: use split with no arguments to split on any
605+ # whitespace including tabs
606+ return Fstab.Entry(*filter(
607+ lambda x: x not in ('', None),
608+ line.strip("\n").split()))
609+
610+ @property
611+ def entries(self):
612+ self.seek(0)
613+ for line in self.readlines():
614+ line = line.decode('us-ascii')
615+ try:
616+ if line.strip() and not line.startswith("#"):
617+ yield self._hydrate_entry(line)
618+ except ValueError:
619+ pass
620+
621+ def get_entry_by_attr(self, attr, value):
622+ for entry in self.entries:
623+ e_attr = getattr(entry, attr)
624+ if e_attr == value:
625+ return entry
626+ return None
627+
628+ def add_entry(self, entry):
629+ if self.get_entry_by_attr('device', entry.device):
630+ return False
631+
632+ self.write((str(entry) + '\n').encode('us-ascii'))
633+ self.truncate()
634+ return entry
635+
636+ def remove_entry(self, entry):
637+ self.seek(0)
638+
639+ lines = [l.decode('us-ascii') for l in self.readlines()]
640+
641+ found = False
642+ for index, line in enumerate(lines):
643+ if not line.startswith("#"):
644+ if self._hydrate_entry(line) == entry:
645+ found = True
646+ break
647+
648+ if not found:
649+ return False
650+
651+ lines.remove(line)
652+
653+ self.seek(0)
654+ self.write(''.join(lines).encode('us-ascii'))
655+ self.truncate()
656+ return True
657+
658+ @classmethod
659+ def remove_by_mountpoint(cls, mountpoint, path=None):
660+ fstab = cls(path=path)
661+ entry = fstab.get_entry_by_attr('mountpoint', mountpoint)
662+ if entry:
663+ return fstab.remove_entry(entry)
664+ return False
665+
666+ @classmethod
667+ def add(cls, device, mountpoint, filesystem, options=None, path=None):
668+ return cls(path=path).add_entry(Fstab.Entry(device,
669+ mountpoint, filesystem,
670+ options=options))
671
672=== modified file 'hooks/charmhelpers/core/hookenv.py'
673--- hooks/charmhelpers/core/hookenv.py 2013-08-21 19:14:32 +0000
674+++ hooks/charmhelpers/core/hookenv.py 2015-02-19 17:05:21 +0000
675@@ -1,3 +1,19 @@
676+# Copyright 2014-2015 Canonical Limited.
677+#
678+# This file is part of charm-helpers.
679+#
680+# charm-helpers is free software: you can redistribute it and/or modify
681+# it under the terms of the GNU Lesser General Public License version 3 as
682+# published by the Free Software Foundation.
683+#
684+# charm-helpers is distributed in the hope that it will be useful,
685+# but WITHOUT ANY WARRANTY; without even the implied warranty of
686+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
687+# GNU Lesser General Public License for more details.
688+#
689+# You should have received a copy of the GNU Lesser General Public License
690+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
691+
692 "Interactions with the Juju environment"
693 # Copyright 2013 Canonical Ltd.
694 #
695@@ -8,7 +24,14 @@
696 import json
697 import yaml
698 import subprocess
699-import UserDict
700+import sys
701+from subprocess import CalledProcessError
702+
703+import six
704+if not six.PY3:
705+ from UserDict import UserDict
706+else:
707+ from collections import UserDict
708
709 CRITICAL = "CRITICAL"
710 ERROR = "ERROR"
711@@ -21,9 +44,9 @@
712
713
714 def cached(func):
715- ''' Cache return values for multiple executions of func + args
716+ """Cache return values for multiple executions of func + args
717
718- For example:
719+ For example::
720
721 @cached
722 def unit_get(attribute):
723@@ -32,7 +55,7 @@
724 unit_get('test')
725
726 will cache the result of unit_get + 'test' for future calls.
727- '''
728+ """
729 def wrapper(*args, **kwargs):
730 global cache
731 key = str((func, args, kwargs))
732@@ -46,8 +69,8 @@
733
734
735 def flush(key):
736- ''' Flushes any entries from function cache where the
737- key is found in the function+args '''
738+ """Flushes any entries from function cache where the
739+ key is found in the function+args """
740 flush_list = []
741 for item in cache:
742 if key in item:
743@@ -57,20 +80,22 @@
744
745
746 def log(message, level=None):
747- "Write a message to the juju log"
748+ """Write a message to the juju log"""
749 command = ['juju-log']
750 if level:
751 command += ['-l', level]
752+ if not isinstance(message, six.string_types):
753+ message = repr(message)
754 command += [message]
755 subprocess.call(command)
756
757
758-class Serializable(UserDict.IterableUserDict):
759- "Wrapper, an object that can be serialized to yaml or json"
760+class Serializable(UserDict):
761+ """Wrapper, an object that can be serialized to yaml or json"""
762
763 def __init__(self, obj):
764 # wrap the object
765- UserDict.IterableUserDict.__init__(self)
766+ UserDict.__init__(self)
767 self.data = obj
768
769 def __getattr__(self, attr):
770@@ -96,11 +121,11 @@
771 self.data = state
772
773 def json(self):
774- "Serialize the object to json"
775+ """Serialize the object to json"""
776 return json.dumps(self.data)
777
778 def yaml(self):
779- "Serialize the object to yaml"
780+ """Serialize the object to yaml"""
781 return yaml.dump(self.data)
782
783
784@@ -119,50 +144,181 @@
785
786
787 def in_relation_hook():
788- "Determine whether we're running in a relation hook"
789+ """Determine whether we're running in a relation hook"""
790 return 'JUJU_RELATION' in os.environ
791
792
793 def relation_type():
794- "The scope for the current relation hook"
795+ """The scope for the current relation hook"""
796 return os.environ.get('JUJU_RELATION', None)
797
798
799 def relation_id():
800- "The relation ID for the current relation hook"
801+ """The relation ID for the current relation hook"""
802 return os.environ.get('JUJU_RELATION_ID', None)
803
804
805 def local_unit():
806- "Local unit ID"
807+ """Local unit ID"""
808 return os.environ['JUJU_UNIT_NAME']
809
810
811 def remote_unit():
812- "The remote unit for the current relation hook"
813+ """The remote unit for the current relation hook"""
814 return os.environ['JUJU_REMOTE_UNIT']
815
816
817 def service_name():
818- "The name service group this unit belongs to"
819+ """The name service group this unit belongs to"""
820 return local_unit().split('/')[0]
821
822
823+def hook_name():
824+ """The name of the currently executing hook"""
825+ return os.path.basename(sys.argv[0])
826+
827+
828+class Config(dict):
829+ """A dictionary representation of the charm's config.yaml, with some
830+ extra features:
831+
832+ - See which values in the dictionary have changed since the previous hook.
833+ - For values that have changed, see what the previous value was.
834+ - Store arbitrary data for use in a later hook.
835+
836+ NOTE: Do not instantiate this object directly - instead call
837+ ``hookenv.config()``, which will return an instance of :class:`Config`.
838+
839+ Example usage::
840+
841+ >>> # inside a hook
842+ >>> from charmhelpers.core import hookenv
843+ >>> config = hookenv.config()
844+ >>> config['foo']
845+ 'bar'
846+ >>> # store a new key/value for later use
847+ >>> config['mykey'] = 'myval'
848+
849+
850+ >>> # user runs `juju set mycharm foo=baz`
851+ >>> # now we're inside subsequent config-changed hook
852+ >>> config = hookenv.config()
853+ >>> config['foo']
854+ 'baz'
855+ >>> # test to see if this val has changed since last hook
856+ >>> config.changed('foo')
857+ True
858+ >>> # what was the previous value?
859+ >>> config.previous('foo')
860+ 'bar'
861+ >>> # keys/values that we add are preserved across hooks
862+ >>> config['mykey']
863+ 'myval'
864+
865+ """
866+ CONFIG_FILE_NAME = '.juju-persistent-config'
867+
868+ def __init__(self, *args, **kw):
869+ super(Config, self).__init__(*args, **kw)
870+ self.implicit_save = True
871+ self._prev_dict = None
872+ self.path = os.path.join(charm_dir(), Config.CONFIG_FILE_NAME)
873+ if os.path.exists(self.path):
874+ self.load_previous()
875+
876+ def __getitem__(self, key):
877+ """For regular dict lookups, check the current juju config first,
878+ then the previous (saved) copy. This ensures that user-saved values
879+ will be returned by a dict lookup.
880+
881+ """
882+ try:
883+ return dict.__getitem__(self, key)
884+ except KeyError:
885+ return (self._prev_dict or {})[key]
886+
887+ def keys(self):
888+ prev_keys = []
889+ if self._prev_dict is not None:
890+ prev_keys = self._prev_dict.keys()
891+ return list(set(prev_keys + list(dict.keys(self))))
892+
893+ def load_previous(self, path=None):
894+ """Load previous copy of config from disk.
895+
896+ In normal usage you don't need to call this method directly - it
897+ is called automatically at object initialization.
898+
899+ :param path:
900+
901+ File path from which to load the previous config. If `None`,
902+ config is loaded from the default location. If `path` is
903+ specified, subsequent `save()` calls will write to the same
904+ path.
905+
906+ """
907+ self.path = path or self.path
908+ with open(self.path) as f:
909+ self._prev_dict = json.load(f)
910+
911+ def changed(self, key):
912+ """Return True if the current value for this key is different from
913+ the previous value.
914+
915+ """
916+ if self._prev_dict is None:
917+ return True
918+ return self.previous(key) != self.get(key)
919+
920+ def previous(self, key):
921+ """Return previous value for this key, or None if there
922+ is no previous value.
923+
924+ """
925+ if self._prev_dict:
926+ return self._prev_dict.get(key)
927+ return None
928+
929+ def save(self):
930+ """Save this config to disk.
931+
932+ If the charm is using the :mod:`Services Framework <services.base>`
933+ or :meth:'@hook <Hooks.hook>' decorator, this
934+ is called automatically at the end of successful hook execution.
935+ Otherwise, it should be called directly by user code.
936+
937+ To disable automatic saves, set ``implicit_save=False`` on this
938+ instance.
939+
940+ """
941+ if self._prev_dict:
942+ for k, v in six.iteritems(self._prev_dict):
943+ if k not in self:
944+ self[k] = v
945+ with open(self.path, 'w') as f:
946+ json.dump(self, f)
947+
948+
949 @cached
950 def config(scope=None):
951- "Juju charm configuration"
952+ """Juju charm configuration"""
953 config_cmd_line = ['config-get']
954 if scope is not None:
955 config_cmd_line.append(scope)
956 config_cmd_line.append('--format=json')
957 try:
958- return json.loads(subprocess.check_output(config_cmd_line))
959+ config_data = json.loads(
960+ subprocess.check_output(config_cmd_line).decode('UTF-8'))
961+ if scope is not None:
962+ return config_data
963+ return Config(config_data)
964 except ValueError:
965 return None
966
967
968 @cached
969 def relation_get(attribute=None, unit=None, rid=None):
970+ """Get relation information"""
971 _args = ['relation-get', '--format=json']
972 if rid:
973 _args.append('-r')
974@@ -171,16 +327,22 @@
975 if unit:
976 _args.append(unit)
977 try:
978- return json.loads(subprocess.check_output(_args))
979+ return json.loads(subprocess.check_output(_args).decode('UTF-8'))
980 except ValueError:
981 return None
982-
983-
984-def relation_set(relation_id=None, relation_settings={}, **kwargs):
985+ except CalledProcessError as e:
986+ if e.returncode == 2:
987+ return None
988+ raise
989+
990+
991+def relation_set(relation_id=None, relation_settings=None, **kwargs):
992+ """Set relation information for the current unit"""
993+ relation_settings = relation_settings if relation_settings else {}
994 relation_cmd_line = ['relation-set']
995 if relation_id is not None:
996 relation_cmd_line.extend(('-r', relation_id))
997- for k, v in (relation_settings.items() + kwargs.items()):
998+ for k, v in (list(relation_settings.items()) + list(kwargs.items())):
999 if v is None:
1000 relation_cmd_line.append('{}='.format(k))
1001 else:
1002@@ -192,28 +354,30 @@
1003
1004 @cached
1005 def relation_ids(reltype=None):
1006- "A list of relation_ids"
1007+ """A list of relation_ids"""
1008 reltype = reltype or relation_type()
1009 relid_cmd_line = ['relation-ids', '--format=json']
1010 if reltype is not None:
1011 relid_cmd_line.append(reltype)
1012- return json.loads(subprocess.check_output(relid_cmd_line)) or []
1013+ return json.loads(
1014+ subprocess.check_output(relid_cmd_line).decode('UTF-8')) or []
1015 return []
1016
1017
1018 @cached
1019 def related_units(relid=None):
1020- "A list of related units"
1021+ """A list of related units"""
1022 relid = relid or relation_id()
1023 units_cmd_line = ['relation-list', '--format=json']
1024 if relid is not None:
1025 units_cmd_line.extend(('-r', relid))
1026- return json.loads(subprocess.check_output(units_cmd_line)) or []
1027+ return json.loads(
1028+ subprocess.check_output(units_cmd_line).decode('UTF-8')) or []
1029
1030
1031 @cached
1032 def relation_for_unit(unit=None, rid=None):
1033- "Get the json represenation of a unit's relation"
1034+ """Get the json represenation of a unit's relation"""
1035 unit = unit or remote_unit()
1036 relation = relation_get(unit=unit, rid=rid)
1037 for key in relation:
1038@@ -225,7 +389,7 @@
1039
1040 @cached
1041 def relations_for_id(relid=None):
1042- "Get relations of a specific relation ID"
1043+ """Get relations of a specific relation ID"""
1044 relation_data = []
1045 relid = relid or relation_ids()
1046 for unit in related_units(relid):
1047@@ -237,7 +401,7 @@
1048
1049 @cached
1050 def relations_of_type(reltype=None):
1051- "Get relations of a specific type"
1052+ """Get relations of a specific type"""
1053 relation_data = []
1054 reltype = reltype or relation_type()
1055 for relid in relation_ids(reltype):
1056@@ -248,22 +412,33 @@
1057
1058
1059 @cached
1060+def metadata():
1061+ """Get the current charm metadata.yaml contents as a python object"""
1062+ with open(os.path.join(charm_dir(), 'metadata.yaml')) as md:
1063+ return yaml.safe_load(md)
1064+
1065+
1066+@cached
1067 def relation_types():
1068- "Get a list of relation types supported by this charm"
1069- charmdir = os.environ.get('CHARM_DIR', '')
1070- mdf = open(os.path.join(charmdir, 'metadata.yaml'))
1071- md = yaml.safe_load(mdf)
1072+ """Get a list of relation types supported by this charm"""
1073 rel_types = []
1074+ md = metadata()
1075 for key in ('provides', 'requires', 'peers'):
1076 section = md.get(key)
1077 if section:
1078 rel_types.extend(section.keys())
1079- mdf.close()
1080 return rel_types
1081
1082
1083 @cached
1084+def charm_name():
1085+ """Get the name of the current charm as is specified on metadata.yaml"""
1086+ return metadata().get('name')
1087+
1088+
1089+@cached
1090 def relations():
1091+ """Get a nested dictionary of relation data for all related units"""
1092 rels = {}
1093 for reltype in relation_types():
1094 relids = {}
1095@@ -277,15 +452,35 @@
1096 return rels
1097
1098
1099+@cached
1100+def is_relation_made(relation, keys='private-address'):
1101+ '''
1102+ Determine whether a relation is established by checking for
1103+ presence of key(s). If a list of keys is provided, they
1104+ must all be present for the relation to be identified as made
1105+ '''
1106+ if isinstance(keys, str):
1107+ keys = [keys]
1108+ for r_id in relation_ids(relation):
1109+ for unit in related_units(r_id):
1110+ context = {}
1111+ for k in keys:
1112+ context[k] = relation_get(k, rid=r_id,
1113+ unit=unit)
1114+ if None not in context.values():
1115+ return True
1116+ return False
1117+
1118+
1119 def open_port(port, protocol="TCP"):
1120- "Open a service network port"
1121+ """Open a service network port"""
1122 _args = ['open-port']
1123 _args.append('{}/{}'.format(port, protocol))
1124 subprocess.check_call(_args)
1125
1126
1127 def close_port(port, protocol="TCP"):
1128- "Close a service network port"
1129+ """Close a service network port"""
1130 _args = ['close-port']
1131 _args.append('{}/{}'.format(port, protocol))
1132 subprocess.check_call(_args)
1133@@ -293,37 +488,69 @@
1134
1135 @cached
1136 def unit_get(attribute):
1137+ """Get the unit ID for the remote unit"""
1138 _args = ['unit-get', '--format=json', attribute]
1139 try:
1140- return json.loads(subprocess.check_output(_args))
1141+ return json.loads(subprocess.check_output(_args).decode('UTF-8'))
1142 except ValueError:
1143 return None
1144
1145
1146 def unit_private_ip():
1147+ """Get this unit's private IP address"""
1148 return unit_get('private-address')
1149
1150
1151 class UnregisteredHookError(Exception):
1152+ """Raised when an undefined hook is called"""
1153 pass
1154
1155
1156 class Hooks(object):
1157- def __init__(self):
1158+ """A convenient handler for hook functions.
1159+
1160+ Example::
1161+
1162+ hooks = Hooks()
1163+
1164+ # register a hook, taking its name from the function name
1165+ @hooks.hook()
1166+ def install():
1167+ pass # your code here
1168+
1169+ # register a hook, providing a custom hook name
1170+ @hooks.hook("config-changed")
1171+ def config_changed():
1172+ pass # your code here
1173+
1174+ if __name__ == "__main__":
1175+ # execute a hook based on the name the program is called by
1176+ hooks.execute(sys.argv)
1177+ """
1178+
1179+ def __init__(self, config_save=True):
1180 super(Hooks, self).__init__()
1181 self._hooks = {}
1182+ self._config_save = config_save
1183
1184 def register(self, name, function):
1185+ """Register a hook"""
1186 self._hooks[name] = function
1187
1188 def execute(self, args):
1189+ """Execute a registered hook based on args[0]"""
1190 hook_name = os.path.basename(args[0])
1191 if hook_name in self._hooks:
1192 self._hooks[hook_name]()
1193+ if self._config_save:
1194+ cfg = config()
1195+ if cfg.implicit_save:
1196+ cfg.save()
1197 else:
1198 raise UnregisteredHookError(hook_name)
1199
1200 def hook(self, *hook_names):
1201+ """Decorator, registering them as hooks"""
1202 def wrapper(decorated):
1203 for hook_name in hook_names:
1204 self.register(hook_name, decorated)
1205@@ -337,4 +564,5 @@
1206
1207
1208 def charm_dir():
1209+ """Return the root directory of the current charm"""
1210 return os.environ.get('CHARM_DIR')
1211
1212=== modified file 'hooks/charmhelpers/core/host.py'
1213--- hooks/charmhelpers/core/host.py 2013-08-21 19:14:32 +0000
1214+++ hooks/charmhelpers/core/host.py 2015-02-19 17:05:21 +0000
1215@@ -1,3 +1,19 @@
1216+# Copyright 2014-2015 Canonical Limited.
1217+#
1218+# This file is part of charm-helpers.
1219+#
1220+# charm-helpers is free software: you can redistribute it and/or modify
1221+# it under the terms of the GNU Lesser General Public License version 3 as
1222+# published by the Free Software Foundation.
1223+#
1224+# charm-helpers is distributed in the hope that it will be useful,
1225+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1226+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1227+# GNU Lesser General Public License for more details.
1228+#
1229+# You should have received a copy of the GNU Lesser General Public License
1230+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1231+
1232 """Tools for working with the host system"""
1233 # Copyright 2012 Canonical Ltd.
1234 #
1235@@ -6,43 +22,58 @@
1236 # Matthew Wedgwood <matthew.wedgwood@canonical.com>
1237
1238 import os
1239+import re
1240 import pwd
1241 import grp
1242 import random
1243 import string
1244 import subprocess
1245 import hashlib
1246-
1247+from contextlib import contextmanager
1248 from collections import OrderedDict
1249
1250-from hookenv import log
1251+import six
1252+
1253+from .hookenv import log
1254+from .fstab import Fstab
1255
1256
1257 def service_start(service_name):
1258- service('start', service_name)
1259+ """Start a system service"""
1260+ return service('start', service_name)
1261
1262
1263 def service_stop(service_name):
1264- service('stop', service_name)
1265+ """Stop a system service"""
1266+ return service('stop', service_name)
1267
1268
1269 def service_restart(service_name):
1270- service('restart', service_name)
1271+ """Restart a system service"""
1272+ return service('restart', service_name)
1273
1274
1275 def service_reload(service_name, restart_on_failure=False):
1276- if not service('reload', service_name) and restart_on_failure:
1277- service('restart', service_name)
1278+ """Reload a system service, optionally falling back to restart if
1279+ reload fails"""
1280+ service_result = service('reload', service_name)
1281+ if not service_result and restart_on_failure:
1282+ service_result = service('restart', service_name)
1283+ return service_result
1284
1285
1286 def service(action, service_name):
1287+ """Control a system service"""
1288 cmd = ['service', service_name, action]
1289 return subprocess.call(cmd) == 0
1290
1291
1292 def service_running(service):
1293+ """Determine whether a system service is running"""
1294 try:
1295- output = subprocess.check_output(['service', service, 'status'])
1296+ output = subprocess.check_output(
1297+ ['service', service, 'status'],
1298+ stderr=subprocess.STDOUT).decode('UTF-8')
1299 except subprocess.CalledProcessError:
1300 return False
1301 else:
1302@@ -52,8 +83,20 @@
1303 return False
1304
1305
1306+def service_available(service_name):
1307+ """Determine whether a system service is available"""
1308+ try:
1309+ subprocess.check_output(
1310+ ['service', service_name, 'status'],
1311+ stderr=subprocess.STDOUT).decode('UTF-8')
1312+ except subprocess.CalledProcessError as e:
1313+ return 'unrecognized service' not in e.output
1314+ else:
1315+ return True
1316+
1317+
1318 def adduser(username, password=None, shell='/bin/bash', system_user=False):
1319- """Add a user"""
1320+ """Add a user to the system"""
1321 try:
1322 user_info = pwd.getpwnam(username)
1323 log('user {0} already exists!'.format(username))
1324@@ -74,6 +117,26 @@
1325 return user_info
1326
1327
1328+def add_group(group_name, system_group=False):
1329+ """Add a group to the system"""
1330+ try:
1331+ group_info = grp.getgrnam(group_name)
1332+ log('group {0} already exists!'.format(group_name))
1333+ except KeyError:
1334+ log('creating group {0}'.format(group_name))
1335+ cmd = ['addgroup']
1336+ if system_group:
1337+ cmd.append('--system')
1338+ else:
1339+ cmd.extend([
1340+ '--group',
1341+ ])
1342+ cmd.append(group_name)
1343+ subprocess.check_call(cmd)
1344+ group_info = grp.getgrnam(group_name)
1345+ return group_info
1346+
1347+
1348 def add_user_to_group(username, group):
1349 """Add a user to a group"""
1350 cmd = [
1351@@ -93,7 +156,7 @@
1352 cmd.append(from_path)
1353 cmd.append(to_path)
1354 log(" ".join(cmd))
1355- return subprocess.check_output(cmd).strip()
1356+ return subprocess.check_output(cmd).decode('UTF-8').strip()
1357
1358
1359 def symlink(source, destination):
1360@@ -108,66 +171,81 @@
1361 subprocess.check_call(cmd)
1362
1363
1364-def mkdir(path, owner='root', group='root', perms=0555, force=False):
1365+def mkdir(path, owner='root', group='root', perms=0o555, force=False):
1366 """Create a directory"""
1367 log("Making dir {} {}:{} {:o}".format(path, owner, group,
1368 perms))
1369 uid = pwd.getpwnam(owner).pw_uid
1370 gid = grp.getgrnam(group).gr_gid
1371 realpath = os.path.abspath(path)
1372- if os.path.exists(realpath):
1373- if force and not os.path.isdir(realpath):
1374+ path_exists = os.path.exists(realpath)
1375+ if path_exists and force:
1376+ if not os.path.isdir(realpath):
1377 log("Removing non-directory file {} prior to mkdir()".format(path))
1378 os.unlink(realpath)
1379- else:
1380+ os.makedirs(realpath, perms)
1381+ elif not path_exists:
1382 os.makedirs(realpath, perms)
1383 os.chown(realpath, uid, gid)
1384-
1385-
1386-def write_file(path, content, owner='root', group='root', perms=0444):
1387- """Create or overwrite a file with the contents of a string"""
1388+ os.chmod(realpath, perms)
1389+
1390+
1391+def write_file(path, content, owner='root', group='root', perms=0o444):
1392+ """Create or overwrite a file with the contents of a byte string."""
1393 log("Writing file {} {}:{} {:o}".format(path, owner, group, perms))
1394 uid = pwd.getpwnam(owner).pw_uid
1395 gid = grp.getgrnam(group).gr_gid
1396- with open(path, 'w') as target:
1397+ with open(path, 'wb') as target:
1398 os.fchown(target.fileno(), uid, gid)
1399 os.fchmod(target.fileno(), perms)
1400 target.write(content)
1401
1402
1403-def mount(device, mountpoint, options=None, persist=False):
1404- '''Mount a filesystem'''
1405+def fstab_remove(mp):
1406+ """Remove the given mountpoint entry from /etc/fstab
1407+ """
1408+ return Fstab.remove_by_mountpoint(mp)
1409+
1410+
1411+def fstab_add(dev, mp, fs, options=None):
1412+ """Adds the given device entry to the /etc/fstab file
1413+ """
1414+ return Fstab.add(dev, mp, fs, options=options)
1415+
1416+
1417+def mount(device, mountpoint, options=None, persist=False, filesystem="ext3"):
1418+ """Mount a filesystem at a particular mountpoint"""
1419 cmd_args = ['mount']
1420 if options is not None:
1421 cmd_args.extend(['-o', options])
1422 cmd_args.extend([device, mountpoint])
1423 try:
1424 subprocess.check_output(cmd_args)
1425- except subprocess.CalledProcessError, e:
1426+ except subprocess.CalledProcessError as e:
1427 log('Error mounting {} at {}\n{}'.format(device, mountpoint, e.output))
1428 return False
1429+
1430 if persist:
1431- # TODO: update fstab
1432- pass
1433+ return fstab_add(device, mountpoint, filesystem, options=options)
1434 return True
1435
1436
1437 def umount(mountpoint, persist=False):
1438- '''Unmount a filesystem'''
1439+ """Unmount a filesystem"""
1440 cmd_args = ['umount', mountpoint]
1441 try:
1442 subprocess.check_output(cmd_args)
1443- except subprocess.CalledProcessError, e:
1444+ except subprocess.CalledProcessError as e:
1445 log('Error unmounting {}\n{}'.format(mountpoint, e.output))
1446 return False
1447+
1448 if persist:
1449- # TODO: update fstab
1450- pass
1451+ return fstab_remove(mountpoint)
1452 return True
1453
1454
1455 def mounts():
1456- '''List of all mounted volumes as [[mountpoint,device],[...]]'''
1457+ """Get a list of all mounted volumes as [[mountpoint,device],[...]]"""
1458 with open('/proc/mounts') as f:
1459 # [['/mount/point','/dev/path'],[...]]
1460 system_mounts = [m[1::-1] for m in [l.strip().split()
1461@@ -175,50 +253,81 @@
1462 return system_mounts
1463
1464
1465-def file_hash(path):
1466- ''' Generate a md5 hash of the contents of 'path' or None if not found '''
1467+def file_hash(path, hash_type='md5'):
1468+ """
1469+ Generate a hash checksum of the contents of 'path' or None if not found.
1470+
1471+ :param str hash_type: Any hash alrgorithm supported by :mod:`hashlib`,
1472+ such as md5, sha1, sha256, sha512, etc.
1473+ """
1474 if os.path.exists(path):
1475- h = hashlib.md5()
1476- with open(path, 'r') as source:
1477- h.update(source.read()) # IGNORE:E1101 - it does have update
1478+ h = getattr(hashlib, hash_type)()
1479+ with open(path, 'rb') as source:
1480+ h.update(source.read())
1481 return h.hexdigest()
1482 else:
1483 return None
1484
1485
1486-def restart_on_change(restart_map):
1487- ''' Restart services based on configuration files changing
1488-
1489- This function is used a decorator, for example
1490+def check_hash(path, checksum, hash_type='md5'):
1491+ """
1492+ Validate a file using a cryptographic checksum.
1493+
1494+ :param str checksum: Value of the checksum used to validate the file.
1495+ :param str hash_type: Hash algorithm used to generate `checksum`.
1496+ Can be any hash alrgorithm supported by :mod:`hashlib`,
1497+ such as md5, sha1, sha256, sha512, etc.
1498+ :raises ChecksumError: If the file fails the checksum
1499+
1500+ """
1501+ actual_checksum = file_hash(path, hash_type)
1502+ if checksum != actual_checksum:
1503+ raise ChecksumError("'%s' != '%s'" % (checksum, actual_checksum))
1504+
1505+
1506+class ChecksumError(ValueError):
1507+ pass
1508+
1509+
1510+def restart_on_change(restart_map, stopstart=False):
1511+ """Restart services based on configuration files changing
1512+
1513+ This function is used a decorator, for example::
1514
1515 @restart_on_change({
1516 '/etc/ceph/ceph.conf': [ 'cinder-api', 'cinder-volume' ]
1517 })
1518 def ceph_client_changed():
1519- ...
1520+ pass # your code here
1521
1522 In this example, the cinder-api and cinder-volume services
1523 would be restarted if /etc/ceph/ceph.conf is changed by the
1524 ceph_client_changed function.
1525- '''
1526+ """
1527 def wrap(f):
1528- def wrapped_f(*args):
1529+ def wrapped_f(*args, **kwargs):
1530 checksums = {}
1531 for path in restart_map:
1532 checksums[path] = file_hash(path)
1533- f(*args)
1534+ f(*args, **kwargs)
1535 restarts = []
1536 for path in restart_map:
1537 if checksums[path] != file_hash(path):
1538 restarts += restart_map[path]
1539- for service_name in list(OrderedDict.fromkeys(restarts)):
1540- service('restart', service_name)
1541+ services_list = list(OrderedDict.fromkeys(restarts))
1542+ if not stopstart:
1543+ for service_name in services_list:
1544+ service('restart', service_name)
1545+ else:
1546+ for action in ['stop', 'start']:
1547+ for service_name in services_list:
1548+ service(action, service_name)
1549 return wrapped_f
1550 return wrap
1551
1552
1553 def lsb_release():
1554- '''Return /etc/lsb-release in a dict'''
1555+ """Return /etc/lsb-release in a dict"""
1556 d = {}
1557 with open('/etc/lsb-release', 'r') as lsb:
1558 for l in lsb:
1559@@ -228,12 +337,110 @@
1560
1561
1562 def pwgen(length=None):
1563- '''Generate a random pasword.'''
1564+ """Generate a random pasword."""
1565 if length is None:
1566 length = random.choice(range(35, 45))
1567 alphanumeric_chars = [
1568- l for l in (string.letters + string.digits)
1569+ l for l in (string.ascii_letters + string.digits)
1570 if l not in 'l0QD1vAEIOUaeiou']
1571 random_chars = [
1572 random.choice(alphanumeric_chars) for _ in range(length)]
1573 return(''.join(random_chars))
1574+
1575+
1576+def list_nics(nic_type):
1577+ '''Return a list of nics of given type(s)'''
1578+ if isinstance(nic_type, six.string_types):
1579+ int_types = [nic_type]
1580+ else:
1581+ int_types = nic_type
1582+ interfaces = []
1583+ for int_type in int_types:
1584+ cmd = ['ip', 'addr', 'show', 'label', int_type + '*']
1585+ ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
1586+ ip_output = (line for line in ip_output if line)
1587+ for line in ip_output:
1588+ if line.split()[1].startswith(int_type):
1589+ matched = re.search('.*: (' + int_type + r'[0-9]+\.[0-9]+)@.*', line)
1590+ if matched:
1591+ interface = matched.groups()[0]
1592+ else:
1593+ interface = line.split()[1].replace(":", "")
1594+ interfaces.append(interface)
1595+
1596+ return interfaces
1597+
1598+
1599+def set_nic_mtu(nic, mtu):
1600+ '''Set MTU on a network interface'''
1601+ cmd = ['ip', 'link', 'set', nic, 'mtu', mtu]
1602+ subprocess.check_call(cmd)
1603+
1604+
1605+def get_nic_mtu(nic):
1606+ cmd = ['ip', 'addr', 'show', nic]
1607+ ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
1608+ mtu = ""
1609+ for line in ip_output:
1610+ words = line.split()
1611+ if 'mtu' in words:
1612+ mtu = words[words.index("mtu") + 1]
1613+ return mtu
1614+
1615+
1616+def get_nic_hwaddr(nic):
1617+ cmd = ['ip', '-o', '-0', 'addr', 'show', nic]
1618+ ip_output = subprocess.check_output(cmd).decode('UTF-8')
1619+ hwaddr = ""
1620+ words = ip_output.split()
1621+ if 'link/ether' in words:
1622+ hwaddr = words[words.index('link/ether') + 1]
1623+ return hwaddr
1624+
1625+
1626+def cmp_pkgrevno(package, revno, pkgcache=None):
1627+ '''Compare supplied revno with the revno of the installed package
1628+
1629+ * 1 => Installed revno is greater than supplied arg
1630+ * 0 => Installed revno is the same as supplied arg
1631+ * -1 => Installed revno is less than supplied arg
1632+
1633+ This function imports apt_cache function from charmhelpers.fetch if
1634+ the pkgcache argument is None. Be sure to add charmhelpers.fetch if
1635+ you call this function, or pass an apt_pkg.Cache() instance.
1636+ '''
1637+ import apt_pkg
1638+ if not pkgcache:
1639+ from charmhelpers.fetch import apt_cache
1640+ pkgcache = apt_cache()
1641+ pkg = pkgcache[package]
1642+ return apt_pkg.version_compare(pkg.current_ver.ver_str, revno)
1643+
1644+
1645+@contextmanager
1646+def chdir(d):
1647+ cur = os.getcwd()
1648+ try:
1649+ yield os.chdir(d)
1650+ finally:
1651+ os.chdir(cur)
1652+
1653+
1654+def chownr(path, owner, group, follow_links=True):
1655+ uid = pwd.getpwnam(owner).pw_uid
1656+ gid = grp.getgrnam(group).gr_gid
1657+ if follow_links:
1658+ chown = os.chown
1659+ else:
1660+ chown = os.lchown
1661+
1662+ for root, dirs, files in os.walk(path):
1663+ for name in dirs + files:
1664+ full = os.path.join(root, name)
1665+ broken_symlink = os.path.lexists(full) and not os.path.exists(full)
1666+ if not broken_symlink:
1667+ chown(full, uid, gid)
1668+
1669+
1670+def lchownr(path, owner, group):
1671+ chownr(path, owner, group, follow_links=False)
1672
1673=== added directory 'hooks/charmhelpers/core/services'
1674=== added file 'hooks/charmhelpers/core/services/__init__.py'
1675--- hooks/charmhelpers/core/services/__init__.py 1970-01-01 00:00:00 +0000
1676+++ hooks/charmhelpers/core/services/__init__.py 2015-02-19 17:05:21 +0000
1677@@ -0,0 +1,18 @@
1678+# Copyright 2014-2015 Canonical Limited.
1679+#
1680+# This file is part of charm-helpers.
1681+#
1682+# charm-helpers is free software: you can redistribute it and/or modify
1683+# it under the terms of the GNU Lesser General Public License version 3 as
1684+# published by the Free Software Foundation.
1685+#
1686+# charm-helpers is distributed in the hope that it will be useful,
1687+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1688+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1689+# GNU Lesser General Public License for more details.
1690+#
1691+# You should have received a copy of the GNU Lesser General Public License
1692+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1693+
1694+from .base import * # NOQA
1695+from .helpers import * # NOQA
1696
1697=== added file 'hooks/charmhelpers/core/services/base.py'
1698--- hooks/charmhelpers/core/services/base.py 1970-01-01 00:00:00 +0000
1699+++ hooks/charmhelpers/core/services/base.py 2015-02-19 17:05:21 +0000
1700@@ -0,0 +1,329 @@
1701+# Copyright 2014-2015 Canonical Limited.
1702+#
1703+# This file is part of charm-helpers.
1704+#
1705+# charm-helpers is free software: you can redistribute it and/or modify
1706+# it under the terms of the GNU Lesser General Public License version 3 as
1707+# published by the Free Software Foundation.
1708+#
1709+# charm-helpers is distributed in the hope that it will be useful,
1710+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1711+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1712+# GNU Lesser General Public License for more details.
1713+#
1714+# You should have received a copy of the GNU Lesser General Public License
1715+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1716+
1717+import os
1718+import re
1719+import json
1720+from collections import Iterable
1721+
1722+from charmhelpers.core import host
1723+from charmhelpers.core import hookenv
1724+
1725+
1726+__all__ = ['ServiceManager', 'ManagerCallback',
1727+ 'PortManagerCallback', 'open_ports', 'close_ports', 'manage_ports',
1728+ 'service_restart', 'service_stop']
1729+
1730+
1731+class ServiceManager(object):
1732+ def __init__(self, services=None):
1733+ """
1734+ Register a list of services, given their definitions.
1735+
1736+ Service definitions are dicts in the following formats (all keys except
1737+ 'service' are optional)::
1738+
1739+ {
1740+ "service": <service name>,
1741+ "required_data": <list of required data contexts>,
1742+ "provided_data": <list of provided data contexts>,
1743+ "data_ready": <one or more callbacks>,
1744+ "data_lost": <one or more callbacks>,
1745+ "start": <one or more callbacks>,
1746+ "stop": <one or more callbacks>,
1747+ "ports": <list of ports to manage>,
1748+ }
1749+
1750+ The 'required_data' list should contain dicts of required data (or
1751+ dependency managers that act like dicts and know how to collect the data).
1752+ Only when all items in the 'required_data' list are populated are the list
1753+ of 'data_ready' and 'start' callbacks executed. See `is_ready()` for more
1754+ information.
1755+
1756+ The 'provided_data' list should contain relation data providers, most likely
1757+ a subclass of :class:`charmhelpers.core.services.helpers.RelationContext`,
1758+ that will indicate a set of data to set on a given relation.
1759+
1760+ The 'data_ready' value should be either a single callback, or a list of
1761+ callbacks, to be called when all items in 'required_data' pass `is_ready()`.
1762+ Each callback will be called with the service name as the only parameter.
1763+ After all of the 'data_ready' callbacks are called, the 'start' callbacks
1764+ are fired.
1765+
1766+ The 'data_lost' value should be either a single callback, or a list of
1767+ callbacks, to be called when a 'required_data' item no longer passes
1768+ `is_ready()`. Each callback will be called with the service name as the
1769+ only parameter. After all of the 'data_lost' callbacks are called,
1770+ the 'stop' callbacks are fired.
1771+
1772+ The 'start' value should be either a single callback, or a list of
1773+ callbacks, to be called when starting the service, after the 'data_ready'
1774+ callbacks are complete. Each callback will be called with the service
1775+ name as the only parameter. This defaults to
1776+ `[host.service_start, services.open_ports]`.
1777+
1778+ The 'stop' value should be either a single callback, or a list of
1779+ callbacks, to be called when stopping the service. If the service is
1780+ being stopped because it no longer has all of its 'required_data', this
1781+ will be called after all of the 'data_lost' callbacks are complete.
1782+ Each callback will be called with the service name as the only parameter.
1783+ This defaults to `[services.close_ports, host.service_stop]`.
1784+
1785+ The 'ports' value should be a list of ports to manage. The default
1786+ 'start' handler will open the ports after the service is started,
1787+ and the default 'stop' handler will close the ports prior to stopping
1788+ the service.
1789+
1790+
1791+ Examples:
1792+
1793+ The following registers an Upstart service called bingod that depends on
1794+ a mongodb relation and which runs a custom `db_migrate` function prior to
1795+ restarting the service, and a Runit service called spadesd::
1796+
1797+ manager = services.ServiceManager([
1798+ {
1799+ 'service': 'bingod',
1800+ 'ports': [80, 443],
1801+ 'required_data': [MongoRelation(), config(), {'my': 'data'}],
1802+ 'data_ready': [
1803+ services.template(source='bingod.conf'),
1804+ services.template(source='bingod.ini',
1805+ target='/etc/bingod.ini',
1806+ owner='bingo', perms=0400),
1807+ ],
1808+ },
1809+ {
1810+ 'service': 'spadesd',
1811+ 'data_ready': services.template(source='spadesd_run.j2',
1812+ target='/etc/sv/spadesd/run',
1813+ perms=0555),
1814+ 'start': runit_start,
1815+ 'stop': runit_stop,
1816+ },
1817+ ])
1818+ manager.manage()
1819+ """
1820+ self._ready_file = os.path.join(hookenv.charm_dir(), 'READY-SERVICES.json')
1821+ self._ready = None
1822+ self.services = {}
1823+ for service in services or []:
1824+ service_name = service['service']
1825+ self.services[service_name] = service
1826+
1827+ def manage(self):
1828+ """
1829+ Handle the current hook by doing The Right Thing with the registered services.
1830+ """
1831+ hook_name = hookenv.hook_name()
1832+ if hook_name == 'stop':
1833+ self.stop_services()
1834+ else:
1835+ self.provide_data()
1836+ self.reconfigure_services()
1837+ cfg = hookenv.config()
1838+ if cfg.implicit_save:
1839+ cfg.save()
1840+
1841+ def provide_data(self):
1842+ """
1843+ Set the relation data for each provider in the ``provided_data`` list.
1844+
1845+ A provider must have a `name` attribute, which indicates which relation
1846+ to set data on, and a `provide_data()` method, which returns a dict of
1847+ data to set.
1848+ """
1849+ hook_name = hookenv.hook_name()
1850+ for service in self.services.values():
1851+ for provider in service.get('provided_data', []):
1852+ if re.match(r'{}-relation-(joined|changed)'.format(provider.name), hook_name):
1853+ data = provider.provide_data()
1854+ _ready = provider._is_ready(data) if hasattr(provider, '_is_ready') else data
1855+ if _ready:
1856+ hookenv.relation_set(None, data)
1857+
1858+ def reconfigure_services(self, *service_names):
1859+ """
1860+ Update all files for one or more registered services, and,
1861+ if ready, optionally restart them.
1862+
1863+ If no service names are given, reconfigures all registered services.
1864+ """
1865+ for service_name in service_names or self.services.keys():
1866+ if self.is_ready(service_name):
1867+ self.fire_event('data_ready', service_name)
1868+ self.fire_event('start', service_name, default=[
1869+ service_restart,
1870+ manage_ports])
1871+ self.save_ready(service_name)
1872+ else:
1873+ if self.was_ready(service_name):
1874+ self.fire_event('data_lost', service_name)
1875+ self.fire_event('stop', service_name, default=[
1876+ manage_ports,
1877+ service_stop])
1878+ self.save_lost(service_name)
1879+
1880+ def stop_services(self, *service_names):
1881+ """
1882+ Stop one or more registered services, by name.
1883+
1884+ If no service names are given, stops all registered services.
1885+ """
1886+ for service_name in service_names or self.services.keys():
1887+ self.fire_event('stop', service_name, default=[
1888+ manage_ports,
1889+ service_stop])
1890+
1891+ def get_service(self, service_name):
1892+ """
1893+ Given the name of a registered service, return its service definition.
1894+ """
1895+ service = self.services.get(service_name)
1896+ if not service:
1897+ raise KeyError('Service not registered: %s' % service_name)
1898+ return service
1899+
1900+ def fire_event(self, event_name, service_name, default=None):
1901+ """
1902+ Fire a data_ready, data_lost, start, or stop event on a given service.
1903+ """
1904+ service = self.get_service(service_name)
1905+ callbacks = service.get(event_name, default)
1906+ if not callbacks:
1907+ return
1908+ if not isinstance(callbacks, Iterable):
1909+ callbacks = [callbacks]
1910+ for callback in callbacks:
1911+ if isinstance(callback, ManagerCallback):
1912+ callback(self, service_name, event_name)
1913+ else:
1914+ callback(service_name)
1915+
1916+ def is_ready(self, service_name):
1917+ """
1918+ Determine if a registered service is ready, by checking its 'required_data'.
1919+
1920+ A 'required_data' item can be any mapping type, and is considered ready
1921+ if `bool(item)` evaluates as True.
1922+ """
1923+ service = self.get_service(service_name)
1924+ reqs = service.get('required_data', [])
1925+ return all(bool(req) for req in reqs)
1926+
1927+ def _load_ready_file(self):
1928+ if self._ready is not None:
1929+ return
1930+ if os.path.exists(self._ready_file):
1931+ with open(self._ready_file) as fp:
1932+ self._ready = set(json.load(fp))
1933+ else:
1934+ self._ready = set()
1935+
1936+ def _save_ready_file(self):
1937+ if self._ready is None:
1938+ return
1939+ with open(self._ready_file, 'w') as fp:
1940+ json.dump(list(self._ready), fp)
1941+
1942+ def save_ready(self, service_name):
1943+ """
1944+ Save an indicator that the given service is now data_ready.
1945+ """
1946+ self._load_ready_file()
1947+ self._ready.add(service_name)
1948+ self._save_ready_file()
1949+
1950+ def save_lost(self, service_name):
1951+ """
1952+ Save an indicator that the given service is no longer data_ready.
1953+ """
1954+ self._load_ready_file()
1955+ self._ready.discard(service_name)
1956+ self._save_ready_file()
1957+
1958+ def was_ready(self, service_name):
1959+ """
1960+ Determine if the given service was previously data_ready.
1961+ """
1962+ self._load_ready_file()
1963+ return service_name in self._ready
1964+
1965+
1966+class ManagerCallback(object):
1967+ """
1968+ Special case of a callback that takes the `ServiceManager` instance
1969+ in addition to the service name.
1970+
1971+ Subclasses should implement `__call__` which should accept three parameters:
1972+
1973+ * `manager` The `ServiceManager` instance
1974+ * `service_name` The name of the service it's being triggered for
1975+ * `event_name` The name of the event that this callback is handling
1976+ """
1977+ def __call__(self, manager, service_name, event_name):
1978+ raise NotImplementedError()
1979+
1980+
1981+class PortManagerCallback(ManagerCallback):
1982+ """
1983+ Callback class that will open or close ports, for use as either
1984+ a start or stop action.
1985+ """
1986+ def __call__(self, manager, service_name, event_name):
1987+ service = manager.get_service(service_name)
1988+ new_ports = service.get('ports', [])
1989+ port_file = os.path.join(hookenv.charm_dir(), '.{}.ports'.format(service_name))
1990+ if os.path.exists(port_file):
1991+ with open(port_file) as fp:
1992+ old_ports = fp.read().split(',')
1993+ for old_port in old_ports:
1994+ if bool(old_port):
1995+ old_port = int(old_port)
1996+ if old_port not in new_ports:
1997+ hookenv.close_port(old_port)
1998+ with open(port_file, 'w') as fp:
1999+ fp.write(','.join(str(port) for port in new_ports))
2000+ for port in new_ports:
2001+ if event_name == 'start':
2002+ hookenv.open_port(port)
2003+ elif event_name == 'stop':
2004+ hookenv.close_port(port)
2005+
2006+
2007+def service_stop(service_name):
2008+ """
2009+ Wrapper around host.service_stop to prevent spurious "unknown service"
2010+ messages in the logs.
2011+ """
2012+ if host.service_running(service_name):
2013+ host.service_stop(service_name)
2014+
2015+
2016+def service_restart(service_name):
2017+ """
2018+ Wrapper around host.service_restart to prevent spurious "unknown service"
2019+ messages in the logs.
2020+ """
2021+ if host.service_available(service_name):
2022+ if host.service_running(service_name):
2023+ host.service_restart(service_name)
2024+ else:
2025+ host.service_start(service_name)
2026+
2027+
2028+# Convenience aliases
2029+open_ports = close_ports = manage_ports = PortManagerCallback()
2030
2031=== added file 'hooks/charmhelpers/core/services/helpers.py'
2032--- hooks/charmhelpers/core/services/helpers.py 1970-01-01 00:00:00 +0000
2033+++ hooks/charmhelpers/core/services/helpers.py 2015-02-19 17:05:21 +0000
2034@@ -0,0 +1,259 @@
2035+# Copyright 2014-2015 Canonical Limited.
2036+#
2037+# This file is part of charm-helpers.
2038+#
2039+# charm-helpers is free software: you can redistribute it and/or modify
2040+# it under the terms of the GNU Lesser General Public License version 3 as
2041+# published by the Free Software Foundation.
2042+#
2043+# charm-helpers is distributed in the hope that it will be useful,
2044+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2045+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2046+# GNU Lesser General Public License for more details.
2047+#
2048+# You should have received a copy of the GNU Lesser General Public License
2049+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2050+
2051+import os
2052+import yaml
2053+from charmhelpers.core import hookenv
2054+from charmhelpers.core import templating
2055+
2056+from charmhelpers.core.services.base import ManagerCallback
2057+
2058+
2059+__all__ = ['RelationContext', 'TemplateCallback',
2060+ 'render_template', 'template']
2061+
2062+
2063+class RelationContext(dict):
2064+ """
2065+ Base class for a context generator that gets relation data from juju.
2066+
2067+ Subclasses must provide the attributes `name`, which is the name of the
2068+ interface of interest, `interface`, which is the type of the interface of
2069+ interest, and `required_keys`, which is the set of keys required for the
2070+ relation to be considered complete. The data for all interfaces matching
2071+ the `name` attribute that are complete will used to populate the dictionary
2072+ values (see `get_data`, below).
2073+
2074+ The generated context will be namespaced under the relation :attr:`name`,
2075+ to prevent potential naming conflicts.
2076+
2077+ :param str name: Override the relation :attr:`name`, since it can vary from charm to charm
2078+ :param list additional_required_keys: Extend the list of :attr:`required_keys`
2079+ """
2080+ name = None
2081+ interface = None
2082+ required_keys = []
2083+
2084+ def __init__(self, name=None, additional_required_keys=None):
2085+ if name is not None:
2086+ self.name = name
2087+ if additional_required_keys is not None:
2088+ self.required_keys.extend(additional_required_keys)
2089+ self.get_data()
2090+
2091+ def __bool__(self):
2092+ """
2093+ Returns True if all of the required_keys are available.
2094+ """
2095+ return self.is_ready()
2096+
2097+ __nonzero__ = __bool__
2098+
2099+ def __repr__(self):
2100+ return super(RelationContext, self).__repr__()
2101+
2102+ def is_ready(self):
2103+ """
2104+ Returns True if all of the `required_keys` are available from any units.
2105+ """
2106+ ready = len(self.get(self.name, [])) > 0
2107+ if not ready:
2108+ hookenv.log('Incomplete relation: {}'.format(self.__class__.__name__), hookenv.DEBUG)
2109+ return ready
2110+
2111+ def _is_ready(self, unit_data):
2112+ """
2113+ Helper method that tests a set of relation data and returns True if
2114+ all of the `required_keys` are present.
2115+ """
2116+ return set(unit_data.keys()).issuperset(set(self.required_keys))
2117+
2118+ def get_data(self):
2119+ """
2120+ Retrieve the relation data for each unit involved in a relation and,
2121+ if complete, store it in a list under `self[self.name]`. This
2122+ is automatically called when the RelationContext is instantiated.
2123+
2124+ The units are sorted lexographically first by the service ID, then by
2125+ the unit ID. Thus, if an interface has two other services, 'db:1'
2126+ and 'db:2', with 'db:1' having two units, 'wordpress/0' and 'wordpress/1',
2127+ and 'db:2' having one unit, 'mediawiki/0', all of which have a complete
2128+ set of data, the relation data for the units will be stored in the
2129+ order: 'wordpress/0', 'wordpress/1', 'mediawiki/0'.
2130+
2131+ If you only care about a single unit on the relation, you can just
2132+ access it as `{{ interface[0]['key'] }}`. However, if you can at all
2133+ support multiple units on a relation, you should iterate over the list,
2134+ like::
2135+
2136+ {% for unit in interface -%}
2137+ {{ unit['key'] }}{% if not loop.last %},{% endif %}
2138+ {%- endfor %}
2139+
2140+ Note that since all sets of relation data from all related services and
2141+ units are in a single list, if you need to know which service or unit a
2142+ set of data came from, you'll need to extend this class to preserve
2143+ that information.
2144+ """
2145+ if not hookenv.relation_ids(self.name):
2146+ return
2147+
2148+ ns = self.setdefault(self.name, [])
2149+ for rid in sorted(hookenv.relation_ids(self.name)):
2150+ for unit in sorted(hookenv.related_units(rid)):
2151+ reldata = hookenv.relation_get(rid=rid, unit=unit)
2152+ if self._is_ready(reldata):
2153+ ns.append(reldata)
2154+
2155+ def provide_data(self):
2156+ """
2157+ Return data to be relation_set for this interface.
2158+ """
2159+ return {}
2160+
2161+
2162+class MysqlRelation(RelationContext):
2163+ """
2164+ Relation context for the `mysql` interface.
2165+
2166+ :param str name: Override the relation :attr:`name`, since it can vary from charm to charm
2167+ :param list additional_required_keys: Extend the list of :attr:`required_keys`
2168+ """
2169+ name = 'db'
2170+ interface = 'mysql'
2171+ required_keys = ['host', 'user', 'password', 'database']
2172+
2173+
2174+class HttpRelation(RelationContext):
2175+ """
2176+ Relation context for the `http` interface.
2177+
2178+ :param str name: Override the relation :attr:`name`, since it can vary from charm to charm
2179+ :param list additional_required_keys: Extend the list of :attr:`required_keys`
2180+ """
2181+ name = 'website'
2182+ interface = 'http'
2183+ required_keys = ['host', 'port']
2184+
2185+ def provide_data(self):
2186+ return {
2187+ 'host': hookenv.unit_get('private-address'),
2188+ 'port': 80,
2189+ }
2190+
2191+
2192+class RequiredConfig(dict):
2193+ """
2194+ Data context that loads config options with one or more mandatory options.
2195+
2196+ Once the required options have been changed from their default values, all
2197+ config options will be available, namespaced under `config` to prevent
2198+ potential naming conflicts (for example, between a config option and a
2199+ relation property).
2200+
2201+ :param list *args: List of options that must be changed from their default values.
2202+ """
2203+
2204+ def __init__(self, *args):
2205+ self.required_options = args
2206+ self['config'] = hookenv.config()
2207+ with open(os.path.join(hookenv.charm_dir(), 'config.yaml')) as fp:
2208+ self.config = yaml.load(fp).get('options', {})
2209+
2210+ def __bool__(self):
2211+ for option in self.required_options:
2212+ if option not in self['config']:
2213+ return False
2214+ current_value = self['config'][option]
2215+ default_value = self.config[option].get('default')
2216+ if current_value == default_value:
2217+ return False
2218+ if current_value in (None, '') and default_value in (None, ''):
2219+ return False
2220+ return True
2221+
2222+ def __nonzero__(self):
2223+ return self.__bool__()
2224+
2225+
2226+class StoredContext(dict):
2227+ """
2228+ A data context that always returns the data that it was first created with.
2229+
2230+ This is useful to do a one-time generation of things like passwords, that
2231+ will thereafter use the same value that was originally generated, instead
2232+ of generating a new value each time it is run.
2233+ """
2234+ def __init__(self, file_name, config_data):
2235+ """
2236+ If the file exists, populate `self` with the data from the file.
2237+ Otherwise, populate with the given data and persist it to the file.
2238+ """
2239+ if os.path.exists(file_name):
2240+ self.update(self.read_context(file_name))
2241+ else:
2242+ self.store_context(file_name, config_data)
2243+ self.update(config_data)
2244+
2245+ def store_context(self, file_name, config_data):
2246+ if not os.path.isabs(file_name):
2247+ file_name = os.path.join(hookenv.charm_dir(), file_name)
2248+ with open(file_name, 'w') as file_stream:
2249+ os.fchmod(file_stream.fileno(), 0o600)
2250+ yaml.dump(config_data, file_stream)
2251+
2252+ def read_context(self, file_name):
2253+ if not os.path.isabs(file_name):
2254+ file_name = os.path.join(hookenv.charm_dir(), file_name)
2255+ with open(file_name, 'r') as file_stream:
2256+ data = yaml.load(file_stream)
2257+ if not data:
2258+ raise OSError("%s is empty" % file_name)
2259+ return data
2260+
2261+
2262+class TemplateCallback(ManagerCallback):
2263+ """
2264+ Callback class that will render a Jinja2 template, for use as a ready
2265+ action.
2266+
2267+ :param str source: The template source file, relative to
2268+ `$CHARM_DIR/templates`
2269+
2270+ :param str target: The target to write the rendered template to
2271+ :param str owner: The owner of the rendered file
2272+ :param str group: The group of the rendered file
2273+ :param int perms: The permissions of the rendered file
2274+ """
2275+ def __init__(self, source, target,
2276+ owner='root', group='root', perms=0o444):
2277+ self.source = source
2278+ self.target = target
2279+ self.owner = owner
2280+ self.group = group
2281+ self.perms = perms
2282+
2283+ def __call__(self, manager, service_name, event_name):
2284+ service = manager.get_service(service_name)
2285+ context = {}
2286+ for ctx in service.get('required_data', []):
2287+ context.update(ctx)
2288+ templating.render(self.source, self.target, context,
2289+ self.owner, self.group, self.perms)
2290+
2291+
2292+# Convenience aliases for templates
2293+render_template = template = TemplateCallback
2294
2295=== added file 'hooks/charmhelpers/core/sysctl.py'
2296--- hooks/charmhelpers/core/sysctl.py 1970-01-01 00:00:00 +0000
2297+++ hooks/charmhelpers/core/sysctl.py 2015-02-19 17:05:21 +0000
2298@@ -0,0 +1,56 @@
2299+#!/usr/bin/env python
2300+# -*- coding: utf-8 -*-
2301+
2302+# Copyright 2014-2015 Canonical Limited.
2303+#
2304+# This file is part of charm-helpers.
2305+#
2306+# charm-helpers is free software: you can redistribute it and/or modify
2307+# it under the terms of the GNU Lesser General Public License version 3 as
2308+# published by the Free Software Foundation.
2309+#
2310+# charm-helpers is distributed in the hope that it will be useful,
2311+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2312+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2313+# GNU Lesser General Public License for more details.
2314+#
2315+# You should have received a copy of the GNU Lesser General Public License
2316+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2317+
2318+__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
2319+
2320+import yaml
2321+
2322+from subprocess import check_call
2323+
2324+from charmhelpers.core.hookenv import (
2325+ log,
2326+ DEBUG,
2327+ ERROR,
2328+)
2329+
2330+
2331+def create(sysctl_dict, sysctl_file):
2332+ """Creates a sysctl.conf file from a YAML associative array
2333+
2334+ :param sysctl_dict: a YAML-formatted string of sysctl options eg "{ 'kernel.max_pid': 1337 }"
2335+ :type sysctl_dict: str
2336+ :param sysctl_file: path to the sysctl file to be saved
2337+ :type sysctl_file: str or unicode
2338+ :returns: None
2339+ """
2340+ try:
2341+ sysctl_dict_parsed = yaml.safe_load(sysctl_dict)
2342+ except yaml.YAMLError:
2343+ log("Error parsing YAML sysctl_dict: {}".format(sysctl_dict),
2344+ level=ERROR)
2345+ return
2346+
2347+ with open(sysctl_file, "w") as fd:
2348+ for key, value in sysctl_dict_parsed.items():
2349+ fd.write("{}={}\n".format(key, value))
2350+
2351+ log("Updating sysctl_file: %s values: %s" % (sysctl_file, sysctl_dict_parsed),
2352+ level=DEBUG)
2353+
2354+ check_call(["sysctl", "-p", sysctl_file])
2355
2356=== added file 'hooks/charmhelpers/core/templating.py'
2357--- hooks/charmhelpers/core/templating.py 1970-01-01 00:00:00 +0000
2358+++ hooks/charmhelpers/core/templating.py 2015-02-19 17:05:21 +0000
2359@@ -0,0 +1,68 @@
2360+# Copyright 2014-2015 Canonical Limited.
2361+#
2362+# This file is part of charm-helpers.
2363+#
2364+# charm-helpers is free software: you can redistribute it and/or modify
2365+# it under the terms of the GNU Lesser General Public License version 3 as
2366+# published by the Free Software Foundation.
2367+#
2368+# charm-helpers is distributed in the hope that it will be useful,
2369+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2370+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2371+# GNU Lesser General Public License for more details.
2372+#
2373+# You should have received a copy of the GNU Lesser General Public License
2374+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2375+
2376+import os
2377+
2378+from charmhelpers.core import host
2379+from charmhelpers.core import hookenv
2380+
2381+
2382+def render(source, target, context, owner='root', group='root',
2383+ perms=0o444, templates_dir=None, encoding='UTF-8'):
2384+ """
2385+ Render a template.
2386+
2387+ The `source` path, if not absolute, is relative to the `templates_dir`.
2388+
2389+ The `target` path should be absolute.
2390+
2391+ The context should be a dict containing the values to be replaced in the
2392+ template.
2393+
2394+ The `owner`, `group`, and `perms` options will be passed to `write_file`.
2395+
2396+ If omitted, `templates_dir` defaults to the `templates` folder in the charm.
2397+
2398+ Note: Using this requires python-jinja2; if it is not installed, calling
2399+ this will attempt to use charmhelpers.fetch.apt_install to install it.
2400+ """
2401+ try:
2402+ from jinja2 import FileSystemLoader, Environment, exceptions
2403+ except ImportError:
2404+ try:
2405+ from charmhelpers.fetch import apt_install
2406+ except ImportError:
2407+ hookenv.log('Could not import jinja2, and could not import '
2408+ 'charmhelpers.fetch to install it',
2409+ level=hookenv.ERROR)
2410+ raise
2411+ apt_install('python-jinja2', fatal=True)
2412+ from jinja2 import FileSystemLoader, Environment, exceptions
2413+
2414+ if templates_dir is None:
2415+ templates_dir = os.path.join(hookenv.charm_dir(), 'templates')
2416+ loader = Environment(loader=FileSystemLoader(templates_dir))
2417+ try:
2418+ source = source
2419+ template = loader.get_template(source)
2420+ except exceptions.TemplateNotFound as e:
2421+ hookenv.log('Could not load template %s from %s.' %
2422+ (source, templates_dir),
2423+ level=hookenv.ERROR)
2424+ raise e
2425+ content = template.render(context)
2426+ host.mkdir(os.path.dirname(target), owner, group, perms=0o755)
2427+ host.write_file(target, content.encode(encoding), owner, group, perms)
2428
2429=== modified file 'hooks/charmhelpers/fetch/__init__.py'
2430--- hooks/charmhelpers/fetch/__init__.py 2013-08-21 19:19:29 +0000
2431+++ hooks/charmhelpers/fetch/__init__.py 2015-02-19 17:05:21 +0000
2432@@ -1,18 +1,39 @@
2433+# Copyright 2014-2015 Canonical Limited.
2434+#
2435+# This file is part of charm-helpers.
2436+#
2437+# charm-helpers is free software: you can redistribute it and/or modify
2438+# it under the terms of the GNU Lesser General Public License version 3 as
2439+# published by the Free Software Foundation.
2440+#
2441+# charm-helpers is distributed in the hope that it will be useful,
2442+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2443+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2444+# GNU Lesser General Public License for more details.
2445+#
2446+# You should have received a copy of the GNU Lesser General Public License
2447+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2448+
2449 import importlib
2450+from tempfile import NamedTemporaryFile
2451+import time
2452 from yaml import safe_load
2453 from charmhelpers.core.host import (
2454 lsb_release
2455 )
2456-from urlparse import (
2457- urlparse,
2458- urlunparse,
2459-)
2460 import subprocess
2461 from charmhelpers.core.hookenv import (
2462 config,
2463 log,
2464 )
2465-import apt_pkg
2466+import os
2467+
2468+import six
2469+if six.PY3:
2470+ from urllib.parse import urlparse, urlunparse
2471+else:
2472+ from urlparse import urlparse, urlunparse
2473+
2474
2475 CLOUD_ARCHIVE = """# Ubuntu Cloud Archive
2476 deb http://ubuntu-cloud.archive.canonical.com/ubuntu {} main
2477@@ -20,12 +41,109 @@
2478 PROPOSED_POCKET = """# Proposed
2479 deb http://archive.ubuntu.com/ubuntu {}-proposed main universe multiverse restricted
2480 """
2481+CLOUD_ARCHIVE_POCKETS = {
2482+ # Folsom
2483+ 'folsom': 'precise-updates/folsom',
2484+ 'precise-folsom': 'precise-updates/folsom',
2485+ 'precise-folsom/updates': 'precise-updates/folsom',
2486+ 'precise-updates/folsom': 'precise-updates/folsom',
2487+ 'folsom/proposed': 'precise-proposed/folsom',
2488+ 'precise-folsom/proposed': 'precise-proposed/folsom',
2489+ 'precise-proposed/folsom': 'precise-proposed/folsom',
2490+ # Grizzly
2491+ 'grizzly': 'precise-updates/grizzly',
2492+ 'precise-grizzly': 'precise-updates/grizzly',
2493+ 'precise-grizzly/updates': 'precise-updates/grizzly',
2494+ 'precise-updates/grizzly': 'precise-updates/grizzly',
2495+ 'grizzly/proposed': 'precise-proposed/grizzly',
2496+ 'precise-grizzly/proposed': 'precise-proposed/grizzly',
2497+ 'precise-proposed/grizzly': 'precise-proposed/grizzly',
2498+ # Havana
2499+ 'havana': 'precise-updates/havana',
2500+ 'precise-havana': 'precise-updates/havana',
2501+ 'precise-havana/updates': 'precise-updates/havana',
2502+ 'precise-updates/havana': 'precise-updates/havana',
2503+ 'havana/proposed': 'precise-proposed/havana',
2504+ 'precise-havana/proposed': 'precise-proposed/havana',
2505+ 'precise-proposed/havana': 'precise-proposed/havana',
2506+ # Icehouse
2507+ 'icehouse': 'precise-updates/icehouse',
2508+ 'precise-icehouse': 'precise-updates/icehouse',
2509+ 'precise-icehouse/updates': 'precise-updates/icehouse',
2510+ 'precise-updates/icehouse': 'precise-updates/icehouse',
2511+ 'icehouse/proposed': 'precise-proposed/icehouse',
2512+ 'precise-icehouse/proposed': 'precise-proposed/icehouse',
2513+ 'precise-proposed/icehouse': 'precise-proposed/icehouse',
2514+ # Juno
2515+ 'juno': 'trusty-updates/juno',
2516+ 'trusty-juno': 'trusty-updates/juno',
2517+ 'trusty-juno/updates': 'trusty-updates/juno',
2518+ 'trusty-updates/juno': 'trusty-updates/juno',
2519+ 'juno/proposed': 'trusty-proposed/juno',
2520+ 'trusty-juno/proposed': 'trusty-proposed/juno',
2521+ 'trusty-proposed/juno': 'trusty-proposed/juno',
2522+ # Kilo
2523+ 'kilo': 'trusty-updates/kilo',
2524+ 'trusty-kilo': 'trusty-updates/kilo',
2525+ 'trusty-kilo/updates': 'trusty-updates/kilo',
2526+ 'trusty-updates/kilo': 'trusty-updates/kilo',
2527+ 'kilo/proposed': 'trusty-proposed/kilo',
2528+ 'trusty-kilo/proposed': 'trusty-proposed/kilo',
2529+ 'trusty-proposed/kilo': 'trusty-proposed/kilo',
2530+}
2531+
2532+# The order of this list is very important. Handlers should be listed in from
2533+# least- to most-specific URL matching.
2534+FETCH_HANDLERS = (
2535+ 'charmhelpers.fetch.archiveurl.ArchiveUrlFetchHandler',
2536+ 'charmhelpers.fetch.bzrurl.BzrUrlFetchHandler',
2537+ 'charmhelpers.fetch.giturl.GitUrlFetchHandler',
2538+)
2539+
2540+APT_NO_LOCK = 100 # The return code for "couldn't acquire lock" in APT.
2541+APT_NO_LOCK_RETRY_DELAY = 10 # Wait 10 seconds between apt lock checks.
2542+APT_NO_LOCK_RETRY_COUNT = 30 # Retry to acquire the lock X times.
2543+
2544+
2545+class SourceConfigError(Exception):
2546+ pass
2547+
2548+
2549+class UnhandledSource(Exception):
2550+ pass
2551+
2552+
2553+class AptLockError(Exception):
2554+ pass
2555+
2556+
2557+class BaseFetchHandler(object):
2558+
2559+ """Base class for FetchHandler implementations in fetch plugins"""
2560+
2561+ def can_handle(self, source):
2562+ """Returns True if the source can be handled. Otherwise returns
2563+ a string explaining why it cannot"""
2564+ return "Wrong source type"
2565+
2566+ def install(self, source):
2567+ """Try to download and unpack the source. Return the path to the
2568+ unpacked files or raise UnhandledSource."""
2569+ raise UnhandledSource("Wrong source type {}".format(source))
2570+
2571+ def parse_url(self, url):
2572+ return urlparse(url)
2573+
2574+ def base_url(self, url):
2575+ """Return url without querystring or fragment"""
2576+ parts = list(self.parse_url(url))
2577+ parts[4:] = ['' for i in parts[4:]]
2578+ return urlunparse(parts)
2579
2580
2581 def filter_installed_packages(packages):
2582 """Returns a list of packages that require installation"""
2583- apt_pkg.init()
2584- cache = apt_pkg.Cache()
2585+ cache = apt_cache()
2586 _pkgs = []
2587 for package in packages:
2588 try:
2589@@ -38,41 +156,74 @@
2590 return _pkgs
2591
2592
2593+def apt_cache(in_memory=True):
2594+ """Build and return an apt cache"""
2595+ import apt_pkg
2596+ apt_pkg.init()
2597+ if in_memory:
2598+ apt_pkg.config.set("Dir::Cache::pkgcache", "")
2599+ apt_pkg.config.set("Dir::Cache::srcpkgcache", "")
2600+ return apt_pkg.Cache()
2601+
2602+
2603 def apt_install(packages, options=None, fatal=False):
2604 """Install one or more packages"""
2605- options = options or []
2606- cmd = ['apt-get', '-y']
2607+ if options is None:
2608+ options = ['--option=Dpkg::Options::=--force-confold']
2609+
2610+ cmd = ['apt-get', '--assume-yes']
2611 cmd.extend(options)
2612 cmd.append('install')
2613- if isinstance(packages, basestring):
2614+ if isinstance(packages, six.string_types):
2615 cmd.append(packages)
2616 else:
2617 cmd.extend(packages)
2618 log("Installing {} with options: {}".format(packages,
2619 options))
2620- if fatal:
2621- subprocess.check_call(cmd)
2622+ _run_apt_command(cmd, fatal)
2623+
2624+
2625+def apt_upgrade(options=None, fatal=False, dist=False):
2626+ """Upgrade all packages"""
2627+ if options is None:
2628+ options = ['--option=Dpkg::Options::=--force-confold']
2629+
2630+ cmd = ['apt-get', '--assume-yes']
2631+ cmd.extend(options)
2632+ if dist:
2633+ cmd.append('dist-upgrade')
2634 else:
2635- subprocess.call(cmd)
2636+ cmd.append('upgrade')
2637+ log("Upgrading with options: {}".format(options))
2638+ _run_apt_command(cmd, fatal)
2639
2640
2641 def apt_update(fatal=False):
2642 """Update local apt cache"""
2643 cmd = ['apt-get', 'update']
2644- if fatal:
2645- subprocess.check_call(cmd)
2646- else:
2647- subprocess.call(cmd)
2648+ _run_apt_command(cmd, fatal)
2649
2650
2651 def apt_purge(packages, fatal=False):
2652 """Purge one or more packages"""
2653- cmd = ['apt-get', '-y', 'purge']
2654- if isinstance(packages, basestring):
2655+ cmd = ['apt-get', '--assume-yes', 'purge']
2656+ if isinstance(packages, six.string_types):
2657 cmd.append(packages)
2658 else:
2659 cmd.extend(packages)
2660 log("Purging {}".format(packages))
2661+ _run_apt_command(cmd, fatal)
2662+
2663+
2664+def apt_hold(packages, fatal=False):
2665+ """Hold one or more packages"""
2666+ cmd = ['apt-mark', 'hold']
2667+ if isinstance(packages, six.string_types):
2668+ cmd.append(packages)
2669+ else:
2670+ cmd.extend(packages)
2671+ log("Holding {}".format(packages))
2672+
2673 if fatal:
2674 subprocess.check_call(cmd)
2675 else:
2676@@ -80,84 +231,145 @@
2677
2678
2679 def add_source(source, key=None):
2680- if ((source.startswith('ppa:') or
2681- source.startswith('http:'))):
2682+ """Add a package source to this system.
2683+
2684+ @param source: a URL or sources.list entry, as supported by
2685+ add-apt-repository(1). Examples::
2686+
2687+ ppa:charmers/example
2688+ deb https://stub:key@private.example.com/ubuntu trusty main
2689+
2690+ In addition:
2691+ 'proposed:' may be used to enable the standard 'proposed'
2692+ pocket for the release.
2693+ 'cloud:' may be used to activate official cloud archive pockets,
2694+ such as 'cloud:icehouse'
2695+ 'distro' may be used as a noop
2696+
2697+ @param key: A key to be added to the system's APT keyring and used
2698+ to verify the signatures on packages. Ideally, this should be an
2699+ ASCII format GPG public key including the block headers. A GPG key
2700+ id may also be used, but be aware that only insecure protocols are
2701+ available to retrieve the actual public key from a public keyserver
2702+ placing your Juju environment at risk. ppa and cloud archive keys
2703+ are securely added automtically, so sould not be provided.
2704+ """
2705+ if source is None:
2706+ log('Source is not present. Skipping')
2707+ return
2708+
2709+ if (source.startswith('ppa:') or
2710+ source.startswith('http') or
2711+ source.startswith('deb ') or
2712+ source.startswith('cloud-archive:')):
2713 subprocess.check_call(['add-apt-repository', '--yes', source])
2714 elif source.startswith('cloud:'):
2715 apt_install(filter_installed_packages(['ubuntu-cloud-keyring']),
2716 fatal=True)
2717 pocket = source.split(':')[-1]
2718+ if pocket not in CLOUD_ARCHIVE_POCKETS:
2719+ raise SourceConfigError(
2720+ 'Unsupported cloud: source option %s' %
2721+ pocket)
2722+ actual_pocket = CLOUD_ARCHIVE_POCKETS[pocket]
2723 with open('/etc/apt/sources.list.d/cloud-archive.list', 'w') as apt:
2724- apt.write(CLOUD_ARCHIVE.format(pocket))
2725+ apt.write(CLOUD_ARCHIVE.format(actual_pocket))
2726 elif source == 'proposed':
2727 release = lsb_release()['DISTRIB_CODENAME']
2728 with open('/etc/apt/sources.list.d/proposed.list', 'w') as apt:
2729 apt.write(PROPOSED_POCKET.format(release))
2730+ elif source == 'distro':
2731+ pass
2732+ else:
2733+ log("Unknown source: {!r}".format(source))
2734+
2735 if key:
2736- subprocess.check_call(['apt-key', 'import', key])
2737-
2738-
2739-class SourceConfigError(Exception):
2740- pass
2741+ if '-----BEGIN PGP PUBLIC KEY BLOCK-----' in key:
2742+ with NamedTemporaryFile('w+') as key_file:
2743+ key_file.write(key)
2744+ key_file.flush()
2745+ key_file.seek(0)
2746+ subprocess.check_call(['apt-key', 'add', '-'], stdin=key_file)
2747+ else:
2748+ # Note that hkp: is in no way a secure protocol. Using a
2749+ # GPG key id is pointless from a security POV unless you
2750+ # absolutely trust your network and DNS.
2751+ subprocess.check_call(['apt-key', 'adv', '--keyserver',
2752+ 'hkp://keyserver.ubuntu.com:80', '--recv',
2753+ key])
2754
2755
2756 def configure_sources(update=False,
2757 sources_var='install_sources',
2758 keys_var='install_keys'):
2759 """
2760- Configure multiple sources from charm configuration
2761+ Configure multiple sources from charm configuration.
2762+
2763+ The lists are encoded as yaml fragments in the configuration.
2764+ The frament needs to be included as a string. Sources and their
2765+ corresponding keys are of the types supported by add_source().
2766
2767 Example config:
2768- install_sources:
2769+ install_sources: |
2770 - "ppa:foo"
2771 - "http://example.com/repo precise main"
2772- install_keys:
2773+ install_keys: |
2774 - null
2775 - "a1b2c3d4"
2776
2777 Note that 'null' (a.k.a. None) should not be quoted.
2778 """
2779- sources = safe_load(config(sources_var))
2780- keys = safe_load(config(keys_var))
2781- if isinstance(sources, basestring) and isinstance(keys, basestring):
2782- add_source(sources, keys)
2783+ sources = safe_load((config(sources_var) or '').strip()) or []
2784+ keys = safe_load((config(keys_var) or '').strip()) or None
2785+
2786+ if isinstance(sources, six.string_types):
2787+ sources = [sources]
2788+
2789+ if keys is None:
2790+ for source in sources:
2791+ add_source(source, None)
2792 else:
2793- if not len(sources) == len(keys):
2794- msg = 'Install sources and keys lists are different lengths'
2795- raise SourceConfigError(msg)
2796- for src_num in range(len(sources)):
2797- add_source(sources[src_num], keys[src_num])
2798+ if isinstance(keys, six.string_types):
2799+ keys = [keys]
2800+
2801+ if len(sources) != len(keys):
2802+ raise SourceConfigError(
2803+ 'Install sources and keys lists are different lengths')
2804+ for source, key in zip(sources, keys):
2805+ add_source(source, key)
2806 if update:
2807 apt_update(fatal=True)
2808
2809-# The order of this list is very important. Handlers should be listed in from
2810-# least- to most-specific URL matching.
2811-FETCH_HANDLERS = (
2812- 'charmhelpers.fetch.archiveurl.ArchiveUrlFetchHandler',
2813- 'charmhelpers.fetch.bzrurl.BzrUrlFetchHandler',
2814-)
2815-
2816-
2817-class UnhandledSource(Exception):
2818- pass
2819-
2820-
2821-def install_remote(source):
2822+
2823+def install_remote(source, *args, **kwargs):
2824 """
2825 Install a file tree from a remote source
2826
2827 The specified source should be a url of the form:
2828 scheme://[host]/path[#[option=value][&...]]
2829
2830- Schemes supported are based on this modules submodules
2831- Options supported are submodule-specific"""
2832+ Schemes supported are based on this modules submodules.
2833+ Options supported are submodule-specific.
2834+ Additional arguments are passed through to the submodule.
2835+
2836+ For example::
2837+
2838+ dest = install_remote('http://example.com/archive.tgz',
2839+ checksum='deadbeef',
2840+ hash_type='sha1')
2841+
2842+ This will download `archive.tgz`, validate it using SHA1 and, if
2843+ the file is ok, extract it and return the directory in which it
2844+ was extracted. If the checksum fails, it will raise
2845+ :class:`charmhelpers.core.host.ChecksumError`.
2846+ """
2847 # We ONLY check for True here because can_handle may return a string
2848 # explaining why it can't handle a given source.
2849 handlers = [h for h in plugins() if h.can_handle(source) is True]
2850 installed_to = None
2851 for handler in handlers:
2852 try:
2853- installed_to = handler.install(source)
2854+ installed_to = handler.install(source, *args, **kwargs)
2855 except UnhandledSource:
2856 pass
2857 if not installed_to:
2858@@ -171,28 +383,6 @@
2859 return install_remote(source)
2860
2861
2862-class BaseFetchHandler(object):
2863- """Base class for FetchHandler implementations in fetch plugins"""
2864- def can_handle(self, source):
2865- """Returns True if the source can be handled. Otherwise returns
2866- a string explaining why it cannot"""
2867- return "Wrong source type"
2868-
2869- def install(self, source):
2870- """Try to download and unpack the source. Return the path to the
2871- unpacked files or raise UnhandledSource."""
2872- raise UnhandledSource("Wrong source type {}".format(source))
2873-
2874- def parse_url(self, url):
2875- return urlparse(url)
2876-
2877- def base_url(self, url):
2878- """Return url without querystring or fragment"""
2879- parts = list(self.parse_url(url))
2880- parts[4:] = ['' for i in parts[4:]]
2881- return urlunparse(parts)
2882-
2883-
2884 def plugins(fetch_handlers=None):
2885 if not fetch_handlers:
2886 fetch_handlers = FETCH_HANDLERS
2887@@ -200,10 +390,50 @@
2888 for handler_name in fetch_handlers:
2889 package, classname = handler_name.rsplit('.', 1)
2890 try:
2891- handler_class = getattr(importlib.import_module(package), classname)
2892+ handler_class = getattr(
2893+ importlib.import_module(package),
2894+ classname)
2895 plugin_list.append(handler_class())
2896 except (ImportError, AttributeError):
2897 # Skip missing plugins so that they can be ommitted from
2898 # installation if desired
2899- log("FetchHandler {} not found, skipping plugin".format(handler_name))
2900+ log("FetchHandler {} not found, skipping plugin".format(
2901+ handler_name))
2902 return plugin_list
2903+
2904+
2905+def _run_apt_command(cmd, fatal=False):
2906+ """
2907+ Run an APT command, checking output and retrying if the fatal flag is set
2908+ to True.
2909+
2910+ :param: cmd: str: The apt command to run.
2911+ :param: fatal: bool: Whether the command's output should be checked and
2912+ retried.
2913+ """
2914+ env = os.environ.copy()
2915+
2916+ if 'DEBIAN_FRONTEND' not in env:
2917+ env['DEBIAN_FRONTEND'] = 'noninteractive'
2918+
2919+ if fatal:
2920+ retry_count = 0
2921+ result = None
2922+
2923+ # If the command is considered "fatal", we need to retry if the apt
2924+ # lock was not acquired.
2925+
2926+ while result is None or result == APT_NO_LOCK:
2927+ try:
2928+ result = subprocess.check_call(cmd, env=env)
2929+ except subprocess.CalledProcessError as e:
2930+ retry_count = retry_count + 1
2931+ if retry_count > APT_NO_LOCK_RETRY_COUNT:
2932+ raise
2933+ result = e.returncode
2934+ log("Couldn't acquire DPKG lock. Will retry in {} seconds."
2935+ "".format(APT_NO_LOCK_RETRY_DELAY))
2936+ time.sleep(APT_NO_LOCK_RETRY_DELAY)
2937+
2938+ else:
2939+ subprocess.call(cmd, env=env)
2940
2941=== modified file 'hooks/charmhelpers/fetch/archiveurl.py'
2942--- hooks/charmhelpers/fetch/archiveurl.py 2013-08-21 19:19:29 +0000
2943+++ hooks/charmhelpers/fetch/archiveurl.py 2015-02-19 17:05:21 +0000
2944@@ -1,5 +1,40 @@
2945+# Copyright 2014-2015 Canonical Limited.
2946+#
2947+# This file is part of charm-helpers.
2948+#
2949+# charm-helpers is free software: you can redistribute it and/or modify
2950+# it under the terms of the GNU Lesser General Public License version 3 as
2951+# published by the Free Software Foundation.
2952+#
2953+# charm-helpers is distributed in the hope that it will be useful,
2954+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2955+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2956+# GNU Lesser General Public License for more details.
2957+#
2958+# You should have received a copy of the GNU Lesser General Public License
2959+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2960+
2961 import os
2962-import urllib2
2963+import hashlib
2964+import re
2965+
2966+import six
2967+if six.PY3:
2968+ from urllib.request import (
2969+ build_opener, install_opener, urlopen, urlretrieve,
2970+ HTTPPasswordMgrWithDefaultRealm, HTTPBasicAuthHandler,
2971+ )
2972+ from urllib.parse import urlparse, urlunparse, parse_qs
2973+ from urllib.error import URLError
2974+else:
2975+ from urllib import urlretrieve
2976+ from urllib2 import (
2977+ build_opener, install_opener, urlopen,
2978+ HTTPPasswordMgrWithDefaultRealm, HTTPBasicAuthHandler,
2979+ URLError
2980+ )
2981+ from urlparse import urlparse, urlunparse, parse_qs
2982+
2983 from charmhelpers.fetch import (
2984 BaseFetchHandler,
2985 UnhandledSource
2986@@ -8,11 +43,37 @@
2987 get_archive_handler,
2988 extract,
2989 )
2990-from charmhelpers.core.host import mkdir
2991+from charmhelpers.core.host import mkdir, check_hash
2992+
2993+
2994+def splituser(host):
2995+ '''urllib.splituser(), but six's support of this seems broken'''
2996+ _userprog = re.compile('^(.*)@(.*)$')
2997+ match = _userprog.match(host)
2998+ if match:
2999+ return match.group(1, 2)
3000+ return None, host
3001+
3002+
3003+def splitpasswd(user):
3004+ '''urllib.splitpasswd(), but six's support of this is missing'''
3005+ _passwdprog = re.compile('^([^:]*):(.*)$', re.S)
3006+ match = _passwdprog.match(user)
3007+ if match:
3008+ return match.group(1, 2)
3009+ return user, None
3010
3011
3012 class ArchiveUrlFetchHandler(BaseFetchHandler):
3013- """Handler for archives via generic URLs"""
3014+ """
3015+ Handler to download archive files from arbitrary URLs.
3016+
3017+ Can fetch from http, https, ftp, and file URLs.
3018+
3019+ Can install either tarballs (.tar, .tgz, .tbz2, etc) or zip files.
3020+
3021+ Installs the contents of the archive in $CHARM_DIR/fetched/.
3022+ """
3023 def can_handle(self, source):
3024 url_parts = self.parse_url(source)
3025 if url_parts.scheme not in ('http', 'https', 'ftp', 'file'):
3026@@ -22,9 +83,28 @@
3027 return False
3028
3029 def download(self, source, dest):
3030+ """
3031+ Download an archive file.
3032+
3033+ :param str source: URL pointing to an archive file.
3034+ :param str dest: Local path location to download archive file to.
3035+ """
3036 # propogate all exceptions
3037 # URLError, OSError, etc
3038- response = urllib2.urlopen(source)
3039+ proto, netloc, path, params, query, fragment = urlparse(source)
3040+ if proto in ('http', 'https'):
3041+ auth, barehost = splituser(netloc)
3042+ if auth is not None:
3043+ source = urlunparse((proto, barehost, path, params, query, fragment))
3044+ username, password = splitpasswd(auth)
3045+ passman = HTTPPasswordMgrWithDefaultRealm()
3046+ # Realm is set to None in add_password to force the username and password
3047+ # to be used whatever the realm
3048+ passman.add_password(None, source, username, password)
3049+ authhandler = HTTPBasicAuthHandler(passman)
3050+ opener = build_opener(authhandler)
3051+ install_opener(opener)
3052+ response = urlopen(source)
3053 try:
3054 with open(dest, 'w') as dest_file:
3055 dest_file.write(response.read())
3056@@ -33,16 +113,49 @@
3057 os.unlink(dest)
3058 raise e
3059
3060- def install(self, source):
3061+ # Mandatory file validation via Sha1 or MD5 hashing.
3062+ def download_and_validate(self, url, hashsum, validate="sha1"):
3063+ tempfile, headers = urlretrieve(url)
3064+ check_hash(tempfile, hashsum, validate)
3065+ return tempfile
3066+
3067+ def install(self, source, dest=None, checksum=None, hash_type='sha1'):
3068+ """
3069+ Download and install an archive file, with optional checksum validation.
3070+
3071+ The checksum can also be given on the `source` URL's fragment.
3072+ For example::
3073+
3074+ handler.install('http://example.com/file.tgz#sha1=deadbeef')
3075+
3076+ :param str source: URL pointing to an archive file.
3077+ :param str dest: Local destination path to install to. If not given,
3078+ installs to `$CHARM_DIR/archives/archive_file_name`.
3079+ :param str checksum: If given, validate the archive file after download.
3080+ :param str hash_type: Algorithm used to generate `checksum`.
3081+ Can be any hash alrgorithm supported by :mod:`hashlib`,
3082+ such as md5, sha1, sha256, sha512, etc.
3083+
3084+ """
3085 url_parts = self.parse_url(source)
3086 dest_dir = os.path.join(os.environ.get('CHARM_DIR'), 'fetched')
3087 if not os.path.exists(dest_dir):
3088- mkdir(dest_dir, perms=0755)
3089+ mkdir(dest_dir, perms=0o755)
3090 dld_file = os.path.join(dest_dir, os.path.basename(url_parts.path))
3091 try:
3092 self.download(source, dld_file)
3093- except urllib2.URLError as e:
3094+ except URLError as e:
3095 raise UnhandledSource(e.reason)
3096 except OSError as e:
3097 raise UnhandledSource(e.strerror)
3098- return extract(dld_file)
3099+ options = parse_qs(url_parts.fragment)
3100+ for key, value in options.items():
3101+ if not six.PY3:
3102+ algorithms = hashlib.algorithms
3103+ else:
3104+ algorithms = hashlib.algorithms_available
3105+ if key in algorithms:
3106+ check_hash(dld_file, value, key)
3107+ if checksum:
3108+ check_hash(dld_file, checksum, hash_type)
3109+ return extract(dld_file, dest)
3110
3111=== modified file 'hooks/charmhelpers/fetch/bzrurl.py'
3112--- hooks/charmhelpers/fetch/bzrurl.py 2013-08-21 19:19:29 +0000
3113+++ hooks/charmhelpers/fetch/bzrurl.py 2015-02-19 17:05:21 +0000
3114@@ -1,11 +1,39 @@
3115+# Copyright 2014-2015 Canonical Limited.
3116+#
3117+# This file is part of charm-helpers.
3118+#
3119+# charm-helpers is free software: you can redistribute it and/or modify
3120+# it under the terms of the GNU Lesser General Public License version 3 as
3121+# published by the Free Software Foundation.
3122+#
3123+# charm-helpers is distributed in the hope that it will be useful,
3124+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3125+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3126+# GNU Lesser General Public License for more details.
3127+#
3128+# You should have received a copy of the GNU Lesser General Public License
3129+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3130+
3131 import os
3132-from bzrlib.branch import Branch
3133 from charmhelpers.fetch import (
3134 BaseFetchHandler,
3135 UnhandledSource
3136 )
3137 from charmhelpers.core.host import mkdir
3138
3139+import six
3140+if six.PY3:
3141+ raise ImportError('bzrlib does not support Python3')
3142+
3143+try:
3144+ from bzrlib.branch import Branch
3145+ from bzrlib import bzrdir, workingtree, errors
3146+except ImportError:
3147+ from charmhelpers.fetch import apt_install
3148+ apt_install("python-bzrlib")
3149+ from bzrlib.branch import Branch
3150+ from bzrlib import bzrdir, workingtree, errors
3151+
3152
3153 class BzrUrlFetchHandler(BaseFetchHandler):
3154 """Handler for bazaar branches via generic and lp URLs"""
3155@@ -25,20 +53,26 @@
3156 from bzrlib.plugin import load_plugins
3157 load_plugins()
3158 try:
3159+ local_branch = bzrdir.BzrDir.create_branch_convenience(dest)
3160+ except errors.AlreadyControlDirError:
3161+ local_branch = Branch.open(dest)
3162+ try:
3163 remote_branch = Branch.open(source)
3164- remote_branch.bzrdir.sprout(dest).open_branch()
3165+ remote_branch.push(local_branch)
3166+ tree = workingtree.WorkingTree.open(dest)
3167+ tree.update()
3168 except Exception as e:
3169 raise e
3170
3171 def install(self, source):
3172 url_parts = self.parse_url(source)
3173 branch_name = url_parts.path.strip("/").split("/")[-1]
3174- dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched", branch_name)
3175+ dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
3176+ branch_name)
3177 if not os.path.exists(dest_dir):
3178- mkdir(dest_dir, perms=0755)
3179+ mkdir(dest_dir, perms=0o755)
3180 try:
3181 self.branch(source, dest_dir)
3182 except OSError as e:
3183 raise UnhandledSource(e.strerror)
3184 return dest_dir
3185-
3186
3187=== added file 'hooks/charmhelpers/fetch/giturl.py'
3188--- hooks/charmhelpers/fetch/giturl.py 1970-01-01 00:00:00 +0000
3189+++ hooks/charmhelpers/fetch/giturl.py 2015-02-19 17:05:21 +0000
3190@@ -0,0 +1,71 @@
3191+# Copyright 2014-2015 Canonical Limited.
3192+#
3193+# This file is part of charm-helpers.
3194+#
3195+# charm-helpers is free software: you can redistribute it and/or modify
3196+# it under the terms of the GNU Lesser General Public License version 3 as
3197+# published by the Free Software Foundation.
3198+#
3199+# charm-helpers is distributed in the hope that it will be useful,
3200+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3201+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3202+# GNU Lesser General Public License for more details.
3203+#
3204+# You should have received a copy of the GNU Lesser General Public License
3205+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3206+
3207+import os
3208+from charmhelpers.fetch import (
3209+ BaseFetchHandler,
3210+ UnhandledSource
3211+)
3212+from charmhelpers.core.host import mkdir
3213+
3214+import six
3215+if six.PY3:
3216+ raise ImportError('GitPython does not support Python 3')
3217+
3218+try:
3219+ from git import Repo
3220+except ImportError:
3221+ from charmhelpers.fetch import apt_install
3222+ apt_install("python-git")
3223+ from git import Repo
3224+
3225+from git.exc import GitCommandError
3226+
3227+
3228+class GitUrlFetchHandler(BaseFetchHandler):
3229+ """Handler for git branches via generic and github URLs"""
3230+ def can_handle(self, source):
3231+ url_parts = self.parse_url(source)
3232+ # TODO (mattyw) no support for ssh git@ yet
3233+ if url_parts.scheme not in ('http', 'https', 'git'):
3234+ return False
3235+ else:
3236+ return True
3237+
3238+ def clone(self, source, dest, branch):
3239+ if not self.can_handle(source):
3240+ raise UnhandledSource("Cannot handle {}".format(source))
3241+
3242+ repo = Repo.clone_from(source, dest)
3243+ repo.git.checkout(branch)
3244+
3245+ def install(self, source, branch="master", dest=None):
3246+ url_parts = self.parse_url(source)
3247+ branch_name = url_parts.path.strip("/").split("/")[-1]
3248+ if dest:
3249+ dest_dir = os.path.join(dest, branch_name)
3250+ else:
3251+ dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
3252+ branch_name)
3253+ if not os.path.exists(dest_dir):
3254+ mkdir(dest_dir, perms=0o755)
3255+ try:
3256+ self.clone(source, dest_dir, branch)
3257+ except GitCommandError as e:
3258+ raise UnhandledSource(e.message)
3259+ except OSError as e:
3260+ raise UnhandledSource(e.strerror)
3261+ return dest_dir
3262
3263=== modified file 'hooks/hooks.py'
3264--- hooks/hooks.py 2015-01-20 09:47:15 +0000
3265+++ hooks/hooks.py 2015-02-19 17:05:21 +0000
3266@@ -9,10 +9,11 @@
3267 import subprocess
3268 import sys
3269 import yaml
3270+import pwd
3271
3272 from itertools import izip, tee, groupby
3273
3274-from charmhelpers.core.host import pwgen
3275+from charmhelpers.core.host import pwgen, lsb_release
3276 from charmhelpers.core.hookenv import (
3277 log,
3278 config as config_get,
3279@@ -30,7 +31,8 @@
3280 from charmhelpers.fetch import (
3281 apt_install,
3282 add_source,
3283- apt_update
3284+ apt_update,
3285+ apt_cache
3286 )
3287
3288 from charmhelpers.contrib.charmsupport import nrpe
3289@@ -46,6 +48,11 @@
3290 metrics_cronjob_path = "/etc/cron.d/haproxy_metrics"
3291 metrics_script_path = "/usr/local/bin/haproxy_to_statsd.sh"
3292 service_affecting_packages = ['haproxy']
3293+apt_backports_template = (
3294+ "deb http://archive.ubuntu.com/ubuntu %(release)s-backports "
3295+ "main restricted universe multiverse")
3296+haproxy_preferences_path = "/etc/apt/preferences.d/haproxy"
3297+
3298
3299 dupe_options = [
3300 "mode tcp",
3301@@ -143,6 +150,9 @@
3302 haproxy_globals.append(" quiet")
3303 haproxy_globals.append(" spread-checks %d" %
3304 config_data['global_spread_checks'])
3305+ if has_ssl_support():
3306+ haproxy_globals.append(" tune.ssl.default-dh-param %d" %
3307+ config_data['global_default_dh_param'])
3308 if config_data['global_stats_socket'] is True:
3309 sock_path = "/var/run/haproxy/haproxy.sock"
3310 haproxy_globals.append(" stats socket %s mode 0600" % sock_path)
3311@@ -221,8 +231,12 @@
3312 listen_stanzas = re.findall(
3313 "listen\s+([^\s]+)\s+([^:]+):(.*)",
3314 haproxy_config)
3315+ # Match bind stanzas like:
3316+ #
3317+ # bind 1.2.3.5:234
3318+ # bind 1.2.3.4:123 ssl crt /foo/bar
3319 bind_stanzas = re.findall(
3320- "\s+bind\s+([^:]+):(\d+)\s*\n\s+default_backend\s+([^\s]+)",
3321+ "\s+bind\s+([^:]+):(\d+).*\n\s+default_backend\s+([^\s]+)",
3322 haproxy_config, re.M)
3323 return (tuple(((service, addr, int(port))
3324 for service, addr, port in listen_stanzas)) +
3325@@ -259,6 +273,30 @@
3326
3327
3328 # -----------------------------------------------------------------------------
3329+# update_ssl_cert: write the default SSL certificate using the values from the
3330+# 'ssl-cert'/'ssl-key' and configuration keys
3331+# -----------------------------------------------------------------------------
3332+def update_ssl_cert(config_data):
3333+ ssl_cert = config_data.get("ssl_cert")
3334+ if not ssl_cert:
3335+ return
3336+ if ssl_cert == "SELFSIGNED":
3337+ log("Using self-signed certificate")
3338+ content = get_selfsigned_cert()
3339+ else:
3340+ ssl_key = config_data.get("ssl_key")
3341+ if not ssl_key:
3342+ log("No ssl_key provided, proceeding without default certificate")
3343+ return
3344+ log("Using config-provided certificate")
3345+ content = base64.b64decode(ssl_cert)
3346+ content += base64.b64decode(ssl_key)
3347+
3348+ pem_path = os.path.join(default_haproxy_lib_dir, "default.pem")
3349+ write_ssl_pem(pem_path, content)
3350+
3351+
3352+# -----------------------------------------------------------------------------
3353 # create_listen_stanza: Function to create a generic listen section in the
3354 # haproxy config
3355 # service_name: Arbitrary service name
3356@@ -274,10 +312,13 @@
3357 # http_status: status to handle
3358 # content: base 64 content for HAProxy to
3359 # write to socket
3360+# crts: List of base 64 contents for SSL certificate
3361+# files that will be used in the bind line.
3362 # -----------------------------------------------------------------------------
3363 def create_listen_stanza(service_name=None, service_ip=None,
3364 service_port=None, service_options=None,
3365- server_entries=None, service_errorfiles=None):
3366+ server_entries=None, service_errorfiles=None,
3367+ service_crts=None):
3368 if service_name is None or service_ip is None or service_port is None:
3369 return None
3370 fe_options = []
3371@@ -302,8 +343,19 @@
3372 service_config = []
3373 unit_name = os.environ["JUJU_UNIT_NAME"].replace("/", "-")
3374 service_config.append("frontend %s-%s" % (unit_name, service_port))
3375- service_config.append(" bind %s:%s" %
3376- (service_ip, service_port))
3377+ bind_stanza = " bind %s:%s" % (service_ip, service_port)
3378+ if service_crts:
3379+ # Enable SSL termination for this frontend, using the given
3380+ # certificates.
3381+ bind_stanza += " ssl"
3382+ for i, crt in enumerate(service_crts):
3383+ if crt == "DEFAULT":
3384+ path = os.path.join(default_haproxy_lib_dir, "default.pem")
3385+ else:
3386+ path = os.path.join(default_haproxy_lib_dir,
3387+ "service_%s" % service_name, "%d.pem" % i)
3388+ bind_stanza += " crt %s" % path
3389+ service_config.append(bind_stanza)
3390 service_config.append(" default_backend %s" % (service_name,))
3391 service_config.extend(" %s" % service_option.strip()
3392 for service_option in fe_options)
3393@@ -634,21 +686,29 @@
3394 # Construct the new haproxy.cfg file
3395 for service_key, service_config in services_dict.items():
3396 log("Service: %s" % service_key)
3397+ service_name = service_config["service_name"]
3398 server_entries = service_config.get('servers')
3399
3400 errorfiles = service_config.get('errorfiles', [])
3401 for errorfile in errorfiles:
3402- service_name = services_dict[service_key]['service_name']
3403- path = os.path.join(default_haproxy_lib_dir,
3404- "service_%s" % service_name)
3405- if not os.path.exists(path):
3406- os.makedirs(path)
3407+ path = get_service_lib_path(service_name)
3408 full_path = os.path.join(
3409 path, "%s.http" % errorfile["http_status"])
3410 with open(full_path, 'w') as f:
3411 f.write(base64.b64decode(errorfile["content"]))
3412
3413- service_name = service_config["service_name"]
3414+ # Write to disk the content of the given SSL certificates
3415+ crts = service_config.get('crts', [])
3416+ for i, crt in enumerate(crts):
3417+ if crt == "DEFAULT":
3418+ continue
3419+ content = base64.b64decode(crt)
3420+ path = get_service_lib_path(service_name)
3421+ full_path = os.path.join(path, "%d.pem" % i)
3422+ write_ssl_pem(full_path, content)
3423+ with open(full_path, 'w') as f:
3424+ f.write(content)
3425+
3426 if not os.path.exists(default_haproxy_service_config_dir):
3427 os.mkdir(default_haproxy_service_config_dir, 0600)
3428 with open(os.path.join(default_haproxy_service_config_dir,
3429@@ -658,7 +718,16 @@
3430 service_config['service_host'],
3431 service_config['service_port'],
3432 service_config['service_options'],
3433- server_entries, errorfiles))
3434+ server_entries, errorfiles, crts))
3435+
3436+
3437+def get_service_lib_path(service_name):
3438+ # Get a service-specific lib path
3439+ path = os.path.join(default_haproxy_lib_dir,
3440+ "service_%s" % service_name)
3441+ if not os.path.exists(path):
3442+ os.makedirs(path)
3443+ return path
3444
3445
3446 # -----------------------------------------------------------------------------
3447@@ -760,9 +829,16 @@
3448 os.mkdir(default_haproxy_service_config_dir, 0600)
3449
3450 config_data = config_get()
3451- add_source(config_data.get('source'), config_data.get('key'))
3452+ source = config_data.get('source')
3453+ if source == 'backports':
3454+ release = lsb_release()['DISTRIB_CODENAME']
3455+ source = apt_backports_template % {'release': release}
3456+ add_backports_preferences(release)
3457+ add_source(source, config_data.get('key'))
3458 apt_update(fatal=True)
3459 apt_install(['haproxy', 'python-jinja2'], fatal=True)
3460+ # Install pyasn1 library and modules for inspecting SSL certificates
3461+ apt_install(['python-pyasn1', 'python-pyasn1-modules'], fatal=False)
3462 ensure_package_status(service_affecting_packages,
3463 config_data['package_status'])
3464 enable_haproxy()
3465@@ -787,6 +863,7 @@
3466 sys.exit()
3467 haproxy_services = load_services()
3468 update_sysctl(config_data)
3469+ update_ssl_cert(config_data)
3470 construct_haproxy_config(haproxy_globals,
3471 haproxy_defaults,
3472 haproxy_monitoring,
3473@@ -978,6 +1055,126 @@
3474 }))
3475
3476
3477+def add_backports_preferences(release):
3478+ with open(haproxy_preferences_path, "w") as preferences:
3479+ preferences.write(
3480+ "Package: haproxy\n"
3481+ "Pin: release a=%(release)s-backports\n"
3482+ "Pin-Priority: 500\n" % {'release': release})
3483+
3484+
3485+def has_ssl_support():
3486+ """Return True if the locally installed haproxy package supports SSL."""
3487+ cache = apt_cache()
3488+ package = cache["haproxy"]
3489+ return package.current_ver.ver_str.split(".")[0:2] >= ["1", "5"]
3490+
3491+
3492+def get_selfsigned_cert():
3493+ """Return the content of the self-signed certificate.
3494+
3495+ If no self-signed certificate is there or the existing one doesn't match
3496+ our unit data, a new one will be created.
3497+ """
3498+ cert_file = os.path.join(default_haproxy_lib_dir, "selfsigned_ca.crt")
3499+ key_file = os.path.join(default_haproxy_lib_dir, "selfsigned.key")
3500+ if is_selfsigned_cert_stale(cert_file, key_file):
3501+ log("Generating self-signed certificate")
3502+ gen_selfsigned_cert(cert_file, key_file)
3503+ content = ""
3504+ for content_file in [cert_file, key_file]:
3505+ with open(content_file, "r") as fd:
3506+ content += fd.read()
3507+ return content
3508+
3509+
3510+# XXX taken from the apache2 charm.
3511+def is_selfsigned_cert_stale(cert_file, key_file):
3512+ """
3513+ Do we need to generate a new self-signed cert?
3514+
3515+ @param cert_file: destination path of generated certificate
3516+ @param key_file: destination path of generated private key
3517+ """
3518+ # Basic Existence Checks
3519+ if not os.path.exists(cert_file):
3520+ return True
3521+ if not os.path.exists(key_file):
3522+ return True
3523+
3524+ # Common Name
3525+ from OpenSSL import crypto
3526+ with open(cert_file) as fd:
3527+ cert = crypto.load_certificate(
3528+ crypto.FILETYPE_PEM, fd.read())
3529+ cn = cert.get_subject().commonName
3530+ if unit_get('public-address') != cn:
3531+ return True
3532+
3533+ # Subject Alternate Name -- only trusty+ support this
3534+ try:
3535+ from pyasn1.codec.der import decoder
3536+ from pyasn1_modules import rfc2459
3537+ except ImportError:
3538+ log('Cannot check subjAltName on <= 12.04, skipping.')
3539+ return False
3540+ cert_addresses = set()
3541+ unit_addresses = set(
3542+ [unit_get('public-address'), unit_get('private-address')])
3543+ for i in range(0, cert.get_extension_count()):
3544+ extension = cert.get_extension(i)
3545+ try:
3546+ names = decoder.decode(
3547+ extension.get_data(), asn1Spec=rfc2459.SubjectAltName())[0]
3548+ for name in names:
3549+ cert_addresses.add(str(name.getComponent()))
3550+ except:
3551+ pass
3552+ if cert_addresses != unit_addresses:
3553+ log('subjAltName: Cert (%s) != Unit (%s), assuming stale' % (
3554+ cert_addresses, unit_addresses))
3555+ return True
3556+
3557+ return False
3558+
3559+
3560+# XXX taken from the apache2 charm.
3561+def gen_selfsigned_cert(cert_file, key_file):
3562+ """
3563+ Create a self-signed certificate.
3564+
3565+ @param cert_file: destination path of generated certificate
3566+ @param key_file: destination path of generated private key
3567+ """
3568+ os.environ['OPENSSL_CN'] = unit_get('public-address')
3569+ os.environ['OPENSSL_PUBLIC'] = unit_get("public-address")
3570+ os.environ['OPENSSL_PRIVATE'] = unit_get("private-address")
3571+ # Set the umask so the child process will inherit it and
3572+ # the generated files will be readable only by root..
3573+ old_mask = os.umask(077)
3574+ subprocess.call(
3575+ ['openssl', 'req', '-new', '-x509', '-nodes', '-config',
3576+ os.path.join(os.environ['CHARM_DIR'], 'data', 'openssl.cnf'),
3577+ '-keyout', key_file, '-out', cert_file],)
3578+ os.umask(old_mask)
3579+ uid = pwd.getpwnam('haproxy').pw_uid
3580+ os.chown(key_file, uid, -1)
3581+ os.chown(cert_file, uid, -1)
3582+
3583+
3584+def write_ssl_pem(path, content):
3585+ """Write an SSL pem file and set permissions on it."""
3586+ # Set the umask so the child process will inherit it and we
3587+ # can make certificate files readable only by the 'haproxy'
3588+ # user (see below).
3589+ old_mask = os.umask(077)
3590+ with open(path, 'w') as f:
3591+ f.write(content)
3592+ os.umask(old_mask)
3593+ uid = pwd.getpwnam('haproxy').pw_uid
3594+ os.chown(path, uid, -1)
3595+
3596+
3597 # #############################################################################
3598 # Main section
3599 # #############################################################################
3600@@ -991,8 +1188,13 @@
3601 config_changed()
3602 update_nrpe_config()
3603 elif hook_name == "config-changed":
3604+ config_data = config_get()
3605+ if config_data.changed("source"):
3606+ install_hook()
3607 config_changed()
3608 update_nrpe_config()
3609+ if config_data.implicit_save:
3610+ config_data.save()
3611 elif hook_name == "start":
3612 start_hook()
3613 elif hook_name == "stop":
3614
3615=== modified file 'hooks/tests/test_config_changed_hooks.py'
3616--- hooks/tests/test_config_changed_hooks.py 2014-05-14 21:32:33 +0000
3617+++ hooks/tests/test_config_changed_hooks.py 2015-02-19 17:05:21 +0000
3618@@ -1,4 +1,6 @@
3619 import sys
3620+import base64
3621+import os
3622
3623 from testtools import TestCase
3624 from mock import patch
3625@@ -27,6 +29,8 @@
3626 "service_haproxy")
3627 self.update_sysctl = self.patch_hook(
3628 "update_sysctl")
3629+ self.update_ssl_cert = self.patch_hook(
3630+ "update_ssl_cert")
3631 self.notify_website = self.patch_hook("notify_website")
3632 self.notify_peer = self.patch_hook("notify_peer")
3633 self.write_metrics_cronjob = self.patch_hook("write_metrics_cronjob")
3634@@ -119,3 +123,27 @@
3635 'foo-defaults\n\n'
3636 )
3637 mock_open.assert_called_with(hooks.default_haproxy_config, 'w')
3638+
3639+ def test_update_ssl_cert_custom_certificate(self):
3640+ config_data = {
3641+ "ssl_cert": base64.b64encode("cert-data\n"),
3642+ "ssl_key": base64.b64encode("key-data\n")}
3643+ with patch("hooks.log"):
3644+ with patch("hooks.write_ssl_pem") as write_ssl_pem_mock:
3645+ hooks.update_ssl_cert(config_data)
3646+ default_pem_path = os.path.join(
3647+ hooks.default_haproxy_lib_dir, "default.pem")
3648+ write_ssl_pem_mock.assert_called_with(
3649+ default_pem_path, "cert-data\nkey-data\n")
3650+
3651+ def test_update_ssl_cert_selfsigned(self):
3652+ config_data = {"ssl_cert": "SELFSIGNED"}
3653+ with patch("hooks.log"):
3654+ with patch("hooks.get_selfsigned_cert") as selfsigned_mock:
3655+ selfsigned_mock.return_value = "data"
3656+ with patch("hooks.write_ssl_pem") as write_ssl_pem_mock:
3657+ hooks.update_ssl_cert(config_data)
3658+ default_pem_path = os.path.join(
3659+ hooks.default_haproxy_lib_dir, "default.pem")
3660+ write_ssl_pem_mock.assert_called_with(
3661+ default_pem_path, "data")
3662
3663=== modified file 'hooks/tests/test_helpers.py'
3664--- hooks/tests/test_helpers.py 2014-01-16 18:59:36 +0000
3665+++ hooks/tests/test_helpers.py 2015-02-19 17:05:21 +0000
3666@@ -13,18 +13,21 @@
3667
3668 class HelpersTest(TestCase):
3669
3670+ @patch('hooks.has_ssl_support')
3671 @patch('hooks.config_get')
3672- def test_creates_haproxy_globals(self, config_get):
3673+ def test_creates_haproxy_globals(self, config_get, has_ssl_support):
3674 config_get.return_value = {
3675 'global_log': 'foo-log, bar-log',
3676 'global_maxconn': 123,
3677 'global_user': 'foo-user',
3678 'global_group': 'foo-group',
3679 'global_spread_checks': 234,
3680+ 'global_default_dh_param': 345,
3681 'global_debug': False,
3682 'global_quiet': False,
3683 'global_stats_socket': True,
3684 }
3685+ has_ssl_support.return_value = True
3686 result = hooks.create_haproxy_globals()
3687
3688 sock_path = "/var/run/haproxy/haproxy.sock"
3689@@ -36,22 +39,27 @@
3690 ' user foo-user',
3691 ' group foo-group',
3692 ' spread-checks 234',
3693+ ' tune.ssl.default-dh-param 345',
3694 ' stats socket %s mode 0600' % sock_path,
3695 ])
3696 self.assertEqual(result, expected)
3697
3698+ @patch('hooks.has_ssl_support')
3699 @patch('hooks.config_get')
3700- def test_creates_haproxy_globals_quietly_with_debug(self, config_get):
3701+ def test_creates_haproxy_globals_quietly_with_debug(
3702+ self, config_get, has_ssl_support):
3703 config_get.return_value = {
3704 'global_log': 'foo-log, bar-log',
3705 'global_maxconn': 123,
3706 'global_user': 'foo-user',
3707 'global_group': 'foo-group',
3708 'global_spread_checks': 234,
3709+ 'global_default_dh_param': 345,
3710 'global_debug': True,
3711 'global_quiet': True,
3712 'global_stats_socket': False,
3713 }
3714+ has_ssl_support.return_value = True
3715 result = hooks.create_haproxy_globals()
3716
3717 expected = '\n'.join([
3718@@ -64,6 +72,37 @@
3719 ' debug',
3720 ' quiet',
3721 ' spread-checks 234',
3722+ ' tune.ssl.default-dh-param 345',
3723+ ])
3724+ self.assertEqual(result, expected)
3725+
3726+ @patch('hooks.has_ssl_support')
3727+ @patch('hooks.config_get')
3728+ def test_creates_haproxy_globals_no_ssl_support(
3729+ self, config_get, has_ssl_support):
3730+ config_get.return_value = {
3731+ 'global_log': 'foo-log, bar-log',
3732+ 'global_maxconn': 123,
3733+ 'global_user': 'foo-user',
3734+ 'global_group': 'foo-group',
3735+ 'global_spread_checks': 234,
3736+ 'global_debug': False,
3737+ 'global_quiet': False,
3738+ 'global_stats_socket': True,
3739+ }
3740+ has_ssl_support.return_value = False
3741+ result = hooks.create_haproxy_globals()
3742+
3743+ sock_path = "/var/run/haproxy/haproxy.sock"
3744+ expected = '\n'.join([
3745+ 'global',
3746+ ' log foo-log',
3747+ ' log bar-log',
3748+ ' maxconn 123',
3749+ ' user foo-user',
3750+ ' group foo-group',
3751+ ' spread-checks 234',
3752+ ' stats socket %s mode 0600' % sock_path,
3753 ])
3754 self.assertEqual(result, expected)
3755
3756@@ -192,6 +231,23 @@
3757 stanzas)
3758
3759 @patch('hooks.load_haproxy_config')
3760+ def test_get_listen_stanzas_with_ssl_frontend(self, load_haproxy_config):
3761+ load_haproxy_config.return_value = '''
3762+ frontend foo-2-123
3763+ bind 1.2.3.4:123 ssl crt /foo/bar
3764+ default_backend foo.internal
3765+ frontend foo-2-234
3766+ bind 1.2.3.5:234
3767+ default_backend bar.internal
3768+ '''
3769+
3770+ stanzas = hooks.get_listen_stanzas()
3771+
3772+ self.assertEqual((('foo.internal', '1.2.3.4', 123),
3773+ ('bar.internal', '1.2.3.5', 234)),
3774+ stanzas)
3775+
3776+ @patch('hooks.load_haproxy_config')
3777 def test_get_empty_tuple_when_no_stanzas(self, load_haproxy_config):
3778 load_haproxy_config.return_value = '''
3779 '''
3780@@ -393,6 +449,35 @@
3781
3782 self.assertEqual(expected, result)
3783
3784+ @patch.dict(os.environ, {"JUJU_UNIT_NAME": "haproxy/2"})
3785+ def test_creates_a_listen_stanza_with_crts(self):
3786+ service_name = 'foo'
3787+ service_ip = '1.2.3.4'
3788+ service_port = 443
3789+ server_entries = [
3790+ ('name-1', 'ip-1', 'port-1', ('foo1', 'bar1')),
3791+ ]
3792+ content = ("-----BEGIN CERTIFICATE-----\n"
3793+ "<data>\n"
3794+ "-----END CERTIFICATE-----\n")
3795+ crts = [base64.b64encode(content)]
3796+
3797+ result = hooks.create_listen_stanza(service_name, service_ip,
3798+ service_port,
3799+ server_entries=server_entries,
3800+ service_crts=crts)
3801+
3802+ expected = '\n'.join((
3803+ 'frontend haproxy-2-443',
3804+ ' bind 1.2.3.4:443 ssl crt /var/lib/haproxy/service_foo/0.pem',
3805+ ' default_backend foo',
3806+ '',
3807+ 'backend foo',
3808+ ' server name-1 ip-1:port-1 foo1 bar1',
3809+ ))
3810+
3811+ self.assertEqual(expected, result)
3812+
3813 def test_doesnt_create_listen_stanza_if_args_not_provided(self):
3814 self.assertIsNone(hooks.create_listen_stanza())
3815
3816
3817=== modified file 'hooks/tests/test_install.py'
3818--- hooks/tests/test_install.py 2015-01-20 09:47:15 +0000
3819+++ hooks/tests/test_install.py 2015-02-19 17:05:21 +0000
3820@@ -44,8 +44,12 @@
3821
3822 def test_install_packages(self):
3823 hooks.install_hook()
3824- self.apt_install.assert_called_once_with(
3825- ['haproxy', 'python-jinja2'], fatal=True)
3826+ calls = self.apt_install.call_args_list
3827+ self.assertEqual((['haproxy', 'python-jinja2'],), calls[0][0])
3828+ self.assertEqual({'fatal': True}, calls[0][1])
3829+ self.assertEqual(
3830+ (['python-pyasn1', 'python-pyasn1-modules'],), calls[1][0])
3831+ self.assertEqual({'fatal': False}, calls[1][1])
3832
3833 def test_add_source(self):
3834 hooks.install_hook()
3835@@ -58,6 +62,21 @@
3836 hooks.install_hook()
3837 self.apt_update.assert_called_once_with(fatal=True)
3838
3839+ def test_add_source_with_backports(self):
3840+ self.config_get.return_value = {
3841+ 'source': 'backports', 'package_status': 'install'}
3842+ with patch("charmhelpers.core.host.lsb_release") as lsb_release:
3843+ lsb_release.return_value = {'DISTRIB_CODENAME': 'trusty'}
3844+ with patch("hooks.add_backports_preferences") as add_apt_prefs:
3845+ add_apt_prefs.assert_called_once()
3846+ hooks.install_hook()
3847+ self.config_get.assert_called_once()
3848+ source = ("deb http://archive.ubuntu.com/ubuntu trusty-backports "
3849+ "main restricted universe multiverse")
3850+ self.add_source.assert_called_once_with(
3851+ source,
3852+ self.config_get.return_value.get("key"))
3853+
3854 def test_ensures_package_status(self):
3855 hooks.install_hook()
3856 self.config_get.assert_called_once()
3857
3858=== modified file 'hooks/tests/test_peer_hooks.py'
3859--- hooks/tests/test_peer_hooks.py 2014-09-08 18:21:01 +0000
3860+++ hooks/tests/test_peer_hooks.py 2015-02-19 17:05:21 +0000
3861@@ -1,6 +1,7 @@
3862 import base64
3863 import os
3864 import yaml
3865+import pwd
3866
3867 from testtools import TestCase
3868 from mock import patch
3869@@ -196,7 +197,7 @@
3870
3871 create_listen_stanza.assert_called_with(
3872 'bar', 'some-host', 'some-port', 'some-options',
3873- (1, 2), [])
3874+ (1, 2), [], [])
3875 mock_open.assert_called_with(
3876 '/var/run/haproxy/bar.service', 'w')
3877 mock_file.write.assert_called_with('some content')
3878@@ -232,3 +233,60 @@
3879 '/var/lib/haproxy/service_bar/403.http', 'w')
3880 mock_file.write.assert_any_call(content)
3881 self.assertTrue(create_listen_stanza.called)
3882+
3883+ @patch('hooks.create_listen_stanza')
3884+ def test_writes_crts(self, create_listen_stanza):
3885+ create_listen_stanza.return_value = 'some content'
3886+
3887+ content = ("-----BEGIN CERTIFICATE-----\n"
3888+ "<data>\n"
3889+ "-----END CERTIFICATE-----\n")
3890+ services_dict = {
3891+ 'foo': {
3892+ 'service_name': 'bar',
3893+ 'service_host': 'some-host',
3894+ 'service_port': 'some-port',
3895+ 'service_options': 'some-options',
3896+ 'servers': (1, 2),
3897+ 'crts': [base64.b64encode(content)]
3898+ },
3899+ }
3900+
3901+ with patch.object(os.path, "exists") as exists:
3902+ exists.return_value = True
3903+ with patch_open() as (mock_open, mock_file):
3904+ with patch.object(pwd, "getpwnam") as getpwnam:
3905+ class DB(object):
3906+ pw_uid = 9999
3907+ getpwnam.return_value = DB()
3908+ with patch.object(os, "chown") as chown:
3909+ hooks.write_service_config(services_dict)
3910+ path = '/var/lib/haproxy/service_bar/0.pem'
3911+ mock_open.assert_any_call(path, 'w')
3912+ mock_file.write.assert_any_call(content)
3913+ chown.assert_called_with(path, 9999, - 1)
3914+ self.assertTrue(create_listen_stanza.called)
3915+
3916+ @patch('hooks.create_listen_stanza')
3917+ def test_skip_crts_default(self, create_listen_stanza):
3918+ create_listen_stanza.return_value = 'some content'
3919+ services_dict = {
3920+ 'foo': {
3921+ 'service_name': 'bar',
3922+ 'service_host': 'some-host',
3923+ 'service_port': 'some-port',
3924+ 'service_options': 'some-options',
3925+ 'servers': (1, 2),
3926+ 'crts': ["DEFAULT"]
3927+ },
3928+ }
3929+
3930+ with patch.object(os.path, "exists") as exists:
3931+ exists.return_value = True
3932+ with patch.object(os, "makedirs"):
3933+ with patch_open() as (mock_open, mock_file):
3934+ hooks.write_service_config(services_dict)
3935+ self.assertNotEqual(
3936+ mock_open.call_args,
3937+ ('/var/lib/haproxy/service_bar/0.pem', 'w'))
3938+ self.assertTrue(create_listen_stanza.called)
3939
3940=== modified file 'tests/10_deploy_test.py'
3941--- tests/10_deploy_test.py 2014-10-31 21:27:09 +0000
3942+++ tests/10_deploy_test.py 2015-02-19 17:05:21 +0000
3943@@ -6,6 +6,8 @@
3944 import amulet
3945 import requests
3946 import base64
3947+import yaml
3948+import time
3949
3950 d = amulet.Deployment(series='trusty')
3951 # Add the haproxy charm to the deployment.
3952@@ -81,5 +83,48 @@
3953 'configuration file.' % apache_private
3954 amulet.raise_status(amulet.FAIL, msg=message)
3955
3956+d.configure('haproxy', {
3957+ 'source': 'backports',
3958+ 'ssl_cert': 'SELFSIGNED',
3959+ 'services': yaml.safe_dump([
3960+ {'service_name': 'apache',
3961+ 'service_host': '0.0.0.0',
3962+ 'service_port': 80,
3963+ 'service_options': [
3964+ 'mode http', 'balance leastconn', 'option httpchk GET / HTTP/1.0'
3965+ ],
3966+ 'servers': [
3967+ ['apache', apache_private, 80, 'maxconn 50']]},
3968+ {'service_name': 'apache-ssl',
3969+ 'service_port': 443,
3970+ 'service_host': '0.0.0.0',
3971+ 'service_options': [
3972+ 'mode http', 'balance leastconn', 'option httpchk GET / HTTP/1.0'
3973+ ],
3974+ 'crts': ['DEFAULT'],
3975+ 'servers': [['apache', apache_private, 80, 'maxconn 50']]}])
3976+})
3977+
3978+# We need a retry loop here, since there's no way to tell when the new
3979+# configuration is in place.
3980+url = 'http://%s/index.html' % haproxy_address
3981+secure_url = 'https://%s/index.html' % haproxy_address
3982+retries = 10
3983+for i in range(retries):
3984+ try:
3985+ page = requests.get(url)
3986+ page.raise_for_status()
3987+ page = requests.get(secure_url, verify=False)
3988+ page.raise_for_status()
3989+ except requests.exceptions.ConnectionError:
3990+ if i == retries - 1:
3991+ # This was the last one, let's fail
3992+ raise
3993+ time.sleep(6)
3994+ else:
3995+ break
3996+
3997+print('Successfully got the Apache2 web page through haproxy SSL termination.')
3998+
3999 # Send a message that the tests are complete.
4000 print('The haproxy tests are complete.')

Subscribers

People subscribed via source and target branches

to all changes: