Juju Charms Collection
haproxy package

Merge lp:~free.ekanayaka/charms/trusty/haproxy/backends-support into lp:charms/trusty/haproxy

  1. Trusty Tahr (14.04)
  2. backends-support
  3. Merge into trunk
Proposed by Free Ekanayaka
Status: Merged
Merged at revision: 89
Proposed branch: lp:~free.ekanayaka/charms/trusty/haproxy/backends-support
Merge into: lp:charms/trusty/haproxy
Diff against target: 443 lines (+227/-19)
6 files modified
README.md (+24/-3)
hooks/hooks.py (+52/-11)
hooks/tests/test_helpers.py (+35/-3)
hooks/tests/test_peer_hooks.py (+1/-1)
hooks/tests/test_reverseproxy_hooks.py (+66/-0)
tests/10_deploy_test.py (+49/-1)
To merge this branch: bzr merge lp:~free.ekanayaka/charms/trusty/haproxy/backends-support
Reviewer Review Type Date Requested Status
Chris Glass (community) Approve
Review Queue (community) automated testing Needs Fixing
Review via email: mp+251908@code.launchpad.net

Description of the change

This branch adds support for configuring extra backends beside the default one. This allows for example to implement URL-based routing and forward traffic to different backends based on URL patterns.

To post a comment you must log in.
lp:~free.ekanayaka/charms/trusty/haproxy/backends-support updated
92. By Free Ekanayaka

Add docstring

93. By Free Ekanayaka

More comments

Revision history for this message
Review Queue (review-queue) wrote :

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11100-results

review: Needs Fixing (automated testing)
Revision history for this message
Review Queue (review-queue) wrote :

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11101-results

review: Needs Fixing (automated testing)
lp:~free.ekanayaka/charms/trusty/haproxy/backends-support updated
94. By Free Ekanayaka

Dummy commit to trigger test bots

95. By Free Ekanayaka

Woraround sentry bug in tests

Revision history for this message
Chris Glass (tribaal) wrote :

Looks good, despite the CI not +1'ing (connection errors or CI errors - not branch problems).

+1, will merge now.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'README.md'
--- README.md 2015-02-09 16:45:29 +0000
+++ README.md 2015-03-18 20:28:50 +0000
@@ -65,6 +65,8 @@
6565
66 relation-set "services=66 relation-set "services=
67 - { service_name: my_web_app,67 - { service_name: my_web_app,
68 service_host: 0.0.0.0,
69 service_port: 80,
68 service_options: [mode http, balance leastconn],70 service_options: [mode http, balance leastconn],
69 servers: [[my_web_app_1, $host, $port, option httpchk GET / HTTP/1.0],71 servers: [[my_web_app_1, $host, $port, option httpchk GET / HTTP/1.0],
70 [... optionally more servers here ...]]}72 [... optionally more servers here ...]]}
@@ -72,9 +74,28 @@
72 "74 "
7375
74Once set, haproxy will union multiple `servers` stanzas from any units76Once set, haproxy will union multiple `servers` stanzas from any units
75joining with the same `service_name` under one listen stanza.77joining with the same `service_name` under one backend stanza, which will be
76`service-options` and `server_options` will be overwritten, so ensure they78the default backend for the service (requests against the given service_port on
77are set uniformly on all services with the same name.79the haproxy unit will be forwarded to that backend). Note that `service-options`
80and `server_options` will be overwritten, so ensure they are set uniformly on
81all services with the same name.
82
83If you need additional backends, possibly handling ACL-filtered requests, you
84can add a 'backends' entry to a service stanza. For example in order to redirect
85to a different backend all requests to URLs starting with '/foo', you could have:
86
87 relation-set "services=
88 - { service_name: my_web_app,
89 service_host: 0.0.0.0,
90 service_port: 80,
91 service_options: [mode http, acl foo path_beg -i /foo, use_backend foo if foo],
92 servers: [[my_web_app_1, $host, $port, option httpchk GET / HTTP/1.0],
93 [... optionally more servers here ...]]
94 backends:
95 - { backend_name: foo,
96 servers: [[my_web_app2, $host, $port2, option httpchk GET / HTTP/1.0],
97 [... optionally more servers here ...]]}}
98
7899
79## Website Relation100## Website Relation
80101
81102
=== modified file 'hooks/hooks.py'
--- hooks/hooks.py 2015-02-19 17:05:11 +0000
+++ hooks/hooks.py 2015-03-18 20:28:50 +0000
@@ -62,6 +62,7 @@
62 ]62 ]
6363
64frontend_only_options = [64frontend_only_options = [
65 "acl",
65 "backlog",66 "backlog",
66 "bind",67 "bind",
67 "capture cookie",68 "capture cookie",
@@ -69,6 +70,7 @@
69 "capture response header",70 "capture response header",
70 "clitimeout",71 "clitimeout",
71 "default_backend",72 "default_backend",
73 "http-request",
72 "maxconn",74 "maxconn",
73 "monitor fail",75 "monitor fail",
74 "monitor-net",76 "monitor-net",
@@ -84,6 +86,7 @@
84 "option socket-stats",86 "option socket-stats",
85 "option tcp-smart-accept",87 "option tcp-smart-accept",
86 "rate-limit sessions",88 "rate-limit sessions",
89 "redirect",
87 "tcp-request content accept",90 "tcp-request content accept",
88 "tcp-request content reject",91 "tcp-request content reject",
89 "tcp-request inspect-delay",92 "tcp-request inspect-delay",
@@ -303,11 +306,14 @@
303# service_ip: IP address to listen for connections306# service_ip: IP address to listen for connections
304# service_port: Port to listen for connections307# service_port: Port to listen for connections
305# service_options: Comma separated list of options308# service_options: Comma separated list of options
306# server_entries: List of tuples309# server_entries: List of tuples
307# server_name310# server_name
308# server_ip311# server_ip
309# server_port312# server_port
310# server_options313# server_options
314# backends: List of dicts
315# backend_name: backend name,
316# servers: list of tuples as in server_entries
311# errorfiles: List of dicts317# errorfiles: List of dicts
312# http_status: status to handle318# http_status: status to handle
313# content: base 64 content for HAProxy to319# content: base 64 content for HAProxy to
@@ -318,7 +324,7 @@
318def create_listen_stanza(service_name=None, service_ip=None,324def create_listen_stanza(service_name=None, service_ip=None,
319 service_port=None, service_options=None,325 service_port=None, service_options=None,
320 server_entries=None, service_errorfiles=None,326 server_entries=None, service_errorfiles=None,
321 service_crts=None):327 service_crts=None, service_backends=None):
322 if service_name is None or service_ip is None or service_port is None:328 if service_name is None or service_ip is None or service_port is None:
323 return None329 return None
324 fe_options = []330 fe_options = []
@@ -359,17 +365,43 @@
359 service_config.append(" default_backend %s" % (service_name,))365 service_config.append(" default_backend %s" % (service_name,))
360 service_config.extend(" %s" % service_option.strip()366 service_config.extend(" %s" % service_option.strip()
361 for service_option in fe_options)367 for service_option in fe_options)
362 service_config.append("")368
363 service_config.append("backend %s" % (service_name,))369 # For now errorfiles are common for all backends, in the future we
364 service_config.extend(" %s" % service_option.strip()370 # might offer support for per-backend error files.
365 for service_option in be_options)371 backend_errorfiles = [] # List of (status, path) tuples
366 if service_errorfiles is not None:372 if service_errorfiles is not None:
367 for errorfile in service_errorfiles:373 for errorfile in service_errorfiles:
368 path = os.path.join(default_haproxy_lib_dir,374 path = os.path.join(default_haproxy_lib_dir,
369 "service_%s" % service_name,375 "service_%s" % service_name,
370 "%s.http" % errorfile["http_status"])376 "%s.http" % errorfile["http_status"])
371 service_config.append(377 backend_errorfiles.append((errorfile["http_status"], path))
372 " errorfile %s %s" % (errorfile["http_status"], path))378
379 # Default backend
380 _append_backend(
381 service_config, service_name, be_options, backend_errorfiles,
382 server_entries)
383
384 # Extra backends
385 if service_backends is not None:
386 for service_backend in service_backends:
387 _append_backend(
388 service_config, service_backend["backend_name"],
389 be_options, backend_errorfiles, service_backend["servers"])
390
391 return '\n'.join(service_config)
392
393
394def _append_backend(service_config, name, options, errorfiles, server_entries):
395 """Append a new backend stanza to the given service_config.
396
397 A backend stanza consists in a 'backend <name>' line followed by option
398 lines, errorfile lines and server line.
399 """
400 service_config.append("")
401 service_config.append("backend %s" % (name,))
402 service_config.extend(" %s" % option.strip() for option in options)
403 for status, path in errorfiles:
404 service_config.append(" errorfile %s %s" % (status, path))
373 if isinstance(server_entries, (list, tuple)):405 if isinstance(server_entries, (list, tuple)):
374 for i, (server_name, server_ip, server_port,406 for i, (server_name, server_ip, server_port,
375 server_options) in enumerate(server_entries):407 server_options) in enumerate(server_entries):
@@ -382,7 +414,6 @@
382 server_line += " " + " ".join(server_options)414 server_line += " " + " ".join(server_options)
383 server_line = server_line.format(i=i)415 server_line = server_line.format(i=i)
384 service_config.append(server_line)416 service_config.append(server_line)
385 return '\n'.join(service_config)
386417
387418
388# -----------------------------------------------------------------------------419# -----------------------------------------------------------------------------
@@ -403,7 +434,7 @@
403 monitoring_config.append("mode http")434 monitoring_config.append("mode http")
404 monitoring_config.append("acl allowed_cidr src %s" %435 monitoring_config.append("acl allowed_cidr src %s" %
405 config_data['monitoring_allowed_cidr'])436 config_data['monitoring_allowed_cidr'])
406 monitoring_config.append("block unless allowed_cidr")437 monitoring_config.append("http-request deny unless allowed_cidr")
407 monitoring_config.append("stats enable")438 monitoring_config.append("stats enable")
408 monitoring_config.append("stats uri /")439 monitoring_config.append("stats uri /")
409 monitoring_config.append("stats realm Haproxy\ Statistics")440 monitoring_config.append("stats realm Haproxy\ Statistics")
@@ -475,11 +506,20 @@
475 service = new_service.copy()506 service = new_service.copy()
476 service.update(old_service)507 service.update(old_service)
477 if "servers" in service:508 if "servers" in service:
509 # Merge all 'servers' entries of the default backend
478 servers = service["servers"]510 servers = service["servers"]
479 if "servers" in new_service:511 if "servers" in new_service:
480 servers.extend(new_service["servers"])512 servers.extend(new_service["servers"])
481 servers.sort()513 servers.sort()
482 service["servers"] = list(x for x, _ in groupby(servers))514 service["servers"] = list(x for x, _ in groupby(servers))
515 if "backends" in service and "backends" in new_service:
516 # Merge all 'servers' entries of the additional backends
517 for i, backend in enumerate(service["backends"]):
518 servers = backend["servers"]
519 servers.extend(new_service["backends"][i]["servers"])
520 servers.sort()
521 service["backends"][i]["servers"] = list(
522 x for x, _ in groupby(servers))
483 return service523 return service
484524
485525
@@ -688,6 +728,7 @@
688 log("Service: %s" % service_key)728 log("Service: %s" % service_key)
689 service_name = service_config["service_name"]729 service_name = service_config["service_name"]
690 server_entries = service_config.get('servers')730 server_entries = service_config.get('servers')
731 backends = service_config.get('backends', [])
691732
692 errorfiles = service_config.get('errorfiles', [])733 errorfiles = service_config.get('errorfiles', [])
693 for errorfile in errorfiles:734 for errorfile in errorfiles:
@@ -718,7 +759,7 @@
718 service_config['service_host'],759 service_config['service_host'],
719 service_config['service_port'],760 service_config['service_port'],
720 service_config['service_options'],761 service_config['service_options'],
721 server_entries, errorfiles, crts))762 server_entries, errorfiles, crts, backends))
722763
723764
724def get_service_lib_path(service_name):765def get_service_lib_path(service_name):
725766
=== modified file 'hooks/tests/test_helpers.py'
--- hooks/tests/test_helpers.py 2015-02-09 11:39:40 +0000
+++ hooks/tests/test_helpers.py 2015-03-18 20:28:50 +0000
@@ -478,6 +478,38 @@
478478
479 self.assertEqual(expected, result)479 self.assertEqual(expected, result)
480480
481 @patch.dict(os.environ, {"JUJU_UNIT_NAME": "haproxy/2"})
482 def test_creates_a_listen_stanza_with_backends(self):
483 service_name = 'foo'
484 service_ip = '1.2.3.4'
485 service_port = 80
486 server_entries = [
487 ('name-1', 'ip-1', 'port-1', ('foo1', 'bar1')),
488 ]
489 service_backends = [
490 {"backend_name": "foo-bar",
491 "servers": [
492 ('bar-name-1', 'bar-ip-1', 'bar-port-1', ('bar2', 'bar3'))
493 ]}
494 ]
495 result = hooks.create_listen_stanza(
496 service_name, service_ip, service_port,
497 server_entries=server_entries, service_backends=service_backends)
498
499 expected = '\n'.join((
500 'frontend haproxy-2-80',
501 ' bind 1.2.3.4:80',
502 ' default_backend foo',
503 '',
504 'backend foo',
505 ' server name-1 ip-1:port-1 foo1 bar1',
506 '',
507 'backend foo-bar',
508 ' server bar-name-1 bar-ip-1:bar-port-1 bar2 bar3',
509 ))
510
511 self.assertEqual(expected, result)
512
481 def test_doesnt_create_listen_stanza_if_args_not_provided(self):513 def test_doesnt_create_listen_stanza_if_args_not_provided(self):
482 self.assertIsNone(hooks.create_listen_stanza())514 self.assertIsNone(hooks.create_listen_stanza())
483515
@@ -504,7 +536,7 @@
504 'some-service', '0.0.0.0', 1234, [536 'some-service', '0.0.0.0', 1234, [
505 'mode http',537 'mode http',
506 'acl allowed_cidr src some-cidr',538 'acl allowed_cidr src some-cidr',
507 'block unless allowed_cidr',539 'http-request deny unless allowed_cidr',
508 'stats enable',540 'stats enable',
509 'stats uri /',541 'stats uri /',
510 'stats realm Haproxy\\ Statistics',542 'stats realm Haproxy\\ Statistics',
@@ -551,7 +583,7 @@
551 'some-service', '0.0.0.0', 1234, [583 'some-service', '0.0.0.0', 1234, [
552 'mode http',584 'mode http',
553 'acl allowed_cidr src some-cidr',585 'acl allowed_cidr src some-cidr',
554 'block unless allowed_cidr',586 'http-request deny unless allowed_cidr',
555 'stats enable',587 'stats enable',
556 'stats uri /',588 'stats uri /',
557 'stats realm Haproxy\\ Statistics',589 'stats realm Haproxy\\ Statistics',
@@ -585,7 +617,7 @@
585 'some-service', '0.0.0.0', 1234, [617 'some-service', '0.0.0.0', 1234, [
586 'mode http',618 'mode http',
587 'acl allowed_cidr src some-cidr',619 'acl allowed_cidr src some-cidr',
588 'block unless allowed_cidr',620 'http-request deny unless allowed_cidr',
589 'stats enable',621 'stats enable',
590 'stats uri /',622 'stats uri /',
591 'stats realm Haproxy\\ Statistics',623 'stats realm Haproxy\\ Statistics',
592624
=== modified file 'hooks/tests/test_peer_hooks.py'
--- hooks/tests/test_peer_hooks.py 2015-02-19 17:05:11 +0000
+++ hooks/tests/test_peer_hooks.py 2015-03-18 20:28:50 +0000
@@ -197,7 +197,7 @@
197197
198 create_listen_stanza.assert_called_with(198 create_listen_stanza.assert_called_with(
199 'bar', 'some-host', 'some-port', 'some-options',199 'bar', 'some-host', 'some-port', 'some-options',
200 (1, 2), [], [])200 (1, 2), [], [], [])
201 mock_open.assert_called_with(201 mock_open.assert_called_with(
202 '/var/run/haproxy/bar.service', 'w')202 '/var/run/haproxy/bar.service', 'w')
203 mock_file.write.assert_called_with('some content')203 mock_file.write.assert_called_with('some content')
204204
=== modified file 'hooks/tests/test_reverseproxy_hooks.py'
--- hooks/tests/test_reverseproxy_hooks.py 2014-01-21 15:46:29 +0000
+++ hooks/tests/test_reverseproxy_hooks.py 2015-03-18 20:28:50 +0000
@@ -393,6 +393,72 @@
393 self.assertEqual(expected, hooks.create_services())393 self.assertEqual(expected, hooks.create_services())
394 self.write_service_config.assert_called_with(expected)394 self.write_service_config.assert_called_with(expected)
395395
396 def test_with_multiple_units_and_backends_in_relation(self):
397 """
398 Have multiple units specifying "services" in the relation
399 using the "backends" option. Make sure data is created correctly
400 with create_services()
401 """
402 self.get_config_services.return_value = {
403 None: {
404 "service_name": "service",
405 },
406 }
407 self.relations_of_type.return_value = [
408 {"port": 4242,
409 "private-address": "1.2.3.4",
410 "__unit__": "foo/0",
411 "services": yaml.safe_dump([{
412 "service_name": "service",
413 "servers": [('foo-0', '1.2.3.4',
414 4242, ["maxconn 4"])],
415 "backends": [
416 {"backend_name": "foo-bar",
417 "servers": [('foo-bar-0', '2.2.2.2',
418 2222, ["maxconn 4"])],
419 },
420 ]
421 }])
422 },
423 {"port": 4242,
424 "private-address": "1.2.3.5",
425 "__unit__": "foo/1",
426 "services": yaml.safe_dump([{
427 "service_name": "service",
428 "servers": [('foo-0', '1.2.3.5',
429 4242, ["maxconn 4"])],
430 "backends": [
431 {"backend_name": "foo-bar",
432 "servers": [('foo-bar-1', '2.2.2.3',
433 3333, ["maxconn 4"])],
434 },
435 ]
436 }])
437 },
438 ]
439
440 expected = {
441 'service': {
442 'service_name': 'service',
443 'service_host': '0.0.0.0',
444 'service_port': 10002,
445 'servers': [
446 ['foo-0', '1.2.3.4', 4242, ["maxconn 4"]],
447 ['foo-0', '1.2.3.5', 4242, ["maxconn 4"]]
448 ],
449 'backends': [
450 {"backend_name": "foo-bar",
451 "servers": [
452 ['foo-bar-0', '2.2.2.2', 2222, ["maxconn 4"]],
453 ['foo-bar-1', '2.2.2.3', 3333, ["maxconn 4"]],
454 ],
455 },
456 ]
457 },
458 }
459 self.assertEqual(expected, hooks.create_services())
460 self.write_service_config.assert_called_with(expected)
461
396 def test_merge_service(self):462 def test_merge_service(self):
397 """ Make sure merge_services maintains "server" entries. """463 """ Make sure merge_services maintains "server" entries. """
398 s1 = {'service_name': 'f', 'servers': [['f', '4', 4, ['maxconn 4']]]}464 s1 = {'service_name': 'f', 'servers': [['f', '4', 4, ['maxconn 4']]]}
399465
=== modified file 'tests/10_deploy_test.py'
--- tests/10_deploy_test.py 2015-02-19 17:05:11 +0000
+++ tests/10_deploy_test.py 2015-03-18 20:28:50 +0000
@@ -12,7 +12,7 @@
12d = amulet.Deployment(series='trusty')12d = amulet.Deployment(series='trusty')
13# Add the haproxy charm to the deployment.13# Add the haproxy charm to the deployment.
14d.add('haproxy')14d.add('haproxy')
15d.add('apache2')15d.add('apache2', units=2)
1616
17# Get the directory this way to load the file when CWD is different.17# Get the directory this way to load the file when CWD is different.
18path = os.path.abspath(os.path.dirname(__file__))18path = os.path.abspath(os.path.dirname(__file__))
@@ -83,6 +83,7 @@
83 'configuration file.' % apache_private83 'configuration file.' % apache_private
84 amulet.raise_status(amulet.FAIL, msg=message)84 amulet.raise_status(amulet.FAIL, msg=message)
8585
86# Test SSL termination
86d.configure('haproxy', {87d.configure('haproxy', {
87 'source': 'backports',88 'source': 'backports',
88 'ssl_cert': 'SELFSIGNED',89 'ssl_cert': 'SELFSIGNED',
@@ -116,6 +117,7 @@
116 page.raise_for_status()117 page.raise_for_status()
117 page = requests.get(secure_url, verify=False)118 page = requests.get(secure_url, verify=False)
118 page.raise_for_status()119 page.raise_for_status()
120 success = True
119 except requests.exceptions.ConnectionError:121 except requests.exceptions.ConnectionError:
120 if i == retries - 1:122 if i == retries - 1:
121 # This was the last one, let's fail123 # This was the last one, let's fail
@@ -126,5 +128,51 @@
126128
127print('Successfully got the Apache2 web page through haproxy SSL termination.')129print('Successfully got the Apache2 web page through haproxy SSL termination.')
128130
131apache_unit2 = d.sentry.unit['apache2/1']
132apache_private2 = apache_unit2.run("unit-get private-address")[0]
133
134# Create a file on the second apache unit's www directory.
135apache_unit2.run("echo foo > /var/www/html/foo")
136
137d.configure('haproxy', {
138 'services': yaml.safe_dump([
139 {'service_name': 'apache',
140 'service_host': '0.0.0.0',
141 'service_port': 80,
142 'service_options': [
143 'mode http', 'balance leastconn', 'option httpchk GET / HTTP/1.0',
144 'acl foo path_beg -i /foo', 'use_backend foo if foo',
145 ],
146 'servers': [
147 ['apache', apache_private, 80, 'maxconn 50']],
148 'backends': [
149 {'backend_name': 'foo',
150 'servers': [
151 ['apache2', apache_private2, 80, 'maxconn 50']]}
152 ]}])
153})
154
155# Let's exercise our URL-based routing by trying to fetch a URL that will
156# only work for the second apache unit (which is configured as server
157# of the extra backend).
158url = 'http://%s/foo' % haproxy_address
159
160# We need a retry loop here, since there's no way to tell when the new
161# configuration is in place.
162retries = 10
163for i in range(retries):
164 try:
165 page = requests.get(url)
166 page.raise_for_status()
167 except:
168 if i == retries - 1:
169 # This was the last one, let's fail
170 raise
171 time.sleep(6)
172 else:
173 break
174
175print('Successfully got the /foo URL from the second Apache unit.')
176
129# Send a message that the tests are complete.177# Send a message that the tests are complete.
130print('The haproxy tests are complete.')178print('The haproxy tests are complete.')

Subscribers

People subscribed via source and target branches

to all changes: