Juju Charms Collection
haproxy package

Merge lp:~free.ekanayaka/charms/trusty/haproxy/backends-support into lp:charms/trusty/haproxy

  1. Trusty Tahr (14.04)
  2. backends-support
  3. Merge into trunk
Proposed by Free Ekanayaka
Status: Merged
Merged at revision: 89
Proposed branch: lp:~free.ekanayaka/charms/trusty/haproxy/backends-support
Merge into: lp:charms/trusty/haproxy
Diff against target: 443 lines (+227/-19)
6 files modified
README.md (+24/-3)
hooks/hooks.py (+52/-11)
hooks/tests/test_helpers.py (+35/-3)
hooks/tests/test_peer_hooks.py (+1/-1)
hooks/tests/test_reverseproxy_hooks.py (+66/-0)
tests/10_deploy_test.py (+49/-1)
To merge this branch: bzr merge lp:~free.ekanayaka/charms/trusty/haproxy/backends-support
Reviewer Review Type Date Requested Status
Chris Glass (community) Approve
Review Queue (community) automated testing Needs Fixing
Review via email: mp+251908@code.launchpad.net

Description of the change

This branch adds support for configuring extra backends beside the default one. This allows for example to implement URL-based routing and forward traffic to different backends based on URL patterns.

To post a comment you must log in.
lp:~free.ekanayaka/charms/trusty/haproxy/backends-support updated
92. By Free Ekanayaka

Add docstring

93. By Free Ekanayaka

More comments

Revision history for this message
Review Queue (review-queue) wrote :

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11100-results

review: Needs Fixing (automated testing)
Revision history for this message
Review Queue (review-queue) wrote :

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11101-results

review: Needs Fixing (automated testing)
lp:~free.ekanayaka/charms/trusty/haproxy/backends-support updated
94. By Free Ekanayaka

Dummy commit to trigger test bots

95. By Free Ekanayaka

Woraround sentry bug in tests

Revision history for this message
Chris Glass (tribaal) wrote :

Looks good, despite the CI not +1'ing (connection errors or CI errors - not branch problems).

+1, will merge now.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'README.md'
2--- README.md 2015-02-09 16:45:29 +0000
3+++ README.md 2015-03-18 20:28:50 +0000
4@@ -65,6 +65,8 @@
5
6 relation-set "services=
7 - { service_name: my_web_app,
8+ service_host: 0.0.0.0,
9+ service_port: 80,
10 service_options: [mode http, balance leastconn],
11 servers: [[my_web_app_1, $host, $port, option httpchk GET / HTTP/1.0],
12 [... optionally more servers here ...]]}
13@@ -72,9 +74,28 @@
14 "
15
16 Once set, haproxy will union multiple `servers` stanzas from any units
17-joining with the same `service_name` under one listen stanza.
18-`service-options` and `server_options` will be overwritten, so ensure they
19-are set uniformly on all services with the same name.
20+joining with the same `service_name` under one backend stanza, which will be
21+the default backend for the service (requests against the given service_port on
22+the haproxy unit will be forwarded to that backend). Note that `service-options`
23+and `server_options` will be overwritten, so ensure they are set uniformly on
24+all services with the same name.
25+
26+If you need additional backends, possibly handling ACL-filtered requests, you
27+can add a 'backends' entry to a service stanza. For example in order to redirect
28+to a different backend all requests to URLs starting with '/foo', you could have:
29+
30+ relation-set "services=
31+ - { service_name: my_web_app,
32+ service_host: 0.0.0.0,
33+ service_port: 80,
34+ service_options: [mode http, acl foo path_beg -i /foo, use_backend foo if foo],
35+ servers: [[my_web_app_1, $host, $port, option httpchk GET / HTTP/1.0],
36+ [... optionally more servers here ...]]
37+ backends:
38+ - { backend_name: foo,
39+ servers: [[my_web_app2, $host, $port2, option httpchk GET / HTTP/1.0],
40+ [... optionally more servers here ...]]}}
41+
42
43 ## Website Relation
44
45
46=== modified file 'hooks/hooks.py'
47--- hooks/hooks.py 2015-02-19 17:05:11 +0000
48+++ hooks/hooks.py 2015-03-18 20:28:50 +0000
49@@ -62,6 +62,7 @@
50 ]
51
52 frontend_only_options = [
53+ "acl",
54 "backlog",
55 "bind",
56 "capture cookie",
57@@ -69,6 +70,7 @@
58 "capture response header",
59 "clitimeout",
60 "default_backend",
61+ "http-request",
62 "maxconn",
63 "monitor fail",
64 "monitor-net",
65@@ -84,6 +86,7 @@
66 "option socket-stats",
67 "option tcp-smart-accept",
68 "rate-limit sessions",
69+ "redirect",
70 "tcp-request content accept",
71 "tcp-request content reject",
72 "tcp-request inspect-delay",
73@@ -303,11 +306,14 @@
74 # service_ip: IP address to listen for connections
75 # service_port: Port to listen for connections
76 # service_options: Comma separated list of options
77-# server_entries: List of tuples
78+# server_entries: List of tuples
79 # server_name
80 # server_ip
81 # server_port
82 # server_options
83+# backends: List of dicts
84+# backend_name: backend name,
85+# servers: list of tuples as in server_entries
86 # errorfiles: List of dicts
87 # http_status: status to handle
88 # content: base 64 content for HAProxy to
89@@ -318,7 +324,7 @@
90 def create_listen_stanza(service_name=None, service_ip=None,
91 service_port=None, service_options=None,
92 server_entries=None, service_errorfiles=None,
93- service_crts=None):
94+ service_crts=None, service_backends=None):
95 if service_name is None or service_ip is None or service_port is None:
96 return None
97 fe_options = []
98@@ -359,17 +365,43 @@
99 service_config.append(" default_backend %s" % (service_name,))
100 service_config.extend(" %s" % service_option.strip()
101 for service_option in fe_options)
102- service_config.append("")
103- service_config.append("backend %s" % (service_name,))
104- service_config.extend(" %s" % service_option.strip()
105- for service_option in be_options)
106+
107+ # For now errorfiles are common for all backends, in the future we
108+ # might offer support for per-backend error files.
109+ backend_errorfiles = [] # List of (status, path) tuples
110 if service_errorfiles is not None:
111 for errorfile in service_errorfiles:
112 path = os.path.join(default_haproxy_lib_dir,
113 "service_%s" % service_name,
114 "%s.http" % errorfile["http_status"])
115- service_config.append(
116- " errorfile %s %s" % (errorfile["http_status"], path))
117+ backend_errorfiles.append((errorfile["http_status"], path))
118+
119+ # Default backend
120+ _append_backend(
121+ service_config, service_name, be_options, backend_errorfiles,
122+ server_entries)
123+
124+ # Extra backends
125+ if service_backends is not None:
126+ for service_backend in service_backends:
127+ _append_backend(
128+ service_config, service_backend["backend_name"],
129+ be_options, backend_errorfiles, service_backend["servers"])
130+
131+ return '\n'.join(service_config)
132+
133+
134+def _append_backend(service_config, name, options, errorfiles, server_entries):
135+ """Append a new backend stanza to the given service_config.
136+
137+ A backend stanza consists in a 'backend <name>' line followed by option
138+ lines, errorfile lines and server line.
139+ """
140+ service_config.append("")
141+ service_config.append("backend %s" % (name,))
142+ service_config.extend(" %s" % option.strip() for option in options)
143+ for status, path in errorfiles:
144+ service_config.append(" errorfile %s %s" % (status, path))
145 if isinstance(server_entries, (list, tuple)):
146 for i, (server_name, server_ip, server_port,
147 server_options) in enumerate(server_entries):
148@@ -382,7 +414,6 @@
149 server_line += " " + " ".join(server_options)
150 server_line = server_line.format(i=i)
151 service_config.append(server_line)
152- return '\n'.join(service_config)
153
154
155 # -----------------------------------------------------------------------------
156@@ -403,7 +434,7 @@
157 monitoring_config.append("mode http")
158 monitoring_config.append("acl allowed_cidr src %s" %
159 config_data['monitoring_allowed_cidr'])
160- monitoring_config.append("block unless allowed_cidr")
161+ monitoring_config.append("http-request deny unless allowed_cidr")
162 monitoring_config.append("stats enable")
163 monitoring_config.append("stats uri /")
164 monitoring_config.append("stats realm Haproxy\ Statistics")
165@@ -475,11 +506,20 @@
166 service = new_service.copy()
167 service.update(old_service)
168 if "servers" in service:
169+ # Merge all 'servers' entries of the default backend
170 servers = service["servers"]
171 if "servers" in new_service:
172 servers.extend(new_service["servers"])
173 servers.sort()
174 service["servers"] = list(x for x, _ in groupby(servers))
175+ if "backends" in service and "backends" in new_service:
176+ # Merge all 'servers' entries of the additional backends
177+ for i, backend in enumerate(service["backends"]):
178+ servers = backend["servers"]
179+ servers.extend(new_service["backends"][i]["servers"])
180+ servers.sort()
181+ service["backends"][i]["servers"] = list(
182+ x for x, _ in groupby(servers))
183 return service
184
185
186@@ -688,6 +728,7 @@
187 log("Service: %s" % service_key)
188 service_name = service_config["service_name"]
189 server_entries = service_config.get('servers')
190+ backends = service_config.get('backends', [])
191
192 errorfiles = service_config.get('errorfiles', [])
193 for errorfile in errorfiles:
194@@ -718,7 +759,7 @@
195 service_config['service_host'],
196 service_config['service_port'],
197 service_config['service_options'],
198- server_entries, errorfiles, crts))
199+ server_entries, errorfiles, crts, backends))
200
201
202 def get_service_lib_path(service_name):
203
204=== modified file 'hooks/tests/test_helpers.py'
205--- hooks/tests/test_helpers.py 2015-02-09 11:39:40 +0000
206+++ hooks/tests/test_helpers.py 2015-03-18 20:28:50 +0000
207@@ -478,6 +478,38 @@
208
209 self.assertEqual(expected, result)
210
211+ @patch.dict(os.environ, {"JUJU_UNIT_NAME": "haproxy/2"})
212+ def test_creates_a_listen_stanza_with_backends(self):
213+ service_name = 'foo'
214+ service_ip = '1.2.3.4'
215+ service_port = 80
216+ server_entries = [
217+ ('name-1', 'ip-1', 'port-1', ('foo1', 'bar1')),
218+ ]
219+ service_backends = [
220+ {"backend_name": "foo-bar",
221+ "servers": [
222+ ('bar-name-1', 'bar-ip-1', 'bar-port-1', ('bar2', 'bar3'))
223+ ]}
224+ ]
225+ result = hooks.create_listen_stanza(
226+ service_name, service_ip, service_port,
227+ server_entries=server_entries, service_backends=service_backends)
228+
229+ expected = '\n'.join((
230+ 'frontend haproxy-2-80',
231+ ' bind 1.2.3.4:80',
232+ ' default_backend foo',
233+ '',
234+ 'backend foo',
235+ ' server name-1 ip-1:port-1 foo1 bar1',
236+ '',
237+ 'backend foo-bar',
238+ ' server bar-name-1 bar-ip-1:bar-port-1 bar2 bar3',
239+ ))
240+
241+ self.assertEqual(expected, result)
242+
243 def test_doesnt_create_listen_stanza_if_args_not_provided(self):
244 self.assertIsNone(hooks.create_listen_stanza())
245
246@@ -504,7 +536,7 @@
247 'some-service', '0.0.0.0', 1234, [
248 'mode http',
249 'acl allowed_cidr src some-cidr',
250- 'block unless allowed_cidr',
251+ 'http-request deny unless allowed_cidr',
252 'stats enable',
253 'stats uri /',
254 'stats realm Haproxy\\ Statistics',
255@@ -551,7 +583,7 @@
256 'some-service', '0.0.0.0', 1234, [
257 'mode http',
258 'acl allowed_cidr src some-cidr',
259- 'block unless allowed_cidr',
260+ 'http-request deny unless allowed_cidr',
261 'stats enable',
262 'stats uri /',
263 'stats realm Haproxy\\ Statistics',
264@@ -585,7 +617,7 @@
265 'some-service', '0.0.0.0', 1234, [
266 'mode http',
267 'acl allowed_cidr src some-cidr',
268- 'block unless allowed_cidr',
269+ 'http-request deny unless allowed_cidr',
270 'stats enable',
271 'stats uri /',
272 'stats realm Haproxy\\ Statistics',
273
274=== modified file 'hooks/tests/test_peer_hooks.py'
275--- hooks/tests/test_peer_hooks.py 2015-02-19 17:05:11 +0000
276+++ hooks/tests/test_peer_hooks.py 2015-03-18 20:28:50 +0000
277@@ -197,7 +197,7 @@
278
279 create_listen_stanza.assert_called_with(
280 'bar', 'some-host', 'some-port', 'some-options',
281- (1, 2), [], [])
282+ (1, 2), [], [], [])
283 mock_open.assert_called_with(
284 '/var/run/haproxy/bar.service', 'w')
285 mock_file.write.assert_called_with('some content')
286
287=== modified file 'hooks/tests/test_reverseproxy_hooks.py'
288--- hooks/tests/test_reverseproxy_hooks.py 2014-01-21 15:46:29 +0000
289+++ hooks/tests/test_reverseproxy_hooks.py 2015-03-18 20:28:50 +0000
290@@ -393,6 +393,72 @@
291 self.assertEqual(expected, hooks.create_services())
292 self.write_service_config.assert_called_with(expected)
293
294+ def test_with_multiple_units_and_backends_in_relation(self):
295+ """
296+ Have multiple units specifying "services" in the relation
297+ using the "backends" option. Make sure data is created correctly
298+ with create_services()
299+ """
300+ self.get_config_services.return_value = {
301+ None: {
302+ "service_name": "service",
303+ },
304+ }
305+ self.relations_of_type.return_value = [
306+ {"port": 4242,
307+ "private-address": "1.2.3.4",
308+ "__unit__": "foo/0",
309+ "services": yaml.safe_dump([{
310+ "service_name": "service",
311+ "servers": [('foo-0', '1.2.3.4',
312+ 4242, ["maxconn 4"])],
313+ "backends": [
314+ {"backend_name": "foo-bar",
315+ "servers": [('foo-bar-0', '2.2.2.2',
316+ 2222, ["maxconn 4"])],
317+ },
318+ ]
319+ }])
320+ },
321+ {"port": 4242,
322+ "private-address": "1.2.3.5",
323+ "__unit__": "foo/1",
324+ "services": yaml.safe_dump([{
325+ "service_name": "service",
326+ "servers": [('foo-0', '1.2.3.5',
327+ 4242, ["maxconn 4"])],
328+ "backends": [
329+ {"backend_name": "foo-bar",
330+ "servers": [('foo-bar-1', '2.2.2.3',
331+ 3333, ["maxconn 4"])],
332+ },
333+ ]
334+ }])
335+ },
336+ ]
337+
338+ expected = {
339+ 'service': {
340+ 'service_name': 'service',
341+ 'service_host': '0.0.0.0',
342+ 'service_port': 10002,
343+ 'servers': [
344+ ['foo-0', '1.2.3.4', 4242, ["maxconn 4"]],
345+ ['foo-0', '1.2.3.5', 4242, ["maxconn 4"]]
346+ ],
347+ 'backends': [
348+ {"backend_name": "foo-bar",
349+ "servers": [
350+ ['foo-bar-0', '2.2.2.2', 2222, ["maxconn 4"]],
351+ ['foo-bar-1', '2.2.2.3', 3333, ["maxconn 4"]],
352+ ],
353+ },
354+ ]
355+ },
356+ }
357+ self.assertEqual(expected, hooks.create_services())
358+ self.write_service_config.assert_called_with(expected)
359+
360 def test_merge_service(self):
361 """ Make sure merge_services maintains "server" entries. """
362 s1 = {'service_name': 'f', 'servers': [['f', '4', 4, ['maxconn 4']]]}
363
364=== modified file 'tests/10_deploy_test.py'
365--- tests/10_deploy_test.py 2015-02-19 17:05:11 +0000
366+++ tests/10_deploy_test.py 2015-03-18 20:28:50 +0000
367@@ -12,7 +12,7 @@
368 d = amulet.Deployment(series='trusty')
369 # Add the haproxy charm to the deployment.
370 d.add('haproxy')
371-d.add('apache2')
372+d.add('apache2', units=2)
373
374 # Get the directory this way to load the file when CWD is different.
375 path = os.path.abspath(os.path.dirname(__file__))
376@@ -83,6 +83,7 @@
377 'configuration file.' % apache_private
378 amulet.raise_status(amulet.FAIL, msg=message)
379
380+# Test SSL termination
381 d.configure('haproxy', {
382 'source': 'backports',
383 'ssl_cert': 'SELFSIGNED',
384@@ -116,6 +117,7 @@
385 page.raise_for_status()
386 page = requests.get(secure_url, verify=False)
387 page.raise_for_status()
388+ success = True
389 except requests.exceptions.ConnectionError:
390 if i == retries - 1:
391 # This was the last one, let's fail
392@@ -126,5 +128,51 @@
393
394 print('Successfully got the Apache2 web page through haproxy SSL termination.')
395
396+apache_unit2 = d.sentry.unit['apache2/1']
397+apache_private2 = apache_unit2.run("unit-get private-address")[0]
398+
399+# Create a file on the second apache unit's www directory.
400+apache_unit2.run("echo foo > /var/www/html/foo")
401+
402+d.configure('haproxy', {
403+ 'services': yaml.safe_dump([
404+ {'service_name': 'apache',
405+ 'service_host': '0.0.0.0',
406+ 'service_port': 80,
407+ 'service_options': [
408+ 'mode http', 'balance leastconn', 'option httpchk GET / HTTP/1.0',
409+ 'acl foo path_beg -i /foo', 'use_backend foo if foo',
410+ ],
411+ 'servers': [
412+ ['apache', apache_private, 80, 'maxconn 50']],
413+ 'backends': [
414+ {'backend_name': 'foo',
415+ 'servers': [
416+ ['apache2', apache_private2, 80, 'maxconn 50']]}
417+ ]}])
418+})
419+
420+# Let's exercise our URL-based routing by trying to fetch a URL that will
421+# only work for the second apache unit (which is configured as server
422+# of the extra backend).
423+url = 'http://%s/foo' % haproxy_address
424+
425+# We need a retry loop here, since there's no way to tell when the new
426+# configuration is in place.
427+retries = 10
428+for i in range(retries):
429+ try:
430+ page = requests.get(url)
431+ page.raise_for_status()
432+ except:
433+ if i == retries - 1:
434+ # This was the last one, let's fail
435+ raise
436+ time.sleep(6)
437+ else:
438+ break
439+
440+print('Successfully got the /foo URL from the second Apache unit.')
441+
442 # Send a message that the tests are complete.
443 print('The haproxy tests are complete.')

Subscribers

People subscribed via source and target branches

to all changes: