Launchpad itself

Merge lp:~cjwatson/launchpad/wsgi-ppa-auth into lp:launchpad

  1. wsgi-ppa-auth
  2. Merge into devel
Proposed by Colin Watson
Status: Rejected
Rejected by: Colin Watson
Proposed branch: lp:~cjwatson/launchpad/wsgi-ppa-auth
Merge into: lp:launchpad
Prerequisite: lp:~cjwatson/launchpad/virtualenv-pip
Diff against target: 1113 lines (+768/-71)
20 files modified
Makefile (+1/-0)
configs/development/local-launchpad-apache (+30/-3)
lib/lp/services/config/schema-lazr.conf (+5/-0)
lib/lp/services/memcache/client.py (+13/-51)
lib/lp/services/memcache/testing.py (+25/-4)
lib/lp/services/memcache/timeline.py (+57/-0)
lib/lp/soyuz/doc/archiveauthtoken.txt (+24/-1)
lib/lp/soyuz/interfaces/archiveapi.py (+46/-0)
lib/lp/soyuz/interfaces/archiveauthtoken.py (+16/-6)
lib/lp/soyuz/model/archiveauthtoken.py (+25/-5)
lib/lp/soyuz/wsgi/archiveauth.py (+83/-0)
lib/lp/soyuz/wsgi/tests/test_archiveauth.py (+151/-0)
lib/lp/soyuz/xmlrpc/archive.py (+62/-0)
lib/lp/soyuz/xmlrpc/tests/test_archive.py (+124/-0)
lib/lp/systemhomes.py (+7/-0)
lib/lp/xmlrpc/application.py (+7/-1)
lib/lp/xmlrpc/configure.zcml (+13/-0)
lib/lp/xmlrpc/interfaces.py (+2/-0)
scripts/wsgi-archive-auth.py (+71/-0)
utilities/rocketfuel-setup (+6/-0)
To merge this branch: bzr merge lp:~cjwatson/launchpad/wsgi-ppa-auth
Reviewer Review Type Date Requested Status
Launchpad code reviewers Pending
Review via email: mp+332125@code.launchpad.net

Commit message

Add a WSGI authenticator for private PPAs.

Description of the change

I got sufficiently annoyed in the process of fixing bug 1722209 to see how hard it would be to fix it properly, since the existing htpasswd scheme has been a thorn in our side for a while now. The answer appears to be "a bit, but not very".

This will let us remove our reliance on htpasswd files, and probably eventually make it easier to do things like issue time-limited tokens (or even macaroons?) to builders.

There are obviously various deployment issues to sort out, and generate-ppa-htaccess still deals with things like deactivation and cancellation emails which we'll need to move elsewhere; it may be possible to just inline the relevant checks into the DB queries in ArchiveAuthTokenSet, since we're always looking at a single subscriber or named token name and so the result set sizes are very small.

The way that mod_wsgi loads the entry point is peculiar. I tried to handle it with the existing buildout-based build system, but that ended up being too fiddly, so I decided to just depend on my virtualenv/pip conversion branch. I considered separating the client code off entirely from Launchpad, but this turns out to be light enough in terms of memory use and startup time that I don't think it's worth the effort at the moment.

To post a comment you must log in.
lp:~cjwatson/launchpad/wsgi-ppa-auth updated
18475. By Colin Watson

Beef up MemcacheFixture a bit to support expiry times and to reject non-str keys.

18476. By Colin Watson

Separate out request-timeline handling so that memcache_client_factory can be used to create a basic client with minimal dependencies.

18477. By Colin Watson

Use memcached instead of timedcache, storing hashed passwords.

18478. By Colin Watson

Merge virtualenv-pip.

Revision history for this message
Colin Watson (cjwatson) wrote :

I've rewritten part of this using memcached rather than timedcache, so the cost of using multiple workers here should now be negligible.

Revision history for this message
Colin Watson (cjwatson) wrote :

Superseded by https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/399617.

Unmerged revisions

18478. By Colin Watson

Merge virtualenv-pip.

18477. By Colin Watson

Use memcached instead of timedcache, storing hashed passwords.

18476. By Colin Watson

Separate out request-timeline handling so that memcache_client_factory can be used to create a basic client with minimal dependencies.

18475. By Colin Watson

Beef up MemcacheFixture a bit to support expiry times and to reject non-str keys.

18474. By Colin Watson

Add a WSGI authenticator for private PPAs.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'Makefile'
2--- Makefile 2017-11-19 12:35:56 +0000
3+++ Makefile 2017-11-19 12:35:56 +0000
4@@ -463,6 +463,7 @@
5 base=local-launchpad; \
6 fi; \
7 sed -e 's,%BRANCH_REWRITE%,$(shell pwd)/scripts/branch-rewrite.py,' \
8+ -e 's,%WSGI_ARCHIVE_AUTH%,$(shell pwd)/scripts/wsgi-archive-auth.py,' \
9 -e 's,%LISTEN_ADDRESS%,$(LISTEN_ADDRESS),' \
10 configs/development/local-launchpad-apache > \
11 /etc/apache2/sites-available/$$base
12
13=== modified file 'configs/development/local-launchpad-apache'
14--- configs/development/local-launchpad-apache 2017-01-10 17:26:29 +0000
15+++ configs/development/local-launchpad-apache 2017-11-19 12:35:56 +0000
16@@ -134,7 +134,6 @@
17
18 <VirtualHost %LISTEN_ADDRESS%:80>
19 ServerName ppa.launchpad.dev
20- ServerAlias private-ppa.launchpad.dev
21 LogLevel debug
22
23 DocumentRoot /var/tmp/ppa
24@@ -147,8 +146,36 @@
25 Deny from all
26 Allow from 127.0.0.0/255.0.0.0
27 </IfVersion>
28- AllowOverride AuthConfig
29- Options Indexes
30+ AllowOverride None
31+ Options Indexes
32+ </Directory>
33+</VirtualHost>
34+
35+<VirtualHost %LISTEN_ADDRESS%:80>
36+ ServerName private-ppa.launchpad.dev
37+ LogLevel debug
38+
39+ DocumentRoot /var/tmp/ppa
40+ <Directory /var/tmp/ppa/>
41+ <IfVersion >= 2.4>
42+ <RequireAll>
43+ Require ip 127.0.0.0/255.0.0.0
44+ Require valid-user
45+ </RequireAll>
46+ </IfVersion>
47+ <IfVersion < 2.4>
48+ Order Deny,Allow
49+ Deny from all
50+ Allow from 127.0.0.0/255.0.0.0
51+ Require valid-user
52+ Satisfy All
53+ </IfVersion>
54+ AllowOverride None
55+ Options Indexes
56+ AuthType Basic
57+ AuthName "Token Required"
58+ AuthBasicProvider wsgi
59+ WSGIAuthUserScript %WSGI_ARCHIVE_AUTH% application-group=lp
60 </Directory>
61 </VirtualHost>
62
63
64=== modified file 'lib/lp/services/config/schema-lazr.conf'
65--- lib/lp/services/config/schema-lazr.conf 2017-09-07 13:25:13 +0000
66+++ lib/lp/services/config/schema-lazr.conf 2017-11-19 12:35:56 +0000
67@@ -1460,6 +1460,11 @@
68 # datatype: boolean
69 require_signing_keys: false
70
71+# The URL to the internal archive API endpoint. This should implement
72+# IArchiveAPI.
73+# datatype: string
74+archive_api_endpoint: http://xmlrpc-private.launchpad.dev:8087/archive
75+
76
77 [ppa_apache_log_parser]
78 logs_root: /srv/ppa.launchpad.net-logs
79
80=== modified file 'lib/lp/services/memcache/client.py'
81--- lib/lp/services/memcache/client.py 2017-09-27 10:59:13 +0000
82+++ lib/lp/services/memcache/client.py 2017-11-19 12:35:56 +0000
83@@ -4,64 +4,26 @@
84 """Launchpad Memcache client."""
85
86 __metaclass__ = type
87-__all__ = []
88+__all__ = [
89+ 'memcache_client_factory',
90+ ]
91
92-import logging
93 import re
94
95-from lazr.restful.utils import get_current_browser_request
96-import memcache
97-
98-from lp.services import features
99 from lp.services.config import config
100-from lp.services.timeline.requesttimeline import get_request_timeline
101-
102-
103-def memcache_client_factory():
104+
105+
106+def memcache_client_factory(timeline=True):
107 """Return a memcache.Client for Launchpad."""
108 servers = [
109 (host, int(weight)) for host, weight in re.findall(
110 r'\((.+?),(\d+)\)', config.memcache.servers)]
111 assert len(servers) > 0, "Invalid memcached server list %r" % (
112 config.memcache.servers,)
113- return TimelineRecordingClient(servers)
114-
115-
116-class TimelineRecordingClient(memcache.Client):
117-
118- def __get_timeline_action(self, suffix, key):
119- request = get_current_browser_request()
120- timeline = get_request_timeline(request)
121- return timeline.start("memcache-%s" % suffix, key)
122-
123- @property
124- def _enabled(self):
125- configured_value = features.getFeatureFlag('memcache')
126- if configured_value is None:
127- return True
128- else:
129- return configured_value
130-
131- def get(self, key):
132- if not self._enabled:
133- return None
134- action = self.__get_timeline_action("get", key)
135- try:
136- return memcache.Client.get(self, key)
137- finally:
138- action.finish()
139-
140- def set(self, key, value, time=0, min_compress_len=0):
141- if not self._enabled:
142- return None
143- action = self.__get_timeline_action("set", key)
144- try:
145- success = memcache.Client.set(self, key, value, time=time,
146- min_compress_len=min_compress_len)
147- if success:
148- logging.debug("Memcache set succeeded for %s", key)
149- else:
150- logging.warn("Memcache set failed for %s", key)
151- return success
152- finally:
153- action.finish()
154+ if timeline:
155+ from lp.services.memcache.timeline import TimelineRecordingClient
156+ client_factory = TimelineRecordingClient
157+ else:
158+ import memcache
159+ client_factory = memcache.Client
160+ return client_factory(servers)
161
162=== modified file 'lib/lp/services/memcache/testing.py'
163--- lib/lp/services/memcache/testing.py 2016-09-07 03:43:36 +0000
164+++ lib/lp/services/memcache/testing.py 2017-11-19 12:35:56 +0000
165@@ -1,4 +1,4 @@
166-# Copyright 2016 Canonical Ltd. This software is licensed under the
167+# Copyright 2016-2017 Canonical Ltd. This software is licensed under the
168 # GNU Affero General Public License version 3 (see the file LICENSE).
169
170 __metaclass__ = type
171@@ -6,6 +6,8 @@
172 'MemcacheFixture',
173 ]
174
175+import time as _time
176+
177 import fixtures
178
179 from lp.services.memcache.interfaces import IMemcacheClient
180@@ -22,14 +24,33 @@
181 super(MemcacheFixture, self).setUp()
182 self.useFixture(ZopeUtilityFixture(self, IMemcacheClient))
183
184+ def check_key(self, key):
185+ # A subset of the checks performed by the real memcache library;
186+ # this one is particularly easy to get wrong in Launchpad code.
187+ if not isinstance(key, str):
188+ raise TypeError("Key must be str.")
189+
190 def get(self, key):
191- return self._cache.get(key)
192+ self.check_key(key)
193+ value, expiry_time = self._cache.get(key, (None, None))
194+ if expiry_time and _time.time() >= expiry_time:
195+ self.delete(key)
196+ return None
197+ else:
198+ return value
199
200- def set(self, key, val):
201- self._cache[key] = val
202+ def set(self, key, val, time=0):
203+ self.check_key(key)
204+ # memcached accepts either delta-seconds from the current time or
205+ # absolute epoch-seconds, and tells them apart using a magic
206+ # threshold. See memcached/memcached.c:realtime.
207+ if time and time <= 60 * 60 * 24 * 30:
208+ time = _time.time() + time
209+ self._cache[key] = (val, time)
210 return 1
211
212 def delete(self, key):
213+ self.check_key(key)
214 self._cache.pop(key, None)
215 return 1
216
217
218=== added file 'lib/lp/services/memcache/timeline.py'
219--- lib/lp/services/memcache/timeline.py 1970-01-01 00:00:00 +0000
220+++ lib/lp/services/memcache/timeline.py 2017-11-19 12:35:56 +0000
221@@ -0,0 +1,57 @@
222+# Copyright 2017 Canonical Ltd. This software is licensed under the
223+# GNU Affero General Public License version 3 (see the file LICENSE).
224+
225+"""Timeline-friendly Launchpad Memcache client."""
226+
227+__metaclass__ = type
228+__all__ = [
229+ 'TimelineRecordingClient',
230+ ]
231+
232+import logging
233+
234+from lazr.restful.utils import get_current_browser_request
235+import memcache
236+
237+from lp.services import features
238+from lp.services.timeline.requesttimeline import get_request_timeline
239+
240+
241+class TimelineRecordingClient(memcache.Client):
242+
243+ def __get_timeline_action(self, suffix, key):
244+ request = get_current_browser_request()
245+ timeline = get_request_timeline(request)
246+ return timeline.start("memcache-%s" % suffix, key)
247+
248+ @property
249+ def _enabled(self):
250+ configured_value = features.getFeatureFlag('memcache')
251+ if configured_value is None:
252+ return True
253+ else:
254+ return configured_value
255+
256+ def get(self, key):
257+ if not self._enabled:
258+ return None
259+ action = self.__get_timeline_action("get", key)
260+ try:
261+ return memcache.Client.get(self, key)
262+ finally:
263+ action.finish()
264+
265+ def set(self, key, value, time=0, min_compress_len=0):
266+ if not self._enabled:
267+ return None
268+ action = self.__get_timeline_action("set", key)
269+ try:
270+ success = memcache.Client.set(self, key, value, time=time,
271+ min_compress_len=min_compress_len)
272+ if success:
273+ logging.debug("Memcache set succeeded for %s", key)
274+ else:
275+ logging.warn("Memcache set failed for %s", key)
276+ return success
277+ finally:
278+ action.finish()
279
280=== modified file 'lib/lp/soyuz/doc/archiveauthtoken.txt'
281--- lib/lp/soyuz/doc/archiveauthtoken.txt 2012-04-10 14:01:17 +0000
282+++ lib/lp/soyuz/doc/archiveauthtoken.txt 2017-11-19 12:35:56 +0000
283@@ -139,12 +139,35 @@
284 ... print token.person.name
285 bradsmith
286
287-Tokens can also be retreived by archive and person:
288+Tokens can also be retrieved by archive and person:
289
290 >>> print token_set.getActiveTokenForArchiveAndPerson(
291 ... new_token.archive, new_token.person).token
292 testtoken
293
294+Or by archive and person name:
295+
296+ >>> print token_set.getActiveTokenForArchiveAndPersonName(
297+ ... new_token.archive, "bradsmith").token
298+ testtoken
299+
300+Tokens are only returned if they match a current subscription:
301+
302+ >>> from zope.security.proxy import removeSecurityProxy
303+ >>> from lp.soyuz.enums import ArchiveSubscriberStatus
304+ >>> removeSecurityProxy(subscription_to_joe_private_ppa).status = (
305+ ... ArchiveSubscriberStatus.EXPIRED)
306+
307+ >>> print token_set.getActiveTokenForArchiveAndPerson(
308+ ... new_token.archive, new_token.person)
309+ None
310+ >>> print token_set.getActiveTokenForArchiveAndPersonName(
311+ ... new_token.archive, "bradsmith")
312+ None
313+
314+ >>> removeSecurityProxy(subscription_to_joe_private_ppa).status = (
315+ ... ArchiveSubscriberStatus.CURRENT)
316+
317
318 == Amending Tokens ==
319
320
321=== added file 'lib/lp/soyuz/interfaces/archiveapi.py'
322--- lib/lp/soyuz/interfaces/archiveapi.py 1970-01-01 00:00:00 +0000
323+++ lib/lp/soyuz/interfaces/archiveapi.py 2017-11-19 12:35:56 +0000
324@@ -0,0 +1,46 @@
325+# Copyright 2017 Canonical Ltd. This software is licensed under the
326+# GNU Affero General Public License version 3 (see the file LICENSE).
327+
328+"""Interfaces for internal archive APIs."""
329+
330+from __future__ import absolute_import, print_function, unicode_literals
331+
332+__metaclass__ = type
333+__all__ = [
334+ 'IArchiveAPI',
335+ 'IArchiveApplication',
336+ ]
337+
338+from zope.interface import Interface
339+
340+from lp.services.webapp.interfaces import ILaunchpadApplication
341+
342+
343+class IArchiveApplication(ILaunchpadApplication):
344+ """Archive application root."""
345+
346+
347+class IArchiveAPI(Interface):
348+ """The Soyuz archive XML-RPC interface to Launchpad.
349+
350+ Published at "archive" on the private XML-RPC server.
351+
352+ PPA frontends use this to check archive authorization tokens.
353+ """
354+
355+ def checkArchiveAuthToken(archive_reference, username, password):
356+ """Check an archive authorization token.
357+
358+ :param archive_reference: The reference form of the archive to check.
359+ :param username: The username sent using HTTP Basic Authentication;
360+ this should either be a `Person.name` or "+" followed by the
361+ name of a named authorization token.
362+ :param password: The password sent using HTTP Basic Authentication;
363+ this should be a corresponding `ArchiveAuthToken.token`.
364+
365+ :returns: A `NotFound` fault if `archive_reference` does not
366+ identify an archive or the username does not identify a valid
367+ token for this archive; an `Unauthorized` fault if the password
368+ is not equal to the selected token for this archive; otherwise
369+ None.
370+ """
371
372=== modified file 'lib/lp/soyuz/interfaces/archiveauthtoken.py'
373--- lib/lp/soyuz/interfaces/archiveauthtoken.py 2016-07-14 16:06:01 +0000
374+++ lib/lp/soyuz/interfaces/archiveauthtoken.py 2017-11-19 12:35:56 +0000
375@@ -96,23 +96,33 @@
376 :return: An object conforming to `IArchiveAuthToken`.
377 """
378
379- def getByArchive(archive):
380+ def getByArchive(archive, valid=False):
381 """Retrieve all the tokens for an archive.
382
383 :param archive: The context archive.
384+ :param valid: If True, only return valid tokens.
385 :return: A result set containing `IArchiveAuthToken`s.
386 """
387
388 def getActiveTokenForArchiveAndPerson(archive, person):
389- """Retrieve an active token for the given archive and person.
390+ """Retrieve a valid active token for the given archive and person.
391
392 :param archive: The archive to which the token corresponds.
393 :param person: The person to which the token corresponds.
394 :return: An `IArchiveAuthToken` or None.
395 """
396
397+ def getActiveTokenForArchiveAndPersonName(archive, person_name):
398+ """Retrieve a valid active token for the given archive and person name.
399+
400+ :param archive: The archive to which the token corresponds.
401+ :param person_name: The name of the person to which the token
402+ corresponds.
403+ :return: An `IArchiveAuthToken` or None.
404+ """
405+
406 def getActiveNamedTokenForArchive(archive, name):
407- """Retrieve an active named token for the given archive and name.
408+ """Retrieve a valid active named token for the given archive and name.
409
410 :param archive: The archive to which the token corresponds.
411 :param name: The name of a named authorization token.
412@@ -120,9 +130,9 @@
413 """
414
415 def getActiveNamedTokensForArchive(archive, names=None):
416- """Retrieve a subset of active named tokens for the given archive if
417- `names` is specified, or all active named tokens for the archive if
418- `names` is null.
419+ """Retrieve a subset of valid active named tokens for the given
420+ archive if `names` is specified, or all valid active named tokens
421+ for the archive if `names` is null.
422
423 :param archive: The archive to which the tokens correspond.
424 :param names: An optional list of token names.
425
426=== modified file 'lib/lp/soyuz/model/archiveauthtoken.py'
427--- lib/lp/soyuz/model/archiveauthtoken.py 2016-07-14 16:06:01 +0000
428+++ lib/lp/soyuz/model/archiveauthtoken.py 2017-11-19 12:35:56 +0000
429@@ -21,8 +21,10 @@
430 from storm.store import Store
431 from zope.interface import implementer
432
433+from lp.registry.model.teammembership import TeamParticipation
434 from lp.services.database.constants import UTC_NOW
435 from lp.services.database.interfaces import IStore
436+from lp.soyuz.enums import ArchiveSubscriberStatus
437 from lp.soyuz.interfaces.archiveauthtoken import (
438 IArchiveAuthToken,
439 IArchiveAuthTokenSet,
440@@ -85,19 +87,37 @@
441 return IStore(ArchiveAuthToken).find(
442 ArchiveAuthToken, ArchiveAuthToken.token == token).one()
443
444- def getByArchive(self, archive):
445+ def getByArchive(self, archive, valid=False):
446 """See `IArchiveAuthTokenSet`."""
447+ # Circular import.
448+ from lp.soyuz.model.archivesubscriber import ArchiveSubscriber
449 store = Store.of(archive)
450- return store.find(
451- ArchiveAuthToken,
452+ clauses = [
453 ArchiveAuthToken.archive == archive,
454- ArchiveAuthToken.date_deactivated == None)
455+ ArchiveAuthToken.date_deactivated == None,
456+ ]
457+ if valid:
458+ clauses.extend([
459+ ArchiveAuthToken.archive_id == ArchiveSubscriber.archive_id,
460+ ArchiveSubscriber.status == ArchiveSubscriberStatus.CURRENT,
461+ ArchiveSubscriber.subscriber_id == TeamParticipation.teamID,
462+ TeamParticipation.personID == ArchiveAuthToken.person_id,
463+ ])
464+ return store.find(ArchiveAuthToken, *clauses)
465
466 def getActiveTokenForArchiveAndPerson(self, archive, person):
467 """See `IArchiveAuthTokenSet`."""
468- return self.getByArchive(archive).find(
469+ return self.getByArchive(archive, valid=True).find(
470 ArchiveAuthToken.person == person).one()
471
472+ def getActiveTokenForArchiveAndPersonName(self, archive, person_name):
473+ """See `IArchiveAuthTokenSet`."""
474+ # Circular import.
475+ from lp.registry.model.person import Person
476+ return self.getByArchive(archive, valid=True).find(
477+ ArchiveAuthToken.person == Person.id,
478+ Person.name == person_name).one()
479+
480 def getActiveNamedTokenForArchive(self, archive, name):
481 """See `IArchiveAuthTokenSet`."""
482 return self.getByArchive(archive).find(
483
484=== added directory 'lib/lp/soyuz/wsgi'
485=== added file 'lib/lp/soyuz/wsgi/__init__.py'
486=== added file 'lib/lp/soyuz/wsgi/archiveauth.py'
487--- lib/lp/soyuz/wsgi/archiveauth.py 1970-01-01 00:00:00 +0000
488+++ lib/lp/soyuz/wsgi/archiveauth.py 2017-11-19 12:35:56 +0000
489@@ -0,0 +1,83 @@
490+# Copyright 2017 Canonical Ltd. This software is licensed under the
491+# GNU Affero General Public License version 3 (see the file LICENSE).
492+
493+"""WSGI archive authorisation provider.
494+
495+This is as lightweight as possible, as it runs on PPA frontends.
496+"""
497+
498+from __future__ import absolute_import, print_function, unicode_literals
499+
500+__metaclass__ = type
501+__all__ = [
502+ 'check_password',
503+ ]
504+
505+import crypt
506+from random import SystemRandom
507+import string
508+import time
509+try:
510+ from xmlrpc.client import (
511+ Fault,
512+ ServerProxy,
513+ )
514+except ImportError:
515+ from xmlrpclib import (
516+ Fault,
517+ ServerProxy,
518+ )
519+
520+from lp.services.config import config
521+from lp.services.memcache.client import memcache_client_factory
522+
523+
524+def _get_archive_reference(environ):
525+ # Reconstruct the relevant part of the URL. We don't care about where
526+ # we're installed.
527+ path = environ.get("SCRIPT_NAME") or "/"
528+ path_info = environ.get("PATH_INFO", "")
529+ path += (path_info if path else path_info[1:])
530+ # Extract the first three segments of the path, and rearrange them to
531+ # form an archive reference.
532+ path_parts = path.lstrip("/").split("/")
533+ if len(path_parts) >= 3:
534+ return "~%s/%s/%s" % (path_parts[0], path_parts[2], path_parts[1])
535+
536+
537+_sr = SystemRandom()
538+
539+
540+def _crypt_sha256(word):
541+ """crypt.crypt(word, crypt.METHOD_SHA256), backported from Python 3.5."""
542+ saltchars = string.ascii_letters + string.digits + './'
543+ salt = '$5$' + ''.join(_sr.choice(saltchars) for _ in range(16))
544+ return crypt.crypt(word, salt)
545+
546+
547+_memcache_client = memcache_client_factory(timeline=False)
548+
549+
550+def check_password(environ, user, password):
551+ archive_reference = _get_archive_reference(environ)
552+ if archive_reference is None:
553+ return None
554+ memcache_key = (
555+ "archive-auth:%s:%s" % (archive_reference, user)).encode("UTF-8")
556+ crypted_password = _memcache_client.get(memcache_key)
557+ if (crypted_password and
558+ crypt.crypt(password, crypted_password) == crypted_password):
559+ return True
560+ proxy = ServerProxy(config.personalpackagearchive.archive_api_endpoint)
561+ try:
562+ proxy.checkArchiveAuthToken(archive_reference, user, password)
563+ # Cache positive responses for a minute to reduce database load.
564+ _memcache_client.set(
565+ memcache_key, _crypt_sha256(password), time.time() + 60)
566+ return True
567+ except Fault as e:
568+ if e.faultCode == 410: # Unauthorized
569+ return False
570+ else:
571+ # Interpret any other fault as NotFound (320).
572+ return None
573
574=== added directory 'lib/lp/soyuz/wsgi/tests'
575=== added file 'lib/lp/soyuz/wsgi/tests/__init__.py'
576=== added file 'lib/lp/soyuz/wsgi/tests/test_archiveauth.py'
577--- lib/lp/soyuz/wsgi/tests/test_archiveauth.py 1970-01-01 00:00:00 +0000
578+++ lib/lp/soyuz/wsgi/tests/test_archiveauth.py 2017-11-19 12:35:56 +0000
579@@ -0,0 +1,151 @@
580+# Copyright 2017 Canonical Ltd. This software is licensed under the
581+# GNU Affero General Public License version 3 (see the file LICENSE).
582+
583+"""Tests for the WSGI archive authorisation provider."""
584+
585+from __future__ import absolute_import, print_function, unicode_literals
586+
587+__metaclass__ = type
588+
589+import crypt
590+import os.path
591+import subprocess
592+import time
593+
594+from fixtures import MonkeyPatch
595+from testtools.matchers import Is
596+import transaction
597+
598+from lp.services.config import config
599+from lp.services.memcache.testing import MemcacheFixture
600+from lp.soyuz.wsgi import archiveauth
601+from lp.testing import TestCaseWithFactory
602+from lp.testing.layers import ZopelessAppServerLayer
603+from lp.xmlrpc import faults
604+
605+
606+class TestWSGIArchiveAuth(TestCaseWithFactory):
607+
608+ layer = ZopelessAppServerLayer
609+
610+ def setUp(self):
611+ super(TestWSGIArchiveAuth, self).setUp()
612+ self.now = time.time()
613+ self.useFixture(MonkeyPatch("time.time", lambda: self.now))
614+ self.memcache_fixture = self.useFixture(MemcacheFixture())
615+ # The WSGI provider doesn't use Zope, so we can't rely on the
616+ # fixture substituting a Zope utility.
617+ self.useFixture(MonkeyPatch(
618+ "lp.soyuz.wsgi.archiveauth._memcache_client",
619+ self.memcache_fixture))
620+
621+ def test_get_archive_reference_short_url(self):
622+ self.assertIsNone(archiveauth._get_archive_reference(
623+ {"SCRIPT_NAME": "/foo"}))
624+
625+ def test_get_archive_reference_archive_base(self):
626+ self.assertEqual(
627+ "~user/ubuntu/ppa",
628+ archiveauth._get_archive_reference(
629+ {"SCRIPT_NAME": "/user/ppa/ubuntu"}))
630+
631+ def test_get_archive_reference_inside_archive(self):
632+ self.assertEqual(
633+ "~user/ubuntu/ppa",
634+ archiveauth._get_archive_reference(
635+ {"SCRIPT_NAME": "/user/ppa/ubuntu/dists"}))
636+
637+ def test_check_password_short_url(self):
638+ self.assertIsNone(archiveauth.check_password(
639+ {"SCRIPT_NAME": "/foo"}, "user", ""))
640+ self.assertEqual({}, self.memcache_fixture._cache)
641+
642+ def test_check_password_not_found(self):
643+ self.assertIsNone(archiveauth.check_password(
644+ {"SCRIPT_NAME": "/nonexistent/bad/unknown"}, "user", ""))
645+ self.assertEqual({}, self.memcache_fixture._cache)
646+
647+ def test_crypt_sha256(self):
648+ crypted_password = archiveauth._crypt_sha256("secret")
649+ self.assertEqual(
650+ crypted_password, crypt.crypt("secret", crypted_password))
651+
652+ def makeArchiveAndToken(self):
653+ archive = self.factory.makeArchive(private=True)
654+ archive_path = "/%s/%s/ubuntu" % (archive.owner.name, archive.name)
655+ subscriber = self.factory.makePerson()
656+ archive.newSubscription(subscriber, archive.owner)
657+ token = archive.newAuthToken(subscriber)
658+ transaction.commit()
659+ return archive, archive_path, subscriber.name, token.token
660+
661+ def test_check_password_unauthorized(self):
662+ _, archive_path, username, password = self.makeArchiveAndToken()
663+ # Test that this returns False, not merely something falsy (e.g.
664+ # None).
665+ self.assertThat(
666+ archiveauth.check_password(
667+ {"SCRIPT_NAME": archive_path}, username, password + "-bad"),
668+ Is(False))
669+ self.assertEqual({}, self.memcache_fixture._cache)
670+
671+ def test_check_password_success(self):
672+ archive, archive_path, username, password = self.makeArchiveAndToken()
673+ self.assertThat(
674+ archiveauth.check_password(
675+ {"SCRIPT_NAME": archive_path}, username, password),
676+ Is(True))
677+ crypted_password = self.memcache_fixture.get(
678+ ("archive-auth:%s:%s" % (archive.reference, username)).encode(
679+ "UTF-8"))
680+ self.assertEqual(
681+ crypted_password, crypt.crypt(password, crypted_password))
682+
683+ def test_check_password_considers_cache(self):
684+ class FakeProxy:
685+ def __init__(self, uri):
686+ pass
687+
688+ def checkArchiveAuthToken(self, archive_reference, username,
689+ password):
690+ raise faults.Unauthorized()
691+
692+ _, archive_path, username, password = self.makeArchiveAndToken()
693+ self.assertThat(
694+ archiveauth.check_password(
695+ {"SCRIPT_NAME": archive_path}, username, password),
696+ Is(True))
697+ self.useFixture(
698+ MonkeyPatch("lp.soyuz.wsgi.archiveauth.ServerProxy", FakeProxy))
699+ # A subsequent check honours the cache.
700+ self.assertThat(
701+ archiveauth.check_password(
702+ {"SCRIPT_NAME": archive_path}, username, password + "-bad"),
703+ Is(False))
704+ self.assertThat(
705+ archiveauth.check_password(
706+ {"SCRIPT_NAME": archive_path}, username, password),
707+ Is(True))
708+ # If we advance time far enough, then the cached result expires.
709+ self.now += 60
710+ self.assertThat(
711+ archiveauth.check_password(
712+ {"SCRIPT_NAME": archive_path}, username, password),
713+ Is(False))
714+
715+ def test_script(self):
716+ _, archive_path, username, password = self.makeArchiveAndToken()
717+ script_path = os.path.join(
718+ config.root, "scripts", "wsgi-archive-auth.py")
719+
720+ def check_via_script(archive_path, username, password):
721+ with open(os.devnull, "w") as devnull:
722+ return subprocess.call(
723+ [script_path, archive_path, username, password],
724+ stderr=devnull)
725+
726+ self.assertEqual(0, check_via_script(archive_path, username, password))
727+ self.assertEqual(
728+ 1, check_via_script(archive_path, username, password + "-bad"))
729+ self.assertEqual(
730+ 2, check_via_script("/nonexistent/bad/unknown", "user", ""))
731
732=== added file 'lib/lp/soyuz/xmlrpc/archive.py'
733--- lib/lp/soyuz/xmlrpc/archive.py 1970-01-01 00:00:00 +0000
734+++ lib/lp/soyuz/xmlrpc/archive.py 2017-11-19 12:35:56 +0000
735@@ -0,0 +1,62 @@
736+# Copyright 2017 Canonical Ltd. This software is licensed under the
737+# GNU Affero General Public License version 3 (see the file LICENSE).
738+
739+"""Implementations of the XML-RPC APIs for Soyuz archives."""
740+
741+from __future__ import absolute_import, print_function, unicode_literals
742+
743+__metaclass__ = type
744+__all__ = [
745+ 'ArchiveAPI',
746+ ]
747+
748+from zope.component import getUtility
749+from zope.interface import implementer
750+from zope.security.proxy import removeSecurityProxy
751+
752+from lp.soyuz.interfaces.archive import IArchiveSet
753+from lp.soyuz.interfaces.archiveapi import IArchiveAPI
754+from lp.soyuz.interfaces.archiveauthtoken import IArchiveAuthTokenSet
755+from lp.services.webapp import LaunchpadXMLRPCView
756+from lp.xmlrpc import faults
757+from lp.xmlrpc.helpers import return_fault
758+
759+
760+BUILDD_USER_NAME = "buildd"
761+
762+
763+@implementer(IArchiveAPI)
764+class ArchiveAPI(LaunchpadXMLRPCView):
765+ """See `IArchiveAPI`."""
766+
767+ @return_fault
768+ def _checkArchiveAuthToken(self, archive_reference, username, password):
769+ archive = getUtility(IArchiveSet).getByReference(archive_reference)
770+ if archive is None:
771+ raise faults.NotFound(
772+ message="No archive found for '%s'." % archive_reference)
773+ archive = removeSecurityProxy(archive)
774+ token_set = getUtility(IArchiveAuthTokenSet)
775+ if username == BUILDD_USER_NAME:
776+ secret = archive.buildd_secret
777+ else:
778+ if username.startswith("+"):
779+ token = token_set.getActiveNamedTokenForArchive(
780+ archive, username[1:])
781+ else:
782+ token = token_set.getActiveTokenForArchiveAndPersonName(
783+ archive, username)
784+ if token is None:
785+ raise faults.NotFound(
786+ message="No valid tokens for '%s' in '%s'." % (
787+ username, archive_reference))
788+ secret = removeSecurityProxy(token).token
789+ if password != secret:
790+ raise faults.Unauthorized()
791+
792+ def checkArchiveAuthToken(self, archive_reference, username, password):
793+ """See `IArchiveAPI`."""
794+ # This thunk exists because you can't use a decorated function as
795+ # the implementation of a method exported over XML-RPC.
796+ return self._checkArchiveAuthToken(
797+ archive_reference, username, password)
798
799=== added directory 'lib/lp/soyuz/xmlrpc/tests'
800=== added file 'lib/lp/soyuz/xmlrpc/tests/__init__.py'
801=== added file 'lib/lp/soyuz/xmlrpc/tests/test_archive.py'
802--- lib/lp/soyuz/xmlrpc/tests/test_archive.py 1970-01-01 00:00:00 +0000
803+++ lib/lp/soyuz/xmlrpc/tests/test_archive.py 2017-11-19 12:35:56 +0000
804@@ -0,0 +1,124 @@
805+# Copyright 2017 Canonical Ltd. This software is licensed under the
806+# GNU Affero General Public License version 3 (see the file LICENSE).
807+
808+"""Tests for the internal Soyuz archive API."""
809+
810+from __future__ import absolute_import, print_function, unicode_literals
811+
812+__metaclass__ = type
813+
814+from zope.security.proxy import removeSecurityProxy
815+
816+from lp.services.features.testing import FeatureFixture
817+from lp.soyuz.interfaces.archive import NAMED_AUTH_TOKEN_FEATURE_FLAG
818+from lp.soyuz.xmlrpc.archive import ArchiveAPI
819+from lp.testing import TestCaseWithFactory
820+from lp.testing.layers import LaunchpadFunctionalLayer
821+from lp.xmlrpc import faults
822+
823+
824+class TestArchiveAPI(TestCaseWithFactory):
825+
826+ layer = LaunchpadFunctionalLayer
827+
828+ def setUp(self):
829+ super(TestArchiveAPI, self).setUp()
830+ self.useFixture(FeatureFixture({NAMED_AUTH_TOKEN_FEATURE_FLAG: "on"}))
831+ self.archive_api = ArchiveAPI(None, None)
832+
833+ def assertNotFound(self, archive_reference, username, password, message):
834+ """Assert that an archive auth token check returns NotFound."""
835+ fault = self.archive_api.checkArchiveAuthToken(
836+ archive_reference, username, password)
837+ self.assertEqual(faults.NotFound(message), fault)
838+
839+ def assertUnauthorized(self, archive_reference, username, password):
840+ """Assert that an archive auth token check returns Unauthorized."""
841+ fault = self.archive_api.checkArchiveAuthToken(
842+ archive_reference, username, password)
843+ self.assertEqual(faults.Unauthorized("Authorisation required."), fault)
844+
845+ def test_checkArchiveAuthToken_unknown_archive(self):
846+ self.assertNotFound(
847+ "~nonexistent/unknown/bad", "user", "",
848+ "No archive found for '~nonexistent/unknown/bad'.")
849+
850+ def test_checkArchiveAuthToken_no_tokens(self):
851+ archive = removeSecurityProxy(self.factory.makeArchive(private=True))
852+ self.assertNotFound(
853+ archive.reference, "nobody", "",
854+ "No valid tokens for 'nobody' in '%s'." % archive.reference)
855+
856+ def test_checkArchiveAuthToken_no_named_tokens(self):
857+ archive = removeSecurityProxy(self.factory.makeArchive(private=True))
858+ self.assertNotFound(
859+ archive.reference, "+missing", "",
860+ "No valid tokens for '+missing' in '%s'." % archive.reference)
861+
862+ def test_checkArchiveAuthToken_buildd_wrong_password(self):
863+ archive = removeSecurityProxy(self.factory.makeArchive(private=True))
864+ self.assertUnauthorized(
865+ archive.reference, "buildd", archive.buildd_secret + "-bad")
866+
867+ def test_checkArchiveAuthToken_buildd_correct_password(self):
868+ archive = removeSecurityProxy(self.factory.makeArchive(private=True))
869+ self.assertIsNone(self.archive_api.checkArchiveAuthToken(
870+ archive.reference, "buildd", archive.buildd_secret))
871+
872+ def test_checkArchiveAuthToken_named_token_wrong_password(self):
873+ archive = removeSecurityProxy(self.factory.makeArchive(private=True))
874+ token = archive.newNamedAuthToken("special")
875+ removeSecurityProxy(token).deactivate()
876+ self.assertNotFound(
877+ archive.reference, "+special", token.token,
878+ "No valid tokens for '+special' in '%s'." % archive.reference)
879+
880+ def test_checkArchiveAuthToken_named_token_deactivated(self):
881+ archive = removeSecurityProxy(self.factory.makeArchive(private=True))
882+ token = archive.newNamedAuthToken("special")
883+ self.assertIsNone(self.archive_api.checkArchiveAuthToken(
884+ archive.reference, "+special", token.token))
885+
886+ def test_checkArchiveAuthToken_named_token_correct_password(self):
887+ archive = removeSecurityProxy(self.factory.makeArchive(private=True))
888+ token = archive.newNamedAuthToken("special")
889+ self.assertIsNone(self.archive_api.checkArchiveAuthToken(
890+ archive.reference, "+special", token.token))
891+
892+ def test_checkArchiveAuthToken_personal_token_wrong_password(self):
893+ archive = removeSecurityProxy(self.factory.makeArchive(private=True))
894+ subscriber = self.factory.makePerson()
895+ archive.newSubscription(subscriber, archive.owner)
896+ token = archive.newAuthToken(subscriber)
897+ self.assertUnauthorized(
898+ archive.reference, subscriber.name, token.token + "-bad")
899+
900+ def test_checkArchiveAuthToken_personal_token_deactivated(self):
901+ archive = removeSecurityProxy(self.factory.makeArchive(private=True))
902+ subscriber = self.factory.makePerson()
903+ archive.newSubscription(subscriber, archive.owner)
904+ token = archive.newAuthToken(subscriber)
905+ removeSecurityProxy(token).deactivate()
906+ self.assertNotFound(
907+ archive.reference, subscriber.name, token.token,
908+ "No valid tokens for '%s' in '%s'." % (
909+ subscriber.name, archive.reference))
910+
911+ def test_checkArchiveAuthToken_personal_token_cancelled(self):
912+ archive = removeSecurityProxy(self.factory.makeArchive(private=True))
913+ subscriber = self.factory.makePerson()
914+ subscription = archive.newSubscription(subscriber, archive.owner)
915+ token = archive.newAuthToken(subscriber)
916+ removeSecurityProxy(subscription).cancel(archive.owner)
917+ self.assertNotFound(
918+ archive.reference, subscriber.name, token.token,
919+ "No valid tokens for '%s' in '%s'." % (
920+ subscriber.name, archive.reference))
921+
922+ def test_checkArchiveAuthToken_personal_token_correct_password(self):
923+ archive = removeSecurityProxy(self.factory.makeArchive(private=True))
924+ subscriber = self.factory.makePerson()
925+ archive.newSubscription(subscriber, archive.owner)
926+ token = archive.newAuthToken(subscriber)
927+ self.assertIsNone(self.archive_api.checkArchiveAuthToken(
928+ archive.reference, subscriber.name, token.token))
929
930=== modified file 'lib/lp/systemhomes.py'
931--- lib/lp/systemhomes.py 2017-05-16 16:33:53 +0000
932+++ lib/lp/systemhomes.py 2017-11-19 12:35:56 +0000
933@@ -72,6 +72,7 @@
934 from lp.services.webapp.publisher import canonical_url
935 from lp.services.webservice.interfaces import IWebServiceApplication
936 from lp.services.worlddata.interfaces.language import ILanguageSet
937+from lp.soyuz.interfaces.archiveapi import IArchiveApplication
938 from lp.testopenid.interfaces.server import ITestOpenIDApplication
939 from lp.translations.interfaces.translationgroup import ITranslationGroupSet
940 from lp.translations.interfaces.translations import IRosettaApplication
941@@ -80,6 +81,12 @@
942 )
943
944
945+@implementer(IArchiveApplication)
946+class ArchiveApplication:
947+
948+ title = "Archive API"
949+
950+
951 @implementer(ICodehostingApplication)
952 class CodehostingApplication:
953 """Codehosting End-Point."""
954
955=== modified file 'lib/lp/xmlrpc/application.py'
956--- lib/lp/xmlrpc/application.py 2015-10-26 14:54:43 +0000
957+++ lib/lp/xmlrpc/application.py 2017-11-19 12:35:56 +0000
958@@ -31,11 +31,12 @@
959 from lp.services.features.xmlrpc import IFeatureFlagApplication
960 from lp.services.webapp import LaunchpadXMLRPCView
961 from lp.services.webapp.interfaces import ILaunchBag
962+from lp.soyuz.interfaces.archiveapi import IArchiveApplication
963 from lp.xmlrpc.interfaces import IPrivateApplication
964
965
966 # NOTE: If you add a traversal here, you should update
967-# the regular expression in utilities/page-performance-report.ini
968+# the regular expression in lp:lp-dev-utils page-performance-report.ini.
969 @implementer(IPrivateApplication)
970 class PrivateApplication:
971
972@@ -45,6 +46,11 @@
973 return getUtility(IMailingListApplication)
974
975 @property
976+ def archive(self):
977+ """See `IPrivateApplication`."""
978+ return getUtility(IArchiveApplication)
979+
980+ @property
981 def authserver(self):
982 """See `IPrivateApplication`."""
983 return getUtility(IAuthServerApplication)
984
985=== modified file 'lib/lp/xmlrpc/configure.zcml'
986--- lib/lp/xmlrpc/configure.zcml 2015-05-04 14:56:58 +0000
987+++ lib/lp/xmlrpc/configure.zcml 2017-11-19 12:35:56 +0000
988@@ -22,6 +22,19 @@
989 />
990
991 <securedutility
992+ class="lp.systemhomes.ArchiveApplication"
993+ provides="lp.soyuz.interfaces.archiveapi.IArchiveApplication">
994+ <allow interface="lp.soyuz.interfaces.archiveapi.IArchiveApplication"/>
995+ </securedutility>
996+
997+ <xmlrpc:view
998+ for="lp.soyuz.interfaces.archiveapi.IArchiveApplication"
999+ interface="lp.soyuz.interfaces.archiveapi.IArchiveAPI"
1000+ class="lp.soyuz.xmlrpc.archive.ArchiveAPI"
1001+ permission="zope.Public"
1002+ />
1003+
1004+ <securedutility
1005 class="lp.systemhomes.CodehostingApplication"
1006 provides="lp.code.interfaces.codehosting.ICodehostingApplication">
1007 <allow interface="lp.code.interfaces.codehosting.ICodehostingApplication"/>
1008
1009=== modified file 'lib/lp/xmlrpc/interfaces.py'
1010--- lib/lp/xmlrpc/interfaces.py 2015-05-04 14:56:58 +0000
1011+++ lib/lp/xmlrpc/interfaces.py 2017-11-19 12:35:56 +0000
1012@@ -17,6 +17,8 @@
1013 class IPrivateApplication(ILaunchpadApplication):
1014 """Launchpad private XML-RPC application root."""
1015
1016+ archive = Attribute("Archive XML-RPC end point.""")
1017+
1018 authserver = Attribute("""Old Authserver API end point.""")
1019
1020 codeimportscheduler = Attribute("""Code import scheduler end point.""")
1021
1022=== added file 'scripts/wsgi-archive-auth.py'
1023--- scripts/wsgi-archive-auth.py 1970-01-01 00:00:00 +0000
1024+++ scripts/wsgi-archive-auth.py 2017-11-19 12:35:56 +0000
1025@@ -0,0 +1,71 @@
1026+#!/usr/bin/python
1027+#
1028+# Copyright 2017 Canonical Ltd. This software is licensed under the
1029+# GNU Affero General Public License version 3 (see the file LICENSE).
1030+
1031+"""WSGI archive authorisation provider entry point.
1032+
1033+Unlike most Launchpad scripts, the #! line of this script does not use -S.
1034+This is because it is only executed (as opposed to imported) for testing,
1035+and mod_wsgi does not disable the automatic import of the site module when
1036+importing this script, so we want the test to imitate mod_wsgi's behaviour
1037+as closely as possible.
1038+"""
1039+
1040+from __future__ import absolute_import, print_function, unicode_literals
1041+
1042+__metaclass__ = type
1043+__all__ = [
1044+ 'check_password',
1045+ ]
1046+
1047+# mod_wsgi imports this file without a useful sys.path, so we need some
1048+# acrobatics to set ourselves up properly.
1049+import os.path
1050+import sys
1051+
1052+scripts_dir = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
1053+if scripts_dir not in sys.path:
1054+ sys.path.insert(0, scripts_dir)
1055+top = os.path.dirname(scripts_dir)
1056+
1057+# We can't stop mod_wsgi importing the site module. Cross fingers and
1058+# arrange for it to be re-imported.
1059+sys.modules.pop("site", None)
1060+sys.modules.pop("sitecustomize", None)
1061+
1062+import _pythonpath
1063+
1064+from lp.soyuz.wsgi.archiveauth import check_password
1065+
1066+
1067+def main():
1068+ # Hook for testing, not used by WSGI.
1069+ from argparse import ArgumentParser
1070+
1071+ from lp.services.memcache.testing import MemcacheFixture
1072+ from lp.soyuz.wsgi import archiveauth
1073+
1074+ parser = ArgumentParser()
1075+ parser.add_argument("archive_path")
1076+ parser.add_argument("username")
1077+ parser.add_argument("password")
1078+ args = parser.parse_args()
1079+ archiveauth._memcache_client = MemcacheFixture()
1080+ result = check_password(
1081+ {"SCRIPT_NAME": args.archive_path}, args.username, args.password)
1082+ if result is None:
1083+ print("Archive or user does not exist.", file=sys.stderr)
1084+ return 2
1085+ elif result is False:
1086+ print("Password does not match.", file=sys.stderr)
1087+ return 1
1088+ elif result is True:
1089+ return 0
1090+ else:
1091+ print("Unexpected result from check_password: %s" % result)
1092+ return 3
1093+
1094+
1095+if __name__ == "__main__":
1096+ sys.exit(main())
1097
1098=== modified file 'utilities/rocketfuel-setup'
1099--- utilities/rocketfuel-setup 2017-01-10 17:24:08 +0000
1100+++ utilities/rocketfuel-setup 2017-11-19 12:35:56 +0000
1101@@ -189,6 +189,12 @@
1102 exit 1
1103 fi
1104
1105+sudo a2enmod wsgi > /dev/null
1106+if [ $? -ne 0 ]; then
1107+ echo "ERROR: Unable to enable wsgi module in Apache2"
1108+ exit 1
1109+fi
1110+
1111 if [ $DO_WORKSPACE == 0 ]; then
1112 cat <<EOT
1113 Branches have not been created, as requested. You will need to do some or all