Launchpad itself

Merge lp:~wgrant/launchpad/no-more-vhost-config into lp:launchpad

  1. no-more-vhost-config
  2. Merge into devel
Proposed by William Grant
Status: Work in progress
Proposed branch: lp:~wgrant/launchpad/no-more-vhost-config
Merge into: lp:launchpad
Diff against target: 1296 lines (+910/-80) (has conflicts)
12 files modified
configs/development/launchpad-lazr.conf (+17/-2)
configs/testrunner-appserver/launchpad-lazr.conf (+5/-0)
configs/testrunner/launchpad-lazr.conf (+4/-0)
lib/devscripts/ec2test/instance.py.OTHER (+695/-0)
lib/lp/services/config/schema-lazr.conf (+28/-19)
lib/lp/services/openid/adapters/openid.py (+23/-0)
lib/lp/services/webapp/doc/webapp-publication.txt (+16/-0)
lib/lp/services/webapp/metazcml.py (+12/-3)
lib/lp/services/webapp/servers.py (+14/-1)
lib/lp/services/webapp/vhosts.py (+58/-53)
lib/lp/testing/layers.py (+29/-1)
lib/lp/testing/tests/test_layers_functional.py (+9/-1)
Text conflict in configs/development/launchpad-lazr.conf
Text conflict in configs/testrunner-appserver/launchpad-lazr.conf
Text conflict in configs/testrunner/launchpad-lazr.conf
Conflict adding files to lib/devscripts/ec2test.  Created directory.
Conflict because lib/devscripts/ec2test is not versioned, but has versioned children.  Versioned directory.
Contents conflict in lib/devscripts/ec2test/instance.py
Text conflict in lib/lp/services/config/schema-lazr.conf
Text conflict in lib/lp/services/openid/adapters/openid.py
Text conflict in lib/lp/services/webapp/doc/webapp-publication.txt
Text conflict in lib/lp/services/webapp/metazcml.py
Text conflict in lib/lp/services/webapp/servers.py
Text conflict in lib/lp/services/webapp/vhosts.py
Text conflict in lib/lp/testing/layers.py
Text conflict in lib/lp/testing/tests/test_layers_functional.py
To merge this branch: bzr merge lp:~wgrant/launchpad/no-more-vhost-config
Reviewer Review Type Date Requested Status
Launchpad code reviewers Pending
Review via email: mp+306686@code.launchpad.net
To post a comment you must log in.

Unmerged revisions

14141. By William Grant

Drop unneeded vhost config sections from the schema.

14140. By William Grant

Fix AppServerLayer.

14139. By William Grant

appserver config now uses a custom port. Doesn't actually work, because the layer relies on config.vhost.mainsite.rooturl.

14138. By William Grant

Comment the config schema, and use a custom portin the vhost config if it's specified.

14137. By William Grant

Inline logic slightly.

14136. By William Grant

Refactor the vhost setup code a bit.

14135. By William Grant

VirtualHostConfig now takes a port, instead of a rooturl.

14134. By William Grant

Rename the xmlrpc_private vhost to xmlrpc-private, to match the subdomain.

14133. By William Grant

Since vhost config is pretty much defined from a base domain and a port, move all that stuff into code.

14132. By William Grant

Our publisher ZCML directive now checks allvhosts, rather than config.vhosts directly.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'configs/development/launchpad-lazr.conf'
--- configs/development/launchpad-lazr.conf 2016-06-30 16:05:11 +0000
+++ configs/development/launchpad-lazr.conf 2016-09-24 06:42:11 +0000
@@ -94,8 +94,12 @@
9494
95[launchpad]95[launchpad]
96enable_test_openid_provider: True96enable_test_openid_provider: True
97<<<<<<< TREE
97openid_canonical_root: https://testopenid.dev/98openid_canonical_root: https://testopenid.dev/
98openid_provider_root: https://testopenid.dev/99openid_provider_root: https://testopenid.dev/
100=======
101openid_provider_root: https://testopenid.dev/
102>>>>>>> MERGE-SOURCE
99code_domain: code.launchpad.dev103code_domain: code.launchpad.dev
100default_batch_size: 5104default_batch_size: 5
101max_attachment_size: 2097152105max_attachment_size: 2097152
@@ -213,15 +217,24 @@
213217
214[vhosts]218[vhosts]
215use_https: True219use_https: True
220base_hostname: launchpad.dev
221vostok_hostname: vostok.dev
222testopenid_hostname: testopenid.dev
223enable_apidoc: True
216224
217[vhost.mainsite]225[vhost.mainsite]
218hostname: launchpad.dev
219althostnames: localhost
220openid_delegate_profile: True226openid_delegate_profile: True
221227
222[vhost.api]228[vhost.api]
229<<<<<<< TREE
223hostname: api.launchpad.dev230hostname: api.launchpad.dev
231=======
232# Turn this on once we've solved cache invalidation problems and are
233# ready to test.
234# enable_server_side_representation_cache: True
235>>>>>>> MERGE-SOURCE
224236
237<<<<<<< TREE
225[vhost.blueprints]238[vhost.blueprints]
226hostname: blueprints.launchpad.dev239hostname: blueprints.launchpad.dev
227240
@@ -249,6 +262,8 @@
249[vhost.feeds]262[vhost.feeds]
250hostname: feeds.launchpad.dev263hostname: feeds.launchpad.dev
251264
265=======
266>>>>>>> MERGE-SOURCE
252[immediate_mail]267[immediate_mail]
253# XXX sinzui 2008-03-26:268# XXX sinzui 2008-03-26:
254# A development box should never send email to the outer world,269# A development box should never send email to the outer world,
255270
=== modified file 'configs/testrunner-appserver/launchpad-lazr.conf'
--- configs/testrunner-appserver/launchpad-lazr.conf 2014-02-27 08:39:44 +0000
+++ configs/testrunner-appserver/launchpad-lazr.conf 2016-09-24 06:42:11 +0000
@@ -22,6 +22,7 @@
22xmlrpc_runner_sleep: 122xmlrpc_runner_sleep: 1
23register_bounces_every: 123register_bounces_every: 1
2424
25<<<<<<< TREE
25[vhost.mainsite]26[vhost.mainsite]
26rooturl: http://launchpad.dev:8085/27rooturl: http://launchpad.dev:8085/
2728
@@ -54,6 +55,10 @@
5455
55[vhost.feeds]56[vhost.feeds]
56rooturl: http://feeds.launchpad.dev:8085/57rooturl: http://feeds.launchpad.dev:8085/
58=======
59[vhosts]
60port: 8085
61>>>>>>> MERGE-SOURCE
5762
58[immediate_mail]63[immediate_mail]
59# BarryWarsaw 04-Dec-2008: AppServerLayer tests should send email to the fake64# BarryWarsaw 04-Dec-2008: AppServerLayer tests should send email to the fake
6065
=== modified file 'configs/testrunner/launchpad-lazr.conf'
--- configs/testrunner/launchpad-lazr.conf 2016-05-18 00:33:18 +0000
+++ configs/testrunner/launchpad-lazr.conf 2016-09-24 06:42:11 +0000
@@ -107,9 +107,13 @@
107# We use the stub Google Service here which maps URL fragment to107# We use the stub Google Service here which maps URL fragment to
108# to static content108# to static content
109homepage_recent_posts_feed: http://launchpad.dev:8092/blog-feed109homepage_recent_posts_feed: http://launchpad.dev:8092/blog-feed
110<<<<<<< TREE
110openid_canonical_root: http://testopenid.dev/111openid_canonical_root: http://testopenid.dev/
111openid_provider_root: http://testopenid.dev/112openid_provider_root: http://testopenid.dev/
112openid_alternate_provider_roots: http://login1.dev/, http://login2.dev/113openid_alternate_provider_roots: http://login1.dev/, http://login2.dev/
114=======
115openid_provider_root: http://testopenid.dev/
116>>>>>>> MERGE-SOURCE
113117
114[launchpad_session]118[launchpad_session]
115cookie: launchpad_tests119cookie: launchpad_tests
116120
=== added directory 'lib/devscripts/ec2test'
=== added file 'lib/devscripts/ec2test/instance.py.OTHER'
--- lib/devscripts/ec2test/instance.py.OTHER 1970-01-01 00:00:00 +0000
+++ lib/devscripts/ec2test/instance.py.OTHER 2016-09-24 06:42:11 +0000
@@ -0,0 +1,695 @@
1# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""Code to represent a single machine instance in EC2."""
5
6__metaclass__ = type
7__all__ = [
8 'EC2Instance',
9 ]
10
11import code
12import errno
13import glob
14import os
15import select
16import socket
17import subprocess
18import sys
19import time
20import traceback
21from datetime import datetime
22from bzrlib.errors import BzrCommandError
23from devscripts.ec2test.session import EC2SessionName
24import paramiko
25
26
27DEFAULT_INSTANCE_TYPE = 'c1.xlarge'
28AVAILABLE_INSTANCE_TYPES = ('m1.large', 'm1.xlarge', 'c1.xlarge')
29
30
31class AcceptAllPolicy:
32 """We accept all unknown host key."""
33
34 def missing_host_key(self, client, hostname, key):
35 # Normally the console output is supposed to contain the Host key but
36 # it doesn't seem to be the case here, so we trust that the host we
37 # are connecting to is the correct one.
38 pass
39
40
41def get_user_key():
42 """Get a SSH key from the agent. Raise an error if no keys were found.
43
44 This key will be used to let the user log in (as $USER) to the instance.
45 """
46 agent = paramiko.Agent()
47 keys = agent.get_keys()
48 if len(keys) == 0:
49 raise BzrCommandError(
50 'You must have an ssh agent running with keys installed that '
51 'will allow the script to access Launchpad and get your '
52 'branch.\n')
53
54 # XXX mars 2010-05-07 bug=577118
55 # Popping the first key off of the stack can create problems if the person
56 # has more than one key in their ssh-agent, but alas, we have no good way
57 # to detect the right key to use. See bug 577118 for a workaround.
58 return keys[0]
59
60
61# Commands to run to turn a blank image into one usable for the rest of the
62# ec2 functionality. They come in two parts, one set that need to be run as
63# root and another that should be run as the 'ec2test' user.
64# Note that the sources from http://us.ec2.archive.ubuntu.com/ubuntu/ are per
65# instructions described in http://is.gd/g1MIT . When we switch to
66# Eucalyptus, we can dump this.
67
68from_scratch_root = """
69# From 'help set':
70# -x Print commands and their arguments as they are executed.
71# -e Exit immediately if a command exits with a non-zero status.
72set -xe
73
74sed -ie 's/main universe/main universe multiverse/' /etc/apt/sources.list
75
76. /etc/lsb-release
77
78cat >> /etc/apt/sources.list << EOF
79deb http://ppa.launchpad.net/launchpad/ubuntu $DISTRIB_CODENAME main
80deb http://ppa.launchpad.net/bzr/ubuntu $DISTRIB_CODENAME main
81deb http://ppa.launchpad.net/bzr-beta-ppa/ubuntu $DISTRIB_CODENAME main
82deb http://us.ec2.archive.ubuntu.com/ubuntu/ $DISTRIB_CODENAME multiverse
83deb-src http://us.ec2.archive.ubuntu.com/ubuntu/ $DISTRIB_CODENAME main
84EOF
85
86# This next part is cribbed from rocketfuel-setup
87dev_host() {
88 sed -i \"s/^127.0.0.88.*$/&\ ${hostname}/\" /etc/hosts
89}
90
91echo 'Adding development hosts on local machine'
92echo '
93# Launchpad virtual domains. This should be on one line.
94127.0.0.88 launchpad.dev
95' >> /etc/hosts
96
97declare -a hostnames
98hostnames=$(cat <<EOF
99 answers.launchpad.dev
100 archive.launchpad.dev
101 api.launchpad.dev
102 bazaar-internal.launchpad.dev
103 blueprints.launchpad.dev
104 bugs.launchpad.dev
105 code.launchpad.dev
106 feeds.launchpad.dev
107 keyserver.launchpad.dev
108 lists.launchpad.dev
109 ppa.launchpad.dev
110 private-ppa.launchpad.dev
111 testopenid.dev
112 translations.launchpad.dev
113 xmlrpc-private.launchpad.dev
114 xmlrpc.launchpad.dev
115EOF
116 )
117
118for hostname in $hostnames; do
119 dev_host;
120done
121
122echo '
123127.0.0.99 bazaar.launchpad.dev
124' >> /etc/hosts
125
126# Add the keys for the three PPAs added to sources.list above.
127apt-key adv --recv-keys --keyserver pool.sks-keyservers.net 2af499cb24ac5f65461405572d1ffb6c0a5174af
128apt-key adv --recv-keys --keyserver pool.sks-keyservers.net ece2800bacf028b31ee3657cd702bf6b8c6c1efd
129apt-key adv --recv-keys --keyserver pool.sks-keyservers.net cbede690576d1e4e813f6bb3ebaf723d37b19b80
130
131aptitude update
132aptitude -y full-upgrade
133
134DEBIAN_FRONTEND=noninteractive apt-get -y install launchpad-developer-dependencies apache2 apache2-mpm-worker
135
136# Create the ec2test user, give them passwordless sudo.
137adduser --gecos "" --disabled-password ec2test
138echo 'ec2test\tALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers
139
140mkdir /home/ec2test/.ssh
141cat > /home/ec2test/.ssh/config << EOF
142CheckHostIP no
143StrictHostKeyChecking no
144EOF
145
146mkdir /var/launchpad
147chown -R ec2test:ec2test /var/www /var/launchpad /home/ec2test/
148"""
149
150
151from_scratch_ec2test = """
152# From 'help set':
153# -x Print commands and their arguments as they are executed.
154# -e Exit immediately if a command exits with a non-zero status.
155set -xe
156
157bzr launchpad-login %(launchpad-login)s
158bzr init-repo --2a /var/launchpad
159bzr branch lp:~launchpad-pqm/launchpad/devel /var/launchpad/test
160bzr branch --standalone lp:lp-source-dependencies /var/launchpad/download-cache
161mkdir /var/launchpad/sourcecode
162/var/launchpad/test/utilities/update-sourcecode /var/launchpad/sourcecode
163"""
164
165
166postmortem_banner = """\
167Postmortem Console. EC2 instance is not yet dead.
168It will shut down when you exit this prompt (CTRL-D)
169
170Tab-completion is enabled.
171EC2Instance is available as `instance`.
172Also try these:
173 http://%(dns)s/current_test.log
174 ssh -A ec2test@%(dns)s
175"""
176
177
178class EC2Instance:
179 """A single EC2 instance."""
180
181 @classmethod
182 def make(cls, name, instance_type, machine_id, demo_networks=None,
183 credentials=None):
184 """Construct an `EC2Instance`.
185
186 :param name: The name to use for the key pair and security group for
187 the instance.
188 :type name: `EC2SessionName`
189 :param instance_type: One of the AVAILABLE_INSTANCE_TYPES.
190 :param machine_id: The AMI to use, or None to do the usual regexp
191 matching. If you put 'based-on:' before the AMI id, it is assumed
192 that the id specifies a blank image that should be made into one
193 suitable for the other ec2 functions (see `from_scratch_root` and
194 `from_scratch_ec2test` above).
195 :param demo_networks: A list of networks to add to the security group
196 to allow access to the instance.
197 :param credentials: An `EC2Credentials` object.
198 """
199 # This import breaks in the test environment. Do it here so
200 # that unit tests (which don't use this factory) can still
201 # import EC2Instance.
202 from bzrlib.plugins.launchpad.account import get_lp_login
203
204 # XXX JeroenVermeulen 2009-11-27 bug=489073: EC2Credentials
205 # imports boto, which isn't necessarily installed in our test
206 # environment. Doing the import here so that unit tests (which
207 # don't use this factory) can still import EC2Instance.
208 from devscripts.ec2test.credentials import EC2Credentials
209
210 assert isinstance(name, EC2SessionName)
211 if instance_type not in AVAILABLE_INSTANCE_TYPES:
212 raise ValueError('unknown instance_type %s' % (instance_type,))
213
214 # We call this here so that it has a chance to complain before the
215 # instance is started (which can take some time).
216 user_key = get_user_key()
217
218 if credentials is None:
219 credentials = EC2Credentials.load_from_file()
220
221 # Make the EC2 connection.
222 account = credentials.connect(name)
223
224 # We do this here because it (1) cleans things up and (2) verifies
225 # that the account is correctly set up. Both of these are appropriate
226 # for initialization.
227 #
228 # We always recreate the keypairs because there is no way to
229 # programmatically retrieve the private key component, unless we
230 # generate it.
231 account.collect_garbage()
232
233 if machine_id and machine_id.startswith('based-on:'):
234 from_scratch = True
235 machine_id = machine_id[len('based-on:'):]
236 else:
237 from_scratch = False
238
239 # get the image
240 image = account.acquire_image(machine_id)
241
242 login = get_lp_login()
243 if not login:
244 raise BzrCommandError(
245 'you must have set your launchpad login in bzr.')
246
247 return EC2Instance(
248 name, image, instance_type, demo_networks, account,
249 from_scratch, user_key, login)
250
251 def __init__(self, name, image, instance_type, demo_networks, account,
252 from_scratch, user_key, launchpad_login):
253 self._name = name
254 self._image = image
255 self._account = account
256 self._instance_type = instance_type
257 self._demo_networks = demo_networks
258 self._boto_instance = None
259 self._from_scratch = from_scratch
260 self._user_key = user_key
261 self._launchpad_login = launchpad_login
262
263 def log(self, msg):
264 """Log a message on stdout, flushing afterwards."""
265 # XXX: JonathanLange 2009-05-31 bug=383076: Should delete this and use
266 # Python logging module instead.
267 sys.stdout.write(msg)
268 sys.stdout.flush()
269
270 def start(self):
271 """Start the instance."""
272 if self._boto_instance is not None:
273 self.log('Instance %s already started' % self._boto_instance.id)
274 return
275 start = time.time()
276 self.private_key = self._account.acquire_private_key()
277 self.security_group = self._account.acquire_security_group(
278 demo_networks=self._demo_networks)
279 reservation = self._image.run(
280 key_name=self._name, security_groups=[self._name],
281 instance_type=self._instance_type)
282 self._boto_instance = reservation.instances[0]
283 self.log('Instance %s starting..' % self._boto_instance.id)
284 while self._boto_instance.state == 'pending':
285 self.log('.')
286 time.sleep(5)
287 self._boto_instance.update()
288 if self._boto_instance.state == 'running':
289 self.log(' started on %s\n' % self.hostname)
290 elapsed = time.time() - start
291 self.log('Started in %d minutes %d seconds\n' %
292 (elapsed // 60, elapsed % 60))
293 self._output = self._boto_instance.get_console_output()
294 self.log(self._output.output)
295 self._ec2test_user_has_keys = False
296 else:
297 raise BzrCommandError(
298 'failed to start: %s\n' % self._boto_instance.state)
299
300 def shutdown(self):
301 """Shut down the instance."""
302 if self._boto_instance is None:
303 self.log('no instance created\n')
304 return
305 self._boto_instance.update()
306 if self._boto_instance.state not in ('shutting-down', 'terminated'):
307 # terminate instance
308 self._boto_instance.stop()
309 self._boto_instance.update()
310 self.log('instance %s\n' % (self._boto_instance.state,))
311
312 @property
313 def hostname(self):
314 if self._boto_instance is None:
315 return None
316 return self._boto_instance.public_dns_name
317
318 def _connect(self, username):
319 """Connect to the instance as `user`. """
320 ssh = paramiko.SSHClient()
321 ssh.set_missing_host_key_policy(AcceptAllPolicy())
322 connect_args = {
323 'username': username,
324 'pkey': self.private_key,
325 'allow_agent': False,
326 'look_for_keys': False,
327 }
328 for count in range(10):
329 try:
330 ssh.connect(self.hostname, **connect_args)
331 except (socket.error, paramiko.AuthenticationException), e:
332 self.log('_connect: %r\n' % (e,))
333 if count < 9:
334 time.sleep(5)
335 self.log('retrying...')
336 else:
337 raise
338 else:
339 break
340 return EC2InstanceConnection(self, username, ssh)
341
342 def _upload_local_key(self, conn, remote_filename):
343 """Upload a key from the local user's agent to `remote_filename`.
344
345 The key will be uploaded in a format suitable for
346 ~/.ssh/authorized_keys.
347 """
348 authorized_keys_file = conn.sftp.open(remote_filename, 'w')
349 authorized_keys_file.write(
350 "%s %s\n" % (
351 self._user_key.get_name(), self._user_key.get_base64()))
352 authorized_keys_file.close()
353
354 def _ensure_ec2test_user_has_keys(self, connection=None):
355 """Make sure that we can connect over ssh as the 'ec2test' user.
356
357 We add both the key that was used to start the instance (so
358 _connect('ec2test') works and a key from the locally running ssh agent
359 (so EC2InstanceConnection.run_with_ssh_agent works).
360 """
361 if not self._ec2test_user_has_keys:
362 if connection is None:
363 connection = self._connect('ubuntu')
364 our_connection = True
365 else:
366 our_connection = False
367 self._upload_local_key(connection, 'local_key')
368 connection.perform(
369 'cat /home/ubuntu/.ssh/authorized_keys local_key '
370 '| sudo tee /home/ec2test/.ssh/authorized_keys > /dev/null'
371 '&& rm local_key')
372 connection.perform('sudo chown -R ec2test:ec2test /home/ec2test/')
373 connection.perform('sudo chmod 644 /home/ec2test/.ssh/*')
374 if our_connection:
375 connection.close()
376 self.log(
377 'You can now use ssh -A ec2test@%s to '
378 'log in the instance.\n' % self.hostname)
379 self._ec2test_user_has_keys = True
380
381 def connect(self):
382 """Connect to the instance as a user with passwordless sudo.
383
384 This may involve first connecting as root and adding SSH keys to the
385 user's account, and in the case of a from scratch image, it will do a
386 lot of set up.
387 """
388 if self._from_scratch:
389 ubuntu_connection = self._connect('ubuntu')
390 self._upload_local_key(ubuntu_connection, 'local_key')
391 ubuntu_connection.perform(
392 'cat local_key >> ~/.ssh/authorized_keys && rm local_key')
393 ubuntu_connection.run_script(from_scratch_root, sudo=True)
394 self._ensure_ec2test_user_has_keys(ubuntu_connection)
395 ubuntu_connection.close()
396 conn = self._connect('ec2test')
397 conn.run_script(
398 from_scratch_ec2test
399 % {'launchpad-login': self._launchpad_login})
400 self._from_scratch = False
401 return conn
402 self._ensure_ec2test_user_has_keys()
403 return self._connect('ec2test')
404
405 def _report_traceback(self):
406 """Print traceback."""
407 traceback.print_exc()
408
409 def set_up_and_run(self, postmortem, shutdown, func, *args, **kw):
410 """Start, run `func` and then maybe shut down.
411
412 :param config: A dictionary specifying details of how the instance
413 should be run:
414 :param postmortem: If true, any exceptions will be caught and an
415 interactive session run to allow debugging the problem.
416 :param shutdown: If true, shut down the instance after `func` and
417 postmortem (if any) are completed.
418 :param func: A callable that will be called when the instance is
419 running and a user account has been set up on it.
420 :param args: Passed to `func`.
421 :param kw: Passed to `func`.
422 """
423 # We ignore the value of the 'shutdown' argument and always shut down
424 # unless `func` returns normally.
425 really_shutdown = True
426 retval = None
427 try:
428 self.start()
429 try:
430 retval = func(*args, **kw)
431 except Exception:
432 # When running in postmortem mode, it is really helpful to see
433 # if there are any exceptions before it waits in the console
434 # (in the finally block), and you can't figure out why it's
435 # broken.
436 self._report_traceback()
437 else:
438 really_shutdown = shutdown
439 finally:
440 try:
441 if postmortem:
442 console = code.InteractiveConsole(locals())
443 console.interact(
444 postmortem_banner % {'dns': self.hostname})
445 print 'Postmortem console closed.'
446 finally:
447 if really_shutdown:
448 self.shutdown()
449 return retval
450
451 def _copy_single_file(self, sftp, local_path, remote_dir):
452 """Copy `local_path` to `remote_dir` on this instance.
453
454 The name in the remote directory will be that of the local file.
455
456 :param sftp: A paramiko SFTP object.
457 :param local_path: The local path.
458 :param remote_dir: The directory on the instance to copy into.
459 """
460 name = os.path.basename(local_path)
461 remote_path = os.path.join(remote_dir, name)
462 remote_file = sftp.open(remote_path, 'w')
463 remote_file.write(open(local_path).read())
464 remote_file.close()
465 return remote_path
466
467 def copy_key_and_certificate_to_image(self, sftp):
468 """Copy the AWS private key and certificate to the image.
469
470 :param sftp: A paramiko SFTP object.
471 """
472 remote_ec2_dir = '/mnt/ec2'
473 remote_pk = self._copy_single_file(
474 sftp, self.local_pk, remote_ec2_dir)
475 remote_cert = self._copy_single_file(
476 sftp, self.local_cert, remote_ec2_dir)
477 return (remote_pk, remote_cert)
478
479 def _check_single_glob_match(self, local_dir, pattern, file_kind):
480 """Check that `pattern` matches one file in `local_dir` and return it.
481
482 :param local_dir: The local directory to look in.
483 :param pattern: The glob patten to match.
484 :param file_kind: The sort of file we're looking for, to be used in
485 error messages.
486 """
487 pattern = os.path.join(local_dir, pattern)
488 matches = glob.glob(pattern)
489 if len(matches) != 1:
490 raise BzrCommandError(
491 '%r must match a single %s file' % (pattern, file_kind))
492 return matches[0]
493
494 def check_bundling_prerequisites(self, name, credentials):
495 """Check, as best we can, that all the files we need to bundle exist.
496 """
497 if subprocess.call(['which', 'ec2-register']):
498 raise BzrCommandError(
499 '`ec2-register` command not found. '
500 'Try `sudo apt-get install ec2-api-tools`.')
501 local_ec2_dir = os.path.expanduser('~/.ec2')
502 if not os.path.exists(local_ec2_dir):
503 raise BzrCommandError(
504 "~/.ec2 must exist and contain aws_user, aws_id, a private "
505 "key file and a certificate.")
506 aws_user_file = os.path.expanduser('~/.ec2/aws_user')
507 if not os.path.exists(aws_user_file):
508 raise BzrCommandError(
509 "~/.ec2/aws_user must exist and contain your numeric AWS id.")
510 self.aws_user = open(aws_user_file).read().strip()
511 self.local_cert = self._check_single_glob_match(
512 local_ec2_dir, 'cert-*.pem', 'certificate')
513 self.local_pk = self._check_single_glob_match(
514 local_ec2_dir, 'pk-*.pem', 'private key')
515 # The bucket `name` needs to exist and be accessible. We create it
516 # here to reserve the name. If the bucket already exists and conforms
517 # to the above requirements, this is a no-op.
518 credentials.connect_s3().create_bucket(name)
519
520 def bundle(self, name, credentials):
521 """Bundle, upload and register the instance as a new AMI.
522
523 :param name: The name-to-be of the new AMI.
524 :param credentials: An `EC2Credentials` object.
525 """
526 connection = self.connect()
527 # See http://is.gd/g1MIT . When we switch to Eucalyptus, we can dump
528 # this installation of the ec2-ami-tools.
529 connection.perform(
530 'sudo env DEBIAN_FRONTEND=noninteractive '
531 'apt-get -y install ec2-ami-tools')
532 connection.perform('rm -f .ssh/authorized_keys')
533 connection.perform('sudo mkdir /mnt/ec2')
534 connection.perform('sudo chown $USER:$USER /mnt/ec2')
535
536 remote_pk, remote_cert = self.copy_key_and_certificate_to_image(
537 connection.sftp)
538
539 bundle_dir = os.path.join('/mnt', name)
540
541 connection.perform('sudo mkdir ' + bundle_dir)
542 connection.perform(' '.join([
543 'sudo ec2-bundle-vol',
544 '-d %s' % bundle_dir,
545 '--batch', # Set batch-mode, which doesn't use prompts.
546 '-k %s' % remote_pk,
547 '-c %s' % remote_cert,
548 '-u %s' % self.aws_user,
549 ]))
550
551 # Assume that the manifest is 'image.manifest.xml', since "image" is
552 # the default prefix.
553 manifest = os.path.join(bundle_dir, 'image.manifest.xml')
554
555 # Best check that the manifest actually exists though.
556 test = 'test -f %s' % manifest
557 connection.perform(test)
558
559 connection.perform(' '.join([
560 'sudo ec2-upload-bundle',
561 '-b %s' % name,
562 '-m %s' % manifest,
563 '-a %s' % credentials.identifier,
564 '-s %s' % credentials.secret,
565 ]))
566
567 connection.close()
568
569 # This is invoked locally.
570 mfilename = os.path.basename(manifest)
571 manifest_path = os.path.join(name, mfilename)
572
573 env = os.environ.copy()
574 if 'JAVA_HOME' not in os.environ:
575 env['JAVA_HOME'] = '/usr/lib/jvm/default-java'
576 now = datetime.strftime(datetime.utcnow(), "%Y-%m-%d %H:%M:%S UTC")
577 description = "Created %s" % now
578 cmd = [
579 'ec2-register',
580 '--private-key=%s' % self.local_pk,
581 '--cert=%s' % self.local_cert,
582 '--name=%s' % (name,),
583 '--description=%s' % description,
584 manifest_path,
585 ]
586 self.log("Executing command: %s" % ' '.join(cmd))
587 subprocess.check_call(cmd, env=env)
588
589
590class EC2InstanceConnection:
591 """An ssh connection to an `EC2Instance`."""
592
593 def __init__(self, instance, username, ssh):
594 self._instance = instance
595 self._username = username
596 self._ssh = ssh
597 self._sftp = None
598
599 @property
600 def sftp(self):
601 if self._sftp is None:
602 self._sftp = self._ssh.open_sftp()
603 return self._sftp
604
605 def perform(self, cmd, ignore_failure=False, out=None, err=None):
606 """Perform 'cmd' on server.
607
608 :param ignore_failure: If False, raise an error on non-zero exit
609 statuses.
610 :param out: A stream to write the output of the remote command to.
611 :param err: A stream to write the error of the remote command to.
612 """
613 if out is None:
614 out = sys.stdout
615 if err is None:
616 err = sys.stderr
617 self._instance.log(
618 '%s@%s$ %s\n'
619 % (self._username, self._instance._boto_instance.id, cmd))
620 session = self._ssh.get_transport().open_session()
621 session.exec_command(cmd)
622 session.shutdown_write()
623 while 1:
624 try:
625 select.select([session], [], [], 0.5)
626 except (IOError, select.error), e:
627 if e.errno == errno.EINTR:
628 continue
629 if session.recv_ready():
630 data = session.recv(4096)
631 if data:
632 out.write(data)
633 out.flush()
634 if session.recv_stderr_ready():
635 data = session.recv_stderr(4096)
636 if data:
637 err.write(data)
638 err.flush()
639 if session.exit_status_ready():
640 break
641 session.close()
642 # XXX: JonathanLange 2009-05-31: If the command is killed by a signal
643 # on the remote server, the SSH protocol does not send an exit_status,
644 # it instead sends a different message with the number of the signal
645 # that killed the process. AIUI, this code will fail confusingly if
646 # that happens.
647 res = session.recv_exit_status()
648 if res and not ignore_failure:
649 raise RuntimeError('Command failed: %s' % (cmd,))
650 return res
651
652 def run_with_ssh_agent(self, cmd, ignore_failure=False):
653 """Run 'cmd' in a subprocess.
654
655 Use this to run commands that require local SSH credentials. For
656 example, getting private branches from Launchpad.
657 """
658 self._instance.log(
659 '%s@%s$ %s\n'
660 % (self._username, self._instance._boto_instance.id, cmd))
661 call = ['ssh', '-A', self._username + '@' + self._instance.hostname,
662 '-o', 'CheckHostIP no',
663 '-o', 'StrictHostKeyChecking no',
664 '-o', 'UserKnownHostsFile ~/.ec2/known_hosts',
665 cmd]
666 res = subprocess.call(call)
667 if res and not ignore_failure:
668 raise RuntimeError('Command failed: %s' % (cmd,))
669 return res
670
671 def run_script(self, script_text, sudo=False):
672 """Upload `script_text` to the instance and run it with bash."""
673 script = self.sftp.open('script.sh', 'w')
674 script.write(script_text)
675 script.close()
676 cmd = '/bin/bash script.sh'
677 if sudo:
678 cmd = 'sudo ' + cmd
679 self.run_with_ssh_agent(cmd)
680 # At least for mwhudson, the paramiko connection often drops while the
681 # script is running. Reconnect just in case.
682 self.reconnect()
683 self.perform('rm script.sh')
684
685 def reconnect(self):
686 """Close the connection and reopen it."""
687 self.close()
688 self._ssh = self._instance._connect(self._username)._ssh
689
690 def close(self):
691 if self._sftp is not None:
692 self._sftp.close()
693 self._sftp = None
694 self._ssh.close()
695 self._ssh = None
0696
=== modified file 'lib/launchpad_loggerhead/app.py'
=== modified file 'lib/lp/services/config/schema-lazr.conf'
--- lib/lp/services/config/schema-lazr.conf 2016-09-02 18:21:07 +0000
+++ lib/lp/services/config/schema-lazr.conf 2016-09-24 06:42:11 +0000
@@ -864,6 +864,7 @@
864# datatype: boolean864# datatype: boolean
865launch: True865launch: True
866866
867<<<<<<< TREE
867# The root URL of the canonical OpenID provider. This is used when constructing868# The root URL of the canonical OpenID provider. This is used when constructing
868# OpenID URLs as user identifiers for external services such as gpgservice.869# OpenID URLs as user identifiers for external services such as gpgservice.
869# Since these user identifiers must remain constant forever, this setting must870# Since these user identifiers must remain constant forever, this setting must
@@ -878,6 +879,11 @@
878# for our accounts' OpenID identifiers. We only store the suffix, and879# for our accounts' OpenID identifiers. We only store the suffix, and
879# the suffix is provided by two providers.880# the suffix is provided by two providers.
880openid_alternate_provider_roots: none881openid_alternate_provider_roots: none
882=======
883# The root URL of the OpenID provider used to log into Launchpad.
884# datatype: string
885openid_provider_root: none
886>>>>>>> MERGE-SOURCE
881887
882# If true, the main template will be styled so that it is888# If true, the main template will be styled so that it is
883# obvious to the end user that they are using a demo system889# obvious to the end user that they are using a demo system
@@ -1700,26 +1706,23 @@
1700# datatype: boolean1706# datatype: boolean
1701use_https: True1707use_https: True
17021708
1703[vhost.template]1709# Base hostname for the Launchpad virtual hosts.
1704# Host name of this virtual host.
1705# This is matched from the incoming Host header, and
1706# also used to put together URLs if rooturl is not provided.
1707# Example: launchpad.net1710# Example: launchpad.net
1708# datatype: string1711base_hostname: none
1709hostname: none1712
17101713# Custom port for all virtual hosts.
1711# Alternative host names to match, in addition to1714# Only used by the test suite.
1712# the one given in hostname, comma separated.1715port: none
1713# Example: wwwww.launchpad.net, www.launchpad.net1716
1714# datatype: string1717# Hostnames for non-Launchpad virtual hosts.
1715althostnames: none1718# Presently only used in development and tests.
17161719vostok_hostname: none
1717# Explicit root URL for this virtual host.1720testopenid_hostname: none
1718# If this is not provided, the root URL is calculated1721
1719# based on the host name.1722# Should the apidoc virtual host be enabled?
1720# Example: https://launchpad.net/1723# This is the internal Zope apidoc, not public lazr.restful API docs.
1721# datatype: string1724enable_apidoc: False
1722rooturl: none1725
17231726
1724[vhost.mainsite]1727[vhost.mainsite]
1725# Can the profile page act as a OpenID delegated identity?1728# Can the profile page act as a OpenID delegated identity?
@@ -1728,6 +1731,7 @@
17281731
1729[vhost.api]1732[vhost.api]
17301733
1734<<<<<<< TREE
1731[vhost.blueprints]1735[vhost.blueprints]
17321736
1733[vhost.code]1737[vhost.code]
@@ -1749,6 +1753,11 @@
17491753
1750# Stubbed Key server for testing purposes. It can serve a restricted set of1754# Stubbed Key server for testing purposes. It can serve a restricted set of
1751# keys in SKS format.1755# keys in SKS format.
1756=======
1757
1758# Stubed Key server for test proposes, it's able to serve
1759# in SKS format, a restricted set of keys.
1760>>>>>>> MERGE-SOURCE
1752[testkeyserver]1761[testkeyserver]
1753# Directory to be created to store the pre-installed key-files1762# Directory to be created to store the pre-installed key-files
1754# datatype: string1763# datatype: string
17551764
=== modified file 'lib/lp/services/openid/adapters/openid.py'
--- lib/lp/services/openid/adapters/openid.py 2016-05-24 04:45:38 +0000
+++ lib/lp/services/openid/adapters/openid.py 2016-09-24 06:42:11 +0000
@@ -10,9 +10,26 @@
10 'OpenIDPersistentIdentity',10 'OpenIDPersistentIdentity',
11 ]11 ]
1212
13<<<<<<< TREE
13from zope.component import adapter14from zope.component import adapter
14from zope.interface import implementer15from zope.interface import implementer
16=======
17from zope.component import (
18 adapter,
19 adapts,
20 )
21from zope.interface import (
22 implementer,
23 implements,
24 )
25>>>>>>> MERGE-SOURCE
1526
27<<<<<<< TREE
28=======
29from canonical.config import config
30from canonical.launchpad.interfaces.account import IAccount
31from canonical.launchpad.interfaces.lpstorm import IStore
32>>>>>>> MERGE-SOURCE
16from lp.registry.interfaces.person import IPerson33from lp.registry.interfaces.person import IPerson
17from lp.services.config import config34from lp.services.config import config
18from lp.services.database.interfaces import IStore35from lp.services.database.interfaces import IStore
@@ -27,6 +44,7 @@
27 @staticmethod44 @staticmethod
28 def getServiceURL():45 def getServiceURL():
29 """The OpenID server URL (/+openid) for the current request."""46 """The OpenID server URL (/+openid) for the current request."""
47<<<<<<< TREE
30 return config.launchpad.openid_provider_root + '+openid'48 return config.launchpad.openid_provider_root + '+openid'
3149
32 @staticmethod50 @staticmethod
@@ -42,6 +60,11 @@
4260
43@adapter(IAccount)61@adapter(IAccount)
44@implementer(IOpenIDPersistentIdentity)62@implementer(IOpenIDPersistentIdentity)
63=======
64 return config.openid_provider_root + '+openid'
65
66
67>>>>>>> MERGE-SOURCE
45class OpenIDPersistentIdentity:68class OpenIDPersistentIdentity:
46 """A persistent OpenID identifier for a user."""69 """A persistent OpenID identifier for a user."""
4770
4871
=== modified file 'lib/lp/services/webapp/doc/webapp-publication.txt'
--- lib/lp/services/webapp/doc/webapp-publication.txt 2016-09-21 02:50:41 +0000
+++ lib/lp/services/webapp/doc/webapp-publication.txt 2016-09-24 06:42:11 +0000
@@ -71,12 +71,24 @@
71 rooturl: http://translations.launchpad.dev/71 rooturl: http://translations.launchpad.dev/
72 althosts:72 althosts:
73 ----73 ----
74<<<<<<< TREE
75=======
76 vostok @ vostok.dev
77 rooturl: http://vostok.dev/
78 althosts:
79 ----
80>>>>>>> MERGE-SOURCE
74 xmlrpc @ xmlrpc.launchpad.dev81 xmlrpc @ xmlrpc.launchpad.dev
75 rooturl: http://xmlrpc.launchpad.dev/82 rooturl: http://xmlrpc.launchpad.dev/
76 althosts:83 althosts:
77 ----84 ----
85<<<<<<< TREE
78 xmlrpc_private @ xmlrpc-private.launchpad.dev86 xmlrpc_private @ xmlrpc-private.launchpad.dev
79 rooturl: http://xmlrpc-private.launchpad.dev/87 rooturl: http://xmlrpc-private.launchpad.dev/
88=======
89 xmlrpc-private @ xmlrpc-private.launchpad.dev
90 rooturl: http://launchpad.dev/
91>>>>>>> MERGE-SOURCE
80 althosts:92 althosts:
81 ----93 ----
8294
@@ -96,6 +108,10 @@
96 localhost108 localhost
97 testopenid.dev109 testopenid.dev
98 translations.launchpad.dev110 translations.launchpad.dev
111<<<<<<< TREE
112=======
113 vostok.dev
114>>>>>>> MERGE-SOURCE
99 xmlrpc-private.launchpad.dev115 xmlrpc-private.launchpad.dev
100 xmlrpc.launchpad.dev116 xmlrpc.launchpad.dev
101117
102118
=== modified file 'lib/lp/services/webapp/login.py'
=== modified file 'lib/lp/services/webapp/metazcml.py'
--- lib/lp/services/webapp/metazcml.py 2015-10-14 15:22:01 +0000
+++ lib/lp/services/webapp/metazcml.py 2016-09-24 06:42:11 +0000
@@ -49,10 +49,15 @@
49from zope.security.proxy import ProxyFactory49from zope.security.proxy import ProxyFactory
50from zope.security.zcml import IPermissionDirective50from zope.security.zcml import IPermissionDirective
5151
52<<<<<<< TREE
52from lp.app.interfaces.security import IAuthorization53from lp.app.interfaces.security import IAuthorization
53from lp.layers import FeedsLayer54from lp.layers import FeedsLayer
54from lp.services.config import config55from lp.services.config import config
55from lp.services.webapp.interfaces import (56from lp.services.webapp.interfaces import (
57=======
58from canonical.launchpad.layers import FeedsLayer
59from canonical.launchpad.webapp.interfaces import (
60>>>>>>> MERGE-SOURCE
56 IApplicationMenu,61 IApplicationMenu,
57 ICanonicalUrlData,62 ICanonicalUrlData,
58 IContextMenu,63 IContextMenu,
@@ -60,7 +65,13 @@
60 IFavicon,65 IFavicon,
61 INavigationMenu,66 INavigationMenu,
62 )67 )
68<<<<<<< TREE
63from lp.services.webapp.publisher import RenamedView69from lp.services.webapp.publisher import RenamedView
70=======
71from canonical.launchpad.webapp.publisher import RenamedView
72from canonical.launchpad.webapp.vhosts import allvhosts
73from lp.app.interfaces.security import IAuthorization
74>>>>>>> MERGE-SOURCE
6475
6576
66class IAuthorizationsDirective(Interface):77class IAuthorizationsDirective(Interface):
@@ -617,9 +628,7 @@
617 # supplied -- we don't care about the priority in Launchpad but it628 # supplied -- we don't care about the priority in Launchpad but it
618 # needs to be unique -- and to do nothing if no hostname is629 # needs to be unique -- and to do nothing if no hostname is
619 # configured for this publisher.630 # configured for this publisher.
620631 if name not in allvhosts.configs:
621 section = getattr(config.vhost, name, None)
622 if section is None or section.hostname is None:
623 return632 return
624 global _arbitrary_priority633 global _arbitrary_priority
625 if priority is None:634 if priority is None:
626635
=== modified file 'lib/lp/services/webapp/servers.py'
--- lib/lp/services/webapp/servers.py 2016-09-14 11:13:06 +0000
+++ lib/lp/services/webapp/servers.py 2016-09-24 06:42:11 +0000
@@ -1434,6 +1434,7 @@
1434 ErrorReportRequest):1434 ErrorReportRequest):
1435 """Request type for doing public XML-RPC in Launchpad."""1435 """Request type for doing public XML-RPC in Launchpad."""
14361436
1437<<<<<<< TREE
1437 def getRootURL(self, rootsite):1438 def getRootURL(self, rootsite):
1438 """See IBasicLaunchpadRequest."""1439 """See IBasicLaunchpadRequest."""
1439 # XML-RPC requests occasionally need to use canonical_url, for1440 # XML-RPC requests occasionally need to use canonical_url, for
@@ -1444,6 +1445,18 @@
1444 rootsite = 'mainsite'1445 rootsite = 'mainsite'
1445 return super(PublicXMLRPCRequest, self).getRootURL(rootsite)1446 return super(PublicXMLRPCRequest, self).getRootURL(rootsite)
14461447
1448=======
1449 def getRootURL(self, rootsite):
1450 """See IBasicLaunchpadRequest."""
1451 # XML-RPC requests occasionally need to use canonical_url, for
1452 # the likes of sending emails. Until these are tracked down and
1453 # fixed to use mainsite explicitly, replace the XML-RPC root
1454 # URLs with mainsite's, so that URLs are meaningful.
1455 if rootsite in (None, 'xmlrpc', 'xmlrpc-private'):
1456 rootsite = 'mainsite'
1457 return super(PublicXMLRPCRequest, self).getRootURL(rootsite)
1458
1459>>>>>>> MERGE-SOURCE
1447 def _createResponse(self):1460 def _createResponse(self):
1448 return PublicXMLRPCResponse()1461 return PublicXMLRPCResponse()
14491462
@@ -1584,7 +1597,7 @@
15841597
1585 if private_port is not None:1598 if private_port is not None:
1586 factories.append(XMLRPCRequestPublicationFactory(1599 factories.append(XMLRPCRequestPublicationFactory(
1587 'xmlrpc_private', PrivateXMLRPCRequest,1600 'xmlrpc-private', PrivateXMLRPCRequest,
1588 PrivateXMLRPCPublication, port=private_port))1601 PrivateXMLRPCPublication, port=private_port))
15891602
1590 # Register those factories, in priority order corresponding to1603 # Register those factories, in priority order corresponding to
15911604
=== modified file 'lib/lp/services/webapp/vhosts.py'
--- lib/lp/services/webapp/vhosts.py 2015-10-14 15:22:01 +0000
+++ lib/lp/services/webapp/vhosts.py 2016-09-24 06:42:11 +0000
@@ -3,56 +3,28 @@
33
4"""Virtual host handling for the Launchpad webapp."""4"""Virtual host handling for the Launchpad webapp."""
55
6__all__ = ['allvhosts']6__all__ = [
7 'allvhosts',
8 'AllVirtualHostsConfiguration',
9 ]
710
811
9class VirtualHostConfig:12class VirtualHostConfig:
10 """The configuration of a single virtual host."""13 """The configuration of a single virtual host."""
1114
12 def __init__(self, hostname, althostnames, rooturl, use_https):15 def __init__(self, hostname, althostnames, use_https, port=None):
13 if althostnames is None:16 if use_https:
14 althostnames = []17 protocol = 'https'
15 else:18 else:
16 althostnames = self._hostnameStrToList(althostnames)19 protocol = 'http'
1720 rooturl = (
18 if rooturl is None:21 '%s://%s%s/' %
19 if use_https:22 (protocol, hostname, (':%d' % port) if port is not None else ''))
20 protocol = 'https'
21 else:
22 protocol = 'http'
23 rooturl = '%s://%s/' % (protocol, hostname)
2423
25 self.hostname = hostname24 self.hostname = hostname
26 self.rooturl = rooturl25 self.rooturl = rooturl
27 self.althostnames = althostnames26 self.althostnames = althostnames
2827
29 @staticmethod
30 def _hostnameStrToList(althostnames):
31 """Return list of hostname strings given a string of althostnames.
32
33 This is to parse althostnames from the launchpad.conf file.
34
35 Basically, it's a comma separated list, but we're quite flexible
36 about what is accepted. See the examples in the following doctest.
37
38 >>> thismethod = VirtualHostConfig._hostnameStrToList
39 >>> thismethod('foo')
40 ['foo']
41 >>> thismethod('foo,bar, baz')
42 ['foo', 'bar', 'baz']
43 >>> thismethod('foo,,bar, ,baz ,')
44 ['foo', 'bar', 'baz']
45 >>> thismethod('')
46 []
47 >>> thismethod(' ')
48 []
49
50 """
51 if not althostnames.strip():
52 return []
53 return [
54 name.strip() for name in althostnames.split(',') if name.strip()]
55
5628
57class AllVirtualHostsConfiguration:29class AllVirtualHostsConfiguration:
58 """A representation of the virtual hosting configuration for30 """A representation of the virtual hosting configuration for
@@ -69,7 +41,7 @@
69 self.hostnames : set of hostnames handled by the vhost config41 self.hostnames : set of hostnames handled by the vhost config
70 """42 """
7143
72 def __init__(self):44 def __init__(self, config=None):
73 """Initialize all virtual host settings from launchpad.conf.45 """Initialize all virtual host settings from launchpad.conf.
7446
75 launchpad_conf_vhosts: The virtual_hosts config item from47 launchpad_conf_vhosts: The virtual_hosts config item from
@@ -77,27 +49,60 @@
7749
78 """50 """
79 self._has_vhost_data = False51 self._has_vhost_data = False
8052 self._config = config
53
54<<<<<<< TREE
55=======
56 def _addVHost(self, vhost, hostname, use_https=True, althostnames=None):
57 self._configs[vhost] = VirtualHostConfig(
58 hostname, althostnames or [], self._use_https and use_https,
59 self._port)
60
61>>>>>>> MERGE-SOURCE
81 def _getVHostData(self):62 def _getVHostData(self):
82 """Parse the vhosts on demand."""63 """Construct the vhosts on demand."""
83 # Avoid the circular imports inherent with the use of canonical.lazr.64 # Avoid the circular imports inherent with the use of canonical.lazr.
84 if self._has_vhost_data:65 if self._has_vhost_data:
85 return66 return
67<<<<<<< TREE
86 from lp.services.config import config68 from lp.services.config import config
69=======
70 if self._config is None:
71 import canonical.config
72 self._config = canonical.config.config
73 config = self._config
74>>>>>>> MERGE-SOURCE
87 self._use_https = config.vhosts.use_https75 self._use_https = config.vhosts.use_https
76 self._port = config.vhosts.port
88 self._configs = {}77 self._configs = {}
89 self._hostnames = set()78 self._hostnames = set()
90 for section in config.getByCategory('vhost'):79
91 if section.hostname is None:80 # Most vhosts are just subdomains of base_hostname.
92 continue81 normal_vhosts = [
93 category, vhost = section.category_and_section_names82 'api', 'xmlrpc', 'xmlrpc-private',
94 self._configs[vhost] = config = VirtualHostConfig(83 'answers', 'blueprints', 'bugs', 'code', 'translations']
95 section.hostname,84 if config.vhosts.enable_apidoc:
96 section.althostnames,85 normal_vhosts.append('apidoc')
97 section.rooturl,86 for vhost in normal_vhosts:
98 self._use_https)87 self._addVHost(
99 self._hostnames.add(config.hostname)88 vhost, '%s.%s' % (vhost, config.vhosts.base_hostname))
100 self._hostnames.update(config.althostnames)89 # mainsite is base_hostname (eg. launchpad.dev), with an optional
90 # www subdomain.
91 self._addVHost(
92 'mainsite', config.vhosts.base_hostname,
93 althostnames=['www.' + config.vhosts.base_hostname, 'localhost'])
94 # feeds never uses SSL, but is otherwise normal.
95 self._addVHost(
96 'feeds', 'feeds.' + config.vhosts.base_hostname, use_https=False)
97
98 if config.vhosts.testopenid_hostname:
99 self._addVHost('testopenid', config.vhosts.testopenid_hostname)
100 if config.vhosts.vostok_hostname:
101 self._addVHost('vostok', config.vhosts.vostok_hostname)
102
103 for conf in self._configs.itervalues():
104 self._hostnames.add(conf.hostname)
105 self._hostnames.update(conf.althostnames)
101 self._has_vhost_data = True106 self._has_vhost_data = True
102107
103 def reload(self):108 def reload(self):
104109
=== modified file 'lib/lp/testing/layers.py'
--- lib/lp/testing/layers.py 2016-02-22 23:43:33 +0000
+++ lib/lp/testing/layers.py 2016-09-24 06:42:11 +0000
@@ -113,8 +113,35 @@
113 ConfigFixture,113 ConfigFixture,
114 ConfigUseFixture,114 ConfigUseFixture,
115 )115 )
116<<<<<<< TREE
116from lp.services.database.interfaces import IStore117from lp.services.database.interfaces import IStore
117from lp.services.database.sqlbase import session_store118from lp.services.database.sqlbase import session_store
119=======
120from canonical.database.sqlbase import session_store
121from canonical.launchpad.scripts import execute_zcml_for_scripts
122from canonical.launchpad.webapp.interfaces import (
123 DEFAULT_FLAVOR,
124 IOpenLaunchBag,
125 IStoreSelector,
126 MAIN_STORE,
127 )
128from canonical.launchpad.webapp.servers import (
129 LaunchpadAccessLogger,
130 register_launchpad_request_publication_factories,
131 )
132import canonical.launchpad.webapp.session
133from canonical.launchpad.webapp.vhosts import AllVirtualHostsConfiguration
134from canonical.lazr import pidfile
135from canonical.lazr.testing.layers import MockRootFolder
136from canonical.lazr.timeout import (
137 get_default_timeout_function,
138 set_default_timeout_function,
139 )
140from canonical.librarian.testing.server import LibrarianServerFixture
141from canonical.testing import reset_logging
142from canonical.testing.profiled import profiled
143from canonical.testing.smtpd import SMTPController
144>>>>>>> MERGE-SOURCE
118from lp.services.googlesearch.tests.googleserviceharness import (145from lp.services.googlesearch.tests.googleserviceharness import (
119 GoogleServiceTestSetup,146 GoogleServiceTestSetup,
120 )147 )
@@ -1850,7 +1877,8 @@
18501877
1851 @classmethod1878 @classmethod
1852 def appserver_root_url(cls):1879 def appserver_root_url(cls):
1853 return cls.appserver_config.vhost.mainsite.rooturl1880 return AllVirtualHostsConfiguration(
1881 cls.appserver_config).configs['mainsite'].rooturl
18541882
1855 @classmethod1883 @classmethod
1856 def _waitUntilAppServerIsReady(cls):1884 def _waitUntilAppServerIsReady(cls):
18571885
=== modified file 'lib/lp/testing/tests/test_layers_functional.py'
--- lib/lp/testing/tests/test_layers_functional.py 2015-10-13 14:01:25 +0000
+++ lib/lp/testing/tests/test_layers_functional.py 2016-09-24 06:42:11 +0000
@@ -32,8 +32,15 @@
32 getUtility,32 getUtility,
33 )33 )
3434
35<<<<<<< TREE
35from lp.services.config import config36from lp.services.config import config
36from lp.services.librarian.client import (37from lp.services.librarian.client import (
38=======
39from canonical.config import config
40from canonical.launchpad.webapp.vhosts import AllVirtualHostsConfiguration
41from canonical.lazr.pidfile import pidfile_path
42from canonical.librarian.client import (
43>>>>>>> MERGE-SOURCE
37 LibrarianClient,44 LibrarianClient,
38 UploadFailed,45 UploadFailed,
39 )46 )
@@ -498,7 +505,8 @@
498505
499 def testAppServerIsAvailable(self):506 def testAppServerIsAvailable(self):
500 # Test that the app server is up and running.507 # Test that the app server is up and running.
501 mainsite = LayerProcessController.appserver_config.vhost.mainsite508 mainsite = AllVirtualHostsConfiguration(
509 LayerProcessController.appserver_config).configs['mainsite']
502 home_page = urlopen(mainsite.rooturl).read()510 home_page = urlopen(mainsite.rooturl).read()
503 self.failUnless(511 self.failUnless(
504 'Is your project registered yet?' in home_page,512 'Is your project registered yet?' in home_page,
505513
=== modified file 'utilities/rocketfuel-setup'