Launchpad itself

Merge lp:~stub/launchpad/replication into lp:launchpad/db-devel

  1. replication
  2. Merge into db-devel
Proposed by Stuart Bishop
Status: Merged
Merged at revision: not available
Proposed branch: lp:~stub/launchpad/replication
Merge into: lp:launchpad/db-devel
Diff against target: 108 lines (+51/-3)
4 files modified
database/replication/Makefile (+3/-1)
database/replication/helpers.py (+20/-1)
database/replication/initialize.py (+2/-1)
database/replication/sync.py (+26/-0)
To merge this branch: bzr merge lp:~stub/launchpad/replication
Reviewer Review Type Date Requested Status
Jonathan Lange (community) Approve
Review via email: mp+16898@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Stuart Bishop (stub) wrote :

A standalone authentication service is being assembled, and needs to be tested on staging (it has to be staging as things are still partially integrated). There are some extra tables and data that is contained in the database dumps, but we need to teach the staging rebuild scripts to not mess with them by trying to stuff them into the Launchpad replication sets.

Revision history for this message
Jonathan Lange (jml) wrote :

Hey Stuart,

The branch looks good to me. A couple of minor points though.

 * In database/replication/helpers.py, the new items in the list should probably be newline separated
 * In the same file, it's not clear how you figured out those table names, nor is it clear how I'd find out how you figured them out. Would it be possible to expand the comment explaining how they were determined?

jml

review: Needs Fixing
Revision history for this message
Jonathan Lange (jml) wrote :

Thanks. You've spelled "separate" incorrectly. Other than that, looks fine.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'database/replication/Makefile'
--- database/replication/Makefile 2009-12-14 13:00:03 +0000
+++ database/replication/Makefile 2010-01-08 05:18:15 +0000
@@ -100,7 +100,7 @@
100 # up when roles don't exist in this cluster, and we rebuild it later100 # up when roles don't exist in this cluster, and we rebuild it later
101 # with security.py anyway.101 # with security.py anyway.
102 pg_restore --dbname=lpmain_staging_new \102 pg_restore --dbname=lpmain_staging_new \
103 --no-acl --no-owner --exit-on-error ${STAGING_DUMP}103 --no-owner --exit-on-error ${STAGING_DUMP}
104 psql -q -d lpmain_staging_new -f authdb_drop.sql104 psql -q -d lpmain_staging_new -f authdb_drop.sql
105 psql -q -d lpmain_staging_new -f authdb_create.sql \105 psql -q -d lpmain_staging_new -f authdb_create.sql \
106 2>&1 | grep -v _sl || true106 2>&1 | grep -v _sl || true
@@ -187,7 +187,9 @@
187 @echo Running fti.py `date`187 @echo Running fti.py `date`
188 ${SHHH} ../schema/fti.py188 ${SHHH} ../schema/fti.py
189 @echo Running security.py `date`189 @echo Running security.py `date`
190 ./slon_ctl.py stop # security.py can deadlock with slony
190 ${SHHH} ../schema/security.py --cluster -U slony191 ${SHHH} ../schema/security.py --cluster -U slony
192 ./slon_ctl.py --lag="0 seconds" start
191 # Migrate tables to the authdb replication set, creating the set193 # Migrate tables to the authdb replication set, creating the set
192 # and subscribing nodes to it as necessary.194 # and subscribing nodes to it as necessary.
193 ./populate_auth_replication_set.py -U slony195 ./populate_auth_replication_set.py -U slony
194196
=== modified file 'database/replication/helpers.py'
--- database/replication/helpers.py 2009-11-30 11:35:04 +0000
+++ database/replication/helpers.py 2010-01-08 05:18:15 +0000
@@ -71,8 +71,27 @@
71 'public.lp_person',71 'public.lp_person',
72 'public.lp_personlocation',72 'public.lp_personlocation',
73 'public.lp_teamparticipation',73 'public.lp_teamparticipation',
74 # Ubuntu SSO database. These tables where created manually by ISD
75 # and the Launchpad scripts should not mess with them. Eventually
76 # these tables will be in a totally separate database.
77 'public.auth_permission',
78 'public.auth_group',
79 'public.auth_user',
80 'public.auth_message',
81 'public.django_content_type',
82 'public.auth_permission',
83 'public.django_session',
84 'public.django_site',
85 'public.django_admin_log',
86 'public.ssoopenidrpconfig',
87 'public.auth_group_permissions',
88 'public.auth_user_groups',
89 'public.auth_user_user_permissions',
74 ])90 ])
7591
92# Calculate IGNORED_SEQUENCES
93IGNORED_SEQUENCES = set('%s_id_seq' % table for table in IGNORED_TABLES)
94
7695
77def slony_installed(con):96def slony_installed(con):
78 """Return True if the connected database is part of a Launchpad Slony-I97 """Return True if the connected database is part of a Launchpad Slony-I
@@ -447,7 +466,7 @@
447466
448 return (467 return (
449 all_tables - replicated_tables - IGNORED_TABLES,468 all_tables - replicated_tables - IGNORED_TABLES,
450 all_sequences - replicated_sequences)469 all_sequences - replicated_sequences - IGNORED_SEQUENCES)
451470
452471
453class ReplicationConfigError(Exception):472class ReplicationConfigError(Exception):
454473
=== modified file 'database/replication/initialize.py'
--- database/replication/initialize.py 2009-10-17 14:06:03 +0000
+++ database/replication/initialize.py 2010-01-08 05:18:15 +0000
@@ -224,7 +224,8 @@
224 fails += 1224 fails += 1
225 for sequence in all_sequences_in_schema(cur, 'public'):225 for sequence in all_sequences_in_schema(cur, 'public'):
226 times_seen = 0226 times_seen = 0
227 for sequence_set in [authdb_sequences, lpmain_sequences]:227 for sequence_set in [
228 authdb_sequences, lpmain_sequences, helpers.IGNORED_SEQUENCES]:
228 if sequence in sequence_set:229 if sequence in sequence_set:
229 times_seen += 1230 times_seen += 1
230 if times_seen == 0:231 if times_seen == 0:
231232
=== added file 'database/replication/sync.py'
--- database/replication/sync.py 1970-01-01 00:00:00 +0000
+++ database/replication/sync.py 2010-01-08 05:18:15 +0000
@@ -0,0 +1,26 @@
1#!/usr/bin/python2.5
2#
3# Copyright 2010 Canonical Ltd. This software is licensed under the
4# GNU Affero General Public License version 3 (see the file LICENSE).
5
6"""Block until the replication cluster synchronizes."""
7
8__metaclass__ = type
9__all__ = []
10
11import _pythonpath
12
13from optparse import OptionParser
14
15from canonical.launchpad.scripts import logger_options, db_options
16from replication.helpers import sync
17
18if __name__ == '__main__':
19 parser = OptionParser()
20 parser.add_option(
21 "-t", "--timeout", dest="timeout", metavar="SECS", type="int",
22 help="Abort if no sync after SECS seconds.", default=0)
23 logger_options(parser)
24 db_options(parser)
25 options, args = parser.parse_args()
26 sync(options.timeout)

Subscribers

People subscribed via source and target branches

to status/vote changes: