Launchpad itself

Merge lp:~stub/launchpad/replication into lp:launchpad/db-devel

  1. replication
  2. Merge into db-devel
Proposed by Stuart Bishop
Status: Merged
Merged at revision: not available
Proposed branch: lp:~stub/launchpad/replication
Merge into: lp:launchpad/db-devel
Diff against target: 108 lines (+51/-3)
4 files modified
database/replication/Makefile (+3/-1)
database/replication/helpers.py (+20/-1)
database/replication/initialize.py (+2/-1)
database/replication/sync.py (+26/-0)
To merge this branch: bzr merge lp:~stub/launchpad/replication
Reviewer Review Type Date Requested Status
Jonathan Lange (community) Approve
Review via email: mp+16898@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Stuart Bishop (stub) wrote :

A standalone authentication service is being assembled, and needs to be tested on staging (it has to be staging as things are still partially integrated). There are some extra tables and data that is contained in the database dumps, but we need to teach the staging rebuild scripts to not mess with them by trying to stuff them into the Launchpad replication sets.

Revision history for this message
Jonathan Lange (jml) wrote :

Hey Stuart,

The branch looks good to me. A couple of minor points though.

 * In database/replication/helpers.py, the new items in the list should probably be newline separated
 * In the same file, it's not clear how you figured out those table names, nor is it clear how I'd find out how you figured them out. Would it be possible to expand the comment explaining how they were determined?

jml

review: Needs Fixing
Revision history for this message
Jonathan Lange (jml) wrote :

Thanks. You've spelled "separate" incorrectly. Other than that, looks fine.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'database/replication/Makefile'
2--- database/replication/Makefile 2009-12-14 13:00:03 +0000
3+++ database/replication/Makefile 2010-01-08 05:18:15 +0000
4@@ -100,7 +100,7 @@
5 # up when roles don't exist in this cluster, and we rebuild it later
6 # with security.py anyway.
7 pg_restore --dbname=lpmain_staging_new \
8- --no-acl --no-owner --exit-on-error ${STAGING_DUMP}
9+ --no-owner --exit-on-error ${STAGING_DUMP}
10 psql -q -d lpmain_staging_new -f authdb_drop.sql
11 psql -q -d lpmain_staging_new -f authdb_create.sql \
12 2>&1 | grep -v _sl || true
13@@ -187,7 +187,9 @@
14 @echo Running fti.py `date`
15 ${SHHH} ../schema/fti.py
16 @echo Running security.py `date`
17+ ./slon_ctl.py stop # security.py can deadlock with slony
18 ${SHHH} ../schema/security.py --cluster -U slony
19+ ./slon_ctl.py --lag="0 seconds" start
20 # Migrate tables to the authdb replication set, creating the set
21 # and subscribing nodes to it as necessary.
22 ./populate_auth_replication_set.py -U slony
23
24=== modified file 'database/replication/helpers.py'
25--- database/replication/helpers.py 2009-11-30 11:35:04 +0000
26+++ database/replication/helpers.py 2010-01-08 05:18:15 +0000
27@@ -71,8 +71,27 @@
28 'public.lp_person',
29 'public.lp_personlocation',
30 'public.lp_teamparticipation',
31+ # Ubuntu SSO database. These tables where created manually by ISD
32+ # and the Launchpad scripts should not mess with them. Eventually
33+ # these tables will be in a totally separate database.
34+ 'public.auth_permission',
35+ 'public.auth_group',
36+ 'public.auth_user',
37+ 'public.auth_message',
38+ 'public.django_content_type',
39+ 'public.auth_permission',
40+ 'public.django_session',
41+ 'public.django_site',
42+ 'public.django_admin_log',
43+ 'public.ssoopenidrpconfig',
44+ 'public.auth_group_permissions',
45+ 'public.auth_user_groups',
46+ 'public.auth_user_user_permissions',
47 ])
48
49+# Calculate IGNORED_SEQUENCES
50+IGNORED_SEQUENCES = set('%s_id_seq' % table for table in IGNORED_TABLES)
51+
52
53 def slony_installed(con):
54 """Return True if the connected database is part of a Launchpad Slony-I
55@@ -447,7 +466,7 @@
56
57 return (
58 all_tables - replicated_tables - IGNORED_TABLES,
59- all_sequences - replicated_sequences)
60+ all_sequences - replicated_sequences - IGNORED_SEQUENCES)
61
62
63 class ReplicationConfigError(Exception):
64
65=== modified file 'database/replication/initialize.py'
66--- database/replication/initialize.py 2009-10-17 14:06:03 +0000
67+++ database/replication/initialize.py 2010-01-08 05:18:15 +0000
68@@ -224,7 +224,8 @@
69 fails += 1
70 for sequence in all_sequences_in_schema(cur, 'public'):
71 times_seen = 0
72- for sequence_set in [authdb_sequences, lpmain_sequences]:
73+ for sequence_set in [
74+ authdb_sequences, lpmain_sequences, helpers.IGNORED_SEQUENCES]:
75 if sequence in sequence_set:
76 times_seen += 1
77 if times_seen == 0:
78
79=== added file 'database/replication/sync.py'
80--- database/replication/sync.py 1970-01-01 00:00:00 +0000
81+++ database/replication/sync.py 2010-01-08 05:18:15 +0000
82@@ -0,0 +1,26 @@
83+#!/usr/bin/python2.5
84+#
85+# Copyright 2010 Canonical Ltd. This software is licensed under the
86+# GNU Affero General Public License version 3 (see the file LICENSE).
87+
88+"""Block until the replication cluster synchronizes."""
89+
90+__metaclass__ = type
91+__all__ = []
92+
93+import _pythonpath
94+
95+from optparse import OptionParser
96+
97+from canonical.launchpad.scripts import logger_options, db_options
98+from replication.helpers import sync
99+
100+if __name__ == '__main__':
101+ parser = OptionParser()
102+ parser.add_option(
103+ "-t", "--timeout", dest="timeout", metavar="SECS", type="int",
104+ help="Abort if no sync after SECS seconds.", default=0)
105+ logger_options(parser)
106+ db_options(parser)
107+ options, args = parser.parse_args()
108+ sync(options.timeout)

Subscribers

People subscribed via source and target branches

to status/vote changes: