Launchpad itself

Merge lp:~stub/launchpad/dbpolicy-syntax into lp:launchpad

  1. dbpolicy-syntax
  2. Merge into devel
Proposed by Stuart Bishop
Status: Merged
Approved by: Aaron Bentley
Approved revision: not available
Merged at revision: not available
Proposed branch: lp:~stub/launchpad/dbpolicy-syntax
Merge into: lp:launchpad
Diff against target: 211 lines (+160/-0)
5 files modified
lib/canonical/launchpad/doc/db-policy.txt (+126/-0)
lib/canonical/launchpad/doc/storm.txt (+5/-0)
lib/canonical/launchpad/ftests/test_system_documentation.py (+4/-0)
lib/canonical/launchpad/webapp/dbpolicy.py (+11/-0)
lib/canonical/launchpad/webapp/interfaces.py (+14/-0)
To merge this branch: bzr merge lp:~stub/launchpad/dbpolicy-syntax
Reviewer Review Type Date Requested Status
Aaron Bentley (community) Approve
Review via email: mp+19854@code.launchpad.net

Commit message

Turn database policies into Python context managers for use with the 'with' statement

To post a comment you must log in.
Revision history for this message
Stuart Bishop (stub) wrote :

People have been asking how to get scripts making use of the slave databases (yay!). This can already be done, but I'm taking the opportunity to improve the syntax of installing database policies and thus nicer, readable documentation (yay!).

lp:~stub/launchpad/dbpolicy-syntax updated
10357. By Stuart Bishop

Interfaces don't have self arguments

10358. By Stuart Bishop

typos

Revision history for this message
Aaron Bentley (abentley) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added file 'lib/canonical/launchpad/doc/db-policy.txt'
2--- lib/canonical/launchpad/doc/db-policy.txt 1970-01-01 00:00:00 +0000
3+++ lib/canonical/launchpad/doc/db-policy.txt 2010-02-22 12:18:22 +0000
4@@ -0,0 +1,126 @@
5+Storm Stores & Database Policies
6+================================
7+
8+Launchpad has multiple master and slave databases. Changes to data are
9+made on the master and replicated asynchronously to the slave
10+databases. Slave databases will usually lag a few seconds behind their
11+master. Under high load they may lag a few minutes behind, during
12+maintenance they may lag a few hours behind and if things explode
13+while admins are on holiday they may lag days behind.
14+
15+If know your code needs to change data, or must have the latest posible
16+information, you retrieve objects from the master databases that stores
17+the data for your database class.
18+
19+ >>> from canonical.launchpad.interfaces.lpstorm import IMasterStore
20+ >>> from lp.registry.model.person import Person
21+ >>> import transaction
22+
23+ >>> writable_janitor = IMasterStore(Person).find(
24+ ... Person, Person.name == 'janitor').one()
25+
26+ >>> writable_janitor.displayname = 'Jack the Janitor'
27+ >>> transaction.commit()
28+
29+Sometimes though we know we will not make changes and don't care much
30+if the information is a little out of date. In these cases you should
31+explicitly retrieve objects from a slave.
32+
33+The more agressively we retrieve objects from slave databases instead
34+of the master, the better the overall performance of Launchpad will be.
35+We can distribute this load over many slave databases but are limited to
36+a single master.
37+
38+ >>> from canonical.launchpad.interfaces.lpstorm import ISlaveStore
39+ >>> ro_janitor = ISlaveStore(Person).find(
40+ ... Person, Person.name == 'janitor').one()
41+ >>> ro_janitor is writable_janitor
42+ False
43+
44+ >>> ro_janitor.displayname = 'Janice the Janitor'
45+ >>> transaction.commit()
46+ Traceback (most recent call last):
47+ ...
48+ InternalError: transaction is read-only
49+
50+ >>> transaction.abort()
51+
52+Much of our code does not know if the objects being retrieved need to be
53+updatable to or have to be absolutely up to date. In this case, we
54+retrieve objects from the default store. What object being returned
55+depends on the currently installed database policy.
56+
57+ >>> from canonical.launchpad.interfaces.lpstorm import IStore
58+ >>> default_janitor = IStore(Person).find(
59+ ... Person, Person.name == 'janitor').one()
60+ >>> default_janitor is writable_janitor
61+ True
62+
63+As you can see, the default database policy retrieves objects from
64+the master database. This allows our code written before database
65+replication was implemented to keep working.
66+
67+To alter this behavior, you can install a different database policy.
68+
69+ >>> from canonical.launchpad.webapp.dbpolicy import SlaveDatabasePolicy
70+ >>> with SlaveDatabasePolicy():
71+ ... default_janitor = IStore(Person).find(
72+ ... Person, Person.name == 'janitor').one()
73+ >>> default_janitor is writable_janitor
74+ False
75+
76+The database policy can also affect what happens when objects are
77+explicitly retrieved from a slave or master database. For example,
78+if we have code that needs to run during database maintenance or
79+code we want to prove only accesses slave database resources, we can
80+raise an exception if an attempt is made to access master database
81+resources.
82+
83+ >>> from canonical.launchpad.webapp.dbpolicy import (
84+ ... SlaveOnlyDatabasePolicy)
85+ >>> with SlaveOnlyDatabasePolicy():
86+ ... whoops = IMasterStore(Person).find(
87+ ... Person, Person.name == 'janitor').one()
88+ Traceback (most recent call last):
89+ ...
90+ DisallowedStore: master
91+
92+We can even ensure no database activity occurs at all, for instance
93+if we need to guarantee a potentially long running call doesn't access
94+the database at all starting a new and potentially long running
95+database transaction.
96+
97+ >>> from canonical.launchpad.webapp.dbpolicy import DatabaseBlockedPolicy
98+ >>> with DatabaseBlockedPolicy():
99+ ... whoops = IStore(Person).find(
100+ ... Person, Person.name == 'janitor').one()
101+ Traceback (most recent call last):
102+ ...
103+ DisallowedStore: ('main', 'default')
104+
105+Database policies can also be installed and uninstalled using the
106+IStoreSelector utility for cases where the 'with' syntax cannot
107+be used.
108+
109+ >>> from canonical.launchpad.webapp.interfaces import IStoreSelector
110+ >>> getUtility(IStoreSelector).push(SlaveDatabasePolicy())
111+ >>> try:
112+ ... default_janitor = IStore(Person).find(
113+ ... Person, Person.name == 'janitor').one()
114+ ... finally:
115+ ... db_policy = getUtility(IStoreSelector).pop()
116+ >>> default_janitor is ro_janitor
117+ True
118+
119+Casting
120+-------
121+
122+If you need to change an object you have a read only copy of, or are
123+unsure if the object is writable or not, you can easily cast it
124+to a writable copy. This is a noop if the object is already writable
125+so is good defensive programming.
126+
127+ >>> from canonical.launchpad.interfaces.lpstorm import IMasterObject
128+ >>> IMasterObject(ro_janitor) is writable_janitor
129+ True
130+
131
132=== modified file 'lib/canonical/launchpad/doc/storm.txt'
133--- lib/canonical/launchpad/doc/storm.txt 2009-08-21 17:43:28 +0000
134+++ lib/canonical/launchpad/doc/storm.txt 2010-02-22 12:18:22 +0000
135@@ -1,3 +1,8 @@
136+Note: A more readable version of this is in db-policy.txt. Most of this
137+doctest will disappear soon when the auth replication set is collapsed
138+back into the main replication set as part of login server seperation.
139+-- StuartBishop 20100222
140+
141 In addition to what Storm provides, we also have some Launchpad
142 specific Storm tools to cope with our master and slave store arrangement.
143
144
145=== modified file 'lib/canonical/launchpad/ftests/test_system_documentation.py'
146--- lib/canonical/launchpad/ftests/test_system_documentation.py 2010-02-10 23:14:56 +0000
147+++ lib/canonical/launchpad/ftests/test_system_documentation.py 2010-02-22 12:18:22 +0000
148@@ -7,6 +7,8 @@
149 """
150 # pylint: disable-msg=C0103
151
152+from __future__ import with_statement
153+
154 import logging
155 import os
156 import unittest
157@@ -391,6 +393,8 @@
158 one_test = LayeredDocFileSuite(
159 path, setUp=setUp, tearDown=tearDown,
160 layer=LaunchpadFunctionalLayer,
161+ # 'icky way of running doctests with __future__ imports
162+ globs={'with_statement': with_statement},
163 stdout_logging_level=logging.WARNING
164 )
165 suite.addTest(one_test)
166
167=== modified file 'lib/canonical/launchpad/webapp/dbpolicy.py'
168--- lib/canonical/launchpad/webapp/dbpolicy.py 2010-01-20 22:09:26 +0000
169+++ lib/canonical/launchpad/webapp/dbpolicy.py 2010-02-22 12:18:22 +0000
170@@ -111,6 +111,17 @@
171 """See `IDatabasePolicy`."""
172 pass
173
174+ def __enter__(self):
175+ """See `IDatabasePolicy`."""
176+ getUtility(IStoreSelector).push(self)
177+
178+ def __exit__(self, exc_type, exc_value, traceback):
179+ """See `IDatabasePolicy`."""
180+ policy = getUtility(IStoreSelector).pop()
181+ assert policy is self, (
182+ "Unexpected database policy %s returned by store selector"
183+ % repr(policy))
184+
185
186 class DatabaseBlockedPolicy(BaseDatabasePolicy):
187 """`IDatabasePolicy` that blocks all access to the database."""
188
189=== modified file 'lib/canonical/launchpad/webapp/interfaces.py'
190--- lib/canonical/launchpad/webapp/interfaces.py 2010-02-17 11:13:06 +0000
191+++ lib/canonical/launchpad/webapp/interfaces.py 2010-02-22 12:18:22 +0000
192@@ -756,6 +756,20 @@
193 The publisher adapts the request to `IDatabasePolicy` to
194 instantiate the policy for the current request.
195 """
196+ def __enter__():
197+ """Standard Python context manager interface.
198+
199+ The IDatabasePolicy will install itself using the IStoreSelector
200+ utility.
201+ """
202+
203+ def __exit__(exc_type, exc_value, traceback):
204+ """Standard Python context manager interface.
205+
206+ The IDatabasePolicy will uninstall itself using the IStoreSelector
207+ utility.
208+ """
209+
210 def getStore(name, flavor):
211 """Retrieve a Store.
212