Launchpad itself

Merge lp:~jtv/launchpad/bug-286237 into lp:launchpad

  1. bug-286237
  2. Merge into devel
Proposed by Jeroen T. Vermeulen
Status: Merged
Approved by: Jeroen T. Vermeulen
Approved revision: no longer in the source branch.
Merged at revision: not available
Proposed branch: lp:~jtv/launchpad/bug-286237
Merge into: lp:launchpad
Diff against target: 754 lines (+279/-94)
14 files modified
lib/lp/translations/browser/tests/test_baseexportview.py (+3/-2)
lib/lp/translations/doc/poexport-queue.txt (+6/-5)
lib/lp/translations/doc/poexport-request-productseries.txt (+3/-2)
lib/lp/translations/doc/poexport-request.txt (+8/-3)
lib/lp/translations/doc/poexportqueue-replication-lag.txt (+89/-0)
lib/lp/translations/doc/potmsgset.txt (+12/-8)
lib/lp/translations/interfaces/poexportrequest.py (+23/-7)
lib/lp/translations/interfaces/potmsgset.py (+14/-8)
lib/lp/translations/model/poexportrequest.py (+68/-31)
lib/lp/translations/model/potmsgset.py (+8/-1)
lib/lp/translations/scripts/po_export_queue.py (+10/-24)
lib/lp/translations/tests/test_potmsgset.py (+23/-1)
lib/lp/translations/tests/test_suggestions.py (+8/-1)
lib/lp/translations/tests/test_translatablemessage.py (+4/-1)
To merge this branch: bzr merge lp:~jtv/launchpad/bug-286237
Reviewer Review Type Date Requested Status
Guilherme Salgado (community) Approve
Canonical Launchpad Engineering code Pending
Review via email: mp+20767@code.launchpad.net

Commit message

Use slave store for suggestions search & translations export.

Description of the change

= Bugs 286237, =

We're still getting some timeouts from the +translate page because fetching external suggestions is costly, and we're still occasionally seeing big, slow exports. Load on the master database peaks higher than Stuart would like, whereas the slaves don't have this problem and can scale out if needed.

This branch moves these tasks to the slave store. This is very easy for external suggestions, and not too hard but still a bit of work for the export queue. (Luckily I'm spending a lot of time waiting for make schema etc. today).

Some background about the export queue: it lives in the POExportRequest table. The app server adds requests to this queue. If often adds many at a time to cover multiple files; simultaneous requests from the same user are bundled. The export script takes a bundle of requests off the queue and produces exports for them based on the translations data we have.

(A few years ago we modified the export script to service just one request at a time. That was a desperation measure against load problems we had at the time, mainly because of OpenOffice.org. In the meantime, exporting full translations for an Ubuntu package has been restricted a bit; database servers have been upgraded; many problems have been addressed; and OpenOffice.org is no longer translated in Ubuntu. So I made the script do all exports at once again. Not waiting 10 minutes or whatever between exports will probably do wonders for perceived speed of the exports.)

In order to prevent double exports caused by replication lag, the export script now first checks what the oldest live request on the master is. It then considers only that request and newer ones on the slave; older ones have presumably already been deleted, but the slave hasn't noticed it yet. When it's done, it deletes the requests it's done with from the master, and so they certainly won't be selected again on the next call. An extra little doctest goes through this.

I split the queue method popRequest into getRequest and removeRequest. This means that the script only tries to delete the request records after it's processed them. This doesn't matter functionally (everything only happens at commit anyway), but shortens the timespan during which the script holds a transaction with pending changes on the master. If the script holds uncommitted deletes on the queue while processing a request, it may block out app servers trying to add to the queue. Doing it this way should make things run a little more smoothly.

The SQL has also been Stormicated. Deleting items wasn't such smooth sailing: Storm informed me that deleting based on is_in or other expressions is not supported yet. Therefore I executed some raw SQL on the master store (but again in the Storm way, not using cursor). The tests need a few more real commits to allow the master and slave stores to synchronize.

I haven't bothered to recover after an error. If something goes wrong, backing off a bit and waiting for the next cron run is probably a sensible thing to do. If it really piles up, the export pages will mention it to users.

No lint. To test:
{{{
./bin/test -vv -t 'translations.*export' -t 'translations.*stories'
}}}

To Q/A, verify that external suggestions still appear on +translate pages. Also, request an export on staging and ask for the export script to be run:
{{{
cronscripts/rosetta-export-queue.py -vvv
}}}

This will print a URL to a Librarian file with the exports. Verify that these are alright, e.g. by comparing to a similar export from production.

Jeroen

To post a comment you must log in.
Revision history for this message
Guilherme Salgado (salgado) wrote :
Download full text (16.3 KiB)

Hi Jeroen,

This was not a trivial one but your cover letter did a very good job at
explaining what you did. Thanks for taking the time to write it.

I have a few comments/suggestions below, but r=me

 review approve

On Fri, 2010-03-05 at 17:37 +0000, Jeroen T. Vermeulen wrote:
> Jeroen T. Vermeulen has proposed merging lp:~jtv/launchpad/bug-286237 into lp:launchpad/devel.

> === modified file 'lib/lp/translations/browser/tests/test_baseexportview.py'
> --- lib/lp/translations/browser/tests/test_baseexportview.py 2010-02-19 17:06:01 +0000
> +++ lib/lp/translations/browser/tests/test_baseexportview.py 2010-03-05 17:38:34 +0000
> @@ -7,6 +7,7 @@
> import transaction
> import unittest
>
> +from canonical.launchpad.interfaces.lpstorm import IMasterStore
> from canonical.launchpad.webapp.servers import LaunchpadTestRequest
> from canonical.testing import ZopelessDatabaseLayer
> from lp.translations.browser.sourcepackage import (
> @@ -15,13 +16,14 @@
> ProductSeriesTranslationsExportView)
> from lp.translations.interfaces.translationfileformat import (
> TranslationFileFormat)
> +from lp.translations.model.poexportrequest import POExportRequest
> from lp.testing import TestCaseWithFactory
>
>
> def wipe_queue(queue):
> """Erase all export queue entries."""
> - while queue.entry_count > 0:
> - queue.popRequest()
> + queue_ids = IMasterStore(POExportRequest).execute(

You don't need to store the queue ids here, do you?

> + "DELETE FROM POExportRequest")
>
>
> class BaseExportViewMixin(TestCaseWithFactory):
>
> === modified file 'lib/lp/translations/doc/poexport-queue.txt'
> --- lib/lp/translations/doc/poexport-queue.txt 2010-02-19 16:02:16 +0000
> +++ lib/lp/translations/doc/poexport-queue.txt 2010-03-05 17:38:34 +0000
> @@ -275,7 +275,8 @@
>
> Once the queue is processed, the queue is empty again.
>
> - >>> process_queue(fake_transaction, logging.getLogger())
> + >>> transaction.commit()
> + >>> process_queue(transaction, logging.getLogger())
> INFO:...Stored file at http://.../po_evolution-2.2.pot
>
> >>> export_request_set.entry_count
> @@ -333,10 +334,10 @@
>
> >>> export_request_set.addRequest(
> ... carlos, pofiles=[pofile], format=TranslationFileFormat.PO)
> - >>> process_queue(fake_transaction, logging.getLogger())
> + >>> transaction.commit()
> + >>> process_queue(transaction, logging.getLogger())
> INFO:root:Stored file at http://...eo.po
>
> - >>> transaction.commit()
> >>> print get_newest_librarian_file().read()
> # Esperanto translation for ...
> ...
> @@ -354,10 +355,10 @@
>
> >>> export_request_set.addRequest(
> ... carlos, pofiles=[pofile], format=TranslationFileFormat.POCHANGED)
> - >>> process_queue(fake_transaction, logging.getLogger())
> + >>> transaction.commit()
> + >>> process_queue(transaction, logging.getLogger())
> INFO:root:Stored file at http://...eo.po
>
> - >>> transaction.commit()
> >>> print get_newest_librarian_file().read()
> # IMPORTANT: This file does NOT contain a complete PO file structure.
> # DO NOT attempt to import this file b...

review: Approve
Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :
Download full text (11.7 KiB)

Thanks for the review! You've brought on some good improvements there:

> > === modified file 'lib/lp/translations/browser/tests/test_baseexportview.py'
> > --- lib/lp/translations/browser/tests/test_baseexportview.py 2010-02-19
> 17:06:01 +0000
> > +++ lib/lp/translations/browser/tests/test_baseexportview.py 2010-03-05
> 17:38:34 +0000

> > @@ -15,13 +16,14 @@
> > ProductSeriesTranslationsExportView)
> > from lp.translations.interfaces.translationfileformat import (
> > TranslationFileFormat)
> > +from lp.translations.model.poexportrequest import POExportRequest
> > from lp.testing import TestCaseWithFactory
> >
> >
> > def wipe_queue(queue):
> > """Erase all export queue entries."""
> > - while queue.entry_count > 0:
> > - queue.popRequest()
> > + queue_ids = IMasterStore(POExportRequest).execute(
>
> You don't need to store the queue ids here, do you?

You're right, I don't, and this can be cleaned up. Just did that.

> > === modified file 'lib/lp/translations/doc/poexport-request.txt'
> > --- lib/lp/translations/doc/poexport-request.txt 2010-02-19 16:54:42
> +0000
> > +++ lib/lp/translations/doc/poexport-request.txt 2010-03-05 17:38:34
> +0000
> > @@ -53,9 +53,10 @@
> >
> > Now we request that the queue be processed.
> >
> > - >>> from lp.testing.faketransaction import FakeTransaction
> > + >>> import transaction
> > >>> from lp.translations.scripts.po_export_queue import process_queue
> > - >>> process_queue(FakeTransaction(), MockLogger())
> > + >>> transaction.commit()
> > + >>> process_queue(transaction, MockLogger())
> > log> Exporting objects for Happy Downloader, related to template pmount
> > in Ubuntu Hoary package "pmount"
> > log> Stored file at http://.../launchpad-export.tar.gz
> > @@ -185,7 +186,8 @@
> > >>> from lp.translations.interfaces.translationfileformat import (
> > ... TranslationFileFormat)
> > >>> request_set.addRequest(person, None, [cs],
> TranslationFileFormat.MO)
> > - >>> process_queue(FakeTransaction(), MockLogger())
> > + >>> transaction.commit()
>
> Do you think it'd be worth adding comments explaining why we need all
> these commits?

Definitely. I added a brief note to the first one.

> > === added file 'lib/lp/translations/doc/poexportqueue-replication-lag.txt'
> > --- lib/lp/translations/doc/poexportqueue-replication-lag.txt 1970-01-01
> 00:00:00 +0000
> > +++ lib/lp/translations/doc/poexportqueue-replication-lag.txt 2010-03-05
> 17:38:34 +0000
> > @@ -0,0 +1,89 @@
> > += Replication Lag and the Export Queue =
> > +
> > +Due to replication lag it's possible for the export queue to see a
> > +request on the slave store that it actually just removed from the master
> > +store.
> > +
> > +We start our story with an empty export queue.
> > +
> > + >>> from datetime import timedelta
> > + >>> import transaction
> > + >>> from zope.component import getUtility
> > + >>> from canonical.launchpad.interfaces.lpstorm import IMasterStore
> > + >>> from lp.translations.interfaces.poexportrequest import (
> > + ... IPOExportRequestSet)
> > + >>> from lp.translations.interfaces.pofil...

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/lp/translations/browser/tests/test_baseexportview.py'
2--- lib/lp/translations/browser/tests/test_baseexportview.py 2010-02-19 17:06:01 +0000
3+++ lib/lp/translations/browser/tests/test_baseexportview.py 2010-03-06 06:29:39 +0000
4@@ -7,6 +7,7 @@
5 import transaction
6 import unittest
7
8+from canonical.launchpad.interfaces.lpstorm import IMasterStore
9 from canonical.launchpad.webapp.servers import LaunchpadTestRequest
10 from canonical.testing import ZopelessDatabaseLayer
11 from lp.translations.browser.sourcepackage import (
12@@ -15,13 +16,13 @@
13 ProductSeriesTranslationsExportView)
14 from lp.translations.interfaces.translationfileformat import (
15 TranslationFileFormat)
16+from lp.translations.model.poexportrequest import POExportRequest
17 from lp.testing import TestCaseWithFactory
18
19
20 def wipe_queue(queue):
21 """Erase all export queue entries."""
22- while queue.entry_count > 0:
23- queue.popRequest()
24+ IMasterStore(POExportRequest).execute("DELETE FROM POExportRequest")
25
26
27 class BaseExportViewMixin(TestCaseWithFactory):
28
29=== modified file 'lib/lp/translations/doc/poexport-queue.txt'
30--- lib/lp/translations/doc/poexport-queue.txt 2010-02-19 16:02:16 +0000
31+++ lib/lp/translations/doc/poexport-queue.txt 2010-03-06 06:29:39 +0000
32@@ -275,7 +275,8 @@
33
34 Once the queue is processed, the queue is empty again.
35
36- >>> process_queue(fake_transaction, logging.getLogger())
37+ >>> transaction.commit()
38+ >>> process_queue(transaction, logging.getLogger())
39 INFO:...Stored file at http://.../po_evolution-2.2.pot
40
41 >>> export_request_set.entry_count
42@@ -333,10 +334,10 @@
43
44 >>> export_request_set.addRequest(
45 ... carlos, pofiles=[pofile], format=TranslationFileFormat.PO)
46- >>> process_queue(fake_transaction, logging.getLogger())
47+ >>> transaction.commit()
48+ >>> process_queue(transaction, logging.getLogger())
49 INFO:root:Stored file at http://...eo.po
50
51- >>> transaction.commit()
52 >>> print get_newest_librarian_file().read()
53 # Esperanto translation for ...
54 ...
55@@ -354,10 +355,10 @@
56
57 >>> export_request_set.addRequest(
58 ... carlos, pofiles=[pofile], format=TranslationFileFormat.POCHANGED)
59- >>> process_queue(fake_transaction, logging.getLogger())
60+ >>> transaction.commit()
61+ >>> process_queue(transaction, logging.getLogger())
62 INFO:root:Stored file at http://...eo.po
63
64- >>> transaction.commit()
65 >>> print get_newest_librarian_file().read()
66 # IMPORTANT: This file does NOT contain a complete PO file structure.
67 # DO NOT attempt to import this file back into Launchpad.
68
69=== modified file 'lib/lp/translations/doc/poexport-request-productseries.txt'
70--- lib/lp/translations/doc/poexport-request-productseries.txt 2010-02-19 16:54:42 +0000
71+++ lib/lp/translations/doc/poexport-request-productseries.txt 2010-03-06 06:29:39 +0000
72@@ -35,11 +35,12 @@
73 Now we request that the queue be processed.
74
75 >>> import logging
76- >>> from lp.testing.faketransaction import FakeTransaction
77+ >>> import transaction
78 >>> from lp.translations.scripts.po_export_queue import process_queue
79 >>> logger = MockLogger()
80 >>> logger.setLevel(logging.DEBUG)
81- >>> process_queue(FakeTransaction(), logger)
82+ >>> transaction.commit()
83+ >>> process_queue(transaction, logger)
84 log> Exporting objects for ..., related to template evolution-2.2 in
85 Evolution trunk
86 log> Exporting objects for ..., related to template evolution-2.2-test in
87
88=== modified file 'lib/lp/translations/doc/poexport-request.txt'
89--- lib/lp/translations/doc/poexport-request.txt 2010-02-19 16:54:42 +0000
90+++ lib/lp/translations/doc/poexport-request.txt 2010-03-06 06:29:39 +0000
91@@ -53,9 +53,13 @@
92
93 Now we request that the queue be processed.
94
95- >>> from lp.testing.faketransaction import FakeTransaction
96+(Commits are needed to make the test requests seep through to the slave
97+database).
98+
99+ >>> import transaction
100 >>> from lp.translations.scripts.po_export_queue import process_queue
101- >>> process_queue(FakeTransaction(), MockLogger())
102+ >>> transaction.commit()
103+ >>> process_queue(transaction, MockLogger())
104 log> Exporting objects for Happy Downloader, related to template pmount
105 in Ubuntu Hoary package "pmount"
106 log> Stored file at http://.../launchpad-export.tar.gz
107@@ -185,7 +189,8 @@
108 >>> from lp.translations.interfaces.translationfileformat import (
109 ... TranslationFileFormat)
110 >>> request_set.addRequest(person, None, [cs], TranslationFileFormat.MO)
111- >>> process_queue(FakeTransaction(), MockLogger())
112+ >>> transaction.commit()
113+ >>> process_queue(transaction, MockLogger())
114 log> Exporting objects for Happy Downloader, related to template pmount
115 in Ubuntu Hoary package "pmount"
116 log> Stored file at http://.../cs_LC_MESSAGES_pmount.mo
117
118=== added file 'lib/lp/translations/doc/poexportqueue-replication-lag.txt'
119--- lib/lp/translations/doc/poexportqueue-replication-lag.txt 1970-01-01 00:00:00 +0000
120+++ lib/lp/translations/doc/poexportqueue-replication-lag.txt 2010-03-06 06:29:39 +0000
121@@ -0,0 +1,89 @@
122+= Replication Lag and the Export Queue =
123+
124+Due to replication lag it's possible for the export queue to see a
125+request on the slave store that it actually just removed from the master
126+store.
127+
128+We start our story with an empty export queue.
129+
130+ >>> from datetime import timedelta
131+ >>> import transaction
132+ >>> from zope.component import getUtility
133+ >>> from canonical.launchpad.interfaces.lpstorm import IMasterStore
134+ >>> from lp.translations.interfaces.poexportrequest import (
135+ ... IPOExportRequestSet)
136+ >>> from lp.translations.interfaces.pofile import IPOFile
137+ >>> from lp.translations.model.poexportrequest import POExportRequest
138+ >>> query = IMasterStore(POExportRequest).execute(
139+ ... "DELETE FROM POExportRequest")
140+
141+ >>> queue = getUtility(IPOExportRequestSet)
142+
143+We have somebody making an export request.
144+
145+ >>> requester = factory.makePersonNoCommit(
146+ ... email='punter@example.com', name='punter')
147+
148+ >>> template1 = factory.makePOTemplate()
149+ >>> pofile1_be = factory.makePOFile('be', potemplate=template1)
150+ >>> pofile1_ja = factory.makePOFile('ja', potemplate=template1)
151+ >>> queue.addRequest(requester, template1, [pofile1_be, pofile1_ja])
152+ >>> query = IMasterStore(POExportRequest).execute(
153+ ... "UPDATE POExportRequest SET date_created = '2010-01-10'::date")
154+
155+Later, a different and separate request follows.
156+
157+ >>> template2 = factory.makePOTemplate()
158+ >>> pofile2_se = factory.makePOFile('se', potemplate=template2)
159+ >>> pofile2_ga = factory.makePOFile('ga', potemplate=template2)
160+ >>> queue.addRequest(requester, template2, [pofile2_se, pofile2_ga])
161+
162+The database is replicated in this state.
163+
164+ >>> transaction.commit()
165+
166+getRequest at this point returns the oldest request.
167+
168+ >>> def summarize_request(request_tuple):
169+ ... """Summarize files in export request."""
170+ ... person, sources, format, request_ids = request_tuple
171+ ... summary = []
172+ ... for source in sources:
173+ ... if IPOFile.providedBy(source):
174+ ... summary.append(source.language.code)
175+ ... else:
176+ ... summary.append('(template)')
177+ ... for entry in sorted(summary):
178+ ... print entry
179+
180+ >>> summarize_request(queue.getRequest())
181+ (template)
182+ be
183+ ja
184+
185+It doesn't modify the queue, so it'd say the same thing again if we
186+were to ask again.
187+
188+ >>> repeated_request = queue.getRequest()
189+ >>> summarize_request(repeated_request)
190+ (template)
191+ be
192+ ja
193+
194+The first request is removed from the master store after processing, but
195+not yet from the slave store. (Since this test is all one session, we
196+can reproduce this by not committing the removal). The second request
197+is still technically on the queue, but no longer "live."
198+
199+ >>> person, sources, format, request_ids = repeated_request
200+ >>> print len(request_ids)
201+ 3
202+ >>> queue.removeRequest(request_ids)
203+
204+In this state, despite the replication lag, getRequest is smart enough
205+to return the second request, not the first.
206+
207+ >>> summarize_request(queue.getRequest())
208+ (template)
209+ ga
210+ se
211
212=== modified file 'lib/lp/translations/doc/potmsgset.txt'
213--- lib/lp/translations/doc/potmsgset.txt 2009-10-23 12:44:32 +0000
214+++ lib/lp/translations/doc/potmsgset.txt 2010-03-06 06:29:39 +0000
215@@ -2,10 +2,14 @@
216
217 POTMsgSet represents messages to translate that a POTemplate file has.
218
219-We need to get a POTMsgSet object to performe this test.
220+In this test we'll be committing a lot to let changes replicate to the
221+slave database.
222+
223+ >>> import transaction
224+
225+We need to get a POTMsgSet object to perform this test.
226
227 >>> from zope.component import getUtility
228- >>> from canonical.database.sqlbase import flush_database_updates
229 >>> from lp.translations.model.translationmessage import (
230 ... TranslationMessage)
231 >>> from lp.translations.interfaces.potmsgset import IPOTMsgSet
232@@ -734,7 +738,7 @@
233 are not available as suggestions anymore:
234
235 >>> evo_distro_template.iscurrent = False
236- >>> flush_database_updates()
237+ >>> transaction.commit()
238 >>> suggestions = (
239 ... evo_product_message.getExternallyUsedTranslationMessages(spanish))
240 >>> len(suggestions)
241@@ -748,7 +752,7 @@
242 # We set the template as current again so we are sure that we don't show
243 # suggestions just due to the change to the official_rosetta flag.
244 >>> evo_distro_template.iscurrent = True
245- >>> flush_database_updates()
246+ >>> transaction.commit()
247 >>> suggestions = (
248 ... evo_product_message.getExternallyUsedTranslationMessages(spanish))
249 >>> len(suggestions)
250@@ -757,7 +761,7 @@
251 And products not using translations officially have the same behaviour.
252
253 >>> evolution.official_rosetta = False
254- >>> flush_database_updates()
255+ >>> transaction.commit()
256 >>> suggestions = evo_distro_message.getExternallyUsedTranslationMessages(
257 ... spanish)
258 >>> len(suggestions)
259@@ -767,7 +771,7 @@
260
261 >>> ubuntu.official_rosetta = True
262 >>> evolution.official_rosetta = True
263- >>> flush_database_updates()
264+ >>> transaction.commit()
265
266
267 == POTMsgSet.getExternallySuggestedTranslationMessages ==
268@@ -850,7 +854,7 @@
269 we get no suggestions.
270
271 >>> potmsgset_translated.potemplate.iscurrent = False
272- >>> flush_database_updates()
273+ >>> transaction.commit()
274
275 >>> wiki_submissions = (
276 ... potmsgset_untranslated.getExternallyUsedTranslationMessages(
277@@ -865,7 +869,7 @@
278 # suggestions just due to the change to the official_rosetta flag.
279 >>> potmsgset_translated.potemplate.iscurrent = True
280 >>> ubuntu.official_rosetta = False
281- >>> flush_database_updates()
282+ >>> transaction.commit()
283
284 >>> wiki_submissions = (
285 ... potmsgset_untranslated.getExternallyUsedTranslationMessages(
286
287=== modified file 'lib/lp/translations/interfaces/poexportrequest.py'
288--- lib/lp/translations/interfaces/poexportrequest.py 2010-02-19 16:02:16 +0000
289+++ lib/lp/translations/interfaces/poexportrequest.py 2010-03-06 06:29:39 +0000
290@@ -11,7 +11,7 @@
291 ]
292
293 from zope.interface import Interface
294-from zope.schema import Int, Object
295+from zope.schema import Datetime, Int, Object
296
297 from lp.registry.interfaces.person import IPerson
298 from lp.translations.interfaces.pofile import IPOFile
299@@ -37,12 +37,25 @@
300 :param pofiles: A list of PO files to export.
301 """
302
303- def popRequest():
304- """Take the next request out of the queue.
305-
306- Returns a 3-tuple containing the person who made the request, the PO
307- template the request was for, and a list of `POTemplate` and `POFile`
308- objects to export.
309+ def getRequest():
310+ """Get the next request from the queue.
311+
312+ Returns a tuple containing:
313+ * The person who made the request.
314+ * A list of POFiles and/or POTemplates that are to be exported.
315+ * The requested `TranslationFileFormat`.
316+ * The list of request record ids making up this request.
317+
318+ The objects are all read-only objects from the slave store. The
319+ request ids list should be passed to `removeRequest` when
320+ processing of the request completes.
321+ """
322+
323+ def removeRequest(self, request_ids):
324+ """Remove a request off the queue.
325+
326+ :param request_ids: A list of request record ids as returned by
327+ `getRequest`.
328 """
329
330
331@@ -51,6 +64,9 @@
332 title=u'The person who made the request.',
333 required=True, readonly=True, schema=IPerson)
334
335+ date_created = Datetime(
336+ title=u"Request's creation timestamp.", required=True, readonly=True)
337+
338 potemplate = Object(
339 title=u'The translation template to which the requested file belong.',
340 required=True, readonly=True, schema=IPOTemplate)
341
342=== modified file 'lib/lp/translations/interfaces/potmsgset.py'
343--- lib/lp/translations/interfaces/potmsgset.py 2009-10-22 15:51:58 +0000
344+++ lib/lp/translations/interfaces/potmsgset.py 2010-03-06 06:29:39 +0000
345@@ -163,19 +163,25 @@
346 """
347
348 def getExternallyUsedTranslationMessages(language):
349- """Returns all externally used translations.
350-
351- External are those on other templates for the same English message.
352- "Used" messages are either current or imported ones.
353+ """Find externally used translations for the same message.
354+
355+ This is used to find suggestions for translating this
356+ `POTMsgSet` that are actually used (i.e. current or imported) in
357+ other templates.
358+
359+ The suggestions are read-only; they come from the slave store.
360
361 :param language: language we want translations for.
362 """
363
364 def getExternallySuggestedTranslationMessages(language):
365- """Return all externally suggested translations.
366-
367- External are those on other templates for the same English message.
368- "Suggested" messages are those which are neither current nor imported.
369+ """Find externally suggested translations for the same message.
370+
371+ This is used to find suggestions for translating this
372+ `POTMsgSet` that were entered in another context, but for the
373+ same English text, and are not in actual use.
374+
375+ The suggestions are read-only; they come from the slave store.
376
377 :param language: language we want translations for.
378 """
379
380=== modified file 'lib/lp/translations/model/poexportrequest.py'
381--- lib/lp/translations/model/poexportrequest.py 2010-02-19 17:06:01 +0000
382+++ lib/lp/translations/model/poexportrequest.py 2010-03-06 06:29:39 +0000
383@@ -15,9 +15,12 @@
384 from zope.component import getUtility
385 from zope.interface import implements
386
387+from canonical.database.constants import DEFAULT
388+from canonical.database.datetimecol import UtcDateTimeCol
389+from canonical.database.enumcol import EnumCol
390 from canonical.database.sqlbase import quote, SQLBase, sqlvalues
391-from canonical.database.enumcol import EnumCol
392
393+from canonical.launchpad.interfaces.lpstorm import IMasterStore, ISlaveStore
394 from lp.translations.interfaces.poexportrequest import (
395 IPOExportRequest, IPOExportRequestSet)
396 from lp.translations.interfaces.potemplate import IPOTemplate
397@@ -112,36 +115,69 @@
398 existing.id IS NULL
399 """ % query_params)
400
401- def popRequest(self):
402- """See `IPOExportRequestSet`."""
403- try:
404- request = POExportRequest.select(limit=1, orderBy='id')[0]
405- except IndexError:
406- return None, None, None
407-
408- person = request.person
409- format = request.format
410-
411- query = """
412- person = %s AND
413- format = %s AND
414- date_created = (
415- SELECT date_created
416- FROM POExportRequest
417- ORDER BY id
418- LIMIT 1)""" % sqlvalues(person, format)
419- requests = POExportRequest.select(query, orderBy='potemplate')
420- objects = []
421-
422- for request in requests:
423- if request.pofile is not None:
424- objects.append(request.pofile)
425- else:
426- objects.append(request.potemplate)
427-
428- POExportRequest.delete(request.id)
429-
430- return person, objects, format
431+ def _getOldestLiveRequest(self):
432+ """Return the oldest live request on the master store.
433+
434+ Due to replication lag, the master store is always a little
435+ ahead of the slave store that exports come from.
436+ """
437+ master_store = IMasterStore(POExportRequest)
438+ sorted_by_id = master_store.find(POExportRequest).order_by(
439+ POExportRequest.id)
440+ return sorted_by_id.first()
441+
442+ def _getHeadRequest(self):
443+ """Return oldest request on the queue."""
444+ # Due to replication lag, it's possible that the slave store
445+ # still has copies of requests that have already been completed
446+ # and deleted from the master store. So first get the oldest
447+ # request that is "live," i.e. still present on the master
448+ # store.
449+ oldest_live = self._getOldestLiveRequest()
450+ if oldest_live is None:
451+ return None
452+ else:
453+ return ISlaveStore(POExportRequest).find(
454+ POExportRequest,
455+ POExportRequest.id == oldest_live.id).one()
456+
457+ def getRequest(self):
458+ """See `IPOExportRequestSet`."""
459+ # Exports happen off the slave store. To ensure that export
460+ # does not happen until requests have been replicated to the
461+ # slave, they are read primarily from the slave even though they
462+ # are deleted on the master afterwards.
463+ head = self._getHeadRequest()
464+ if head is None:
465+ return None, None, None, None
466+
467+ requests = ISlaveStore(POExportRequest).find(
468+ POExportRequest,
469+ POExportRequest.person == head.person,
470+ POExportRequest.format == head.format,
471+ POExportRequest.date_created == head.date_created).order_by(
472+ POExportRequest.potemplateID)
473+
474+ summary = [
475+ (request.id, request.pofile or request.potemplate)
476+ for request in requests
477+ ]
478+
479+ sources = [source for request_id, source in summary]
480+ request_ids = [request_id for request_id, source in summary]
481+
482+ return head.person, sources, head.format, request_ids
483+
484+ def removeRequest(self, request_ids):
485+ """See `IPOExportRequestSet`."""
486+ if len(request_ids) > 0:
487+ # Storm 0.15 does not have direct support for deleting based
488+ # on is_in expressions and such, so do it the hard way.
489+ ids_string = ', '.join(sqlvalues(*request_ids))
490+ IMasterStore(POExportRequest).execute("""
491+ DELETE FROM POExportRequest
492+ WHERE id in (%s)
493+ """ % ids_string)
494
495
496 class POExportRequest(SQLBase):
497@@ -152,6 +188,7 @@
498 person = ForeignKey(
499 dbName='person', foreignKey='Person',
500 storm_validator=validate_public_person, notNull=True)
501+ date_created = UtcDateTimeCol(dbName='date_created', default=DEFAULT)
502 potemplate = ForeignKey(dbName='potemplate', foreignKey='POTemplate',
503 notNull=True)
504 pofile = ForeignKey(dbName='pofile', foreignKey='POFile')
505
506=== modified file 'lib/lp/translations/model/potmsgset.py'
507--- lib/lp/translations/model/potmsgset.py 2010-01-28 09:53:45 +0000
508+++ lib/lp/translations/model/potmsgset.py 2010-03-06 06:29:39 +0000
509@@ -15,6 +15,7 @@
510 from zope.component import getUtility
511
512 from sqlobject import ForeignKey, IntCol, StringCol, SQLObjectNotFound
513+from storm.expr import SQL
514 from storm.store import EmptyResultSet, Store
515
516 from canonical.config import config
517@@ -27,6 +28,7 @@
518 from canonical.launchpad.interfaces.launchpad import ILaunchpadCelebrities
519 from canonical.launchpad.readonly import is_read_only
520 from canonical.launchpad.webapp.interfaces import UnexpectedFormData
521+from canonical.launchpad.interfaces.lpstorm import ISlaveStore
522 from lp.translations.interfaces.pofile import IPOFileSet
523 from lp.translations.interfaces.potmsgset import (
524 BrokenTextError,
525@@ -324,6 +326,9 @@
526
527 A message is used if it's either imported or current, and unused
528 otherwise.
529+
530+ Suggestions are read-only, so these objects come from the slave
531+ store.
532 """
533 if not config.rosetta.global_suggestions_enabled:
534 return []
535@@ -387,7 +392,9 @@
536 ORDER BY %(msgstrs)s, date_created DESC
537 ''' % ids_query_params
538
539- result = TranslationMessage.select('id IN (%s)' % ids_query)
540+ result = ISlaveStore(TranslationMessage).find(
541+ TranslationMessage,
542+ TranslationMessage.id.is_in(SQL(ids_query)))
543
544 return shortlist(result, longest_expected=100, hardlimit=2000)
545
546
547=== modified file 'lib/lp/translations/scripts/po_export_queue.py'
548--- lib/lp/translations/scripts/po_export_queue.py 2009-08-17 13:42:00 +0000
549+++ lib/lp/translations/scripts/po_export_queue.py 2010-03-06 06:29:39 +0000
550@@ -373,32 +373,18 @@
551
552
553 def process_queue(transaction_manager, logger):
554- """Process each request in the translation export queue.
555+ """Process all requests in the translation export queue.
556
557- Each item is removed from the queue as it is processed, we only handle
558- one request with each function call.
559+ Each item is removed from the queue as it is processed.
560 """
561 request_set = getUtility(IPOExportRequestSet)
562-
563- request = request_set.popRequest()
564-
565- if None in request:
566- # Any value is None and we must have all values as not None to have
567- # something to process...
568- return
569-
570- person, objects, format = request
571-
572- try:
573+ no_request = (None, None, None, None)
574+
575+ request = request_set.getRequest()
576+ while request != no_request:
577+ person, objects, format, request_ids = request
578 process_request(person, objects, format, logger)
579- except psycopg2.Error:
580- # We had a DB error, we don't try to recover it here, just exit
581- # from the script and next run will retry the export.
582- logger.error(
583- "A DB exception was raised when exporting files for %s" % (
584- person.displayname),
585- exc_info=True)
586- transaction_manager.abort()
587- else:
588- # Apply all changes.
589+ request_set.removeRequest(request_ids)
590 transaction_manager.commit()
591+
592+ request = request_set.getRequest()
593
594=== modified file 'lib/lp/translations/tests/test_potmsgset.py'
595--- lib/lp/translations/tests/test_potmsgset.py 2009-11-25 15:08:26 +0000
596+++ lib/lp/translations/tests/test_potmsgset.py 2010-03-06 06:29:39 +0000
597@@ -1,4 +1,4 @@
598-# Copyright 2009 Canonical Ltd. This software is licensed under the
599+# Copyright 2009-2010 Canonical Ltd. This software is licensed under the
600 # GNU Affero General Public License version 3 (see the file LICENSE).
601
602 # pylint: disable-msg=C0102
603@@ -329,6 +329,8 @@
604 external_pofile = self.factory.makePOFile('sr', external_template)
605 serbian = external_pofile.language
606
607+ transaction.commit()
608+
609 # When there is no translation for the external POTMsgSet,
610 # no externally used suggestions are returned.
611 self.assertEquals(
612@@ -340,6 +342,9 @@
613 external_suggestion = self.factory.makeSharedTranslationMessage(
614 pofile=external_pofile, potmsgset=external_potmsgset,
615 suggestion=True)
616+
617+ transaction.commit()
618+
619 self.assertEquals(
620 self.potmsgset.getExternallyUsedTranslationMessages(serbian),
621 [])
622@@ -350,6 +355,9 @@
623 pofile=external_pofile, potmsgset=external_potmsgset,
624 suggestion=False, is_imported=True)
625 imported_translation.is_current = False
626+
627+ transaction.commit()
628+
629 self.assertEquals(
630 self.potmsgset.getExternallyUsedTranslationMessages(serbian),
631 [imported_translation])
632@@ -359,6 +367,9 @@
633 current_translation = self.factory.makeSharedTranslationMessage(
634 pofile=external_pofile, potmsgset=external_potmsgset,
635 suggestion=False, is_imported=False)
636+
637+ transaction.commit()
638+
639 self.assertEquals(
640 self.potmsgset.getExternallyUsedTranslationMessages(serbian),
641 [imported_translation, current_translation])
642@@ -377,6 +388,8 @@
643 external_pofile = self.factory.makePOFile('sr', external_template)
644 serbian = external_pofile.language
645
646+ transaction.commit()
647+
648 # When there is no translation for the external POTMsgSet,
649 # no externally used suggestions are returned.
650 self.assertEquals(
651@@ -388,6 +401,9 @@
652 external_suggestion = self.factory.makeSharedTranslationMessage(
653 pofile=external_pofile, potmsgset=external_potmsgset,
654 suggestion=True)
655+
656+ transaction.commit()
657+
658 self.assertEquals(
659 self.potmsgset.getExternallySuggestedTranslationMessages(serbian),
660 [external_suggestion])
661@@ -398,6 +414,9 @@
662 pofile=external_pofile, potmsgset=external_potmsgset,
663 suggestion=False, is_imported=True)
664 imported_translation.is_current = False
665+
666+ transaction.commit()
667+
668 self.assertEquals(
669 self.potmsgset.getExternallySuggestedTranslationMessages(serbian),
670 [external_suggestion])
671@@ -407,6 +426,9 @@
672 current_translation = self.factory.makeSharedTranslationMessage(
673 pofile=external_pofile, potmsgset=external_potmsgset,
674 suggestion=False, is_imported=False)
675+
676+ transaction.commit()
677+
678 self.assertEquals(
679 self.potmsgset.getExternallySuggestedTranslationMessages(serbian),
680 [external_suggestion])
681
682=== modified file 'lib/lp/translations/tests/test_suggestions.py'
683--- lib/lp/translations/tests/test_suggestions.py 2009-07-17 00:26:05 +0000
684+++ lib/lp/translations/tests/test_suggestions.py 2010-03-06 06:29:39 +0000
685@@ -1,10 +1,11 @@
686-# Copyright 2009 Canonical Ltd. This software is licensed under the
687+# Copyright 2009-2010 Canonical Ltd. This software is licensed under the
688 # GNU Affero General Public License version 3 (see the file LICENSE).
689
690 __metaclass__ = type
691
692 from datetime import datetime, timedelta
693 from pytz import timezone
694+import transaction
695 import unittest
696
697 import gettextpo
698@@ -66,6 +67,8 @@
699 ["foutmelding 936"], is_imported=False,
700 lock_timestamp=None)
701
702+ transaction.commit()
703+
704 used_suggestions = foomsg.getExternallyUsedTranslationMessages(
705 self.nl)
706 other_suggestions = foomsg.getExternallySuggestedTranslationMessages(
707@@ -88,6 +91,8 @@
708 ["foutmelding 936"], is_imported=False,
709 lock_timestamp=None)
710
711+ transaction.commit()
712+
713 # There is a global (externally used) suggestion.
714 used_suggestions = foomsg.getExternallyUsedTranslationMessages(
715 self.nl)
716@@ -117,6 +122,8 @@
717 is_imported=False, lock_timestamp=None)
718 suggestion.is_current = False
719
720+ transaction.commit()
721+
722 used_suggestions = foomsg.getExternallyUsedTranslationMessages(
723 self.nl)
724 other_suggestions = foomsg.getExternallySuggestedTranslationMessages(
725
726=== modified file 'lib/lp/translations/tests/test_translatablemessage.py'
727--- lib/lp/translations/tests/test_translatablemessage.py 2009-08-26 16:24:02 +0000
728+++ lib/lp/translations/tests/test_translatablemessage.py 2010-03-06 06:29:39 +0000
729@@ -1,4 +1,4 @@
730-# Copyright 2009 Canonical Ltd. This software is licensed under the
731+# Copyright 2009-2010 Canonical Ltd. This software is licensed under the
732 # GNU Affero General Public License version 3 (see the file LICENSE).
733
734 """Unit tests for `TranslatableMessage`."""
735@@ -7,6 +7,7 @@
736
737 from datetime import datetime, timedelta
738 import pytz
739+import transaction
740 from unittest import TestLoader
741
742 from lp.testing import TestCaseWithFactory
743@@ -147,10 +148,12 @@
744 self.message = TranslatableMessage(self.potmsgset, self.pofile)
745
746 def test_getExternalTranslations(self):
747+ transaction.commit()
748 externals = self.message.getExternalTranslations()
749 self.assertContentEqual([self.external_current], externals)
750
751 def test_getExternalSuggestions(self):
752+ transaction.commit()
753 externals = self.message.getExternalSuggestions()
754 self.assertContentEqual([self.external_suggestion], externals)
755