Launchpad itself

Merge lp:~jtv/launchpad/bug-286237 into lp:launchpad

  1. bug-286237
  2. Merge into devel
Proposed by Jeroen T. Vermeulen
Status: Merged
Approved by: Jeroen T. Vermeulen
Approved revision: no longer in the source branch.
Merged at revision: not available
Proposed branch: lp:~jtv/launchpad/bug-286237
Merge into: lp:launchpad
Diff against target: 754 lines (+279/-94)
14 files modified
lib/lp/translations/browser/tests/test_baseexportview.py (+3/-2)
lib/lp/translations/doc/poexport-queue.txt (+6/-5)
lib/lp/translations/doc/poexport-request-productseries.txt (+3/-2)
lib/lp/translations/doc/poexport-request.txt (+8/-3)
lib/lp/translations/doc/poexportqueue-replication-lag.txt (+89/-0)
lib/lp/translations/doc/potmsgset.txt (+12/-8)
lib/lp/translations/interfaces/poexportrequest.py (+23/-7)
lib/lp/translations/interfaces/potmsgset.py (+14/-8)
lib/lp/translations/model/poexportrequest.py (+68/-31)
lib/lp/translations/model/potmsgset.py (+8/-1)
lib/lp/translations/scripts/po_export_queue.py (+10/-24)
lib/lp/translations/tests/test_potmsgset.py (+23/-1)
lib/lp/translations/tests/test_suggestions.py (+8/-1)
lib/lp/translations/tests/test_translatablemessage.py (+4/-1)
To merge this branch: bzr merge lp:~jtv/launchpad/bug-286237
Reviewer Review Type Date Requested Status
Guilherme Salgado (community) Approve
Canonical Launchpad Engineering code Pending
Review via email: mp+20767@code.launchpad.net

Commit message

Use slave store for suggestions search & translations export.

Description of the change

= Bugs 286237, =

We're still getting some timeouts from the +translate page because fetching external suggestions is costly, and we're still occasionally seeing big, slow exports. Load on the master database peaks higher than Stuart would like, whereas the slaves don't have this problem and can scale out if needed.

This branch moves these tasks to the slave store. This is very easy for external suggestions, and not too hard but still a bit of work for the export queue. (Luckily I'm spending a lot of time waiting for make schema etc. today).

Some background about the export queue: it lives in the POExportRequest table. The app server adds requests to this queue. If often adds many at a time to cover multiple files; simultaneous requests from the same user are bundled. The export script takes a bundle of requests off the queue and produces exports for them based on the translations data we have.

(A few years ago we modified the export script to service just one request at a time. That was a desperation measure against load problems we had at the time, mainly because of OpenOffice.org. In the meantime, exporting full translations for an Ubuntu package has been restricted a bit; database servers have been upgraded; many problems have been addressed; and OpenOffice.org is no longer translated in Ubuntu. So I made the script do all exports at once again. Not waiting 10 minutes or whatever between exports will probably do wonders for perceived speed of the exports.)

In order to prevent double exports caused by replication lag, the export script now first checks what the oldest live request on the master is. It then considers only that request and newer ones on the slave; older ones have presumably already been deleted, but the slave hasn't noticed it yet. When it's done, it deletes the requests it's done with from the master, and so they certainly won't be selected again on the next call. An extra little doctest goes through this.

I split the queue method popRequest into getRequest and removeRequest. This means that the script only tries to delete the request records after it's processed them. This doesn't matter functionally (everything only happens at commit anyway), but shortens the timespan during which the script holds a transaction with pending changes on the master. If the script holds uncommitted deletes on the queue while processing a request, it may block out app servers trying to add to the queue. Doing it this way should make things run a little more smoothly.

The SQL has also been Stormicated. Deleting items wasn't such smooth sailing: Storm informed me that deleting based on is_in or other expressions is not supported yet. Therefore I executed some raw SQL on the master store (but again in the Storm way, not using cursor). The tests need a few more real commits to allow the master and slave stores to synchronize.

I haven't bothered to recover after an error. If something goes wrong, backing off a bit and waiting for the next cron run is probably a sensible thing to do. If it really piles up, the export pages will mention it to users.

No lint. To test:
{{{
./bin/test -vv -t 'translations.*export' -t 'translations.*stories'
}}}

To Q/A, verify that external suggestions still appear on +translate pages. Also, request an export on staging and ask for the export script to be run:
{{{
cronscripts/rosetta-export-queue.py -vvv
}}}

This will print a URL to a Librarian file with the exports. Verify that these are alright, e.g. by comparing to a similar export from production.

Jeroen

To post a comment you must log in.
Revision history for this message
Guilherme Salgado (salgado) wrote :
Download full text (16.3 KiB)

Hi Jeroen,

This was not a trivial one but your cover letter did a very good job at
explaining what you did. Thanks for taking the time to write it.

I have a few comments/suggestions below, but r=me

 review approve

On Fri, 2010-03-05 at 17:37 +0000, Jeroen T. Vermeulen wrote:
> Jeroen T. Vermeulen has proposed merging lp:~jtv/launchpad/bug-286237 into lp:launchpad/devel.

> === modified file 'lib/lp/translations/browser/tests/test_baseexportview.py'
> --- lib/lp/translations/browser/tests/test_baseexportview.py 2010-02-19 17:06:01 +0000
> +++ lib/lp/translations/browser/tests/test_baseexportview.py 2010-03-05 17:38:34 +0000
> @@ -7,6 +7,7 @@
> import transaction
> import unittest
>
> +from canonical.launchpad.interfaces.lpstorm import IMasterStore
> from canonical.launchpad.webapp.servers import LaunchpadTestRequest
> from canonical.testing import ZopelessDatabaseLayer
> from lp.translations.browser.sourcepackage import (
> @@ -15,13 +16,14 @@
> ProductSeriesTranslationsExportView)
> from lp.translations.interfaces.translationfileformat import (
> TranslationFileFormat)
> +from lp.translations.model.poexportrequest import POExportRequest
> from lp.testing import TestCaseWithFactory
>
>
> def wipe_queue(queue):
> """Erase all export queue entries."""
> - while queue.entry_count > 0:
> - queue.popRequest()
> + queue_ids = IMasterStore(POExportRequest).execute(

You don't need to store the queue ids here, do you?

> + "DELETE FROM POExportRequest")
>
>
> class BaseExportViewMixin(TestCaseWithFactory):
>
> === modified file 'lib/lp/translations/doc/poexport-queue.txt'
> --- lib/lp/translations/doc/poexport-queue.txt 2010-02-19 16:02:16 +0000
> +++ lib/lp/translations/doc/poexport-queue.txt 2010-03-05 17:38:34 +0000
> @@ -275,7 +275,8 @@
>
> Once the queue is processed, the queue is empty again.
>
> - >>> process_queue(fake_transaction, logging.getLogger())
> + >>> transaction.commit()
> + >>> process_queue(transaction, logging.getLogger())
> INFO:...Stored file at http://.../po_evolution-2.2.pot
>
> >>> export_request_set.entry_count
> @@ -333,10 +334,10 @@
>
> >>> export_request_set.addRequest(
> ... carlos, pofiles=[pofile], format=TranslationFileFormat.PO)
> - >>> process_queue(fake_transaction, logging.getLogger())
> + >>> transaction.commit()
> + >>> process_queue(transaction, logging.getLogger())
> INFO:root:Stored file at http://...eo.po
>
> - >>> transaction.commit()
> >>> print get_newest_librarian_file().read()
> # Esperanto translation for ...
> ...
> @@ -354,10 +355,10 @@
>
> >>> export_request_set.addRequest(
> ... carlos, pofiles=[pofile], format=TranslationFileFormat.POCHANGED)
> - >>> process_queue(fake_transaction, logging.getLogger())
> + >>> transaction.commit()
> + >>> process_queue(transaction, logging.getLogger())
> INFO:root:Stored file at http://...eo.po
>
> - >>> transaction.commit()
> >>> print get_newest_librarian_file().read()
> # IMPORTANT: This file does NOT contain a complete PO file structure.
> # DO NOT attempt to import this file b...

review: Approve
Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :
Download full text (11.7 KiB)

Thanks for the review! You've brought on some good improvements there:

> > === modified file 'lib/lp/translations/browser/tests/test_baseexportview.py'
> > --- lib/lp/translations/browser/tests/test_baseexportview.py 2010-02-19
> 17:06:01 +0000
> > +++ lib/lp/translations/browser/tests/test_baseexportview.py 2010-03-05
> 17:38:34 +0000

> > @@ -15,13 +16,14 @@
> > ProductSeriesTranslationsExportView)
> > from lp.translations.interfaces.translationfileformat import (
> > TranslationFileFormat)
> > +from lp.translations.model.poexportrequest import POExportRequest
> > from lp.testing import TestCaseWithFactory
> >
> >
> > def wipe_queue(queue):
> > """Erase all export queue entries."""
> > - while queue.entry_count > 0:
> > - queue.popRequest()
> > + queue_ids = IMasterStore(POExportRequest).execute(
>
> You don't need to store the queue ids here, do you?

You're right, I don't, and this can be cleaned up. Just did that.

> > === modified file 'lib/lp/translations/doc/poexport-request.txt'
> > --- lib/lp/translations/doc/poexport-request.txt 2010-02-19 16:54:42
> +0000
> > +++ lib/lp/translations/doc/poexport-request.txt 2010-03-05 17:38:34
> +0000
> > @@ -53,9 +53,10 @@
> >
> > Now we request that the queue be processed.
> >
> > - >>> from lp.testing.faketransaction import FakeTransaction
> > + >>> import transaction
> > >>> from lp.translations.scripts.po_export_queue import process_queue
> > - >>> process_queue(FakeTransaction(), MockLogger())
> > + >>> transaction.commit()
> > + >>> process_queue(transaction, MockLogger())
> > log> Exporting objects for Happy Downloader, related to template pmount
> > in Ubuntu Hoary package "pmount"
> > log> Stored file at http://.../launchpad-export.tar.gz
> > @@ -185,7 +186,8 @@
> > >>> from lp.translations.interfaces.translationfileformat import (
> > ... TranslationFileFormat)
> > >>> request_set.addRequest(person, None, [cs],
> TranslationFileFormat.MO)
> > - >>> process_queue(FakeTransaction(), MockLogger())
> > + >>> transaction.commit()
>
> Do you think it'd be worth adding comments explaining why we need all
> these commits?

Definitely. I added a brief note to the first one.

> > === added file 'lib/lp/translations/doc/poexportqueue-replication-lag.txt'
> > --- lib/lp/translations/doc/poexportqueue-replication-lag.txt 1970-01-01
> 00:00:00 +0000
> > +++ lib/lp/translations/doc/poexportqueue-replication-lag.txt 2010-03-05
> 17:38:34 +0000
> > @@ -0,0 +1,89 @@
> > += Replication Lag and the Export Queue =
> > +
> > +Due to replication lag it's possible for the export queue to see a
> > +request on the slave store that it actually just removed from the master
> > +store.
> > +
> > +We start our story with an empty export queue.
> > +
> > + >>> from datetime import timedelta
> > + >>> import transaction
> > + >>> from zope.component import getUtility
> > + >>> from canonical.launchpad.interfaces.lpstorm import IMasterStore
> > + >>> from lp.translations.interfaces.poexportrequest import (
> > + ... IPOExportRequestSet)
> > + >>> from lp.translations.interfaces.pofil...

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/lp/translations/browser/tests/test_baseexportview.py'
--- lib/lp/translations/browser/tests/test_baseexportview.py 2010-02-19 17:06:01 +0000
+++ lib/lp/translations/browser/tests/test_baseexportview.py 2010-03-06 06:29:39 +0000
@@ -7,6 +7,7 @@
7import transaction7import transaction
8import unittest8import unittest
99
10from canonical.launchpad.interfaces.lpstorm import IMasterStore
10from canonical.launchpad.webapp.servers import LaunchpadTestRequest11from canonical.launchpad.webapp.servers import LaunchpadTestRequest
11from canonical.testing import ZopelessDatabaseLayer12from canonical.testing import ZopelessDatabaseLayer
12from lp.translations.browser.sourcepackage import (13from lp.translations.browser.sourcepackage import (
@@ -15,13 +16,13 @@
15 ProductSeriesTranslationsExportView)16 ProductSeriesTranslationsExportView)
16from lp.translations.interfaces.translationfileformat import (17from lp.translations.interfaces.translationfileformat import (
17 TranslationFileFormat)18 TranslationFileFormat)
19from lp.translations.model.poexportrequest import POExportRequest
18from lp.testing import TestCaseWithFactory20from lp.testing import TestCaseWithFactory
1921
2022
21def wipe_queue(queue):23def wipe_queue(queue):
22 """Erase all export queue entries."""24 """Erase all export queue entries."""
23 while queue.entry_count > 0:25 IMasterStore(POExportRequest).execute("DELETE FROM POExportRequest")
24 queue.popRequest()
2526
2627
27class BaseExportViewMixin(TestCaseWithFactory):28class BaseExportViewMixin(TestCaseWithFactory):
2829
=== modified file 'lib/lp/translations/doc/poexport-queue.txt'
--- lib/lp/translations/doc/poexport-queue.txt 2010-02-19 16:02:16 +0000
+++ lib/lp/translations/doc/poexport-queue.txt 2010-03-06 06:29:39 +0000
@@ -275,7 +275,8 @@
275275
276Once the queue is processed, the queue is empty again.276Once the queue is processed, the queue is empty again.
277277
278 >>> process_queue(fake_transaction, logging.getLogger())278 >>> transaction.commit()
279 >>> process_queue(transaction, logging.getLogger())
279 INFO:...Stored file at http://.../po_evolution-2.2.pot280 INFO:...Stored file at http://.../po_evolution-2.2.pot
280281
281 >>> export_request_set.entry_count282 >>> export_request_set.entry_count
@@ -333,10 +334,10 @@
333334
334 >>> export_request_set.addRequest(335 >>> export_request_set.addRequest(
335 ... carlos, pofiles=[pofile], format=TranslationFileFormat.PO)336 ... carlos, pofiles=[pofile], format=TranslationFileFormat.PO)
336 >>> process_queue(fake_transaction, logging.getLogger())337 >>> transaction.commit()
338 >>> process_queue(transaction, logging.getLogger())
337 INFO:root:Stored file at http://...eo.po339 INFO:root:Stored file at http://...eo.po
338340
339 >>> transaction.commit()
340 >>> print get_newest_librarian_file().read()341 >>> print get_newest_librarian_file().read()
341 # Esperanto translation for ...342 # Esperanto translation for ...
342 ...343 ...
@@ -354,10 +355,10 @@
354355
355 >>> export_request_set.addRequest(356 >>> export_request_set.addRequest(
356 ... carlos, pofiles=[pofile], format=TranslationFileFormat.POCHANGED)357 ... carlos, pofiles=[pofile], format=TranslationFileFormat.POCHANGED)
357 >>> process_queue(fake_transaction, logging.getLogger())358 >>> transaction.commit()
359 >>> process_queue(transaction, logging.getLogger())
358 INFO:root:Stored file at http://...eo.po360 INFO:root:Stored file at http://...eo.po
359361
360 >>> transaction.commit()
361 >>> print get_newest_librarian_file().read()362 >>> print get_newest_librarian_file().read()
362 # IMPORTANT: This file does NOT contain a complete PO file structure.363 # IMPORTANT: This file does NOT contain a complete PO file structure.
363 # DO NOT attempt to import this file back into Launchpad.364 # DO NOT attempt to import this file back into Launchpad.
364365
=== modified file 'lib/lp/translations/doc/poexport-request-productseries.txt'
--- lib/lp/translations/doc/poexport-request-productseries.txt 2010-02-19 16:54:42 +0000
+++ lib/lp/translations/doc/poexport-request-productseries.txt 2010-03-06 06:29:39 +0000
@@ -35,11 +35,12 @@
35Now we request that the queue be processed.35Now we request that the queue be processed.
3636
37 >>> import logging37 >>> import logging
38 >>> from lp.testing.faketransaction import FakeTransaction38 >>> import transaction
39 >>> from lp.translations.scripts.po_export_queue import process_queue39 >>> from lp.translations.scripts.po_export_queue import process_queue
40 >>> logger = MockLogger()40 >>> logger = MockLogger()
41 >>> logger.setLevel(logging.DEBUG)41 >>> logger.setLevel(logging.DEBUG)
42 >>> process_queue(FakeTransaction(), logger)42 >>> transaction.commit()
43 >>> process_queue(transaction, logger)
43 log> Exporting objects for ..., related to template evolution-2.2 in44 log> Exporting objects for ..., related to template evolution-2.2 in
44 Evolution trunk45 Evolution trunk
45 log> Exporting objects for ..., related to template evolution-2.2-test in46 log> Exporting objects for ..., related to template evolution-2.2-test in
4647
=== modified file 'lib/lp/translations/doc/poexport-request.txt'
--- lib/lp/translations/doc/poexport-request.txt 2010-02-19 16:54:42 +0000
+++ lib/lp/translations/doc/poexport-request.txt 2010-03-06 06:29:39 +0000
@@ -53,9 +53,13 @@
5353
54Now we request that the queue be processed.54Now we request that the queue be processed.
5555
56 >>> from lp.testing.faketransaction import FakeTransaction56(Commits are needed to make the test requests seep through to the slave
57database).
58
59 >>> import transaction
57 >>> from lp.translations.scripts.po_export_queue import process_queue60 >>> from lp.translations.scripts.po_export_queue import process_queue
58 >>> process_queue(FakeTransaction(), MockLogger())61 >>> transaction.commit()
62 >>> process_queue(transaction, MockLogger())
59 log> Exporting objects for Happy Downloader, related to template pmount63 log> Exporting objects for Happy Downloader, related to template pmount
60 in Ubuntu Hoary package "pmount"64 in Ubuntu Hoary package "pmount"
61 log> Stored file at http://.../launchpad-export.tar.gz65 log> Stored file at http://.../launchpad-export.tar.gz
@@ -185,7 +189,8 @@
185 >>> from lp.translations.interfaces.translationfileformat import (189 >>> from lp.translations.interfaces.translationfileformat import (
186 ... TranslationFileFormat)190 ... TranslationFileFormat)
187 >>> request_set.addRequest(person, None, [cs], TranslationFileFormat.MO)191 >>> request_set.addRequest(person, None, [cs], TranslationFileFormat.MO)
188 >>> process_queue(FakeTransaction(), MockLogger())192 >>> transaction.commit()
193 >>> process_queue(transaction, MockLogger())
189 log> Exporting objects for Happy Downloader, related to template pmount194 log> Exporting objects for Happy Downloader, related to template pmount
190 in Ubuntu Hoary package "pmount"195 in Ubuntu Hoary package "pmount"
191 log> Stored file at http://.../cs_LC_MESSAGES_pmount.mo196 log> Stored file at http://.../cs_LC_MESSAGES_pmount.mo
192197
=== added file 'lib/lp/translations/doc/poexportqueue-replication-lag.txt'
--- lib/lp/translations/doc/poexportqueue-replication-lag.txt 1970-01-01 00:00:00 +0000
+++ lib/lp/translations/doc/poexportqueue-replication-lag.txt 2010-03-06 06:29:39 +0000
@@ -0,0 +1,89 @@
1= Replication Lag and the Export Queue =
2
3Due to replication lag it's possible for the export queue to see a
4request on the slave store that it actually just removed from the master
5store.
6
7We start our story with an empty export queue.
8
9 >>> from datetime import timedelta
10 >>> import transaction
11 >>> from zope.component import getUtility
12 >>> from canonical.launchpad.interfaces.lpstorm import IMasterStore
13 >>> from lp.translations.interfaces.poexportrequest import (
14 ... IPOExportRequestSet)
15 >>> from lp.translations.interfaces.pofile import IPOFile
16 >>> from lp.translations.model.poexportrequest import POExportRequest
17 >>> query = IMasterStore(POExportRequest).execute(
18 ... "DELETE FROM POExportRequest")
19
20 >>> queue = getUtility(IPOExportRequestSet)
21
22We have somebody making an export request.
23
24 >>> requester = factory.makePersonNoCommit(
25 ... email='punter@example.com', name='punter')
26
27 >>> template1 = factory.makePOTemplate()
28 >>> pofile1_be = factory.makePOFile('be', potemplate=template1)
29 >>> pofile1_ja = factory.makePOFile('ja', potemplate=template1)
30 >>> queue.addRequest(requester, template1, [pofile1_be, pofile1_ja])
31 >>> query = IMasterStore(POExportRequest).execute(
32 ... "UPDATE POExportRequest SET date_created = '2010-01-10'::date")
33
34Later, a different and separate request follows.
35
36 >>> template2 = factory.makePOTemplate()
37 >>> pofile2_se = factory.makePOFile('se', potemplate=template2)
38 >>> pofile2_ga = factory.makePOFile('ga', potemplate=template2)
39 >>> queue.addRequest(requester, template2, [pofile2_se, pofile2_ga])
40
41The database is replicated in this state.
42
43 >>> transaction.commit()
44
45getRequest at this point returns the oldest request.
46
47 >>> def summarize_request(request_tuple):
48 ... """Summarize files in export request."""
49 ... person, sources, format, request_ids = request_tuple
50 ... summary = []
51 ... for source in sources:
52 ... if IPOFile.providedBy(source):
53 ... summary.append(source.language.code)
54 ... else:
55 ... summary.append('(template)')
56 ... for entry in sorted(summary):
57 ... print entry
58
59 >>> summarize_request(queue.getRequest())
60 (template)
61 be
62 ja
63
64It doesn't modify the queue, so it'd say the same thing again if we
65were to ask again.
66
67 >>> repeated_request = queue.getRequest()
68 >>> summarize_request(repeated_request)
69 (template)
70 be
71 ja
72
73The first request is removed from the master store after processing, but
74not yet from the slave store. (Since this test is all one session, we
75can reproduce this by not committing the removal). The second request
76is still technically on the queue, but no longer "live."
77
78 >>> person, sources, format, request_ids = repeated_request
79 >>> print len(request_ids)
80 3
81 >>> queue.removeRequest(request_ids)
82
83In this state, despite the replication lag, getRequest is smart enough
84to return the second request, not the first.
85
86 >>> summarize_request(queue.getRequest())
87 (template)
88 ga
89 se
090
=== modified file 'lib/lp/translations/doc/potmsgset.txt'
--- lib/lp/translations/doc/potmsgset.txt 2009-10-23 12:44:32 +0000
+++ lib/lp/translations/doc/potmsgset.txt 2010-03-06 06:29:39 +0000
@@ -2,10 +2,14 @@
22
3POTMsgSet represents messages to translate that a POTemplate file has.3POTMsgSet represents messages to translate that a POTemplate file has.
44
5We need to get a POTMsgSet object to performe this test.5In this test we'll be committing a lot to let changes replicate to the
6slave database.
7
8 >>> import transaction
9
10We need to get a POTMsgSet object to perform this test.
611
7 >>> from zope.component import getUtility12 >>> from zope.component import getUtility
8 >>> from canonical.database.sqlbase import flush_database_updates
9 >>> from lp.translations.model.translationmessage import (13 >>> from lp.translations.model.translationmessage import (
10 ... TranslationMessage)14 ... TranslationMessage)
11 >>> from lp.translations.interfaces.potmsgset import IPOTMsgSet15 >>> from lp.translations.interfaces.potmsgset import IPOTMsgSet
@@ -734,7 +738,7 @@
734are not available as suggestions anymore:738are not available as suggestions anymore:
735739
736 >>> evo_distro_template.iscurrent = False740 >>> evo_distro_template.iscurrent = False
737 >>> flush_database_updates()741 >>> transaction.commit()
738 >>> suggestions = (742 >>> suggestions = (
739 ... evo_product_message.getExternallyUsedTranslationMessages(spanish))743 ... evo_product_message.getExternallyUsedTranslationMessages(spanish))
740 >>> len(suggestions)744 >>> len(suggestions)
@@ -748,7 +752,7 @@
748 # We set the template as current again so we are sure that we don't show752 # We set the template as current again so we are sure that we don't show
749 # suggestions just due to the change to the official_rosetta flag.753 # suggestions just due to the change to the official_rosetta flag.
750 >>> evo_distro_template.iscurrent = True754 >>> evo_distro_template.iscurrent = True
751 >>> flush_database_updates()755 >>> transaction.commit()
752 >>> suggestions = (756 >>> suggestions = (
753 ... evo_product_message.getExternallyUsedTranslationMessages(spanish))757 ... evo_product_message.getExternallyUsedTranslationMessages(spanish))
754 >>> len(suggestions)758 >>> len(suggestions)
@@ -757,7 +761,7 @@
757And products not using translations officially have the same behaviour.761And products not using translations officially have the same behaviour.
758762
759 >>> evolution.official_rosetta = False763 >>> evolution.official_rosetta = False
760 >>> flush_database_updates()764 >>> transaction.commit()
761 >>> suggestions = evo_distro_message.getExternallyUsedTranslationMessages(765 >>> suggestions = evo_distro_message.getExternallyUsedTranslationMessages(
762 ... spanish)766 ... spanish)
763 >>> len(suggestions)767 >>> len(suggestions)
@@ -767,7 +771,7 @@
767771
768 >>> ubuntu.official_rosetta = True772 >>> ubuntu.official_rosetta = True
769 >>> evolution.official_rosetta = True773 >>> evolution.official_rosetta = True
770 >>> flush_database_updates()774 >>> transaction.commit()
771775
772776
773== POTMsgSet.getExternallySuggestedTranslationMessages ==777== POTMsgSet.getExternallySuggestedTranslationMessages ==
@@ -850,7 +854,7 @@
850we get no suggestions.854we get no suggestions.
851855
852 >>> potmsgset_translated.potemplate.iscurrent = False856 >>> potmsgset_translated.potemplate.iscurrent = False
853 >>> flush_database_updates()857 >>> transaction.commit()
854858
855 >>> wiki_submissions = (859 >>> wiki_submissions = (
856 ... potmsgset_untranslated.getExternallyUsedTranslationMessages(860 ... potmsgset_untranslated.getExternallyUsedTranslationMessages(
@@ -865,7 +869,7 @@
865 # suggestions just due to the change to the official_rosetta flag.869 # suggestions just due to the change to the official_rosetta flag.
866 >>> potmsgset_translated.potemplate.iscurrent = True870 >>> potmsgset_translated.potemplate.iscurrent = True
867 >>> ubuntu.official_rosetta = False871 >>> ubuntu.official_rosetta = False
868 >>> flush_database_updates()872 >>> transaction.commit()
869873
870 >>> wiki_submissions = (874 >>> wiki_submissions = (
871 ... potmsgset_untranslated.getExternallyUsedTranslationMessages(875 ... potmsgset_untranslated.getExternallyUsedTranslationMessages(
872876
=== modified file 'lib/lp/translations/interfaces/poexportrequest.py'
--- lib/lp/translations/interfaces/poexportrequest.py 2010-02-19 16:02:16 +0000
+++ lib/lp/translations/interfaces/poexportrequest.py 2010-03-06 06:29:39 +0000
@@ -11,7 +11,7 @@
11 ]11 ]
1212
13from zope.interface import Interface13from zope.interface import Interface
14from zope.schema import Int, Object14from zope.schema import Datetime, Int, Object
1515
16from lp.registry.interfaces.person import IPerson16from lp.registry.interfaces.person import IPerson
17from lp.translations.interfaces.pofile import IPOFile17from lp.translations.interfaces.pofile import IPOFile
@@ -37,12 +37,25 @@
37 :param pofiles: A list of PO files to export.37 :param pofiles: A list of PO files to export.
38 """38 """
3939
40 def popRequest():40 def getRequest():
41 """Take the next request out of the queue.41 """Get the next request from the queue.
4242
43 Returns a 3-tuple containing the person who made the request, the PO43 Returns a tuple containing:
44 template the request was for, and a list of `POTemplate` and `POFile`44 * The person who made the request.
45 objects to export.45 * A list of POFiles and/or POTemplates that are to be exported.
46 * The requested `TranslationFileFormat`.
47 * The list of request record ids making up this request.
48
49 The objects are all read-only objects from the slave store. The
50 request ids list should be passed to `removeRequest` when
51 processing of the request completes.
52 """
53
54 def removeRequest(self, request_ids):
55 """Remove a request off the queue.
56
57 :param request_ids: A list of request record ids as returned by
58 `getRequest`.
46 """59 """
4760
4861
@@ -51,6 +64,9 @@
51 title=u'The person who made the request.',64 title=u'The person who made the request.',
52 required=True, readonly=True, schema=IPerson)65 required=True, readonly=True, schema=IPerson)
5366
67 date_created = Datetime(
68 title=u"Request's creation timestamp.", required=True, readonly=True)
69
54 potemplate = Object(70 potemplate = Object(
55 title=u'The translation template to which the requested file belong.',71 title=u'The translation template to which the requested file belong.',
56 required=True, readonly=True, schema=IPOTemplate)72 required=True, readonly=True, schema=IPOTemplate)
5773
=== modified file 'lib/lp/translations/interfaces/potmsgset.py'
--- lib/lp/translations/interfaces/potmsgset.py 2009-10-22 15:51:58 +0000
+++ lib/lp/translations/interfaces/potmsgset.py 2010-03-06 06:29:39 +0000
@@ -163,19 +163,25 @@
163 """163 """
164164
165 def getExternallyUsedTranslationMessages(language):165 def getExternallyUsedTranslationMessages(language):
166 """Returns all externally used translations.166 """Find externally used translations for the same message.
167167
168 External are those on other templates for the same English message.168 This is used to find suggestions for translating this
169 "Used" messages are either current or imported ones.169 `POTMsgSet` that are actually used (i.e. current or imported) in
170 other templates.
171
172 The suggestions are read-only; they come from the slave store.
170173
171 :param language: language we want translations for.174 :param language: language we want translations for.
172 """175 """
173176
174 def getExternallySuggestedTranslationMessages(language):177 def getExternallySuggestedTranslationMessages(language):
175 """Return all externally suggested translations.178 """Find externally suggested translations for the same message.
176179
177 External are those on other templates for the same English message.180 This is used to find suggestions for translating this
178 "Suggested" messages are those which are neither current nor imported.181 `POTMsgSet` that were entered in another context, but for the
182 same English text, and are not in actual use.
183
184 The suggestions are read-only; they come from the slave store.
179185
180 :param language: language we want translations for.186 :param language: language we want translations for.
181 """187 """
182188
=== modified file 'lib/lp/translations/model/poexportrequest.py'
--- lib/lp/translations/model/poexportrequest.py 2010-02-19 17:06:01 +0000
+++ lib/lp/translations/model/poexportrequest.py 2010-03-06 06:29:39 +0000
@@ -15,9 +15,12 @@
15from zope.component import getUtility15from zope.component import getUtility
16from zope.interface import implements16from zope.interface import implements
1717
18from canonical.database.constants import DEFAULT
19from canonical.database.datetimecol import UtcDateTimeCol
20from canonical.database.enumcol import EnumCol
18from canonical.database.sqlbase import quote, SQLBase, sqlvalues21from canonical.database.sqlbase import quote, SQLBase, sqlvalues
19from canonical.database.enumcol import EnumCol
2022
23from canonical.launchpad.interfaces.lpstorm import IMasterStore, ISlaveStore
21from lp.translations.interfaces.poexportrequest import (24from lp.translations.interfaces.poexportrequest import (
22 IPOExportRequest, IPOExportRequestSet)25 IPOExportRequest, IPOExportRequestSet)
23from lp.translations.interfaces.potemplate import IPOTemplate26from lp.translations.interfaces.potemplate import IPOTemplate
@@ -112,36 +115,69 @@
112 existing.id IS NULL115 existing.id IS NULL
113 """ % query_params)116 """ % query_params)
114117
115 def popRequest(self):118 def _getOldestLiveRequest(self):
116 """See `IPOExportRequestSet`."""119 """Return the oldest live request on the master store.
117 try:120
118 request = POExportRequest.select(limit=1, orderBy='id')[0]121 Due to replication lag, the master store is always a little
119 except IndexError:122 ahead of the slave store that exports come from.
120 return None, None, None123 """
121124 master_store = IMasterStore(POExportRequest)
122 person = request.person125 sorted_by_id = master_store.find(POExportRequest).order_by(
123 format = request.format126 POExportRequest.id)
124127 return sorted_by_id.first()
125 query = """128
126 person = %s AND129 def _getHeadRequest(self):
127 format = %s AND130 """Return oldest request on the queue."""
128 date_created = (131 # Due to replication lag, it's possible that the slave store
129 SELECT date_created132 # still has copies of requests that have already been completed
130 FROM POExportRequest133 # and deleted from the master store. So first get the oldest
131 ORDER BY id134 # request that is "live," i.e. still present on the master
132 LIMIT 1)""" % sqlvalues(person, format)135 # store.
133 requests = POExportRequest.select(query, orderBy='potemplate')136 oldest_live = self._getOldestLiveRequest()
134 objects = []137 if oldest_live is None:
135138 return None
136 for request in requests:139 else:
137 if request.pofile is not None:140 return ISlaveStore(POExportRequest).find(
138 objects.append(request.pofile)141 POExportRequest,
139 else:142 POExportRequest.id == oldest_live.id).one()
140 objects.append(request.potemplate)143
141144 def getRequest(self):
142 POExportRequest.delete(request.id)145 """See `IPOExportRequestSet`."""
143146 # Exports happen off the slave store. To ensure that export
144 return person, objects, format147 # does not happen until requests have been replicated to the
148 # slave, they are read primarily from the slave even though they
149 # are deleted on the master afterwards.
150 head = self._getHeadRequest()
151 if head is None:
152 return None, None, None, None
153
154 requests = ISlaveStore(POExportRequest).find(
155 POExportRequest,
156 POExportRequest.person == head.person,
157 POExportRequest.format == head.format,
158 POExportRequest.date_created == head.date_created).order_by(
159 POExportRequest.potemplateID)
160
161 summary = [
162 (request.id, request.pofile or request.potemplate)
163 for request in requests
164 ]
165
166 sources = [source for request_id, source in summary]
167 request_ids = [request_id for request_id, source in summary]
168
169 return head.person, sources, head.format, request_ids
170
171 def removeRequest(self, request_ids):
172 """See `IPOExportRequestSet`."""
173 if len(request_ids) > 0:
174 # Storm 0.15 does not have direct support for deleting based
175 # on is_in expressions and such, so do it the hard way.
176 ids_string = ', '.join(sqlvalues(*request_ids))
177 IMasterStore(POExportRequest).execute("""
178 DELETE FROM POExportRequest
179 WHERE id in (%s)
180 """ % ids_string)
145181
146182
147class POExportRequest(SQLBase):183class POExportRequest(SQLBase):
@@ -152,6 +188,7 @@
152 person = ForeignKey(188 person = ForeignKey(
153 dbName='person', foreignKey='Person',189 dbName='person', foreignKey='Person',
154 storm_validator=validate_public_person, notNull=True)190 storm_validator=validate_public_person, notNull=True)
191 date_created = UtcDateTimeCol(dbName='date_created', default=DEFAULT)
155 potemplate = ForeignKey(dbName='potemplate', foreignKey='POTemplate',192 potemplate = ForeignKey(dbName='potemplate', foreignKey='POTemplate',
156 notNull=True)193 notNull=True)
157 pofile = ForeignKey(dbName='pofile', foreignKey='POFile')194 pofile = ForeignKey(dbName='pofile', foreignKey='POFile')
158195
=== modified file 'lib/lp/translations/model/potmsgset.py'
--- lib/lp/translations/model/potmsgset.py 2010-01-28 09:53:45 +0000
+++ lib/lp/translations/model/potmsgset.py 2010-03-06 06:29:39 +0000
@@ -15,6 +15,7 @@
15from zope.component import getUtility15from zope.component import getUtility
1616
17from sqlobject import ForeignKey, IntCol, StringCol, SQLObjectNotFound17from sqlobject import ForeignKey, IntCol, StringCol, SQLObjectNotFound
18from storm.expr import SQL
18from storm.store import EmptyResultSet, Store19from storm.store import EmptyResultSet, Store
1920
20from canonical.config import config21from canonical.config import config
@@ -27,6 +28,7 @@
27from canonical.launchpad.interfaces.launchpad import ILaunchpadCelebrities28from canonical.launchpad.interfaces.launchpad import ILaunchpadCelebrities
28from canonical.launchpad.readonly import is_read_only29from canonical.launchpad.readonly import is_read_only
29from canonical.launchpad.webapp.interfaces import UnexpectedFormData30from canonical.launchpad.webapp.interfaces import UnexpectedFormData
31from canonical.launchpad.interfaces.lpstorm import ISlaveStore
30from lp.translations.interfaces.pofile import IPOFileSet32from lp.translations.interfaces.pofile import IPOFileSet
31from lp.translations.interfaces.potmsgset import (33from lp.translations.interfaces.potmsgset import (
32 BrokenTextError,34 BrokenTextError,
@@ -324,6 +326,9 @@
324326
325 A message is used if it's either imported or current, and unused327 A message is used if it's either imported or current, and unused
326 otherwise.328 otherwise.
329
330 Suggestions are read-only, so these objects come from the slave
331 store.
327 """332 """
328 if not config.rosetta.global_suggestions_enabled:333 if not config.rosetta.global_suggestions_enabled:
329 return []334 return []
@@ -387,7 +392,9 @@
387 ORDER BY %(msgstrs)s, date_created DESC392 ORDER BY %(msgstrs)s, date_created DESC
388 ''' % ids_query_params393 ''' % ids_query_params
389394
390 result = TranslationMessage.select('id IN (%s)' % ids_query)395 result = ISlaveStore(TranslationMessage).find(
396 TranslationMessage,
397 TranslationMessage.id.is_in(SQL(ids_query)))
391398
392 return shortlist(result, longest_expected=100, hardlimit=2000)399 return shortlist(result, longest_expected=100, hardlimit=2000)
393400
394401
=== modified file 'lib/lp/translations/scripts/po_export_queue.py'
--- lib/lp/translations/scripts/po_export_queue.py 2009-08-17 13:42:00 +0000
+++ lib/lp/translations/scripts/po_export_queue.py 2010-03-06 06:29:39 +0000
@@ -373,32 +373,18 @@
373373
374374
375def process_queue(transaction_manager, logger):375def process_queue(transaction_manager, logger):
376 """Process each request in the translation export queue.376 """Process all requests in the translation export queue.
377377
378 Each item is removed from the queue as it is processed, we only handle378 Each item is removed from the queue as it is processed.
379 one request with each function call.
380 """379 """
381 request_set = getUtility(IPOExportRequestSet)380 request_set = getUtility(IPOExportRequestSet)
382381 no_request = (None, None, None, None)
383 request = request_set.popRequest()382
384383 request = request_set.getRequest()
385 if None in request:384 while request != no_request:
386 # Any value is None and we must have all values as not None to have385 person, objects, format, request_ids = request
387 # something to process...
388 return
389
390 person, objects, format = request
391
392 try:
393 process_request(person, objects, format, logger)386 process_request(person, objects, format, logger)
394 except psycopg2.Error:387 request_set.removeRequest(request_ids)
395 # We had a DB error, we don't try to recover it here, just exit
396 # from the script and next run will retry the export.
397 logger.error(
398 "A DB exception was raised when exporting files for %s" % (
399 person.displayname),
400 exc_info=True)
401 transaction_manager.abort()
402 else:
403 # Apply all changes.
404 transaction_manager.commit()388 transaction_manager.commit()
389
390 request = request_set.getRequest()
405391
=== modified file 'lib/lp/translations/tests/test_potmsgset.py'
--- lib/lp/translations/tests/test_potmsgset.py 2009-11-25 15:08:26 +0000
+++ lib/lp/translations/tests/test_potmsgset.py 2010-03-06 06:29:39 +0000
@@ -1,4 +1,4 @@
1# Copyright 2009 Canonical Ltd. This software is licensed under the1# Copyright 2009-2010 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4# pylint: disable-msg=C01024# pylint: disable-msg=C0102
@@ -329,6 +329,8 @@
329 external_pofile = self.factory.makePOFile('sr', external_template)329 external_pofile = self.factory.makePOFile('sr', external_template)
330 serbian = external_pofile.language330 serbian = external_pofile.language
331331
332 transaction.commit()
333
332 # When there is no translation for the external POTMsgSet,334 # When there is no translation for the external POTMsgSet,
333 # no externally used suggestions are returned.335 # no externally used suggestions are returned.
334 self.assertEquals(336 self.assertEquals(
@@ -340,6 +342,9 @@
340 external_suggestion = self.factory.makeSharedTranslationMessage(342 external_suggestion = self.factory.makeSharedTranslationMessage(
341 pofile=external_pofile, potmsgset=external_potmsgset,343 pofile=external_pofile, potmsgset=external_potmsgset,
342 suggestion=True)344 suggestion=True)
345
346 transaction.commit()
347
343 self.assertEquals(348 self.assertEquals(
344 self.potmsgset.getExternallyUsedTranslationMessages(serbian),349 self.potmsgset.getExternallyUsedTranslationMessages(serbian),
345 [])350 [])
@@ -350,6 +355,9 @@
350 pofile=external_pofile, potmsgset=external_potmsgset,355 pofile=external_pofile, potmsgset=external_potmsgset,
351 suggestion=False, is_imported=True)356 suggestion=False, is_imported=True)
352 imported_translation.is_current = False357 imported_translation.is_current = False
358
359 transaction.commit()
360
353 self.assertEquals(361 self.assertEquals(
354 self.potmsgset.getExternallyUsedTranslationMessages(serbian),362 self.potmsgset.getExternallyUsedTranslationMessages(serbian),
355 [imported_translation])363 [imported_translation])
@@ -359,6 +367,9 @@
359 current_translation = self.factory.makeSharedTranslationMessage(367 current_translation = self.factory.makeSharedTranslationMessage(
360 pofile=external_pofile, potmsgset=external_potmsgset,368 pofile=external_pofile, potmsgset=external_potmsgset,
361 suggestion=False, is_imported=False)369 suggestion=False, is_imported=False)
370
371 transaction.commit()
372
362 self.assertEquals(373 self.assertEquals(
363 self.potmsgset.getExternallyUsedTranslationMessages(serbian),374 self.potmsgset.getExternallyUsedTranslationMessages(serbian),
364 [imported_translation, current_translation])375 [imported_translation, current_translation])
@@ -377,6 +388,8 @@
377 external_pofile = self.factory.makePOFile('sr', external_template)388 external_pofile = self.factory.makePOFile('sr', external_template)
378 serbian = external_pofile.language389 serbian = external_pofile.language
379390
391 transaction.commit()
392
380 # When there is no translation for the external POTMsgSet,393 # When there is no translation for the external POTMsgSet,
381 # no externally used suggestions are returned.394 # no externally used suggestions are returned.
382 self.assertEquals(395 self.assertEquals(
@@ -388,6 +401,9 @@
388 external_suggestion = self.factory.makeSharedTranslationMessage(401 external_suggestion = self.factory.makeSharedTranslationMessage(
389 pofile=external_pofile, potmsgset=external_potmsgset,402 pofile=external_pofile, potmsgset=external_potmsgset,
390 suggestion=True)403 suggestion=True)
404
405 transaction.commit()
406
391 self.assertEquals(407 self.assertEquals(
392 self.potmsgset.getExternallySuggestedTranslationMessages(serbian),408 self.potmsgset.getExternallySuggestedTranslationMessages(serbian),
393 [external_suggestion])409 [external_suggestion])
@@ -398,6 +414,9 @@
398 pofile=external_pofile, potmsgset=external_potmsgset,414 pofile=external_pofile, potmsgset=external_potmsgset,
399 suggestion=False, is_imported=True)415 suggestion=False, is_imported=True)
400 imported_translation.is_current = False416 imported_translation.is_current = False
417
418 transaction.commit()
419
401 self.assertEquals(420 self.assertEquals(
402 self.potmsgset.getExternallySuggestedTranslationMessages(serbian),421 self.potmsgset.getExternallySuggestedTranslationMessages(serbian),
403 [external_suggestion])422 [external_suggestion])
@@ -407,6 +426,9 @@
407 current_translation = self.factory.makeSharedTranslationMessage(426 current_translation = self.factory.makeSharedTranslationMessage(
408 pofile=external_pofile, potmsgset=external_potmsgset,427 pofile=external_pofile, potmsgset=external_potmsgset,
409 suggestion=False, is_imported=False)428 suggestion=False, is_imported=False)
429
430 transaction.commit()
431
410 self.assertEquals(432 self.assertEquals(
411 self.potmsgset.getExternallySuggestedTranslationMessages(serbian),433 self.potmsgset.getExternallySuggestedTranslationMessages(serbian),
412 [external_suggestion])434 [external_suggestion])
413435
=== modified file 'lib/lp/translations/tests/test_suggestions.py'
--- lib/lp/translations/tests/test_suggestions.py 2009-07-17 00:26:05 +0000
+++ lib/lp/translations/tests/test_suggestions.py 2010-03-06 06:29:39 +0000
@@ -1,10 +1,11 @@
1# Copyright 2009 Canonical Ltd. This software is licensed under the1# Copyright 2009-2010 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4__metaclass__ = type4__metaclass__ = type
55
6from datetime import datetime, timedelta6from datetime import datetime, timedelta
7from pytz import timezone7from pytz import timezone
8import transaction
8import unittest9import unittest
910
10import gettextpo11import gettextpo
@@ -66,6 +67,8 @@
66 ["foutmelding 936"], is_imported=False,67 ["foutmelding 936"], is_imported=False,
67 lock_timestamp=None)68 lock_timestamp=None)
6869
70 transaction.commit()
71
69 used_suggestions = foomsg.getExternallyUsedTranslationMessages(72 used_suggestions = foomsg.getExternallyUsedTranslationMessages(
70 self.nl)73 self.nl)
71 other_suggestions = foomsg.getExternallySuggestedTranslationMessages(74 other_suggestions = foomsg.getExternallySuggestedTranslationMessages(
@@ -88,6 +91,8 @@
88 ["foutmelding 936"], is_imported=False,91 ["foutmelding 936"], is_imported=False,
89 lock_timestamp=None)92 lock_timestamp=None)
9093
94 transaction.commit()
95
91 # There is a global (externally used) suggestion.96 # There is a global (externally used) suggestion.
92 used_suggestions = foomsg.getExternallyUsedTranslationMessages(97 used_suggestions = foomsg.getExternallyUsedTranslationMessages(
93 self.nl)98 self.nl)
@@ -117,6 +122,8 @@
117 is_imported=False, lock_timestamp=None)122 is_imported=False, lock_timestamp=None)
118 suggestion.is_current = False123 suggestion.is_current = False
119124
125 transaction.commit()
126
120 used_suggestions = foomsg.getExternallyUsedTranslationMessages(127 used_suggestions = foomsg.getExternallyUsedTranslationMessages(
121 self.nl)128 self.nl)
122 other_suggestions = foomsg.getExternallySuggestedTranslationMessages(129 other_suggestions = foomsg.getExternallySuggestedTranslationMessages(
123130
=== modified file 'lib/lp/translations/tests/test_translatablemessage.py'
--- lib/lp/translations/tests/test_translatablemessage.py 2009-08-26 16:24:02 +0000
+++ lib/lp/translations/tests/test_translatablemessage.py 2010-03-06 06:29:39 +0000
@@ -1,4 +1,4 @@
1# Copyright 2009 Canonical Ltd. This software is licensed under the1# Copyright 2009-2010 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4"""Unit tests for `TranslatableMessage`."""4"""Unit tests for `TranslatableMessage`."""
@@ -7,6 +7,7 @@
77
8from datetime import datetime, timedelta8from datetime import datetime, timedelta
9import pytz9import pytz
10import transaction
10from unittest import TestLoader11from unittest import TestLoader
1112
12from lp.testing import TestCaseWithFactory13from lp.testing import TestCaseWithFactory
@@ -147,10 +148,12 @@
147 self.message = TranslatableMessage(self.potmsgset, self.pofile)148 self.message = TranslatableMessage(self.potmsgset, self.pofile)
148149
149 def test_getExternalTranslations(self):150 def test_getExternalTranslations(self):
151 transaction.commit()
150 externals = self.message.getExternalTranslations()152 externals = self.message.getExternalTranslations()
151 self.assertContentEqual([self.external_current], externals)153 self.assertContentEqual([self.external_current], externals)
152154
153 def test_getExternalSuggestions(self):155 def test_getExternalSuggestions(self):
156 transaction.commit()
154 externals = self.message.getExternalSuggestions()157 externals = self.message.getExternalSuggestions()
155 self.assertContentEqual([self.external_suggestion], externals)158 self.assertContentEqual([self.external_suggestion], externals)
156159