Launchpad itself

Merge lp:~jtv/launchpad/bug-408206 into lp:launchpad

  1. bug-408206
  2. Merge into devel
Proposed by Jeroen T. Vermeulen
Status: Merged
Approved by: Jeroen T. Vermeulen
Approved revision: no longer in the source branch.
Merged at revision: not available
Proposed branch: lp:~jtv/launchpad/bug-408206
Merge into: lp:launchpad
Diff against target: None lines
To merge this branch: bzr merge lp:~jtv/launchpad/bug-408206
Reviewer Review Type Date Requested Status
Stuart Bishop (community) Approve
Review via email: mp+9579@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

= Bugs 408067, 408319 =

It seems that ddtp-ubuntu is using export-to-branch now. These are the
Ubuntu package descriptions, and by far our largest translation files.
Whatever new ground this project steps on with Translations, it finds
some quicksand.

The export takes too long, and gets killed by a database watchdog. The
script is built to resume after such a failure, but the iteration of
productseries to service then breaks around item #10. Apparently it's
getting batched, with the transaction the result is coming from already
broken.

This branch fixes three issues exposed by this failure:

 * The transaction gets too long. This fix commits after every
   translation file as well as after every productseries.

 * Failures are logged as Warning, not as Error. Failures like this one
   shouldn't be fatal to the script, but it starts logging them as
   errors anyway.

 * After the error the script keeps iterating over a result set it first
   acquired before the abort. That becomes a shortlist, limited to a
   reasonable number of productseries for the script to hold in memory.

None of these are particularly testworthy. To QA, re-enable ddtp-ubuntu
exports and see what breaks next. :-)

Jeroen

Revision history for this message
Stuart Bishop (stub) wrote :

Fine.

Consider using transaction.commit() instead of self.txn.commit() - we are slowly killing off that old Zopeless stuff.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/lp/translations/scripts/translations_to_branch.py'
2--- lib/lp/translations/scripts/translations_to_branch.py 2009-07-17 18:46:25 +0000
3+++ lib/lp/translations/scripts/translations_to_branch.py 2009-08-03 11:34:35 +0000
4@@ -12,6 +12,7 @@
5
6 from storm.expr import Join, SQL
7
8+from canonical.launchpad.helpers import shortlist
9 from lp.codehosting.vfs import get_multi_server
10 from lp.translations.interfaces.potemplate import IPOTemplateSet
11 from canonical.launchpad.webapp.interfaces import (
12@@ -80,6 +81,12 @@
13
14 committer.writeFile(pofile_path, pofile_contents)
15
16+ # We're not actually writing any changes to the
17+ # database, but it's not polite to stay in one
18+ # transaction for too long.
19+ if self.txn:
20+ self.txn.commit()
21+
22 self._commit(source, committer)
23 finally:
24 committer.unlock()
25@@ -88,20 +95,20 @@
26 """Loop over `productseries_iter` and export their translations."""
27 items_done = 0
28 items_failed = 0
29- for source in productseries_iter:
30+
31+ productseries = shortlist(productseries_iter, longest_expected=2000)
32+
33+ for source in productseries:
34 try:
35 self._exportToBranch(source)
36
37- # We're not actually writing any changes to the
38- # database, but it's not polite to stay in one
39- # transaction for too long.
40 if self.txn:
41 self.txn.commit()
42 except (KeyboardInterrupt, SystemExit):
43 raise
44 except Exception, e:
45 items_failed += 1
46- self.logger.warn("Failure: %s" % e)
47+ self.logger.error("Failure: %s" % e)
48 if self.txn:
49 self.txn.abort()
50