Merge lp:~jtv/launchpad/bug-409330 into lp:launchpad

= Bug 409330 =

The nightly cron job that verifies the cached statistics for POFiles is
taking far too long. The last run pushed the nightly script as a whole
over 24 hours, which obviously causes trouble.

With message sharing, this script has become more important than it was.
A change to one template or translation can have all sorts of influence
on other templates and/or translations, and we leave it to the script to
set things straight offline.

Message sharing also makes it slightly too complex to find exactly the
POFiles whose statistics most need updating. We may still do that later
but for now we have an immediate problem to solve.

The only remaining way is to use the starting-ID option. This tells the
job to consider only POFiles with ids starting at a given number. We
can run the script less frequently on older POFiles, on the assumption
that newer POFiles are more likely to be updated more frequently. (It's
a widely effective optimization assumption that new data is likely to be
hot data).

Looking at recent logs, the sweet spot seems to lie between ids 870,000
and 880,000. Starting there covers almost all updates that are usually
made by the script; it's the point where the changes start to become
dense. On the most recent script run, it would have saved slightly over
half the script's run time while still doing almost all of the same job.

We can tweak this number later if needed, or come up with a better
mechanism to approximate the right set of POFiles.

== Implementation ==

The option to start verifying at a given POFile id was already present
in the lower levels of this script's implementation, but the script
object did not expose it to the command line yet. This branch adds the
corresponding command-line option.

It also takes the stats script run out of nightly.sh, so that that
script can stay nicely within 24 hours of run time. The script is to be
run daily as:

    python2.4 rosetta-pofile-stats.py --start-id 875000

And less frequently, say weekly, we'll want a slow but complete run:

    python2.4 rosetta-pofile-stats.py

Finally, the script is made to use DBLoopTuner instead of LoopTuner to
make it a bit more mindful of other clients on the database.

== Tests ==

{{{
./bin/test -vv -t pofile-verify-stats
}}}

== Demo and Q/A ==

After the change, perform a partial script run:

    python2.4 rosetta-pofile-stats.py --start-id 875000

This should take at least 3 hours, probably more, and update a lot of
POFiles. It's hard to predict the number, but at a guess I'd expect it
to be close to a thousand. (Though it may also be as low as a hundred).

Now perform a full script run:

    python2.4 rosetta-pofile-stats.py

This should take very roughly twice as long, but update much fewer
incorrect statistics. The partial run took care of most.

== lint ==

No lint complaints.

Jeroen

Jeroen,

Thanks for investigating and find a quick solution for this production bug.

You need a new test for VerifyPOFileStats, to certify 'start_id' is handled as you expect. Other than that the code changes are good and harmless.

I'm assuming you are coordinating these changes with LOSAs, so they can add the new cronjob with the right parameters in production once it's CPed.