Builder history timing out

Bug #616331 reported by Michael Nelson
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
High
Michael Nelson

Bug Description

Currently on dogfood (and apparently briefly on staging, but I don't know the details) the following page is timing out:

https://dogfood.launchpad.net/builders/rubidium/+history

Based on the call-site, this seems to have been triggered now by a combination of (1) a change in the previous release (10.07) which updated the history to display different types of builds in the history, although this didn't cause problems as there were no other types of BuildFarmJobs other than Binary ones, and (2) a change in the current release (10.08) which re-works Source package recipe builds so that they are now BuildFarmJobs as well.

Query plan issued on dogfood by Julian: http://pastebin.ubuntu.com/476428/

Example raw oops from dogfood: https://pastebin.canonical.com/35713/

Note: I'm creating this bug based on email snippets I was forwarded... if you've better info, please update the description

Related branches

Revision history for this message
Michael Nelson (michael.nelson) wrote :

Stuart is going to analyse the query and I'll look at an option to simply return all builds (rather than filtering private ones out via complex queries), and displaying 'Private build' in the history where appropriate.

tags: added: buildfarm oops timeout
Revision history for this message
Stuart Bishop (stub) wrote :

I can't reproduce this on production as the data is different.

I'd be interested in seeing how the following rewritten query performs on dogfood:

explain analyze SELECT COUNT(*) FROM BuildFarmJob
JOIN PackageBuild ON PackageBuild.build_farm_job = BuildFarmJob.id
JOIN Archive ON PackageBuild.archive = Archive.id
WHERE BuildFarmJob.builder = 106;

Revision history for this message
Stuart Bishop (stub) wrote :

The rewritten query is fast. Looks like the odd join syntax was forcing a bad query plan.

Revision history for this message
Michael Nelson (michael.nelson) wrote :

Here's an oops when accessing the history page as a non-dogfood admin: https://pastebin.canonical.com/35735/

Revision history for this message
Michael Nelson (michael.nelson) wrote :

Just FTR, the remainder of the pre-imp. discussion:

{{{
17:01 < stub> explain analyze
17:01 < stub> SELECT COUNT(DISTINCT BuildFarmJob.id)
17:01 < stub> FROM BuildFarmJob
17:01 < stub> LEFT OUTER JOIN PackageBuild ON PackageBuild.build_farm_job = BuildFarmJob.id
17:01 < stub> LEFT OUTER JOIN Archive ON PackageBuild.archive = Archive.id
17:01 < stub> LEFT OUTER JOIN TeamParticipation ON TeamParticipation.team = Archive.owner
17:01 < stub> WHERE
17:01 < stub> BuildFarmJob.builder = 106
17:01 < stub> AND (Archive.private IS FALSE OR TeamParticipation.person = 1)
17:01 < stub> (I'm not sure of who a suitable owner would be, so using '1')
17:01 < jtv> (note to self: I was talking nonsense earlier)
17:02 < bigjools> noodles775: I am guessing that we won't be able to make a branch for this in the next few minutes.
17:02 < noodles775> Thanks stub: I'll try the storm equivalent.
17:02 < noodles775> bigjools: no.
17:02 < bigjools> noodles775: ok, we'll have to release with this broken, and do a re-roll or CP
}}}

Changed in soyuz:
assignee: nobody → Michael Nelson (michael.nelson)
Revision history for this message
Michael Nelson (michael.nelson) wrote :

I've applied the linked branch on dogfood and verified that the history now displays fine:

https://dogfood.launchpad.net/builders/rubidium/+history

description: updated
Revision history for this message
Robert Collins (lifeless) wrote :

Tweaked priority as per zero-oops-policy ( I know its being worked on :))

tags: removed: oops
Changed in soyuz:
importance: Critical → High
Changed in soyuz:
status: Triaged → In Progress
Changed in soyuz:
status: In Progress → Fix Released
Revision history for this message
Launchpad QA Bot (lpqabot) wrote : Bug fixed by a commit
tags: added: qa-needstesting
tags: added: qa-ok
removed: qa-needstesting
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.