Merge lp:~al-maisan/launchpad/xx-next-builder-502927 into lp:launchpad

Hello there!

This is the next step in getting us to a point where we can estimate build
farm job dispatch times irrespective of job type (binary build, "create source
package from recipe" job, translation job).

The dispatch time estimation requires us to know when the next builder capable
of running the job of interest (JOI) will become available. The branch at hand
adds that logic.

Tests to run:

    bin/test -vv -t test_buildqueue

No "make lint" errors or warnings.

Hi Muharem,

Great branch this; I'm really impressed with the whole thing. Couple of
minor nitpicks but otherwise this is r=me.

> === modified file 'lib/lp/soyuz/model/buildqueue.py'
> --- lib/lp/soyuz/model/buildqueue.py 2010-01-04 14:30:52 +0000
> +++ lib/lp/soyuz/model/buildqueue.py 2010-01-06 14:11:14 +0000
> @@ -212,6 +212,83 @@
> free_builders = result_set.get_one()[0]
> return free_builders
>
> + def _estimateTimeToNextBuilder(
> + self, head_job_processor, head_job_virtualized):
> + """Estimate time until next builder becomes available.
> +
> + For the purpose of estimating the dispatch time of the job of interest
> + (JOI) we need to know how long it will take until the job at the head
> + of JOI's queue is dispatched.
> +
> + There are two cases to consider here: the head job is
> +
> + - processor dependent: only builders with the matching
> + processor/virtualization combination should be considered.
> + - *not* processor dependent: all builders should be considered.
> +
> + :param head_job_processor: The processor required by the job at the
> + head of the queue.
> + :param head_job_virtualized: The virtualization setting required by
> + the job at the head of the queue.
> + :return: The estimated number of seconds untils a builder capable of
> + running the head job becomes available or None if no such builder
> + exists.
> + """
> + store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
> +
> + # First check whether we have free builders.
> + free_builders = self._freeBuildersCount(
> + head_job_processor, head_job_virtualized)
> +
> + if free_builders > 0:
> + # We have free builders for the given processor/virtualization
> + # combination -> zero delay
> + return 0
> +
> + extra_clauses = ''
> + if head_job_processor is not None:
> + # Only look at builders with specific processor types.
> + extra_clauses += """
> + AND Builder.processor = %s
> + AND Builder.virtualized = %s
> + """ % sqlvalues(head_job_processor, head_job_virtualized)
> +
> + params = sqlvalues(JobStatus.RUNNING) + (extra_clauses,)
> +
> + delay_query = """
> + SELECT MIN(
> + CASE WHEN
> + EXTRACT(EPOCH FROM
> + (BuildQueue.estimated_duration -
> + (((now() AT TIME ZONE 'UTC') - Job.date_started)))) >= 0
> + THEN
> + EXTRACT(EPOCH FROM
> + (BuildQueue.estimated_duration -
> + (((now() AT TIME ZONE 'UTC') - Job.date_started))))
> + ELSE
> + -- Assume that jobs that have overdrawn their estimated
> + -- duration time budget will complete within 2 minutes.
> + -- This is a wild guess but has worked well so far.

As discussed on IRC, this comment should indicate that nothing will
actually break if this wild guess i...

Graham Binns wrote:
> Review: Approve code
> Hi Muharem,
>
> Great branch this; I'm really impressed with the whole thing. Couple of
> minor nitpicks but otherwise this is r=me.

Hello Graham,

thank you for being so courteous and reviewing this branch on request.
I have revised the branch to accommodate all your suggestions. Please
find the incremental diff enclosed.

>
>> === modified file 'lib/lp/soyuz/model/buildqueue.py'
>> --- lib/lp/soyuz/model/buildqueue.py 2010-01-04 14:30:52 +0000
>> +++ lib/lp/soyuz/model/buildqueue.py 2010-01-06 14:11:14 +0000
>> @@ -212,6 +212,83 @@
>> free_builders = result_set.get_one()[0]
>> return free_builders
>>
>> + def _estimateTimeToNextBuilder(
>> + self, head_job_processor, head_job_virtualized):
>> + """Estimate time until next builder becomes available.
>> +
>> + For the purpose of estimating the dispatch time of the job of interest
>> + (JOI) we need to know how long it will take until the job at the head
>> + of JOI's queue is dispatched.
>> +
>> + There are two cases to consider here: the head job is
>> +
>> + - processor dependent: only builders with the matching
>> + processor/virtualization combination should be considered.
>> + - *not* processor dependent: all builders should be considered.
>> +
>> + :param head_job_processor: The processor required by the job at the
>> + head of the queue.
>> + :param head_job_virtualized: The virtualization setting required by
>> + the job at the head of the queue.
>> + :return: The estimated number of seconds untils a builder capable of
>> + running the head job becomes available or None if no such builder
>> + exists.
>> + """
>> + store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
>> +
>> + # First check whether we have free builders.
>> + free_builders = self._freeBuildersCount(
>> + head_job_processor, head_job_virtualized)
>> +
>> + if free_builders > 0:
>> + # We have free builders for the given processor/virtualization
>> + # combination -> zero delay
>> + return 0
>> +
>> + extra_clauses = ''
>> + if head_job_processor is not None:
>> + # Only look at builders with specific processor types.
>> + extra_clauses += """
>> + AND Builder.processor = %s
>> + AND Builder.virtualized = %s
>> + """ % sqlvalues(head_job_processor, head_job_virtualized)
>> +
>> + params = sqlvalues(JobStatus.RUNNING) + (extra_clauses,)
>> +
>> + delay_query = """
>> + SELECT MIN(
>> + CASE WHEN
>> + EXTRACT(EPOCH FROM
>> + (BuildQueue.estimated_duration -
>> + (((now() AT TIME ZONE 'UTC') - Job.date_started)))) >= 0
>> + THEN
>> + EXTRACT(EPOCH FROM
>> + (BuildQueue.estimated_duration -
>> + (((now() AT TIME ZONE 'UTC') - Job.date_started))))
>> + ELSE
>> + ...