Merge lp:~al-maisan/launchpad/spec-job-class-503839 into lp:launchpad
- spec-job-class-503839
- Merge into devel
| Status: | Merged |
|---|---|
| Merged at revision: | not available |
| Proposed branch: | lp:~al-maisan/launchpad/spec-job-class-503839 |
| Merge into: | lp:launchpad |
| Diff against target: |
182 lines (+114/-3) 4 files modified
lib/lp/soyuz/configure.zcml (+10/-0) lib/lp/soyuz/interfaces/buildqueue.py (+4/-0) lib/lp/soyuz/model/buildqueue.py (+22/-3) lib/lp/soyuz/tests/test_buildqueue.py (+78/-0) |
| To merge this branch: | bzr merge lp:~al-maisan/launchpad/spec-job-class-503839 |
| Related bugs: |
| Reviewer | Review Type | Date Requested | Status |
|---|---|---|---|
| Michael Nelson (community) | code | Approve | |
| Björn Tillenius | code | Pending | |
|
Review via email:
|
|||
Commit message
Description of the change
| Muharem Hrnjadovic (al-maisan) wrote : | # |
| Michael Nelson (michael.nelson) wrote : | # |
Hi Muharem!
It's great seeing the pieces of the generalisation falling into place!
I've got one rather large question about this change: Why can't you just
use a normal adapter? It seems that you've gone the long way around to
implement adapter-like functionality.
That is, you've got a BuildFarmJobType specified on IBuildQueue.
and you want to find the corresponding class. I don't know whether it is
the best solution, but you *could* in this case, adapt the build_queue
entry itself to the correct IBuildFarmJob implementation?
IBuildFarmJob(
which would return the specifc build farm job.
I'm wondering if it's really that you need specific_
work in the build estimations, rather than specifically for this branch. That
is, originally when you were chatting with gary about using the class manager,
I had understood that you needed to *iterate* over the different
classes to build a query... in that case you would certainly need something
like the specific_
not needing to iterate them at all for this branch, and looking at this branch
in isolation would suggest that a normal adapter would be more intuitive.
If it is the case that you will need specific_
then it may indeed make sense to *not* use adaption here and re-use the
specific_
would need to define both the utility *and* the adapter. I'd like to get
BjornT's thoughts on that point, hence marking as needs information.
Some comments in line below.
> === modified file 'lib/lp/
> --- lib/lp/
> +++ lib/lp/
> @@ -897,6 +897,16 @@
> <allow
> interface=
> </class>
> + <!--
> + The registration below is used to discover all build farm job classes
> + that implement the `IBuildFarmJob` interface. Please see bug #503839
> + for more detail.
> + The 'name' attribute needs to be set to the appropriate
> + `BuildFarmJobType` enumeration value.
> + -->
> + <utility component=
> + name="PACKAGEBUILD"
> + provides=
>
> <!-- BinaryPackageBu
> <adapter
>
> === modified file 'lib/lp/
> --- lib/lp/
> +++ lib/lp/
> @@ -55,6 +55,10 @@
> title=_('The builder behavior required to run this job.'),
> required=False, readonly=True)
>
> + specific_
> + title=_('Job classes that may run on the build farm.'),
> + required=True, readonly=True)
> +
> estimated_duration = Timedelta(
> title=_("Estimated Job Duration"), required=True,
> description=
>
> === modified file 'lib/lp/soyuz/...
| Muharem Hrnjadovic (al-maisan) wrote : | # |
Michael Nelson wrote:
> Review: Needs Information code
> Hi Muharem!
>
> It's great seeing the pieces of the generalisation falling into place!
>
> I've got one rather large question about this change: Why can't you just
> use a normal adapter? It seems that you've gone the long way around to
> implement adapter-like functionality.
>
> That is, you've got a BuildFarmJobType specified on IBuildQueue.
> and you want to find the corresponding class. I don't know whether it is
> the best solution, but you *could* in this case, adapt the build_queue
> entry itself to the correct IBuildFarmJob implementation?
>
> IBuildFarmJob(
>
> which would return the specifc build farm job.
>
> I'm wondering if it's really that you need specific_
> work in the build estimations, rather than specifically for this branch. That
> is, originally when you were chatting with gary about using the class manager,
> I had understood that you needed to *iterate* over the different
> classes to build a query... in that case you would certainly need something
> like the specific_
> not needing to iterate them at all for this branch, and looking at this branch
> in isolation would suggest that a normal adapter would be more intuitive.
>
> If it is the case that you will need specific_
> then it may indeed make sense to *not* use adaption here and re-use the
> specific_
> would need to define both the utility *and* the adapter. I'd like to get
> BjornT's thoughts on that point, hence marking as needs information.
Hello Michael, sorry for the terse answer but, yes, I need to iterate
over the farm build job classes.
See e.g. lines 48-54 in the enclosed diff which is an excerpt from
lp:~al-maisan/launchpad/job-delays-504086, the branch I am currently
working on.
> Some comments in line below.
>
>> === modified file 'lib/lp/
>> --- lib/lp/
>> +++ lib/lp/
>> @@ -897,6 +897,16 @@
>> <allow
>> interface=
>> </class>
>> + <!--
>> + The registration below is used to discover all build farm job classes
>> + that implement the `IBuildFarmJob` interface. Please see bug #503839
>> + for more detail.
>> + The 'name' attribute needs to be set to the appropriate
>> + `BuildFarmJobType` enumeration value.
>> + -->
>> + <utility component=
>> + name="PACKAGEBUILD"
>> + provides=
>>
>> <!-- BinaryPackageBu
>> <adapter
>>
>> === modified file 'lib/lp/
>> --- lib/lp/
>> +++ lib/lp/
>> @@ -55,6 +55,10 @@
>> title=_('The builder behavior required to run this ...
| 1 | === modified file 'lib/lp/soyuz/model/buildqueue.py' | |||
| 2 | --- lib/lp/soyuz/model/buildqueue.py 2010-01-07 06:49:04 +0000 | |||
| 3 | +++ lib/lp/soyuz/model/buildqueue.py 2010-01-07 09:58:59 +0000 | |||
| 4 | @@ -211,6 +211,9 @@ | |||
| 5 | 211 | # The job of interest (JOI) is processor independent. | 211 | # The job of interest (JOI) is processor independent. |
| 6 | 212 | builders_for_job = builders_in_total | 212 | builders_for_job = builders_in_total |
| 7 | 213 | 213 | ||
| 8 | 214 | # All builders can take on platform-independent jobs. | ||
| 9 | 215 | builder_stats[(None,None)] = builders_in_total | ||
| 10 | 216 | |||
| 11 | 214 | return (builders_in_total, builders_for_job, builder_stats) | 217 | return (builders_in_total, builders_for_job, builder_stats) |
| 12 | 215 | 218 | ||
| 13 | 216 | def _freeBuildersCount(self, processor, virtualized): | 219 | def _freeBuildersCount(self, processor, virtualized): |
| 14 | @@ -313,6 +316,123 @@ | |||
| 15 | 313 | else: | 316 | else: |
| 16 | 314 | return int(head_job_delay) | 317 | return int(head_job_delay) |
| 17 | 315 | 318 | ||
| 18 | 319 | def _estimateJobDelay(self, builders_in_total, builder_stats): | ||
| 19 | 320 | """Sum of estimated durations for *pending* jobs ahead in queue. | ||
| 20 | 321 | |||
| 21 | 322 | For the purpose of estimating the dispatch time of the job of | ||
| 22 | 323 | interest (JOI) we need to know the delay caused by all the pending | ||
| 23 | 324 | jobs that are ahead of the JOI in the queue and that compete with it | ||
| 24 | 325 | for builders. | ||
| 25 | 326 | |||
| 26 | 327 | :param builders_in_total: How many active builders are available | ||
| 27 | 328 | altogether? | ||
| 28 | 329 | :param builder_stats: A dictionary with builder counts where the | ||
| 29 | 330 | key is a (processor, virtualized) combination (aka "platform") and | ||
| 30 | 331 | the value is the number of builders that can take on jobs | ||
| 31 | 332 | requiring that combination. | ||
| 32 | 333 | :return: A (sum_of_delays, head_job_platform) tuple where | ||
| 33 | 334 | * 'sum_of_delays' is the estimated delay in seconds and | ||
| 34 | 335 | * 'head_job_platform' is the platform ((processor, virtualized) | ||
| 35 | 336 | combination) required by the job at the head of the queue. | ||
| 36 | 337 | """ | ||
| 37 | 338 | # Please note: this method will send only one request to the database. | ||
| 38 | 339 | store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR) | ||
| 39 | 340 | my_processor = self.specific_job.processor | ||
| 40 | 341 | my_virtualized = self.specific_job.virtualized | ||
| 41 | 342 | my_platform = (my_processor, my_virtualized) | ||
| 42 | 343 | |||
| 43 | 344 | # Ask all build farm job type classes for a plain SQL SELECT query | ||
| 44 | 345 | # that will yield the pending jobs whose score is equal or better | ||
| 45 | 346 | # than the score of the job of interest (JOI). | ||
| 46 | 347 | # Chain these queries in "SELECT .. UNION SELECT .." fashion and | ||
| 47 | 348 | # use them to obtain the jobs that compete with the JOI for builders. | ||
| 48 | 349 | queries = [] | ||
| 49 | 350 | for job_class in self.specific_job_classes.values(): | ||
| 50 | 351 | queries.append( | ||
| 51 | 352 | job_class.composePendingJobsQuery( | ||
| 52 | 353 | self.lastscore, my_processor, my_virtualized)) | ||
| 53 | 354 | query = '%s ORDER BY lastscore DESC, job' % ' UNION '.join(queries) | ||
| 54 | 355 | job_queue = store.execute(query).get_all() | ||
| 55 | 356 | |||
| 56 | 357 | sum_of_delays = 0 | ||
| 57 | 358 | |||
| 58 | 359 | # This will be used to capture per-platform delay totals. | ||
| 59 | 360 | delays = dict() | ||
| 60 | 361 | # This will be used to capture per-platform job counts. | ||
| 61 | 362 | job_counts = dict() | ||
| 62 | 363 | |||
| 63 | 364 | # Which platform is the job at the head of the queue requiring? | ||
| 64 | 365 | head_job_platform = None | ||
| 65 | 366 | |||
| 66 | 367 | # Apply weights to the estimated duration of the jobs as follows: | ||
| 67 | 368 | # - if a job is tied to a processor TP then divide the estimated | ||
| 68 | 369 | # duration of that job by the number of builders that target TP | ||
| 69 | 370 | # since only these can build the job. | ||
| 70 | 371 | # - if the job is processor-independent then divide its estimated | ||
| 71 | 372 | # duration by the total number of builders because any one of | ||
| 72 | 373 | # them may run it. | ||
| 73 | 374 | for job, score, duration, processor, virtualized in job_queue: | ||
| 74 | 375 | if job == self.job.id: | ||
| 75 | 376 | # We have seen all jobs that are ahead of us in the queue | ||
| 76 | 377 | # and can stop now. | ||
| 77 | 378 | # This is guaranteed by the "ORDER BY lastscore DESC.." | ||
| 78 | 379 | # clause above. | ||
| 79 | 380 | break | ||
| 80 | 381 | |||
| 81 | 382 | # For the purpose of estimating the delay for dispatching the job | ||
| 82 | 383 | # at the head of the queue to a builder we need to capture the | ||
| 83 | 384 | # platform it targets. | ||
| 84 | 385 | if head_job_platform is None: | ||
| 85 | 386 | platform = (processor, virtualized) | ||
| 86 | 387 | if my_processor is None: | ||
| 87 | 388 | # The JOI is platform-independent i.e. the highest scored | ||
| 88 | 389 | # job will be the head job. | ||
| 89 | 390 | head_job_platform = platform | ||
| 90 | 391 | else: | ||
| 91 | 392 | # The JOI targets a specific platform. The head job is | ||
| 92 | 393 | # thus the highest scored job that either targets the | ||
| 93 | 394 | # same platform or is platform-independent. | ||
| 94 | 395 | if (my_platform == platform or processor is None): | ||
| 95 | 396 | head_job_platform = platform | ||
| 96 | 397 | |||
| 97 | 398 | if processor is None: | ||
| 98 | 399 | # Processor independent job, can be run on any builder. | ||
| 99 | 400 | builder_count = builders_in_total | ||
| 100 | 401 | else: | ||
| 101 | 402 | # Tied to a particular processor, see how many builders | ||
| 102 | 403 | # can run it. | ||
| 103 | 404 | builder_count = builder_stats.get((processor, virtualized), 0) | ||
| 104 | 405 | |||
| 105 | 406 | if builder_count == 0: | ||
| 106 | 407 | # There is no builder that can run this job, ignore it | ||
| 107 | 408 | # for the purpose of dispatch time estimation. | ||
| 108 | 409 | continue | ||
| 109 | 410 | |||
| 110 | 411 | # Accumulate the delays, and count the number of jobs causing them | ||
| 111 | 412 | # on a (processor, virtualized) basis. | ||
| 112 | 413 | duration = duration.seconds | ||
| 113 | 414 | delays[(processor, virtualized)] = ( | ||
| 114 | 415 | delays.setdefault((processor, virtualized), 0) + duration) | ||
| 115 | 416 | job_counts[(processor, virtualized)] = ( | ||
| 116 | 417 | job_counts.setdefault((processor, virtualized), 0) + 1) | ||
| 117 | 418 | |||
| 118 | 419 | # Now weight/average the delays based on a jobs/builders comparison. | ||
| 119 | 420 | for platform, duration in delays.iteritems(): | ||
| 120 | 421 | jobs = job_counts[platform] | ||
| 121 | 422 | builders = builder_stats[platform] | ||
| 122 | 423 | if jobs < builders: | ||
| 123 | 424 | # There are less jobs than builders that can take them on, | ||
| 124 | 425 | # the delays should be averaged/divided by the number of jobs. | ||
| 125 | 426 | denominator = jobs | ||
| 126 | 427 | else: | ||
| 127 | 428 | denominator = builders | ||
| 128 | 429 | if denominator > 1: | ||
| 129 | 430 | duration = int(duration/float(denominator)) | ||
| 130 | 431 | |||
| 131 | 432 | sum_of_delays += duration | ||
| 132 | 433 | |||
| 133 | 434 | return (sum_of_delays, head_job_platform) | ||
| 134 | 435 | |||
| 135 | 316 | 436 | ||
| 136 | 317 | class BuildQueueSet(object): | 437 | class BuildQueueSet(object): |
| 137 | 318 | """Utility to deal with BuildQueue content class.""" | 438 | """Utility to deal with BuildQueue content class.""" |
| 138 | 319 | 439 | ||
| 139 | === modified file 'lib/lp/soyuz/tests/test_buildqueue.py' | |||
| 140 | --- lib/lp/soyuz/tests/test_buildqueue.py 2010-01-07 06:49:04 +0000 | |||
| 141 | +++ lib/lp/soyuz/tests/test_buildqueue.py 2010-01-07 10:12:09 +0000 | |||
| 142 | @@ -552,7 +552,7 @@ | |||
| 143 | 552 | score = 1000 | 552 | score = 1000 |
| 144 | 553 | duration = 0 | 553 | duration = 0 |
| 145 | 554 | for build in self.builds: | 554 | for build in self.builds: |
| 147 | 555 | score += 1 | 555 | score += getattr(self, 'score_increment', 1) |
| 148 | 556 | duration += 60 | 556 | duration += 60 |
| 149 | 557 | bq = build.buildqueue_record | 557 | bq = build.buildqueue_record |
| 150 | 558 | bq.lastscore = score | 558 | bq.lastscore = score |
| 151 | @@ -740,3 +740,96 @@ | |||
| 152 | 740 | self.assertEqual( | 740 | self.assertEqual( |
| 153 | 741 | bq.specific_job_classes[BuildFarmJobType.BRANCHBUILD], | 741 | bq.specific_job_classes[BuildFarmJobType.BRANCHBUILD], |
| 154 | 742 | FakeBranchBuild) | 742 | FakeBranchBuild) |
| 155 | 743 | |||
| 156 | 744 | |||
| 157 | 745 | class TestMultiArchJobDelayEstimation(MultiArchBuildsBase): | ||
| 158 | 746 | """Test estimated job delays with various processors.""" | ||
| 159 | 747 | score_increment = 2 | ||
| 160 | 748 | def setUp(self): | ||
| 161 | 749 | """Set up a fake 'BRANCHBUILD' build farm job class. | ||
| 162 | 750 | |||
| 163 | 751 | The two platform-independent jobs will have a score of 1017 and 1033 | ||
| 164 | 752 | respectively. | ||
| 165 | 753 | |||
| 166 | 754 | gedit, p: hppa, v:False e:0:01:00 *** s: 1002 | ||
| 167 | 755 | gedit, p: 386, v:False e:0:02:00 *** s: 1004 | ||
| 168 | 756 | firefox, p: hppa, v:False e:0:03:00 *** s: 1006 | ||
| 169 | 757 | firefox, p: 386, v:False e:0:04:00 *** s: 1008 | ||
| 170 | 758 | apg, p: hppa, v:False e:0:05:00 *** s: 1010 | ||
| 171 | 759 | apg, p: 386, v:False e:0:06:00 *** s: 1012 | ||
| 172 | 760 | vim, p: hppa, v:False e:0:07:00 *** s: 1014 | ||
| 173 | 761 | vim, p: 386, v:False e:0:08:00 *** s: 1016 | ||
| 174 | 762 | --> fake1, none none 0:00:22 1017 | ||
| 175 | 763 | gcc, p: hppa, v:False e:0:09:00 *** s: 1018 | ||
| 176 | 764 | gcc, p: 386, v:False e:0:10:00 *** s: 1020 | ||
| 177 | 765 | bison, p: hppa, v:False e:0:11:00 *** s: 1022 | ||
| 178 | 766 | bison, p: 386, v:False e:0:12:00 *** s: 1024 | ||
| 179 | 767 | flex, p: hppa, v:False e:0:13:00 *** s: 1026 | ||
| 180 | 768 | flex, p: 386, v:False e:0:14:00 *** s: 1028 | ||
| 181 | 769 | postgres, p: hppa, v:False e:0:15:00 *** s: 1030 | ||
| 182 | 770 | postgres, p: 386, v:False e:0:16:00 *** s: 1032 | ||
| 183 | 771 | --> fake2, none none 0:03:42 1033 | ||
| 184 | 772 | |||
| 185 | 773 | p=processor, v=virtualized, e=estimated_duration, s=score | ||
| 186 | 774 | """ | ||
| 187 | 775 | super(TestMultiArchJobDelayEstimation, self).setUp() | ||
| 188 | 776 | |||
| 189 | 777 | from zope import component | ||
| 190 | 778 | from zope.interface import implements | ||
| 191 | 779 | from lp.buildmaster.interfaces.buildfarmjob import IBuildFarmJob | ||
| 192 | 780 | |||
| 193 | 781 | class FakeBranchBuild: | ||
| 194 | 782 | implements(IBuildFarmJob) | ||
| 195 | 783 | def score(self): | ||
| 196 | 784 | return -9999 | ||
| 197 | 785 | def getLogFileName(self): | ||
| 198 | 786 | return 'buildlog_fake-branch-build.txt' | ||
| 199 | 787 | def getName(self): | ||
| 200 | 788 | return 'fake-branch-build' | ||
| 201 | 789 | @staticmethod | ||
| 202 | 790 | def composePendingJobsQuery(min_score, processor, virtualized): | ||
| 203 | 791 | return """ | ||
| 204 | 792 | (SELECT | ||
| 205 | 793 | 1000001::integer AS job, | ||
| 206 | 794 | 1017::integer AS lastscore, | ||
| 207 | 795 | '22 seconds'::interval AS estimated_duration, | ||
| 208 | 796 | NULL::integer AS processor, | ||
| 209 | 797 | NULL::boolean AS virtualized | ||
| 210 | 798 | UNION | ||
| 211 | 799 | SELECT | ||
| 212 | 800 | 1000002::integer AS job, | ||
| 213 | 801 | 1033::integer AS lastscore, | ||
| 214 | 802 | '222 seconds'::interval AS estimated_duration, | ||
| 215 | 803 | NULL::integer AS processor, | ||
| 216 | 804 | NULL::boolean AS virtualized) | ||
| 217 | 805 | """ | ||
| 218 | 806 | |||
| 219 | 807 | # Pretend that our `FakeBranchBuild` class implements the | ||
| 220 | 808 | # `IBuildFarmJob` interface. | ||
| 221 | 809 | component.provideUtility( | ||
| 222 | 810 | FakeBranchBuild, IBuildFarmJob, 'BRANCHBUILD') | ||
| 223 | 811 | |||
| 224 | 812 | def test_job_delay(self): | ||
| 225 | 813 | processor_fam = ProcessorFamilySet().getByName('hppa') | ||
| 226 | 814 | hppa_proc = processor_fam.processors[0] | ||
| 227 | 815 | |||
| 228 | 816 | # One of four builders for the 'apg' build is immediately available. | ||
| 229 | 817 | flex_build, flex_job = find_job(self, 'flex', 'hppa') | ||
| 230 | 818 | check_mintime_to_builder(self, flex_job, hppa_proc, False, 0) | ||
| 231 | 819 | |||
| 232 | 820 | builder_data = flex_job._getBuilderData() | ||
| 233 | 821 | builders_in_total, builders_for_job, builder_stats = builder_data | ||
| 234 | 822 | |||
| 235 | 823 | # The delay is 900 (= 15*60) + 222 seconds, the head job is | ||
| 236 | 824 | # platform-independent. | ||
| 237 | 825 | self.assertEqual( | ||
| 238 | 826 | flex_job._estimateJobDelay(builders_in_total, builder_stats), | ||
| 239 | 827 | (1122, (None, None))) | ||
| 240 | 828 | |||
| 241 | 829 | # Assign the postgres job to a builder. | ||
| 242 | 830 | assign_to_builder(self, 'postgres', 1, 'hppa') | ||
| 243 | 831 | # Now only the 222 seconds (the estimated duration of the | ||
| 244 | 832 | # platform-independent job) should be returned. | ||
| 245 | 833 | self.assertEqual( | ||
| 246 | 834 | flex_job._estimateJobDelay(builders_in_total, builder_stats), | ||
| 247 | 835 | (222, (None, None))) | ||
| Muharem Hrnjadovic (al-maisan) wrote : | # |
Michael Nelson wrote:
> Review: Needs Information code
[..]
> I've got one rather large question about this change: Why can't you
> just use a normal adapter? It seems that you've gone the long way
> around to implement adapter-like functionality.
>
> That is, you've got a BuildFarmJobType specified on
> IBuildQueue.
> don't know whether it is the best solution, but you *could* in this
> case, adapt the build_queue entry itself to the correct IBuildFarmJob
> implementation?
>
> IBuildFarmJob(
How would that actually work? IBuildFarmJob is not what we want here.
IBuildQueue.
BuildPackageJob instance.
> which would return the specifc build farm job.
>
> I'm wondering if it's really that you need specific_
> further work in the build estimations, rather than specifically for
That's the case indeed, sorry for not explaining that better.
> this branch. That is, originally when you were chatting with gary
> about using the class manager, I had understood that you needed to
> *iterate* over the different classes to build a query... in that case
> you would certainly need something like the specific_
> you've provided below. But as it is, you are not needing to iterate
> them at all for this branch, and looking at this branch in isolation
> would suggest that a normal adapter would be more intuitive.
Sorry for not explaining in more detail why this is needed please see my
previous email for an example where I do need to iterate over the build
farm job classes.
I also expect that that's something that will be needed when we
generalize the candidate job selection.
> If it is the case that you will need specific_
> branch, then it may indeed make sense to *not* use adaption here and
> re-use the specific_
As stated in the previous reply I do require the iteration over the build
farm job classes for the purpose of job dispatch time estimation.
Being able to register build farm classes in the way suggested makes it
also much easier to test the code.
Please see the end of the diff (lines 193-222) enclosed in my previous
reply for an example.
[..]
Best regards
--
Muharem Hrnjadovic <email address hidden>
Public key id : B2BBFCFC
Key fingerprint : A5A3 CC67 2B87 D641 103F 5602 219F 6B60 B2BB FCFC
| Michael Nelson (michael.nelson) wrote : | # |
Great, thanks for the info Muharem.
So given that you need the class iteration, I'm keen to approve this now (assuming you address the issues I mentioned above inline).
Thanks!
| Muharem Hrnjadovic (al-maisan) wrote : | # |
Michael Nelson wrote:
[..]
Hello Michael,
thanks again for taking the time to review this branch despite your own
busy schedule. That's very much appreciated :)
> Some comments in line below.
Please see my replies as well as the enclosed incremental diff.
>> === modified file 'lib/lp/
>> --- lib/lp/
>> +++ lib/lp/
>> @@ -897,6 +897,16 @@
>> <allow
>> interface=
>> </class>
>> + <!--
>> + The registration below is used to discover all build farm job classes
>> + that implement the `IBuildFarmJob` interface. Please see bug #503839
>> + for more detail.
>> + The 'name' attribute needs to be set to the appropriate
>> + `BuildFarmJobType` enumeration value.
>> + -->
>> + <utility component=
>> + name="PACKAGEBUILD"
>> + provides=
>>
>> <!-- BinaryPackageBu
>> <adapter
>>
>> === modified file 'lib/lp/
>> --- lib/lp/
>> +++ lib/lp/
>> @@ -55,6 +55,10 @@
>> title=_('The builder behavior required to run this job.'),
>> required=False, readonly=True)
>>
>> + specific_
>> + title=_('Job classes that may run on the build farm.'),
>> + required=True, readonly=True)
>> +
>> estimated_duration = Timedelta(
>> title=_("Estimated Job Duration"), required=True,
>> description=
>>
>> === modified file 'lib/lp/
>> --- lib/lp/
>> +++ lib/lp/
>> @@ -57,11 +57,31 @@
>> return IBuildFarmJobBe
>>
>> @property
>> + def specific_
>> + """See `IBuildQueue`."""
>> + from zope.component import getSiteManager
>> + from lp.buildmaster.
>
> Is there a reason for these imports to be inline? If so, please add a
> comment or otherwise move them as per the guidelines.
No other code in the module uses them. I have added a comment to that
effect.
>> +
>> + job_classes = []
>> + components = getSiteManager()
>> + # Get all components that implement the `IBuildFarmJob` interface.
>> + implementations = sorted(
>> + # The above yields a collection of 2-tuples where the first element
>> + # is the name of the `BuildFarmJobType` enum and the second element
>> + # is the implementing class respectively.
>> + for job_enum_name, job_class in implementations:
>> + job_enum = getattr(
>
> I was originally going to say that we should be checking first with
> safe_hasat...
| 1 | === modified file 'lib/lp/soyuz/model/buildqueue.py' | |||
| 2 | --- lib/lp/soyuz/model/buildqueue.py 2010-01-07 05:36:53 +0000 | |||
| 3 | +++ lib/lp/soyuz/model/buildqueue.py 2010-01-07 18:41:24 +0000 | |||
| 4 | @@ -59,21 +59,23 @@ | |||
| 5 | 59 | @property | 59 | @property |
| 6 | 60 | def specific_job_classes(self): | 60 | def specific_job_classes(self): |
| 7 | 61 | """See `IBuildQueue`.""" | 61 | """See `IBuildQueue`.""" |
| 8 | 62 | # The reason for keeping the imports below local is that they are not | ||
| 9 | 63 | # of interest to other methods in this module. | ||
| 10 | 62 | from zope.component import getSiteManager | 64 | from zope.component import getSiteManager |
| 11 | 63 | from lp.buildmaster.interfaces.buildfarmjob import IBuildFarmJob | 65 | from lp.buildmaster.interfaces.buildfarmjob import IBuildFarmJob |
| 12 | 64 | 66 | ||
| 14 | 65 | job_classes = [] | 67 | job_classes = dict() |
| 15 | 68 | # Get all components that implement the `IBuildFarmJob` interface. | ||
| 16 | 66 | components = getSiteManager() | 69 | components = getSiteManager() |
| 17 | 67 | # Get all components that implement the `IBuildFarmJob` interface. | ||
| 18 | 68 | implementations = sorted(components.getUtilitiesFor(IBuildFarmJob)) | 70 | implementations = sorted(components.getUtilitiesFor(IBuildFarmJob)) |
| 19 | 69 | # The above yields a collection of 2-tuples where the first element | 71 | # The above yields a collection of 2-tuples where the first element |
| 20 | 70 | # is the name of the `BuildFarmJobType` enum and the second element | 72 | # is the name of the `BuildFarmJobType` enum and the second element |
| 21 | 71 | # is the implementing class respectively. | 73 | # is the implementing class respectively. |
| 22 | 72 | for job_enum_name, job_class in implementations: | 74 | for job_enum_name, job_class in implementations: |
| 23 | 73 | job_enum = getattr(BuildFarmJobType, job_enum_name) | 75 | job_enum = getattr(BuildFarmJobType, job_enum_name) |
| 25 | 74 | job_classes.append((job_enum, job_class)) | 76 | job_classes[job_enum] = job_class |
| 26 | 75 | 77 | ||
| 28 | 76 | return dict(job_classes) | 78 | return job_classes |
| 29 | 77 | 79 | ||
| 30 | 78 | @property | 80 | @property |
| 31 | 79 | def specific_job(self): | 81 | def specific_job(self): |
| Michael Nelson (michael.nelson) wrote : | # |
On Thu, Jan 7, 2010 at 8:57 PM, Muharem Hrnjadovic
<email address hidden> wrote:
> Michael Nelson wrote:
> [..]
>
> Hello Michael,
>
> thanks again for taking the time to review this branch despite your own
> busy schedule. That's very much appreciated :)
No probs!
>
>> Some comments in line below.
> Please see my replies as well as the enclosed incremental diff.
>
>>> === modified file 'lib/lp/
>>> --- lib/lp/
>>> +++ lib/lp/
>>> @@ -57,11 +57,31 @@
>>> return IBuildFarmJobBe
>>>
>>> @property
>>> + def specific_
>>> + """See `IBuildQueue`."""
>>> + from zope.component import getSiteManager
>>> + from lp.buildmaster.
>>
>> Is there a reason for these imports to be inline? If so, please add a
>> comment or otherwise move them as per the guidelines.
> No other code in the module uses them. I have added a comment to that
> effect.
Erm... I didn't realise that was a valid reason? I thought we only did
so if we *had* to (ie. circular imports etc.). Please check our
styleguide... I'm going from memory but I thought the imports should
always be at the top even if no other code in the module uses them.
Cheers, and happy flying!
-Michael
| Muharem Hrnjadovic (al-maisan) wrote : | # |
Michael Nelson wrote:
> On Thu, Jan 7, 2010 at 8:57 PM, Muharem Hrnjadovic
> <email address hidden> wrote:
>> Michael Nelson wrote:
>> [..]
>>
>> Hello Michael,
>>
>> thanks again for taking the time to review this branch despite your own
>> busy schedule. That's very much appreciated :)
>
> No probs!
>
>>> Some comments in line below.
>> Please see my replies as well as the enclosed incremental diff.
>>
>>>> === modified file 'lib/lp/
>>>> --- lib/lp/
>>>> +++ lib/lp/
>>>> @@ -57,11 +57,31 @@
>>>> return IBuildFarmJobBe
>>>>
>>>> @property
>>>> + def specific_
>>>> + """See `IBuildQueue`."""
>>>> + from zope.component import getSiteManager
>>>> + from lp.buildmaster.
>>> Is there a reason for these imports to be inline? If so, please add a
>>> comment or otherwise move them as per the guidelines.
>> No other code in the module uses them. I have added a comment to that
>> effect.
>
> Erm... I didn't realise that was a valid reason? I thought we only did
> so if we *had* to (ie. circular imports etc.). Please check our
> styleguide... I'm going from memory but I thought the imports should
> always be at the top even if no other code in the module uses them.
Whoops .. sorry .. and thanks for catching this :)
> Cheers, and happy flying!
Thank you! Have a safe journey as well :)
Best regards
--
Muharem Hrnjadovic <email address hidden>
Public key id : B2BBFCFC
Key fingerprint : A5A3 CC67 2B87 D641 103F 5602 219F 6B60 B2BB FCFC
Preview Diff
| 1 | === modified file 'lib/lp/soyuz/configure.zcml' | |||
| 2 | --- lib/lp/soyuz/configure.zcml 2010-01-04 21:15:46 +0000 | |||
| 3 | +++ lib/lp/soyuz/configure.zcml 2010-01-09 02:49:17 +0000 | |||
| 4 | @@ -897,6 +897,16 @@ | |||
| 5 | 897 | <allow | 897 | <allow |
| 6 | 898 | interface="lp.soyuz.interfaces.buildpackagejob.IBuildPackageJob"/> | 898 | interface="lp.soyuz.interfaces.buildpackagejob.IBuildPackageJob"/> |
| 7 | 899 | </class> | 899 | </class> |
| 8 | 900 | <!-- | ||
| 9 | 901 | The registration below is used to discover all build farm job classes | ||
| 10 | 902 | that implement the `IBuildFarmJob` interface. Please see bug #503839 | ||
| 11 | 903 | for more detail. | ||
| 12 | 904 | The 'name' attribute needs to be set to the appropriate | ||
| 13 | 905 | `BuildFarmJobType` enumeration value. | ||
| 14 | 906 | --> | ||
| 15 | 907 | <utility component="lp.soyuz.model.buildpackagejob.BuildPackageJob" | ||
| 16 | 908 | name="PACKAGEBUILD" | ||
| 17 | 909 | provides="lp.buildmaster.interfaces.buildfarmjob.IBuildFarmJob"/> | ||
| 18 | 900 | 910 | ||
| 19 | 901 | <!-- BinaryPackageBuildBehavior --> | 911 | <!-- BinaryPackageBuildBehavior --> |
| 20 | 902 | <adapter | 912 | <adapter |
| 21 | 903 | 913 | ||
| 22 | === modified file 'lib/lp/soyuz/interfaces/buildqueue.py' | |||
| 23 | --- lib/lp/soyuz/interfaces/buildqueue.py 2009-11-27 13:33:55 +0000 | |||
| 24 | +++ lib/lp/soyuz/interfaces/buildqueue.py 2010-01-09 02:49:17 +0000 | |||
| 25 | @@ -55,6 +55,10 @@ | |||
| 26 | 55 | title=_('The builder behavior required to run this job.'), | 55 | title=_('The builder behavior required to run this job.'), |
| 27 | 56 | required=False, readonly=True) | 56 | required=False, readonly=True) |
| 28 | 57 | 57 | ||
| 29 | 58 | specific_job_classes = Field( | ||
| 30 | 59 | title=_('Job classes that may run on the build farm.'), | ||
| 31 | 60 | required=True, readonly=True) | ||
| 32 | 61 | |||
| 33 | 58 | estimated_duration = Timedelta( | 62 | estimated_duration = Timedelta( |
| 34 | 59 | title=_("Estimated Job Duration"), required=True, | 63 | title=_("Estimated Job Duration"), required=True, |
| 35 | 60 | description=_("Estimated job duration interval.")) | 64 | description=_("Estimated job duration interval.")) |
| 36 | 61 | 65 | ||
| 37 | === modified file 'lib/lp/soyuz/model/buildqueue.py' | |||
| 38 | --- lib/lp/soyuz/model/buildqueue.py 2010-01-06 15:58:41 +0000 | |||
| 39 | +++ lib/lp/soyuz/model/buildqueue.py 2010-01-09 02:49:17 +0000 | |||
| 40 | @@ -12,7 +12,8 @@ | |||
| 41 | 12 | 12 | ||
| 42 | 13 | import logging | 13 | import logging |
| 43 | 14 | 14 | ||
| 45 | 15 | from zope.component import getUtility | 15 | from zope.component import getSiteManager, getUtility |
| 46 | 16 | |||
| 47 | 16 | from zope.interface import implements | 17 | from zope.interface import implements |
| 48 | 17 | 18 | ||
| 49 | 18 | from sqlobject import ( | 19 | from sqlobject import ( |
| 50 | @@ -24,7 +25,8 @@ | |||
| 51 | 24 | from canonical.database.enumcol import EnumCol | 25 | from canonical.database.enumcol import EnumCol |
| 52 | 25 | from canonical.database.sqlbase import SQLBase, sqlvalues | 26 | from canonical.database.sqlbase import SQLBase, sqlvalues |
| 53 | 26 | from canonical.launchpad.webapp.interfaces import NotFoundError | 27 | from canonical.launchpad.webapp.interfaces import NotFoundError |
| 55 | 27 | from lp.buildmaster.interfaces.buildfarmjob import BuildFarmJobType | 28 | from lp.buildmaster.interfaces.buildfarmjob import ( |
| 56 | 29 | BuildFarmJobType, IBuildFarmJob) | ||
| 57 | 28 | from lp.buildmaster.interfaces.buildfarmjobbehavior import ( | 30 | from lp.buildmaster.interfaces.buildfarmjobbehavior import ( |
| 58 | 29 | IBuildFarmJobBehavior) | 31 | IBuildFarmJobBehavior) |
| 59 | 30 | from lp.services.job.interfaces.job import JobStatus | 32 | from lp.services.job.interfaces.job import JobStatus |
| 60 | @@ -57,11 +59,28 @@ | |||
| 61 | 57 | return IBuildFarmJobBehavior(self.specific_job) | 59 | return IBuildFarmJobBehavior(self.specific_job) |
| 62 | 58 | 60 | ||
| 63 | 59 | @property | 61 | @property |
| 64 | 62 | def specific_job_classes(self): | ||
| 65 | 63 | """See `IBuildQueue`.""" | ||
| 66 | 64 | job_classes = dict() | ||
| 67 | 65 | # Get all components that implement the `IBuildFarmJob` interface. | ||
| 68 | 66 | components = getSiteManager() | ||
| 69 | 67 | implementations = sorted(components.getUtilitiesFor(IBuildFarmJob)) | ||
| 70 | 68 | # The above yields a collection of 2-tuples where the first element | ||
| 71 | 69 | # is the name of the `BuildFarmJobType` enum and the second element | ||
| 72 | 70 | # is the implementing class respectively. | ||
| 73 | 71 | for job_enum_name, job_class in implementations: | ||
| 74 | 72 | job_enum = getattr(BuildFarmJobType, job_enum_name) | ||
| 75 | 73 | job_classes[job_enum] = job_class | ||
| 76 | 74 | |||
| 77 | 75 | return job_classes | ||
| 78 | 76 | |||
| 79 | 77 | @property | ||
| 80 | 60 | def specific_job(self): | 78 | def specific_job(self): |
| 81 | 61 | """See `IBuildQueue`.""" | 79 | """See `IBuildQueue`.""" |
| 82 | 80 | specific_class = self.specific_job_classes[self.job_type] | ||
| 83 | 62 | store = Store.of(self) | 81 | store = Store.of(self) |
| 84 | 63 | result_set = store.find( | 82 | result_set = store.find( |
| 86 | 64 | BuildPackageJob, BuildPackageJob.job == self.job) | 83 | specific_class, specific_class.job == self.job) |
| 87 | 65 | return result_set.one() | 84 | return result_set.one() |
| 88 | 66 | 85 | ||
| 89 | 67 | @property | 86 | @property |
| 90 | 68 | 87 | ||
| 91 | === modified file 'lib/lp/soyuz/tests/test_buildqueue.py' | |||
| 92 | --- lib/lp/soyuz/tests/test_buildqueue.py 2010-01-06 15:58:41 +0000 | |||
| 93 | +++ lib/lp/soyuz/tests/test_buildqueue.py 2010-01-09 02:49:17 +0000 | |||
| 94 | @@ -13,6 +13,7 @@ | |||
| 95 | 13 | IStoreSelector, MAIN_STORE, DEFAULT_FLAVOR) | 13 | IStoreSelector, MAIN_STORE, DEFAULT_FLAVOR) |
| 96 | 14 | from canonical.testing import LaunchpadZopelessLayer | 14 | from canonical.testing import LaunchpadZopelessLayer |
| 97 | 15 | 15 | ||
| 98 | 16 | from lp.buildmaster.interfaces.buildfarmjob import BuildFarmJobType | ||
| 99 | 16 | from lp.soyuz.interfaces.archive import ArchivePurpose | 17 | from lp.soyuz.interfaces.archive import ArchivePurpose |
| 100 | 17 | from lp.soyuz.interfaces.build import BuildStatus | 18 | from lp.soyuz.interfaces.build import BuildStatus |
| 101 | 18 | from lp.soyuz.interfaces.builder import IBuilderSet | 19 | from lp.soyuz.interfaces.builder import IBuilderSet |
| 102 | @@ -662,3 +663,80 @@ | |||
| 103 | 662 | 663 | ||
| 104 | 663 | # That builder should be available immediately since it's idle. | 664 | # That builder should be available immediately since it's idle. |
| 105 | 664 | check_mintime_to_builder(self, apg_job, None, None, 0) | 665 | check_mintime_to_builder(self, apg_job, None, None, 0) |
| 106 | 666 | |||
| 107 | 667 | |||
| 108 | 668 | class TestJobClasses(TestCaseWithFactory): | ||
| 109 | 669 | """Tests covering build farm job type classes.""" | ||
| 110 | 670 | layer = LaunchpadZopelessLayer | ||
| 111 | 671 | def setUp(self): | ||
| 112 | 672 | """Set up a native x86 build for the test archive.""" | ||
| 113 | 673 | super(TestJobClasses, self).setUp() | ||
| 114 | 674 | |||
| 115 | 675 | self.publisher = SoyuzTestPublisher() | ||
| 116 | 676 | self.publisher.prepareBreezyAutotest() | ||
| 117 | 677 | |||
| 118 | 678 | # First mark all builds in the sample data as already built. | ||
| 119 | 679 | store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR) | ||
| 120 | 680 | sample_data = store.find(Build) | ||
| 121 | 681 | for build in sample_data: | ||
| 122 | 682 | build.buildstate = BuildStatus.FULLYBUILT | ||
| 123 | 683 | store.flush() | ||
| 124 | 684 | |||
| 125 | 685 | # We test builds that target a primary archive. | ||
| 126 | 686 | self.non_ppa = self.factory.makeArchive( | ||
| 127 | 687 | name="primary", purpose=ArchivePurpose.PRIMARY) | ||
| 128 | 688 | self.non_ppa.require_virtualized = False | ||
| 129 | 689 | |||
| 130 | 690 | self.builds = [] | ||
| 131 | 691 | self.builds.extend( | ||
| 132 | 692 | self.publisher.getPubSource( | ||
| 133 | 693 | sourcename="gedit", status=PackagePublishingStatus.PUBLISHED, | ||
| 134 | 694 | archive=self.non_ppa).createMissingBuilds()) | ||
| 135 | 695 | |||
| 136 | 696 | def test_BuildPackageJob(self): | ||
| 137 | 697 | """`BuildPackageJob` is one of the job type classes.""" | ||
| 138 | 698 | from lp.soyuz.model.buildpackagejob import BuildPackageJob | ||
| 139 | 699 | _build, bq = find_job(self, 'gedit') | ||
| 140 | 700 | |||
| 141 | 701 | # This is a binary package build. | ||
| 142 | 702 | self.assertEqual( | ||
| 143 | 703 | bq.job_type, BuildFarmJobType.PACKAGEBUILD, | ||
| 144 | 704 | "This is a binary package build") | ||
| 145 | 705 | |||
| 146 | 706 | # The class registered for 'PACKAGEBUILD' is `BuildPackageJob`. | ||
| 147 | 707 | self.assertEqual( | ||
| 148 | 708 | bq.specific_job_classes[BuildFarmJobType.PACKAGEBUILD], | ||
| 149 | 709 | BuildPackageJob, | ||
| 150 | 710 | "The class registered for 'PACKAGEBUILD' is `BuildPackageJob`") | ||
| 151 | 711 | |||
| 152 | 712 | # The 'specific_job' object associated with this `BuildQueue` | ||
| 153 | 713 | # instance is of type `BuildPackageJob`. | ||
| 154 | 714 | self.assertTrue(bq.specific_job is not None) | ||
| 155 | 715 | self.assertEqual( | ||
| 156 | 716 | bq.specific_job.__class__, BuildPackageJob, | ||
| 157 | 717 | "The 'specific_job' object associated with this `BuildQueue` " | ||
| 158 | 718 | "instance is of type `BuildPackageJob`") | ||
| 159 | 719 | |||
| 160 | 720 | def test_OtherTypeClasses(self): | ||
| 161 | 721 | """Other job type classes are picked up as well.""" | ||
| 162 | 722 | from zope import component | ||
| 163 | 723 | from lp.buildmaster.interfaces.buildfarmjob import IBuildFarmJob | ||
| 164 | 724 | class FakeBranchBuild: | ||
| 165 | 725 | pass | ||
| 166 | 726 | |||
| 167 | 727 | _build, bq = find_job(self, 'gedit') | ||
| 168 | 728 | # First make sure that we don't have a job type class registered for | ||
| 169 | 729 | # 'BRANCHBUILD' yet. | ||
| 170 | 730 | self.assertTrue( | ||
| 171 | 731 | bq.specific_job_classes.get(BuildFarmJobType.BRANCHBUILD) is None) | ||
| 172 | 732 | |||
| 173 | 733 | # Pretend that our `FakeBranchBuild` class implements the | ||
| 174 | 734 | # `IBuildFarmJob` interface. | ||
| 175 | 735 | component.provideUtility( | ||
| 176 | 736 | FakeBranchBuild, IBuildFarmJob, 'BRANCHBUILD') | ||
| 177 | 737 | |||
| 178 | 738 | # Now we should see the `FakeBranchBuild` class "registered" in the | ||
| 179 | 739 | # `specific_job_classes` dictionary under the 'BRANCHBUILD' key. | ||
| 180 | 740 | self.assertEqual( | ||
| 181 | 741 | bq.specific_job_classes[BuildFarmJobType.BRANCHBUILD], | ||
| 182 | 742 | FakeBranchBuild) | ||
Hello there!
This is another step in the renovation of the (Soyuz) build farm. Now that
we're introducing additional build farm job types we need to be able to figure
out what these are without hardcoding that information.
This is mostly required by the general build farm infrastructure i.e. for the
purpose of job dispatch time estimation and candidate job selection.
Pre-implementation discussion (via email) with Gary Poster and others.
Tests to run:
bin/test -vv -t TestJobClasses
No pertinent "make lint" errors or warnings.
Please see also bug #503839 for more detail.