Launchpad itself

Merge lp:~wgrant/launchpad/get-out-of-here-p-a-s into lp:launchpad

  1. get-out-of-here-p-a-s
  2. Merge into devel
Proposed by William Grant
Status: Merged
Approved by: Henning Eggers
Approved revision: no longer in the source branch.
Merged at revision: not available
Proposed branch: lp:~wgrant/launchpad/get-out-of-here-p-a-s
Merge into: lp:launchpad
Diff against target: 304 lines (+104/-100)
6 files modified
lib/lp/buildmaster/master.py (+1/-95)
lib/lp/soyuz/doc/package-arch-specific.txt (+2/-3)
lib/lp/soyuz/model/publishing.py (+1/-1)
lib/lp/soyuz/model/queue.py (+1/-1)
lib/lp/soyuz/pas.py (+94/-0)
lib/lp/soyuz/tests/test_doc.py (+5/-0)
To merge this branch: bzr merge lp:~wgrant/launchpad/get-out-of-here-p-a-s
Reviewer Review Type Date Requested Status
Julian Edwards (community) Approve
Henning Eggers (community) code Approve
Review via email: mp+21312@code.launchpad.net

Commit message

Move P-a-s from lp.buildmaster to lp.soyuz. Landed by henninge.

Description of the change

This branch just moves the Packages-architecture-specific code from lp.buildmaster to lp.soyuz. It's only used to select the Soyuz binary builds that need to be created, so it belongs in lp.soyuz.

To post a comment you must log in.
Revision history for this message
Henning Eggers (henninge) wrote :

Thanks. ;-)

review: Approve (code)
Revision history for this message
Julian Edwards (julian-edwards) wrote :

Looks ok, but I think pas.py should come out of model/ and into soyuz/

Cheers

Revision history for this message
Julian Edwards (julian-edwards) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/lp/buildmaster/master.py'
--- lib/lp/buildmaster/master.py 2010-03-08 12:53:59 +0000
+++ lib/lp/buildmaster/master.py 2010-03-15 10:39:26 +0000
@@ -14,7 +14,6 @@
1414
1515
16import logging16import logging
17import operator
1817
19from zope.component import getUtility18from zope.component import getUtility
2019
@@ -24,102 +23,9 @@
24from lp.archivepublisher.utils import process_in_batches23from lp.archivepublisher.utils import process_in_batches
25from lp.buildmaster.interfaces.buildbase import BuildStatus24from lp.buildmaster.interfaces.buildbase import BuildStatus
26from lp.buildmaster.interfaces.buildqueue import IBuildQueueSet25from lp.buildmaster.interfaces.buildqueue import IBuildQueueSet
27from lp.buildmaster.pas import BuildDaemonPackagesArchSpecific
28from lp.buildmaster.buildergroup import BuilderGroup26from lp.buildmaster.buildergroup import BuilderGroup
29from lp.soyuz.interfaces.archive import ArchivePurpose
30from lp.soyuz.interfaces.build import IBuildSet27from lp.soyuz.interfaces.build import IBuildSet
3128from lp.soyuz.pas import BuildDaemonPackagesArchSpecific
32
33def determineArchitecturesToBuild(pubrec, legal_archseries,
34 distroseries, pas_verify=None):
35 """Return a list of architectures for which this publication should build.
36
37 This function answers the question: given a publication, what
38 architectures should we build it for? It takes a set of legal
39 distroarchseries and the distribution series for which we are
40 building, and optionally a BuildDaemonPackagesArchSpecific
41 (informally known as 'P-a-s') instance.
42
43 The P-a-s component contains a list of forbidden architectures for
44 each source package, which should be respected regardless of which
45 architectures have been requested in the source package metadata,
46 for instance:
47
48 * 'aboot' should only build on powerpc
49 * 'mozilla-firefox' should not build on sparc
50
51 This black/white list is an optimization to suppress temporarily
52 known-failures build attempts and thus saving build-farm time.
53
54 For PPA publications we only consider architectures supported by PPA
55 subsystem (`DistroArchSeries`.supports_virtualized flag) and P-a-s is turned
56 off to give the users the chance to test their fixes for upstream
57 problems.
58
59 :param: pubrec: `ISourcePackagePublishingHistory` representing the
60 source publication.
61 :param: legal_archseries: a list of all initialized `DistroArchSeries`
62 to be considered.
63 :param: distroseries: the context `DistroSeries`.
64 :param: pas_verify: optional P-a-s verifier object/component.
65 :return: a list of `DistroArchSeries` for which the source publication in
66 question should be built.
67 """
68 hint_string = pubrec.sourcepackagerelease.architecturehintlist
69
70 assert hint_string, 'Missing arch_hint_list'
71
72 # Ignore P-a-s for PPA publications.
73 if pubrec.archive.purpose == ArchivePurpose.PPA:
74 pas_verify = None
75
76 # The 'PPA supported' flag only applies to virtualized archives
77 if pubrec.archive.require_virtualized:
78 legal_archseries = [
79 arch for arch in legal_archseries if arch.supports_virtualized]
80 # Cope with no virtualization support at all. It usually happens when
81 # a distroseries is created and initialized, by default no
82 # architecture supports its. Distro-team might take some time to
83 # decide which architecture will be allowed for PPAs and queue-builder
84 # will continue to work meanwhile.
85 if not legal_archseries:
86 return []
87
88 legal_arch_tags = set(arch.architecturetag for arch in legal_archseries)
89
90 if hint_string == 'any':
91 package_tags = legal_arch_tags
92 elif hint_string == 'all':
93 nominated_arch = distroseries.nominatedarchindep
94 legal_archseries_ids = [arch.id for arch in legal_archseries]
95 assert nominated_arch.id in legal_archseries_ids, (
96 'nominatedarchindep is not present in legal_archseries: %s' %
97 ' '.join(legal_arch_tags))
98 package_tags = set([nominated_arch.architecturetag])
99 else:
100 my_archs = hint_string.split()
101 # Allow any-foo or linux-foo to mean foo. See bug 73761.
102 my_archs = [arch.replace("any-", "") for arch in my_archs]
103 my_archs = [arch.replace("linux-", "") for arch in my_archs]
104 my_archs = set(my_archs)
105 package_tags = my_archs.intersection(legal_arch_tags)
106
107 if pas_verify:
108 build_tags = set()
109 for tag in package_tags:
110 sourcepackage_name = pubrec.sourcepackagerelease.name
111 if sourcepackage_name in pas_verify.permit:
112 permitted = pas_verify.permit[sourcepackage_name]
113 if tag not in permitted:
114 continue
115 build_tags.add(tag)
116 else:
117 build_tags = package_tags
118
119 sorted_archseries = sorted(legal_archseries,
120 key=operator.attrgetter('architecturetag'))
121 return [arch for arch in sorted_archseries
122 if arch.architecturetag in build_tags]
12329
12430
125class BuilddMaster:31class BuilddMaster:
12632
=== renamed file 'lib/lp/buildmaster/tests/package-arch-specific.txt' => 'lib/lp/soyuz/doc/package-arch-specific.txt'
--- lib/lp/buildmaster/tests/package-arch-specific.txt 2009-05-14 09:28:47 +0000
+++ lib/lp/soyuz/doc/package-arch-specific.txt 2010-03-15 10:39:26 +0000
@@ -95,8 +95,8 @@
95See if the code which determines archs to build does the right thing for95See if the code which determines archs to build does the right thing for
96each of these options:96each of these options:
9797
98 >>> from lp.buildmaster.master import (98 >>> from lp.soyuz.pas import (
99 ... determineArchitecturesToBuild)99 ... BuildDaemonPackagesArchSpecific, determineArchitecturesToBuild)
100100
101 >>> def print_build_architectures(pub, pas_verify=None):101 >>> def print_build_architectures(pub, pas_verify=None):
102 ... allowed_architectures = determineArchitecturesToBuild(102 ... allowed_architectures = determineArchitecturesToBuild(
@@ -182,7 +182,6 @@
182 >>> import os182 >>> import os
183 >>> import tempfile183 >>> import tempfile
184 >>> import shutil184 >>> import shutil
185 >>> from lp.buildmaster.pas import BuildDaemonPackagesArchSpecific
186185
187 >>> def getPASVerifier(pas_string):186 >>> def getPASVerifier(pas_string):
188 ... """Build and return a PAS verifier based on the string provided."""187 ... """Build and return a PAS verifier based on the string provided."""
189188
=== modified file 'lib/lp/soyuz/model/publishing.py'
--- lib/lp/soyuz/model/publishing.py 2010-03-06 04:57:40 +0000
+++ lib/lp/soyuz/model/publishing.py 2010-03-15 10:39:26 +0000
@@ -41,7 +41,6 @@
41 IStoreSelector, MAIN_STORE, DEFAULT_FLAVOR)41 IStoreSelector, MAIN_STORE, DEFAULT_FLAVOR)
42from canonical.launchpad.webapp.interfaces import NotFoundError42from canonical.launchpad.webapp.interfaces import NotFoundError
43from lp.buildmaster.interfaces.buildbase import BuildStatus43from lp.buildmaster.interfaces.buildbase import BuildStatus
44from lp.buildmaster.master import determineArchitecturesToBuild
45from lp.registry.interfaces.person import validate_public_person44from lp.registry.interfaces.person import validate_public_person
46from lp.registry.interfaces.pocket import PackagePublishingPocket45from lp.registry.interfaces.pocket import PackagePublishingPocket
47from lp.soyuz.model.binarypackagename import BinaryPackageName46from lp.soyuz.model.binarypackagename import BinaryPackageName
@@ -62,6 +61,7 @@
62 PackagePublishingPriority, PackagePublishingStatus,61 PackagePublishingPriority, PackagePublishingStatus,
63 PoolFileOverwriteError)62 PoolFileOverwriteError)
64from lp.soyuz.interfaces.queue import PackageUploadStatus63from lp.soyuz.interfaces.queue import PackageUploadStatus
64from lp.soyuz.pas import determineArchitecturesToBuild
65from lp.soyuz.scripts.changeoverride import ArchiveOverriderError65from lp.soyuz.scripts.changeoverride import ArchiveOverriderError
6666
6767
6868
=== modified file 'lib/lp/soyuz/model/queue.py'
--- lib/lp/soyuz/model/queue.py 2010-03-12 16:52:21 +0000
+++ lib/lp/soyuz/model/queue.py 2010-03-15 10:39:26 +0000
@@ -35,7 +35,6 @@
35from lp.archiveuploader.changesfile import ChangesFile35from lp.archiveuploader.changesfile import ChangesFile
36from lp.archiveuploader.tagfiles import parse_tagfile_lines36from lp.archiveuploader.tagfiles import parse_tagfile_lines
37from lp.archiveuploader.utils import safe_fix_maintainer37from lp.archiveuploader.utils import safe_fix_maintainer
38from lp.buildmaster.pas import BuildDaemonPackagesArchSpecific
39from canonical.cachedproperty import cachedproperty38from canonical.cachedproperty import cachedproperty
40from canonical.config import config39from canonical.config import config
41from canonical.database.constants import UTC_NOW40from canonical.database.constants import UTC_NOW
@@ -60,6 +59,7 @@
60 NonBuildableSourceUploadError, QueueBuildAcceptError,59 NonBuildableSourceUploadError, QueueBuildAcceptError,
61 QueueInconsistentStateError, QueueSourceAcceptError,60 QueueInconsistentStateError, QueueSourceAcceptError,
62 QueueStateWriteProtectedError)61 QueueStateWriteProtectedError)
62from lp.soyuz.pas import BuildDaemonPackagesArchSpecific
63from canonical.launchpad.mail import (63from canonical.launchpad.mail import (
64 format_address, signed_message_from_string, sendmail)64 format_address, signed_message_from_string, sendmail)
65from lp.soyuz.scripts.processaccepted import (65from lp.soyuz.scripts.processaccepted import (
6666
=== renamed file 'lib/lp/buildmaster/pas.py' => 'lib/lp/soyuz/pas.py'
--- lib/lp/buildmaster/pas.py 2009-06-25 00:46:29 +0000
+++ lib/lp/soyuz/pas.py 2010-03-15 10:39:26 +0000
@@ -2,9 +2,12 @@
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4import os4import os
5import operator
56
6from sqlobject import SQLObjectNotFound7from sqlobject import SQLObjectNotFound
78
9from lp.soyuz.interfaces.archive import ArchivePurpose
10
8class BuildDaemonPackagesArchSpecific:11class BuildDaemonPackagesArchSpecific:
9 """Parse and implement "PackagesArchSpecific"."""12 """Parse and implement "PackagesArchSpecific"."""
1013
@@ -106,3 +109,94 @@
106109
107 return source_name, arch_tags110 return source_name, arch_tags
108111
112
113def determineArchitecturesToBuild(pubrec, legal_archseries,
114 distroseries, pas_verify=None):
115 """Return a list of architectures for which this publication should build.
116
117 This function answers the question: given a publication, what
118 architectures should we build it for? It takes a set of legal
119 distroarchseries and the distribution series for which we are
120 building, and optionally a BuildDaemonPackagesArchSpecific
121 (informally known as 'P-a-s') instance.
122
123 The P-a-s component contains a list of forbidden architectures for
124 each source package, which should be respected regardless of which
125 architectures have been requested in the source package metadata,
126 for instance:
127
128 * 'aboot' should only build on powerpc
129 * 'mozilla-firefox' should not build on sparc
130
131 This black/white list is an optimization to suppress temporarily
132 known-failures build attempts and thus saving build-farm time.
133
134 For PPA publications we only consider architectures supported by PPA
135 subsystem (`DistroArchSeries`.supports_virtualized flag) and P-a-s is turned
136 off to give the users the chance to test their fixes for upstream
137 problems.
138
139 :param: pubrec: `ISourcePackagePublishingHistory` representing the
140 source publication.
141 :param: legal_archseries: a list of all initialized `DistroArchSeries`
142 to be considered.
143 :param: distroseries: the context `DistroSeries`.
144 :param: pas_verify: optional P-a-s verifier object/component.
145 :return: a list of `DistroArchSeries` for which the source publication in
146 question should be built.
147 """
148 hint_string = pubrec.sourcepackagerelease.architecturehintlist
149
150 assert hint_string, 'Missing arch_hint_list'
151
152 # Ignore P-a-s for PPA publications.
153 if pubrec.archive.purpose == ArchivePurpose.PPA:
154 pas_verify = None
155
156 # The 'PPA supported' flag only applies to virtualized archives
157 if pubrec.archive.require_virtualized:
158 legal_archseries = [
159 arch for arch in legal_archseries if arch.supports_virtualized]
160 # Cope with no virtualization support at all. It usually happens when
161 # a distroseries is created and initialized, by default no
162 # architecture supports its. Distro-team might take some time to
163 # decide which architecture will be allowed for PPAs and queue-builder
164 # will continue to work meanwhile.
165 if not legal_archseries:
166 return []
167
168 legal_arch_tags = set(arch.architecturetag for arch in legal_archseries)
169
170 if hint_string == 'any':
171 package_tags = legal_arch_tags
172 elif hint_string == 'all':
173 nominated_arch = distroseries.nominatedarchindep
174 legal_archseries_ids = [arch.id for arch in legal_archseries]
175 assert nominated_arch.id in legal_archseries_ids, (
176 'nominatedarchindep is not present in legal_archseries: %s' %
177 ' '.join(legal_arch_tags))
178 package_tags = set([nominated_arch.architecturetag])
179 else:
180 my_archs = hint_string.split()
181 # Allow any-foo or linux-foo to mean foo. See bug 73761.
182 my_archs = [arch.replace("any-", "") for arch in my_archs]
183 my_archs = [arch.replace("linux-", "") for arch in my_archs]
184 my_archs = set(my_archs)
185 package_tags = my_archs.intersection(legal_arch_tags)
186
187 if pas_verify:
188 build_tags = set()
189 for tag in package_tags:
190 sourcepackage_name = pubrec.sourcepackagerelease.name
191 if sourcepackage_name in pas_verify.permit:
192 permitted = pas_verify.permit[sourcepackage_name]
193 if tag not in permitted:
194 continue
195 build_tags.add(tag)
196 else:
197 build_tags = package_tags
198
199 sorted_archseries = sorted(legal_archseries,
200 key=operator.attrgetter('architecturetag'))
201 return [arch for arch in sorted_archseries
202 if arch.architecturetag in build_tags]
109203
=== modified file 'lib/lp/soyuz/tests/test_doc.py'
--- lib/lp/soyuz/tests/test_doc.py 2009-06-30 16:56:07 +0000
+++ lib/lp/soyuz/tests/test_doc.py 2010-03-15 10:39:26 +0000
@@ -200,6 +200,11 @@
200 setUp=builddmasterSetUp,200 setUp=builddmasterSetUp,
201 layer=LaunchpadZopelessLayer,201 layer=LaunchpadZopelessLayer,
202 ),202 ),
203 'package-arch-specific.txt': LayeredDocFileSuite(
204 '../doc/package-arch-specific.txt',
205 setUp=builddmasterSetUp,
206 layer=LaunchpadZopelessLayer,
207 ),
203 }208 }
204209
205210