Launchpad itself

Merge lp:~wgrant/launchpad/bug-598345-restrict-dep-contexts into lp:launchpad

  1. bug-598345-restrict-dep-contexts
  2. Merge into devel
Proposed by William Grant
Status: Merged
Approved by: Robert Collins
Approved revision: no longer in the source branch.
Merged at revision: 11175
Proposed branch: lp:~wgrant/launchpad/bug-598345-restrict-dep-contexts
Merge into: lp:launchpad
Diff against target: 770 lines (+326/-182)
10 files modified
lib/lp/soyuz/adapters/archivedependencies.py (+83/-55)
lib/lp/soyuz/doc/archive-dependencies.txt (+3/-3)
lib/lp/soyuz/doc/archive.txt (+1/-67)
lib/lp/soyuz/doc/binarypackagebuild.txt (+5/-3)
lib/lp/soyuz/interfaces/archive.py (+18/-5)
lib/lp/soyuz/model/archive.py (+19/-15)
lib/lp/soyuz/model/binarypackagebuild.py (+16/-25)
lib/lp/soyuz/tests/test_archive.py (+138/-3)
lib/lp/soyuz/tests/test_binarypackagebuild.py (+39/-4)
lib/lp/soyuz/tests/test_publishing.py (+4/-2)
To merge this branch: bzr merge lp:~wgrant/launchpad/bug-598345-restrict-dep-contexts
Reviewer Review Type Date Requested Status
Robert Collins (community) Approve
Review via email: mp+30203@code.launchpad.net

Commit message

Only retry depwait builds if the candidate is in an available pocket and component.

Description of the change

This branches fixes bug #52698 and bug #598345. At present some depwait builds are retried prematurely, as the candidate query in Archive.findDepCandidateByName constrains just the archive and distroarchseries context -- not the pocket or component, which are also included in the sources.list entries and restrict the available packages. A notable case is that of Hardy PPA packages dependent on debhelper 7: while a sufficient version is present in hardy-backports, most PPAs are not configured to use -backports. This results in such builds being retried every time the script runs, using vast amounts of buildd time to no effect.

A related (and also fixed here) issue is bug #606789: findDepCandidateByName() returned just the latest publication (by BPPH.id), and BinaryPackageBuild._isDependencySatisfied() checked the version constraint against that. But this ignores the fact that apt will look at *all* available versions across all archives and pockets, so _isDependencySatisfied needs to check that at least one of *any* of the available versions satisfies the version constraint.

I've fixed the first problem by refactoring the archive dependency expansion logic in lp.soyuz.adapters.archivedependencies, creating a new expand_dependencies() which returns a list of archives with the available pockets and components. The existing get_sources_list_for_building() wraps that, converting each dependency into a sources.list line, and adding any custom external dependencies as before. findDepCandidateByName() then uses expand_dependencies() to create the context restriction clause (archive, pocket and component). This is the same logic used to generate sources.list itself -- so it results in a completely accurate query.

For the second issue, findDepCandidateByName() has been replaced with findDepCandidates(), which returns a sequence of all matching binaries, latest first. _isDependencySatisfied() has been tweaked to look through all of them, returning True when a satisfying candidate is identified.

I've also removed some manual ogre-model logic from _isDependencySatisfied(), since that's now handled by findDepCandidates().

The findDepCandidateByName() doctests made me sad, so they've been replaced with expanded unit tests in test_archive. test_binarypackagebuild has two new tests: one checking that versioned dependencies work (a hole in the existing suite), and another verifying that more than just the latest binary is considered (new behaviour).

To post a comment you must log in.
Revision history for this message
Robert Collins (lifeless) wrote :

Please say /why/ here:
+ # We want the external dependency lines to show up second: after the
174 + # archive itself, but before any other dependencies.
175 + return [sources_list_lines[0]] + external_dep_lines + \

e.g. 'for an unknown reason We want ...

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/lp/soyuz/adapters/archivedependencies.py'
--- lib/lp/soyuz/adapters/archivedependencies.py 2010-05-10 19:49:02 +0000
+++ lib/lp/soyuz/adapters/archivedependencies.py 2010-07-20 09:26:10 +0000
@@ -12,8 +12,9 @@
1212
13Auxiliary functions exposed for testing purposes:13Auxiliary functions exposed for testing purposes:
1414
15 * get_components_for_building: return the corresponding component15 * get_components_for_context: return the corresponding component
16 dependencies for a build, this result is known as 'ogre_components';16 dependencies for a component and pocket, this result is known as
17 'ogre_components';
17 * get_primary_current_component: return the component name where the18 * get_primary_current_component: return the component name where the
18 building source is published in the primary archive.19 building source is published in the primary archive.
1920
@@ -30,7 +31,8 @@
30 'component_dependencies',31 'component_dependencies',
31 'default_component_dependency_name',32 'default_component_dependency_name',
32 'default_pocket_dependency',33 'default_pocket_dependency',
33 'get_components_for_building',34 'expand_dependencies',
35 'get_components_for_context',
34 'get_primary_current_component',36 'get_primary_current_component',
35 'get_sources_list_for_building',37 'get_sources_list_for_building',
36 'pocket_dependencies',38 'pocket_dependencies',
@@ -86,19 +88,20 @@
86default_component_dependency_name = 'multiverse'88default_component_dependency_name = 'multiverse'
8789
8890
89def get_components_for_building(build):91def get_components_for_context(component, pocket):
90 """Return the components allowed to be used in the build context.92 """Return the components allowed to be used in the build context.
9193
92 :param build: a context `IBuild`.94 :param component: the context `IComponent`.
95 :param pocket: the context `IPocket`.
93 :return: a list of component names.96 :return: a list of component names.
94 """97 """
95 # BACKPORTS should be able to fetch build dependencies from any98 # BACKPORTS should be able to fetch build dependencies from any
96 # component in order to cope with component changes occurring99 # component in order to cope with component changes occurring
97 # accross distroseries. See bug #198936 for further information.100 # across distroseries. See bug #198936 for further information.
98 if build.pocket == PackagePublishingPocket.BACKPORTS:101 if pocket == PackagePublishingPocket.BACKPORTS:
99 return component_dependencies['multiverse']102 return component_dependencies['multiverse']
100103
101 return component_dependencies[build.current_component.name]104 return component_dependencies[component.name]
102105
103106
104def get_primary_current_component(archive, distroseries, sourcepackagename):107def get_primary_current_component(archive, distroseries, sourcepackagename):
@@ -119,6 +122,54 @@
119 return 'universe'122 return 'universe'
120123
121124
125def expand_dependencies(archive, distro_series, pocket, component,
126 source_package_name):
127 """Return the set of dependency archives, pockets and components.
128
129 :param archive: the context `IArchive`.
130 :param distro_series: the context `IDistroSeries`.
131 :param pocket: the context `PackagePublishingPocket`.
132 :param component: the context `IComponent`.
133 :param source_package_name: A source package name (as text)
134 :return: a list of (archive, pocket, [component]), representing the
135 dependencies defined by the given build context.
136 """
137 deps = []
138
139 # Add implicit self-dependency for non-primary contexts.
140 if archive.purpose in ALLOW_RELEASE_BUILDS:
141 deps.append((
142 archive, PackagePublishingPocket.RELEASE,
143 get_components_for_context(component, pocket)))
144
145 primary_component = get_primary_current_component(
146 archive, distro_series, source_package_name)
147 # Consider user-selected archive dependencies.
148 for archive_dependency in archive.dependencies:
149 # When the dependency component is undefined, we should use
150 # the component where the source is published in the primary
151 # archive.
152 if archive_dependency.component is None:
153 components = component_dependencies[primary_component]
154 else:
155 components = component_dependencies[
156 archive_dependency.component.name]
157 # Follow pocket dependencies.
158 for pocket in pocket_dependencies[archive_dependency.pocket]:
159 deps.append(
160 (archive_dependency.dependency, pocket, components))
161
162 # Consider primary archive dependency override. Add the default
163 # primary archive dependencies if it's not present.
164 if archive.getArchiveDependency(
165 archive.distribution.main_archive) is None:
166 primary_dependencies = _get_default_primary_dependencies(
167 archive, component, pocket)
168 deps.extend(primary_dependencies)
169
170 return deps
171
172
122def get_sources_list_for_building(build, distroarchseries, sourcepackagename):173def get_sources_list_for_building(build, distroarchseries, sourcepackagename):
123 """Return the sources_list entries required to build the given item.174 """Return the sources_list entries required to build the given item.
124175
@@ -133,17 +184,13 @@
133 :param sourcepackagename: A source package name (as text)184 :param sourcepackagename: A source package name (as text)
134 :return: a deb sources_list entries (lines).185 :return: a deb sources_list entries (lines).
135 """186 """
136 deps = []187 deps = expand_dependencies(
137 sources_list_lines = []188 build.archive, distroarchseries.distroseries, build.pocket,
138189 build.current_component, sourcepackagename)
139 # Add implicit self-dependency for non-primary contexts.190 sources_list_lines = \
140 if build.archive.purpose in ALLOW_RELEASE_BUILDS:191 _get_sources_list_for_dependencies(deps, distroarchseries)
141 self_dep = [(192
142 build.archive, PackagePublishingPocket.RELEASE,193 external_dep_lines = []
143 get_components_for_building(build))]
144 sources_list_lines = _get_sources_list_for_dependencies(
145 self_dep, distroarchseries)
146
147 # Append external sources_list lines for this archive if it's194 # Append external sources_list lines for this archive if it's
148 # specified in the configuration.195 # specified in the configuration.
149 try:196 try:
@@ -152,7 +199,7 @@
152 for archive_dep in dependencies.splitlines():199 for archive_dep in dependencies.splitlines():
153 line = archive_dep % (200 line = archive_dep % (
154 {'series': distroarchseries.distroseries.name})201 {'series': distroarchseries.distroseries.name})
155 sources_list_lines.append(line)202 external_dep_lines.append(line)
156 except StandardError, e:203 except StandardError, e:
157 # Malformed external dependencies can incapacitate the build farm204 # Malformed external dependencies can incapacitate the build farm
158 # manager (lp:516169). That's obviously not acceptable.205 # manager (lp:516169). That's obviously not acceptable.
@@ -166,34 +213,13 @@
166 if build.archive.enabled == True:213 if build.archive.enabled == True:
167 build.archive.disable()214 build.archive.disable()
168215
169 # Consider user-selected archive dependencies.216 # For an unknown reason (perhaps because OEM has archives with
170 primary_component = get_primary_current_component(217 # binaries that need to override primary binaries of the same
171 build.archive, build.distro_series, sourcepackagename)218 # version), we want the external dependency lines to show up second:
172 for archive_dependency in build.archive.dependencies:219 # after the archive itself, but before any other dependencies.
173 # When the dependency component is undefined, we should use220 return [sources_list_lines[0]] + external_dep_lines + \
174 # the component where the source is published in the primary221 sources_list_lines[1:]
175 # archive.222
176 if archive_dependency.component is None:
177 components = component_dependencies[primary_component]
178 else:
179 components = component_dependencies[
180 archive_dependency.component.name]
181 # Follow pocket dependencies.
182 for pocket in pocket_dependencies[archive_dependency.pocket]:
183 deps.append(
184 (archive_dependency.dependency, pocket, components)
185 )
186
187 # Consider primary archive dependency override. Add the default
188 # primary archive dependencies if it's not present.
189 if build.archive.getArchiveDependency(
190 build.archive.distribution.main_archive) is None:
191 primary_dependencies = _get_default_primary_dependencies(build)
192 deps.extend(primary_dependencies)
193
194 sources_list_lines.extend(
195 _get_sources_list_for_dependencies(deps, distroarchseries))
196 return sources_list_lines
197223
198def _has_published_binaries(archive, distroarchseries, pocket):224def _has_published_binaries(archive, distroarchseries, pocket):
199 """Whether or not the archive dependency has published binaries."""225 """Whether or not the archive dependency has published binaries."""
@@ -252,27 +278,29 @@
252 return sources_list_lines278 return sources_list_lines
253279
254280
255def _get_default_primary_dependencies(build):281def _get_default_primary_dependencies(archive, component, pocket):
256 """Return the default primary dependencies for a given build.282 """Return the default primary dependencies for a given context.
257283
258 :param build: the `IBuild` context;284 :param archive: the context `IArchive`.
285 :param component: the context `IComponent`.
286 :param pocket: the context `PackagePublishingPocket`.
259287
260 :return: a list containing the default dependencies to primary288 :return: a list containing the default dependencies to primary
261 archive.289 archive.
262 """290 """
263 if build.archive.purpose in ALLOW_RELEASE_BUILDS:291 if archive.purpose in ALLOW_RELEASE_BUILDS:
264 primary_pockets = pocket_dependencies[292 primary_pockets = pocket_dependencies[
265 default_pocket_dependency]293 default_pocket_dependency]
266 primary_components = component_dependencies[294 primary_components = component_dependencies[
267 default_component_dependency_name]295 default_component_dependency_name]
268 else:296 else:
269 primary_pockets = pocket_dependencies[build.pocket]297 primary_pockets = pocket_dependencies[pocket]
270 primary_components = get_components_for_building(build)298 primary_components = get_components_for_context(component, pocket)
271299
272 primary_dependencies = []300 primary_dependencies = []
273 for pocket in primary_pockets:301 for pocket in primary_pockets:
274 primary_dependencies.append(302 primary_dependencies.append(
275 (build.distro_series.distribution.main_archive, pocket,303 (archive.distribution.main_archive, pocket,
276 primary_components))304 primary_components))
277305
278 return primary_dependencies306 return primary_dependencies
279307
=== modified file 'lib/lp/soyuz/doc/archive-dependencies.txt'
--- lib/lp/soyuz/doc/archive-dependencies.txt 2010-05-10 19:49:02 +0000
+++ lib/lp/soyuz/doc/archive-dependencies.txt 2010-07-20 09:26:10 +0000
@@ -97,7 +97,7 @@
97 ... 'main', 'restricted', 'universe', 'multiverse', 'partner']97 ... 'main', 'restricted', 'universe', 'multiverse', 'partner']
9898
99 >>> from lp.soyuz.adapters.archivedependencies import (99 >>> from lp.soyuz.adapters.archivedependencies import (
100 ... get_components_for_building)100 ... get_components_for_context)
101101
102 >>> ogre_pub = test_publisher.getPubSource(sourcename='ogre')102 >>> ogre_pub = test_publisher.getPubSource(sourcename='ogre')
103 >>> [ogre_build] = ogre_pub.createMissingBuilds()103 >>> [ogre_build] = ogre_pub.createMissingBuilds()
@@ -111,7 +111,8 @@
111 ... syncUpdate(ogre_pub)111 ... syncUpdate(ogre_pub)
112 ... flush_database_caches()112 ... flush_database_caches()
113 ... components_term = " ".join(113 ... components_term = " ".join(
114 ... get_components_for_building(ogre_build))114 ... get_components_for_context(
115 ... ogre_build.current_component, ogre_build.pocket))
115 ... print '%10s | %s' % (ogre_build.current_component.name,116 ... print '%10s | %s' % (ogre_build.current_component.name,
116 ... components_term)117 ... components_term)
117118
@@ -566,4 +567,3 @@
566 main restricted universe multiverse567 main restricted universe multiverse
567 deb http://ftpmaster.internal/ubuntu hoary-updates568 deb http://ftpmaster.internal/ubuntu hoary-updates
568 main restricted universe multiverse569 main restricted universe multiverse
569
570570
=== modified file 'lib/lp/soyuz/doc/archive.txt'
--- lib/lp/soyuz/doc/archive.txt 2010-07-02 21:25:36 +0000
+++ lib/lp/soyuz/doc/archive.txt 2010-07-20 09:26:10 +0000
@@ -1285,72 +1285,6 @@
1285 ...1285 ...
1286 AssertionError: This dependency does not exist.1286 AssertionError: This dependency does not exist.
12871287
1288== Find binary package dependency candidates ==
1289
1290Archive allows a lookup on a single binary package dependency
1291candidate by its name, via the `PublishedPackage` table:
1292
1293 >>> warty_i386 = warty['i386']
1294
1295 >>> candidate = ubuntu.main_archive.findDepCandidateByName(
1296 ... warty_i386, "pmount")
1297 >>> print candidate.binarypackagerelease.binarypackagename.name
1298 pmount
1299
1300 >>> candidate = cprov.archive.findDepCandidateByName(
1301 ... warty_i386, "pmount")
1302 >>> print candidate.binarypackagerelease.binarypackagename.name
1303 pmount
1304
1305Since 'python2.4' isn't available in our sampledata (not even
1306published), None is returned:
1307
1308 >>> print ubuntu.main_archive.findDepCandidateByName(
1309 ... warty_i386, "python2.4")
1310 None
1311
1312 >>> print cprov.archive.findDepCandidateByName(
1313 ... warty_i386, "python2.4")
1314 None
1315
1316This method is aware of the archive dependency tree. So, even when a
1317package is not published on the context PPA but is available somewhere
1318in the archive dependency domain it will be found.
1319
1320We also add another archive dependency here to exercise findDepCandidateByName
1321a little more.
1322
1323 >>> joe = factory.makePerson(email='joe@example.com')
1324 >>> second_ppa = factory.makeArchive(name="secondppa", owner=joe)
1325 >>> second_archive_dep = cprov.archive.addArchiveDependency(
1326 ... second_ppa, release_pocket, main_component)
1327
1328'at' binary package is not present in Celso's PPA.
1329
1330 >>> cprov_archive.getAllPublishedBinaries(name='at').count()
1331 0
1332
1333But it is available in PRIMARY ubuntu archive.
1334
1335 >>> primary_candidate = ubuntu.main_archive.findDepCandidateByName(
1336 ... warty_i386, "at")
1337 >>> primary_candidate is not None
1338 True
1339
1340Then a lookup on Celso's PPA will find it.
1341
1342 >>> ppa_candidate = cprov.archive.findDepCandidateByName(
1343 ... warty_i386, "at")
1344 >>> ppa_candidate is not None
1345 True
1346
1347 >>> primary_candidate == ppa_candidate
1348 True
1349
1350And clean-up after ourselves:
1351
1352 >>> ignore = cprov.archive.removeArchiveDependency(second_ppa)
1353
1354== Creating a package copy request from an IArchive ==1288== Creating a package copy request from an IArchive ==
13551289
1356The IArchive interface includes a convenience method for creating a1290The IArchive interface includes a convenience method for creating a
@@ -1529,7 +1463,7 @@
15291463
1530 >>> archive_purposes = [archive.purpose.name for archive in archive_set]1464 >>> archive_purposes = [archive.purpose.name for archive in archive_set]
1531 >>> len(archive_purposes)1465 >>> len(archive_purposes)
1532 191466 18
15331467
1534 >>> print sorted(set(archive_purposes))1468 >>> print sorted(set(archive_purposes))
1535 ['COPY', 'DEBUG', 'PARTNER', 'PPA', 'PRIMARY']1469 ['COPY', 'DEBUG', 'PARTNER', 'PPA', 'PRIMARY']
15361470
=== modified file 'lib/lp/soyuz/doc/binarypackagebuild.txt'
--- lib/lp/soyuz/doc/binarypackagebuild.txt 2010-06-10 22:12:08 +0000
+++ lib/lp/soyuz/doc/binarypackagebuild.txt 2010-07-20 09:26:10 +0000
@@ -987,8 +987,9 @@
987 main987 main
988988
989 >>> from lp.soyuz.adapters.archivedependencies import (989 >>> from lp.soyuz.adapters.archivedependencies import (
990 ... get_components_for_building)990 ... get_components_for_context)
991 >>> print get_components_for_building(depwait_build)991 >>> print get_components_for_context(
992 ... depwait_build.current_component, depwait_build.pocket)
992 ['main']993 ['main']
993994
994Thus the 'pmount' dependency remains unsatisfied.995Thus the 'pmount' dependency remains unsatisfied.
@@ -1036,7 +1037,8 @@
1036 >>> flush_database_caches()1037 >>> flush_database_caches()
1037 >>> login(ANONYMOUS)1038 >>> login(ANONYMOUS)
10381039
1039 >>> print get_components_for_building(depwait_build)1040 >>> print get_components_for_context(
1041 ... depwait_build.current_component, depwait_build.pocket)
1040 ['main']1042 ['main']
10411043
1042 >>> print pmount_pub.component.name1044 >>> print pmount_pub.component.name
10431045
=== modified file 'lib/lp/soyuz/interfaces/archive.py'
--- lib/lp/soyuz/interfaces/archive.py 2010-07-12 08:45:32 +0000
+++ lib/lp/soyuz/interfaces/archive.py 2010-07-20 09:26:10 +0000
@@ -443,11 +443,24 @@
443 Person table indexes while searching.443 Person table indexes while searching.
444 """444 """
445445
446 def findDepCandidateByName(distroarchseries, name):446 def findDepCandidates(distro_arch_series, pocket, component,
447 """Return the last published binarypackage by given name.447 source_package_name, dep_name):
448448 """Return matching binaries in this archive and its dependencies.
449 Return the `BinaryPackagePublishingHistory` record by distroarchseries449
450 and name, or None if not found.450 Return all published `IBinaryPackagePublishingHistory` records with
451 the given name, in this archive and dependencies as specified by the
452 given build context, using the usual archive dependency rules.
453
454 We can't just use the first, since there may be other versions
455 published in other dependency archives.
456
457 :param distro_arch_series: the context `IDistroArchSeries`.
458 :param pocket: the context `PackagePublishingPocket`.
459 :param component: the context `IComponent`.
460 :param source_package_name: the context source package name (as text).
461 :param dep_name: the name of the binary package to look up.
462 :return: a sequence of matching `IBinaryPackagePublishingHistory`
463 records.
451 """464 """
452465
453 def removeArchiveDependency(dependency):466 def removeArchiveDependency(dependency):
454467
=== modified file 'lib/lp/soyuz/model/archive.py'
--- lib/lp/soyuz/model/archive.py 2010-07-12 08:45:32 +0000
+++ lib/lp/soyuz/model/archive.py 2010-07-20 09:26:10 +0000
@@ -37,6 +37,7 @@
37from lp.buildmaster.model.buildfarmjob import BuildFarmJob37from lp.buildmaster.model.buildfarmjob import BuildFarmJob
38from lp.buildmaster.model.packagebuild import PackageBuild38from lp.buildmaster.model.packagebuild import PackageBuild
39from lp.services.job.interfaces.job import JobStatus39from lp.services.job.interfaces.job import JobStatus
40from lp.soyuz.adapters.archivedependencies import expand_dependencies
40from lp.soyuz.adapters.packagelocation import PackageLocation41from lp.soyuz.adapters.packagelocation import PackageLocation
41from canonical.launchpad.components.tokens import (42from canonical.launchpad.components.tokens import (
42 create_unique_token_for_table)43 create_unique_token_for_table)
@@ -47,6 +48,7 @@
47from lp.soyuz.model.binarypackagerelease import (48from lp.soyuz.model.binarypackagerelease import (
48 BinaryPackageReleaseDownloadCount)49 BinaryPackageReleaseDownloadCount)
49from lp.soyuz.model.binarypackagebuild import BinaryPackageBuild50from lp.soyuz.model.binarypackagebuild import BinaryPackageBuild
51from lp.soyuz.model.component import Component
50from lp.soyuz.model.distributionsourcepackagecache import (52from lp.soyuz.model.distributionsourcepackagecache import (
51 DistributionSourcePackageCache)53 DistributionSourcePackageCache)
52from lp.soyuz.model.distroseriespackagecache import DistroSeriesPackageCache54from lp.soyuz.model.distroseriespackagecache import DistroSeriesPackageCache
@@ -807,30 +809,32 @@
807 self.sources_cached = sources_cached.count()809 self.sources_cached = sources_cached.count()
808 self.binaries_cached = binaries_cached.count()810 self.binaries_cached = binaries_cached.count()
809811
810 def findDepCandidateByName(self, distroarchseries, name):812 def findDepCandidates(self, distro_arch_series, pocket, component,
813 source_package_name, dep_name):
811 """See `IArchive`."""814 """See `IArchive`."""
812 archives = []815 deps = expand_dependencies(
813 if self.is_ppa:816 self, distro_arch_series.distroseries, pocket, component,
814 archives.append(self.distribution.main_archive.id)817 source_package_name)
815 archives.append(self.id)818 archive_clause = Or([And(
816 archives.extend(819 BinaryPackagePublishingHistory.archiveID == archive.id,
817 IResultSet(self.dependencies).values(820 BinaryPackagePublishingHistory.pocket == pocket,
818 ArchiveDependency.dependencyID))821 Component.name.is_in(components))
822 for (archive, pocket, components) in deps])
819823
820 store = ISlaveStore(BinaryPackagePublishingHistory)824 store = ISlaveStore(BinaryPackagePublishingHistory)
821 candidate = store.find(825 return store.find(
822 BinaryPackagePublishingHistory,826 BinaryPackagePublishingHistory,
823 BinaryPackageName.name == name,827 BinaryPackageName.name == dep_name,
824 BinaryPackageRelease.binarypackagename == BinaryPackageName.id,828 BinaryPackageRelease.binarypackagename == BinaryPackageName.id,
825 BinaryPackagePublishingHistory.binarypackagerelease ==829 BinaryPackagePublishingHistory.binarypackagerelease ==
826 BinaryPackageRelease.id,830 BinaryPackageRelease.id,
827 BinaryPackagePublishingHistory.distroarchseries ==831 BinaryPackagePublishingHistory.distroarchseries ==
828 distroarchseries,832 distro_arch_series,
829 In(BinaryPackagePublishingHistory.archiveID, archives),
830 BinaryPackagePublishingHistory.status ==833 BinaryPackagePublishingHistory.status ==
831 PackagePublishingStatus.PUBLISHED834 PackagePublishingStatus.PUBLISHED,
832 ).order_by(Desc(BinaryPackagePublishingHistory.id))835 BinaryPackagePublishingHistory.componentID == Component.id,
833 return candidate.first()836 archive_clause).order_by(
837 Desc(BinaryPackagePublishingHistory.id))
834838
835 def getArchiveDependency(self, dependency):839 def getArchiveDependency(self, dependency):
836 """See `IArchive`."""840 """See `IArchive`."""
837841
=== modified file 'lib/lp/soyuz/model/binarypackagebuild.py'
--- lib/lp/soyuz/model/binarypackagebuild.py 2010-06-15 11:11:27 +0000
+++ lib/lp/soyuz/model/binarypackagebuild.py 2010-07-20 09:26:10 +0000
@@ -50,7 +50,6 @@
50from lp.buildmaster.model.packagebuild import (50from lp.buildmaster.model.packagebuild import (
51 PackageBuild, PackageBuildDerived)51 PackageBuild, PackageBuildDerived)
52from lp.services.job.model.job import Job52from lp.services.job.model.job import Job
53from lp.soyuz.adapters.archivedependencies import get_components_for_building
54from lp.soyuz.interfaces.archive import ArchivePurpose53from lp.soyuz.interfaces.archive import ArchivePurpose
55from lp.soyuz.interfaces.binarypackagebuild import (54from lp.soyuz.interfaces.binarypackagebuild import (
56 BuildSetStatus, CannotBeRescored, IBinaryPackageBuild,55 BuildSetStatus, CannotBeRescored, IBinaryPackageBuild,
@@ -387,33 +386,25 @@
387 def _isDependencySatisfied(self, token):386 def _isDependencySatisfied(self, token):
388 """Check if the given dependency token is satisfied.387 """Check if the given dependency token is satisfied.
389388
390 Check if the dependency exists, if its version constraint is389 Check if the dependency exists and that its version constraint is
391 satisfied and if it is reachable in the build context.390 satisfied.
392 """391 """
393 name, version, relation = self._parseDependencyToken(token)392 name, version, relation = self._parseDependencyToken(token)
394393
395 dep_candidate = self.archive.findDepCandidateByName(394 # There may be several published versions in the available
396 self.distro_arch_series, name)395 # archives and pockets. If any one of them satisifies our
397396 # constraints, the dependency is satisfied.
398 if not dep_candidate:397 dep_candidates = self.archive.findDepCandidates(
399 return False398 self.distro_arch_series, self.pocket, self.current_component,
400399 self.source_package_release.sourcepackagename.name, name)
401 if not self._checkDependencyVersion(400
402 dep_candidate.binarypackagerelease.version, version, relation):401 for dep_candidate in dep_candidates:
403 return False402 if self._checkDependencyVersion(
404403 dep_candidate.binarypackagerelease.version, version,
405 # Only PRIMARY archive build dependencies should be restricted404 relation):
406 # to the ogre_components. Both PARTNER and PPA can reach405 return True
407 # dependencies from all components in the PRIMARY archive.406
408 # Moreover, PARTNER and PPA component domain is single, i.e,407 return False
409 # PARTNER only contains packages in 'partner' component and PPAs
410 # only contains packages in 'main' component.
411 ogre_components = get_components_for_building(self)
412 if (self.archive.purpose == ArchivePurpose.PRIMARY and
413 dep_candidate.component.name not in ogre_components):
414 return False
415
416 return True
417408
418 def _toAptFormat(self, token):409 def _toAptFormat(self, token):
419 """Rebuild dependencies line in apt format."""410 """Rebuild dependencies line in apt format."""
420411
=== modified file 'lib/lp/soyuz/tests/test_archive.py'
--- lib/lp/soyuz/tests/test_archive.py 2010-07-12 13:32:53 +0000
+++ lib/lp/soyuz/tests/test_archive.py 2010-07-20 09:26:10 +0000
@@ -4,9 +4,10 @@
4"""Test Archive features."""4"""Test Archive features."""
55
6from datetime import date, datetime, timedelta6from datetime import date, datetime, timedelta
7import unittest
8
7import pytz9import pytz
8import unittest10import transaction
9
10from zope.component import getUtility11from zope.component import getUtility
11from zope.security.interfaces import Unauthorized12from zope.security.interfaces import Unauthorized
12from zope.security.proxy import removeSecurityProxy13from zope.security.proxy import removeSecurityProxy
@@ -1099,7 +1100,141 @@
1099 login("commercial-member@canonical.com")1100 login("commercial-member@canonical.com")
1100 self.setCommercial(self.archive, True)1101 self.setCommercial(self.archive, True)
1101 self.assertTrue(self.archive.commercial)1102 self.assertTrue(self.archive.commercial)
1102 1103
1104
1105class TestFindDepCandidates(TestCaseWithFactory):
1106 """Tests for Archive.findDepCandidates."""
1107
1108 layer = LaunchpadZopelessLayer
1109
1110 def setUp(self):
1111 super(TestFindDepCandidates, self).setUp()
1112 self.archive = self.factory.makeArchive()
1113 self.publisher = SoyuzTestPublisher()
1114 login('admin@canonical.com')
1115 self.publisher.prepareBreezyAutotest()
1116
1117 def assertDep(self, arch_tag, name, expected, archive=None,
1118 pocket=PackagePublishingPocket.RELEASE, component=None,
1119 source_package_name='something-new'):
1120 """Helper to check that findDepCandidates works.
1121
1122 Searches for the given dependency name in the given architecture and
1123 archive, and compares it to the given expected value.
1124 The archive defaults to self.archive.
1125
1126 Also commits, since findDepCandidates uses the slave store.
1127 """
1128 transaction.commit()
1129
1130 if component is None:
1131 component = getUtility(IComponentSet)['main']
1132 if archive is None:
1133 archive = self.archive
1134
1135 self.assertEquals(
1136 list(
1137 archive.findDepCandidates(
1138 self.publisher.distroseries[arch_tag], pocket, component,
1139 source_package_name, name)),
1140 expected)
1141
1142 def test_finds_candidate_in_same_archive(self):
1143 # A published candidate in the same archive should be found.
1144 bins = self.publisher.getPubBinaries(
1145 binaryname='foo', archive=self.archive,
1146 status=PackagePublishingStatus.PUBLISHED)
1147 self.assertDep('i386', 'foo', [bins[0]])
1148 self.assertDep('hppa', 'foo', [bins[1]])
1149
1150 def test_does_not_find_pending_publication(self):
1151 # A pending candidate in the same archive should not be found.
1152 bins = self.publisher.getPubBinaries(
1153 binaryname='foo', archive=self.archive)
1154 self.assertDep('i386', 'foo', [])
1155
1156 def test_ppa_searches_primary_archive(self):
1157 # PPA searches implicitly look in the primary archive too.
1158 self.assertEquals(self.archive.purpose, ArchivePurpose.PPA)
1159 self.assertDep('i386', 'foo', [])
1160
1161 bins = self.publisher.getPubBinaries(
1162 binaryname='foo', archive=self.archive.distribution.main_archive,
1163 status=PackagePublishingStatus.PUBLISHED)
1164
1165 self.assertDep('i386', 'foo', [bins[0]])
1166
1167 def test_searches_dependencies(self):
1168 # Candidates from archives on which the target explicitly depends
1169 # should be found.
1170 bins = self.publisher.getPubBinaries(
1171 binaryname='foo', archive=self.archive,
1172 status=PackagePublishingStatus.PUBLISHED)
1173 other_archive = self.factory.makeArchive()
1174 self.assertDep('i386', 'foo', [], archive=other_archive)
1175
1176 other_archive.addArchiveDependency(
1177 self.archive, PackagePublishingPocket.RELEASE)
1178 self.assertDep('i386', 'foo', [bins[0]], archive=other_archive)
1179
1180 def test_obeys_dependency_pockets(self):
1181 # Only packages published in a pocket matching the dependency should
1182 # be found.
1183 release_bins = self.publisher.getPubBinaries(
1184 binaryname='foo-release', archive=self.archive,
1185 status=PackagePublishingStatus.PUBLISHED)
1186 updates_bins = self.publisher.getPubBinaries(
1187 binaryname='foo-updates', archive=self.archive,
1188 status=PackagePublishingStatus.PUBLISHED,
1189 pocket=PackagePublishingPocket.UPDATES)
1190 proposed_bins = self.publisher.getPubBinaries(
1191 binaryname='foo-proposed', archive=self.archive,
1192 status=PackagePublishingStatus.PUBLISHED,
1193 pocket=PackagePublishingPocket.PROPOSED)
1194
1195 # Temporarily turn our test PPA into a copy archive, so we can
1196 # add non-RELEASE dependencies on it.
1197 removeSecurityProxy(self.archive).purpose = ArchivePurpose.COPY
1198
1199 other_archive = self.factory.makeArchive()
1200 other_archive.addArchiveDependency(
1201 self.archive, PackagePublishingPocket.UPDATES)
1202 self.assertDep(
1203 'i386', 'foo-release', [release_bins[0]], archive=other_archive)
1204 self.assertDep(
1205 'i386', 'foo-updates', [updates_bins[0]], archive=other_archive)
1206 self.assertDep('i386', 'foo-proposed', [], archive=other_archive)
1207
1208 other_archive.removeArchiveDependency(self.archive)
1209 other_archive.addArchiveDependency(
1210 self.archive, PackagePublishingPocket.PROPOSED)
1211 self.assertDep(
1212 'i386', 'foo-proposed', [proposed_bins[0]], archive=other_archive)
1213
1214 def test_obeys_dependency_components(self):
1215 # Only packages published in a component matching the dependency
1216 # should be found.
1217 primary = self.archive.distribution.main_archive
1218 main_bins = self.publisher.getPubBinaries(
1219 binaryname='foo-main', archive=primary, component='main',
1220 status=PackagePublishingStatus.PUBLISHED)
1221 universe_bins = self.publisher.getPubBinaries(
1222 binaryname='foo-universe', archive=primary,
1223 component='universe',
1224 status=PackagePublishingStatus.PUBLISHED)
1225
1226 self.archive.addArchiveDependency(
1227 primary, PackagePublishingPocket.RELEASE,
1228 component=getUtility(IComponentSet)['main'])
1229 self.assertDep('i386', 'foo-main', [main_bins[0]])
1230 self.assertDep('i386', 'foo-universe', [])
1231
1232 self.archive.removeArchiveDependency(primary)
1233 self.archive.addArchiveDependency(
1234 primary, PackagePublishingPocket.RELEASE,
1235 component=getUtility(IComponentSet)['universe'])
1236 self.assertDep('i386', 'foo-main', [main_bins[0]])
1237 self.assertDep('i386', 'foo-universe', [universe_bins[0]])
11031238
11041239
1105def test_suite():1240def test_suite():
11061241
=== modified file 'lib/lp/soyuz/tests/test_binarypackagebuild.py'
--- lib/lp/soyuz/tests/test_binarypackagebuild.py 2010-06-21 07:26:51 +0000
+++ lib/lp/soyuz/tests/test_binarypackagebuild.py 2010-07-20 09:26:10 +0000
@@ -164,13 +164,13 @@
164 Return an `IBinaryPackageBuild` in MANUALDEWAIT state and depending on a164 Return an `IBinaryPackageBuild` in MANUALDEWAIT state and depending on a
165 binary that exists and is reachable.165 binary that exists and is reachable.
166 """166 """
167 test_publisher = SoyuzTestPublisher()167 self.publisher = SoyuzTestPublisher()
168 test_publisher.prepareBreezyAutotest()168 self.publisher.prepareBreezyAutotest()
169169
170 depwait_source = test_publisher.getPubSource(170 depwait_source = self.publisher.getPubSource(
171 sourcename='depwait-source')171 sourcename='depwait-source')
172172
173 test_publisher.getPubBinaries(173 self.publisher.getPubBinaries(
174 binaryname='dep-bin',174 binaryname='dep-bin',
175 status=PackagePublishingStatus.PUBLISHED)175 status=PackagePublishingStatus.PUBLISHED)
176176
@@ -273,6 +273,41 @@
273 depwait_build.updateDependencies()273 depwait_build.updateDependencies()
274 self.assertEquals(depwait_build.dependencies, '')274 self.assertEquals(depwait_build.dependencies, '')
275275
276 def testVersionedDependencies(self):
277 # `IBinaryPackageBuild.updateDependencies` supports versioned
278 # dependencies. A build will not be retried unless the candidate
279 # complies with the version restriction.
280 # In this case, dep-bin 666 is available. >> 666 isn't
281 # satisified, but >= 666 is.
282 depwait_build = self._setupSimpleDepwaitContext()
283 self.layer.txn.commit()
284
285 depwait_build.dependencies = u'dep-bin (>> 666)'
286 depwait_build.updateDependencies()
287 self.assertEquals(depwait_build.dependencies, u'dep-bin (>> 666)')
288 depwait_build.dependencies = u'dep-bin (>= 666)'
289 depwait_build.updateDependencies()
290 self.assertEquals(depwait_build.dependencies, u'')
291
292 def testVersionedDependencyOnOldPublication(self):
293 # `IBinaryPackageBuild.updateDependencies` doesn't just consider
294 # the latest publication. There may be older publications which
295 # satisfy the version constraints (in other archives or pockets).
296 # In this case, dep-bin 666 and 999 are available, so both = 666
297 # and = 999 are satisfied.
298 depwait_build = self._setupSimpleDepwaitContext()
299 self.publisher.getPubBinaries(
300 binaryname='dep-bin', version='999',
301 status=PackagePublishingStatus.PUBLISHED)
302 self.layer.txn.commit()
303
304 depwait_build.dependencies = u'dep-bin (= 666)'
305 depwait_build.updateDependencies()
306 self.assertEquals(depwait_build.dependencies, u'')
307 depwait_build.dependencies = u'dep-bin (= 999)'
308 depwait_build.updateDependencies()
309 self.assertEquals(depwait_build.dependencies, u'')
310
276311
277class BaseTestCaseWithThreeBuilds(TestCaseWithFactory):312class BaseTestCaseWithThreeBuilds(TestCaseWithFactory):
278313
279314
=== modified file 'lib/lp/soyuz/tests/test_publishing.py'
--- lib/lp/soyuz/tests/test_publishing.py 2010-07-20 09:16:14 +0000
+++ lib/lp/soyuz/tests/test_publishing.py 2010-07-20 09:26:10 +0000
@@ -267,7 +267,8 @@
267 pub_source=None,267 pub_source=None,
268 version='666',268 version='666',
269 architecturespecific=False,269 architecturespecific=False,
270 builder=None):270 builder=None,
271 component='main'):
271 """Return a list of binary publishing records."""272 """Return a list of binary publishing records."""
272 if distroseries is None:273 if distroseries is None:
273 distroseries = self.distroseries274 distroseries = self.distroseries
@@ -285,7 +286,8 @@
285 pub_source = self.getPubSource(286 pub_source = self.getPubSource(
286 sourcename=sourcename, status=status, pocket=pocket,287 sourcename=sourcename, status=status, pocket=pocket,
287 archive=archive, distroseries=distroseries,288 archive=archive, distroseries=distroseries,
288 version=version, architecturehintlist=architecturehintlist)289 version=version, architecturehintlist=architecturehintlist,
290 component=component)
289 else:291 else:
290 archive = pub_source.archive292 archive = pub_source.archive
291293