Launchpad itself

Merge ~cjwatson/launchpad:built-using-guard-copying into launchpad:master

  1. Git
  2. lp:~cjwatson/launchpad
  3. built-using-guard-copying
  4. Merge into master
Proposed by Colin Watson
Status: Needs review
Proposed branch: ~cjwatson/launchpad:built-using-guard-copying
Merge into: launchpad:master
Prerequisite: ~cjwatson/launchpad:built-using-guard-deletion
Diff against target: 311 lines (+226/-3)
5 files modified
lib/lp/soyuz/interfaces/binarysourcereference.py (+17/-0)
lib/lp/soyuz/model/binarysourcereference.py (+29/-1)
lib/lp/soyuz/scripts/packagecopier.py (+26/-2)
lib/lp/soyuz/scripts/tests/test_copypackage.py (+95/-0)
lib/lp/soyuz/tests/test_binarysourcereference.py (+59/-0)
Reviewer Review Type Date Requested Status
Ioana Lasc (community) Approve
Review via email: mp+382792@code.launchpad.net

Commit message

Guard copies of binaries with Built-Using references

Description of the change

If binaries have Built-Using references, then we need to make sure that we can resolve those references and keep the corresponding sources published while the binaries are published. Prevent copies of binaries if any such references can't be resolved in the target publishing context.

To post a comment you must log in.
Revision history for this message
Ioana Lasc (ilasc) wrote :

Looks good.

review: Approve
Reply

Unmerged commits

9c6dce7... by Colin Watson

Guard copies of binaries with Built-Using references

If binaries have Built-Using references, then we need to make sure that
we can resolve those references and keep the corresponding sources
published while the binaries are published. Prevent copies of binaries
if any such references can't be resolved in the target publishing
context.

LP: #1868558

06ccc3e... by Colin Watson

Simplify tests using createFromSourcePackageReleases

c0c4aa2... by Colin Watson

Expand deletion guard to other pockets

It now checks all pockets that could legitimately depend on the one from
which the publication is being deleted.

d7fbcfd... by Colin Watson

Guard removal of sources referenced by Built-Using

Prevent SourcePackagePublishingHistory.requestDeletion from deleting
source publications that have Built-Using references from active binary
publications in the same archive and suite.

This isn't necessarily complete: in particular, it can miss references
from other pockets, and in any case it might race with a build still in
progress. The intent of this is not to ensure integrity, but to avoid
some easily-detectable mistakes that could cause confusion.

LP: #1868558

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/lib/lp/soyuz/interfaces/binarysourcereference.py b/lib/lp/soyuz/interfaces/binarysourcereference.py
2index e625cf9..f702959 100644
3--- a/lib/lp/soyuz/interfaces/binarysourcereference.py
4+++ b/lib/lp/soyuz/interfaces/binarysourcereference.py
5@@ -99,3 +99,20 @@ class IBinarySourceReferenceSet(Interface):
6 pointing to any of this sequence of `ISourcePackageRelease`s.
7 :return: A `ResultSet` of matching `IBinarySourceReference`s.
8 """
9+
10+ def findMissingSources(archive, distroseries, pockets, reference_type,
11+ binary_package_releases=None):
12+ """Find references to unpublished sources in a given context.
13+
14+ :param archive: An `IArchive` to search for source publications.
15+ :param distroseries: An `IDistroSeries` to search for source
16+ publications.
17+ :param pockets: A sequence of `PackagePublishingPocket`s to search
18+ for source publications.
19+ :param reference_type: A `BinarySourceReferenceType` to search for.
20+ :param binary_package_releases: Only return references from any of
21+ this sequence of `IBinaryPackageRelease`s.
22+ :return: A `ResultSet` of matching `IBinarySourceReference`s where
23+ the `source_package_release` is not published in the given
24+ publishing context.
25+ """
26diff --git a/lib/lp/soyuz/model/binarysourcereference.py b/lib/lp/soyuz/model/binarysourcereference.py
27index ca3563b..c2b41e0 100644
28--- a/lib/lp/soyuz/model/binarysourcereference.py
29+++ b/lib/lp/soyuz/model/binarysourcereference.py
30@@ -14,7 +14,9 @@ __all__ = [
31 import warnings
32
33 from debian.deb822 import PkgRelation
34+from storm.expr import LeftJoin
35 from storm.locals import (
36+ And,
37 Int,
38 Reference,
39 )
40@@ -35,7 +37,10 @@ from lp.soyuz.interfaces.binarysourcereference import (
41 UnparsableBuiltUsing,
42 )
43 from lp.soyuz.model.distroarchseries import DistroArchSeries
44-from lp.soyuz.model.publishing import BinaryPackagePublishingHistory
45+from lp.soyuz.model.publishing import (
46+ BinaryPackagePublishingHistory,
47+ SourcePackagePublishingHistory,
48+ )
49
50
51 @implementer(IBinarySourceReference)
52@@ -175,3 +180,26 @@ class BinarySourceReferenceSet:
53 spr.id for spr in source_package_releases))
54 return IStore(BinarySourceReference).find(
55 BinarySourceReference, *clauses).config(distinct=True)
56+
57+ @classmethod
58+ def findMissingSources(cls, archive, distroseries, pockets, reference_type,
59+ binary_package_releases):
60+ """See `IBinarySourceReferenceSet`."""
61+ origin = [
62+ BinarySourceReference,
63+ LeftJoin(
64+ SourcePackagePublishingHistory,
65+ And(
66+ BinarySourceReference.source_package_release_id ==
67+ SourcePackagePublishingHistory.sourcepackagereleaseID,
68+ SourcePackagePublishingHistory.archive == archive,
69+ SourcePackagePublishingHistory.distroseries ==
70+ distroseries,
71+ SourcePackagePublishingHistory.pocket.is_in(pockets))),
72+ ]
73+ return IStore(BinarySourceReference).using(*origin).find(
74+ BinarySourceReference,
75+ BinarySourceReference.binary_package_release_id.is_in(
76+ bpr.id for bpr in binary_package_releases),
77+ BinarySourceReference.reference_type == reference_type,
78+ SourcePackagePublishingHistory.id == None).config(distinct=True)
79diff --git a/lib/lp/soyuz/scripts/packagecopier.py b/lib/lp/soyuz/scripts/packagecopier.py
80index 3891af5..03f4c77 100644
81--- a/lib/lp/soyuz/scripts/packagecopier.py
82+++ b/lib/lp/soyuz/scripts/packagecopier.py
83@@ -1,4 +1,4 @@
84-# Copyright 2009-2015 Canonical Ltd. This software is licensed under the
85+# Copyright 2009-2020 Canonical Ltd. This software is licensed under the
86 # GNU Affero General Public License version 3 (see the file LICENSE).
87
88 """Package copying utilities."""
89@@ -19,10 +19,17 @@ from zope.component import getUtility
90 from zope.security.proxy import removeSecurityProxy
91
92 from lp.services.database.bulk import load_related
93+from lp.soyuz.adapters.archivedependencies import pocket_dependencies
94 from lp.soyuz.adapters.overrides import SourceOverride
95-from lp.soyuz.enums import SourcePackageFormat
96+from lp.soyuz.enums import (
97+ BinarySourceReferenceType,
98+ SourcePackageFormat,
99+ )
100 from lp.soyuz.interfaces.archive import CannotCopy
101 from lp.soyuz.interfaces.binarypackagebuild import BuildSetStatus
102+from lp.soyuz.interfaces.binarysourcereference import (
103+ IBinarySourceReferenceSet,
104+ )
105 from lp.soyuz.interfaces.publishing import (
106 active_publishing_status,
107 IBinaryPackagePublishingHistory,
108@@ -439,6 +446,7 @@ class CopyChecker:
109 built_binaries = source.getBuiltBinaries(want_files=True)
110 if len(built_binaries) == 0:
111 raise CannotCopy("source has no binaries to be copied")
112+
113 # Deny copies of binary publications containing files with
114 # expiration date set. We only set such value for immediate
115 # expiration of old superseded binaries, so no point in
116@@ -449,6 +457,22 @@ class CopyChecker:
117 if binary_file.libraryfile.expires is not None:
118 raise CannotCopy('source has expired binaries')
119
120+ # Deny copies of binary publications that contain Built-Using
121+ # references to sources that do not exist in the target. The
122+ # dominator will not be able to rectify the situation.
123+ bsr_set = getUtility(IBinarySourceReferenceSet)
124+ missing_sources = bsr_set.findMissingSources(
125+ self.archive, series, pocket_dependencies[pocket],
126+ BinarySourceReferenceType.BUILT_USING,
127+ [binary_pub.binarypackagerelease
128+ for binary_pub in built_binaries])
129+ if not missing_sources.is_empty():
130+ # XXX cjwatson 2020-04-19: It may also be useful to show the
131+ # specific Built-Using references that don't exist.
132+ raise CannotCopy(
133+ 'source has binaries with Built-Using references that do '
134+ 'not exist in the target')
135+
136 # Check if there is already a source with the same name and version
137 # published in the destination archive.
138 self._checkArchiveConflicts(source, series)
139diff --git a/lib/lp/soyuz/scripts/tests/test_copypackage.py b/lib/lp/soyuz/scripts/tests/test_copypackage.py
140index 388fcf7..a41cfb1 100644
141--- a/lib/lp/soyuz/scripts/tests/test_copypackage.py
142+++ b/lib/lp/soyuz/scripts/tests/test_copypackage.py
143@@ -1046,6 +1046,101 @@ class CopyCheckerTestCase(TestCaseWithFactory):
144 copied_from_archive=binary.archive),
145 ]))
146
147+ def test_checkCopy_cannot_copy_dangling_built_using_references(self):
148+ # checkCopy() raises CannotCopy if the copy includes binaries and
149+ # the binaries contain Built-Using references to sources that do not
150+ # exist in the target.
151+
152+ # Create testing sources and binaries.
153+ source = self.test_publisher.getPubSource()
154+ built_using_source = self.test_publisher.getPubSource(
155+ sourcename='built-using')
156+ built_using_relationship = '%s (= %s)' % (
157+ built_using_source.sourcepackagerelease.name,
158+ built_using_source.sourcepackagerelease.version)
159+ self.test_publisher.getPubBinaries(
160+ pub_source=source, built_using=built_using_relationship)
161+
162+ # Create a fresh PPA which will be the destination copy.
163+ archive = self.factory.makeArchive(
164+ distribution=self.test_publisher.ubuntutest,
165+ purpose=ArchivePurpose.PPA)
166+ series = source.distroseries
167+ pocket = source.pocket
168+
169+ # Now source-only copies are allowed.
170+ copy_checker = CopyChecker(archive, include_binaries=False)
171+ self.assertIsNone(
172+ copy_checker.checkCopy(
173+ source, series, pocket, check_permissions=False))
174+
175+ # Copies with binaries are denied.
176+ copy_checker = CopyChecker(archive, include_binaries=True)
177+ self.assertRaisesWithContent(
178+ CannotCopy,
179+ 'source has binaries with Built-Using references that do not '
180+ 'exist in the target',
181+ copy_checker.checkCopy,
182+ source, series, pocket, check_permissions=False)
183+
184+ def test_checkCopy_can_copy_resolvable_built_using_references(self):
185+ # checkCopy() allows copying binaries with Built-Using references to
186+ # sources that exist in the target, even if no longer published or
187+ # in a pocket that the target depends on.
188+
189+ # Create testing sources and binaries.
190+ source = self.test_publisher.getPubSource()
191+ published_source = self.test_publisher.getPubSource(
192+ sourcename='published')
193+ superseded_source = self.test_publisher.getPubSource(
194+ sourcename='superseded')
195+ release_pocket_source = self.test_publisher.getPubSource(
196+ sourcename='release-pocket')
197+ relationships = [
198+ '%s (= %s)' % (
199+ spph.sourcepackagerelease.name,
200+ spph.sourcepackagerelease.version)
201+ for spph in (
202+ published_source, superseded_source, release_pocket_source)]
203+ self.test_publisher.getPubBinaries(
204+ built_using=', '.join(relationships),
205+ status=PackagePublishingStatus.PUBLISHED, pub_source=source)
206+ target_series = self.factory.makeDistroSeries(
207+ distribution=source.distroseries.distribution)
208+ getUtility(ISourcePackageFormatSelectionSet).add(
209+ target_series, SourcePackageFormat.FORMAT_1_0)
210+ self.factory.makeSourcePackagePublishingHistory(
211+ distroseries=target_series, archive=source.archive,
212+ sourcepackagerelease=published_source.sourcepackagerelease,
213+ pocket=PackagePublishingPocket.PROPOSED,
214+ status=PackagePublishingStatus.PUBLISHED)
215+ self.factory.makeSourcePackagePublishingHistory(
216+ distroseries=target_series, archive=source.archive,
217+ sourcepackagerelease=superseded_source.sourcepackagerelease,
218+ pocket=PackagePublishingPocket.PROPOSED,
219+ status=PackagePublishingStatus.SUPERSEDED)
220+ self.factory.makeSourcePackagePublishingHistory(
221+ distroseries=target_series, archive=source.archive,
222+ sourcepackagerelease=release_pocket_source.sourcepackagerelease,
223+ pocket=PackagePublishingPocket.RELEASE,
224+ status=PackagePublishingStatus.PUBLISHED)
225+
226+ # Copies of binaries are permitted.
227+ copy_checker = CopyChecker(source.archive, include_binaries=True)
228+ copy_checker.checkCopy(
229+ source, target_series, PackagePublishingPocket.PROPOSED,
230+ check_permissions=False)
231+
232+ # Since some of the sources were only published in PROPOSED, copies
233+ # of binaries to RELEASE that refer to them are denied.
234+ self.assertRaisesWithContent(
235+ CannotCopy,
236+ 'source has binaries with Built-Using references that do not '
237+ 'exist in the target',
238+ copy_checker.checkCopy,
239+ source, target_series, PackagePublishingPocket.RELEASE,
240+ check_permissions=False)
241+
242
243 class BaseDoCopyTests:
244
245diff --git a/lib/lp/soyuz/tests/test_binarysourcereference.py b/lib/lp/soyuz/tests/test_binarysourcereference.py
246index 0c4db7f..b8ebcef 100644
247--- a/lib/lp/soyuz/tests/test_binarysourcereference.py
248+++ b/lib/lp/soyuz/tests/test_binarysourcereference.py
249@@ -306,3 +306,62 @@ class TestBinarySourceReference(TestCaseWithFactory):
250 source_package_release=bsr.source_package_release,
251 reference_type=BinarySourceReferenceType.BUILT_USING)
252 for bsr in [bsrs[0], bsrs[2]])))
253+
254+ def test_findMissingSources(self):
255+ # findMissingSources finds references whose source publications
256+ # aren't present in the given publishing context.
257+ archive = self.factory.makeArchive()
258+ distroseries = archive.distribution.currentseries
259+ pockets = (
260+ PackagePublishingPocket.RELEASE, PackagePublishingPocket.PROPOSED)
261+ spphs = [
262+ self.factory.makeSourcePackagePublishingHistory(
263+ archive=archive, distroseries=distroseries, pocket=pocket)
264+ for pocket in pockets]
265+ bprs = []
266+ for spph in spphs:
267+ build = self.factory.makeBinaryPackageBuild(
268+ distroarchseries=self.factory.makeDistroArchSeries(
269+ distroseries=spph.distroseries),
270+ archive=spph.archive, pocket=spph.pocket)
271+ bprs.append(self.factory.makeBinaryPackageRelease(build=build))
272+ for bpr, spph in zip(bprs, spphs):
273+ self.reference_set.createFromSourcePackageReleases(
274+ bpr, [spph.sourcepackagerelease],
275+ BinarySourceReferenceType.BUILT_USING)
276+ self.assertTrue(
277+ self.reference_set.findMissingSources(
278+ archive, distroseries, pockets,
279+ BinarySourceReferenceType.BUILT_USING, bprs).is_empty())
280+ # Try searching with slight mismatches; findMissingSources should
281+ # return appropriate results since the necessary SPPHs aren't
282+ # present.
283+ self.assertThat(
284+ self.reference_set.findMissingSources(
285+ self.factory.makeArchive(), distroseries, pockets,
286+ BinarySourceReferenceType.BUILT_USING, bprs),
287+ MatchesSetwise(*(
288+ MatchesStructure.byEquality(
289+ binary_package_release=bpr,
290+ source_package_release=spph.sourcepackagerelease,
291+ reference_type=BinarySourceReferenceType.BUILT_USING)
292+ for bpr, spph in zip(bprs, spphs))))
293+ self.assertThat(
294+ self.reference_set.findMissingSources(
295+ archive, self.factory.makeDistroSeries(), pockets,
296+ BinarySourceReferenceType.BUILT_USING, bprs),
297+ MatchesSetwise(*(
298+ MatchesStructure.byEquality(
299+ binary_package_release=bpr,
300+ source_package_release=spph.sourcepackagerelease,
301+ reference_type=BinarySourceReferenceType.BUILT_USING)
302+ for bpr, spph in zip(bprs, spphs))))
303+ self.assertThat(
304+ self.reference_set.findMissingSources(
305+ archive, distroseries, [PackagePublishingPocket.PROPOSED],
306+ BinarySourceReferenceType.BUILT_USING, bprs),
307+ MatchesSetwise(
308+ MatchesStructure.byEquality(
309+ binary_package_release=bprs[0],
310+ source_package_release=spphs[0].sourcepackagerelease,
311+ reference_type=BinarySourceReferenceType.BUILT_USING)))

Subscribers

People subscribed via source and target branches

to status/vote changes: