Launchpad itself

Merge lp:~james-w/launchpad/copy-archive-package-sets into lp:launchpad/db-devel

  1. copy-archive-package-sets
  2. Merge into db-devel
Proposed by James Westby
Status: Merged
Approved by: Graham Binns
Approved revision: no longer in the source branch.
Merged at revision: 9495
Proposed branch: lp:~james-w/launchpad/copy-archive-package-sets
Merge into: lp:launchpad/db-devel
Prerequisite: lp:~james-w/launchpad/copy-archive-test-refactor
Diff against target: 446 lines (+212/-15)
5 files modified
lib/lp/soyuz/adapters/packagelocation.py (+26/-4)
lib/lp/soyuz/adapters/tests/test_packagelocation.py (+54/-2)
lib/lp/soyuz/model/packagecloner.py (+13/-1)
lib/lp/soyuz/scripts/populate_archive.py (+17/-6)
lib/lp/soyuz/scripts/tests/test_populatearchive.py (+102/-2)
To merge this branch: bzr merge lp:~james-w/launchpad/copy-archive-package-sets
Reviewer Review Type Date Requested Status
Graham Binns (community) code Approve
Review via email: mp+28165@code.launchpad.net

Commit message

--package-set can now be passed to populate_archive to limit the packages copied.

Description of the change

Summary

We want to be able to create copy archives that aren't the full archive.

A way to subset the archive in LP is by packageset, so we make use of that.

Proposed fix

This merge proposal adds a --package-set option to populate_archive.py,
which can be specified multiple times, and then only copies the packages
contained within the named sets.

Pre-implementation notes

Spoke briefly with Julian.

Implementation details

First PackageLocation was modified to store a list of package sets for the
source. build_package_location was also changed to lookup package sets based
on names.

We then hooked that up to the --package-set option to get the list of package
sets the user wants.

The final step is to hook that in to the query that copies the sources. Here we
take care to include the transitive closure of packages across the DAG of package
sets.

Tests

You can run them with

./bin/test -s lp.soyuz.scripts -m test_populatearchive
and
./bin/test -s lp.soyuz.adapters

Demo and Q/A

None.

lint

= Launchpad lint =

Checking for conflicts. and issues in doctests and templates.
Running jslint, xmllint, pyflakes, and pylint.
Using normal rules.

Linting changed files:
  lib/lp/soyuz/adapters/packagelocation.py
  lib/lp/soyuz/adapters/tests/__init__.py
  lib/lp/soyuz/adapters/tests/test_packagelocation.py
  lib/lp/soyuz/model/packagecloner.py
  lib/lp/soyuz/scripts/populate_archive.py
  lib/lp/soyuz/scripts/tests/test_populatearchive.py

== Pyflakes notices ==

lib/lp/soyuz/scripts/populate_archive.py
    112: local variable 'ignore_this' is assigned to but never used
    287: local variable 'ignore_result' is assigned to but never used

To post a comment you must log in.
Revision history for this message
Graham Binns (gmb) wrote :

Hi James,

One minor nitpick but otherwise this branch is r=me.

> === modified file 'lib/lp/soyuz/adapters/packagelocation.py'
> --- lib/lp/soyuz/adapters/packagelocation.py 2009-08-28 06:39:38 +0000
> +++ lib/lp/soyuz/adapters/packagelocation.py 2010-06-22 12:11:56 +0000
> @@ -63,6 +66,10 @@
> if self.component is not None:
> result += ' (%s)' % self.component.name
>
> + if self.packagesets:

We should explicitly test len() here, since we know it's going to be a
list even if it's empty.

> + result += ' [%s]' % (
> + ", ".join([str(p.name) for p in self.packagesets]),)
> +
> return result
>
>

review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/lp/soyuz/adapters/packagelocation.py'
2--- lib/lp/soyuz/adapters/packagelocation.py 2009-08-28 06:39:38 +0000
3+++ lib/lp/soyuz/adapters/packagelocation.py 2010-06-22 12:51:37 +0000
4@@ -28,22 +28,25 @@
5 distroseries = None
6 pocket = None
7 component = None
8+ packagesets = None
9
10 def __init__(self, archive, distribution, distroseries, pocket,
11- component=None):
12+ component=None, packagesets=None):
13 """Initialize the PackageLocation from the given parameters."""
14 self.archive = archive
15 self.distribution = distribution
16 self.distroseries = distroseries
17 self.pocket = pocket
18 self.component = component
19+ self.packagesets = packagesets or []
20
21 def __eq__(self, other):
22 if (self.distribution == other.distribution and
23 self.archive == other.archive and
24 self.distroseries == other.distroseries and
25 self.component == other.component and
26- self.pocket == other.pocket):
27+ self.pocket == other.pocket and
28+ self.packagesets == other.packagesets):
29 return True
30 return False
31
32@@ -63,6 +66,10 @@
33 if self.component is not None:
34 result += ' (%s)' % self.component.name
35
36+ if len(self.packagesets) > 0:
37+ result += ' [%s]' % (
38+ ", ".join([str(p.name) for p in self.packagesets]),)
39+
40 return result
41
42
43@@ -71,7 +78,8 @@
44
45
46 def build_package_location(distribution_name, suite=None, purpose=None,
47- person_name=None, archive_name=None):
48+ person_name=None, archive_name=None,
49+ packageset_names=None):
50 """Convenience function to build PackageLocation objects."""
51
52 # XXX kiko 2007-10-24:
53@@ -87,6 +95,7 @@
54 # Avoid circular imports.
55 from lp.registry.interfaces.distribution import IDistributionSet
56 from lp.soyuz.interfaces.archive import ArchivePurpose, IArchiveSet
57+ from lp.soyuz.interfaces.packageset import IPackagesetSet
58 from lp.registry.interfaces.pocket import PackagePublishingPocket
59
60 try:
61@@ -144,4 +153,17 @@
62 distroseries = distribution.currentseries
63 pocket = PackagePublishingPocket.RELEASE
64
65- return PackageLocation(archive, distribution, distroseries, pocket)
66+ packagesets = []
67+ if packageset_names:
68+ packageset_set = getUtility(IPackagesetSet)
69+ for packageset_name in packageset_names:
70+ try:
71+ packageset = packageset_set.getByName(
72+ packageset_name, distroseries=distroseries)
73+ except NotFoundError, err:
74+ raise PackageLocationError(
75+ "Could not find packageset %s" % err)
76+ packagesets.append(packageset)
77+
78+ return PackageLocation(archive, distribution, distroseries, pocket,
79+ packagesets=packagesets)
80
81=== added file 'lib/lp/soyuz/adapters/tests/__init__.py'
82=== modified file 'lib/lp/soyuz/adapters/tests/test_packagelocation.py'
83--- lib/lp/soyuz/adapters/tests/test_packagelocation.py 2009-06-25 04:06:00 +0000
84+++ lib/lp/soyuz/adapters/tests/test_packagelocation.py 2010-06-22 12:51:37 +0000
85@@ -20,10 +20,11 @@
86
87 def getPackageLocation(self, distribution_name='ubuntu', suite=None,
88 purpose=None, person_name=None,
89- archive_name=None):
90+ archive_name=None, packageset_names=None):
91 """Use a helper method to setup a `PackageLocation` object."""
92 return build_package_location(
93- distribution_name, suite, purpose, person_name, archive_name)
94+ distribution_name, suite, purpose, person_name, archive_name,
95+ packageset_names=packageset_names)
96
97 def testSetupLocationForCOPY(self):
98 """`PackageLocation` for COPY archives."""
99@@ -45,6 +46,7 @@
100 self.assertEqual(location.pocket.name, 'RELEASE')
101 self.assertEqual(location.archive.displayname,
102 'Copy archive now-comes-the-mystery for Mysteryman')
103+ self.assertEqual([], location.packagesets)
104
105 def testSetupLocationForPRIMARY(self):
106 """`PackageLocation` for PRIMARY archives."""
107@@ -54,6 +56,7 @@
108 self.assertEqual(location.pocket.name, 'RELEASE')
109 self.assertEqual(location.archive.displayname,
110 'Primary Archive for Ubuntu Linux')
111+ self.assertEqual([], location.packagesets)
112
113 def testSetupLocationForPPA(self):
114 """`PackageLocation` for PPA archives."""
115@@ -65,6 +68,7 @@
116 self.assertEqual(location.pocket.name, 'RELEASE')
117 self.assertEqual(location.archive.displayname,
118 'PPA for Celso Providelo')
119+ self.assertEqual([], location.packagesets)
120
121 def testSetupLocationForPARTNER(self):
122 """`PackageLocation` for PARTNER archives."""
123@@ -74,6 +78,16 @@
124 self.assertEqual(location.pocket.name, 'RELEASE')
125 self.assertEqual(location.archive.displayname,
126 'Partner Archive for Ubuntu Linux')
127+ self.assertEqual([], location.packagesets)
128+
129+ def testSetupLocationWithPackagesets(self):
130+ packageset_name1 = u"foo-packageset"
131+ packageset_name2 = u"bar-packageset"
132+ packageset1 = self.factory.makePackageset(name=packageset_name1)
133+ packageset2 = self.factory.makePackageset(name=packageset_name2)
134+ location = self.getPackageLocation(
135+ packageset_names=[packageset_name1, packageset_name2])
136+ self.assertEqual([packageset1, packageset2], location.packagesets)
137
138 def testSetupLocationUnknownDistribution(self):
139 """`PackageLocationError` is raised on unknown distribution."""
140@@ -115,6 +129,24 @@
141 distribution_name='debian',
142 purpose=ArchivePurpose.PARTNER)
143
144+ def test_build_package_location_when_packageset_unknown(self):
145+ """`PackageLocationError` is raised on unknown packageset."""
146+ self.assertRaises(
147+ PackageLocationError,
148+ self.getPackageLocation,
149+ distribution_name='debian',
150+ packageset_names=[u"unknown"])
151+
152+ def test_build_package_location_when_one_packageset_unknown(self):
153+ """Test that with one of two packagesets unknown."""
154+ packageset_name = u"foo-packageset"
155+ self.factory.makePackageset(name=packageset_name)
156+ self.assertRaises(
157+ PackageLocationError,
158+ self.getPackageLocation,
159+ distribution_name='debian',
160+ packageset_names=[packageset_name, u"unknown"])
161+
162 def testSetupLocationPPANotMatchingDistribution(self):
163 """`PackageLocationError` is raised when PPA does not match the
164 distribution."""
165@@ -167,6 +199,18 @@
166 distribution_name='ubuntu', purpose=ArchivePurpose.PARTNER)
167 self.assertNotEqual(location_ubuntu_partner, location_cprov_ppa)
168
169+ def testComparePackagesets(self):
170+ location_ubuntu_hoary = self.getPackageLocation()
171+ location_ubuntu_hoary_again = self.getPackageLocation()
172+ packageset = self.factory.makePackageset()
173+ location_ubuntu_hoary.packagesets = [packageset]
174+ self.assertNotEqual(
175+ location_ubuntu_hoary, location_ubuntu_hoary_again)
176+
177+ location_ubuntu_hoary_again.packagesets = [packageset]
178+ self.assertEqual(
179+ location_ubuntu_hoary, location_ubuntu_hoary_again)
180+
181 def testRepresentation(self):
182 """Check if PackageLocation is represented correctly."""
183 location_ubuntu_hoary = self.getPackageLocation()
184@@ -204,6 +248,14 @@
185 str(location_ubuntu_partner),
186 'Partner Archive for Ubuntu Linux: hoary-RELEASE')
187
188+ self.factory.makePackageset(name=u"foo-packageset")
189+ location_ubuntu_packageset = self.getPackageLocation(
190+ packageset_names=[u"foo-packageset"])
191+ self.assertEqual(
192+ str(location_ubuntu_packageset),
193+ 'Primary Archive for Ubuntu Linux: '
194+ 'hoary-RELEASE [foo-packageset]')
195+
196
197 def test_suite():
198 return unittest.TestLoader().loadTestsFromName(__name__)
199
200=== modified file 'lib/lp/soyuz/model/packagecloner.py'
201--- lib/lp/soyuz/model/packagecloner.py 2010-01-12 08:36:58 +0000
202+++ lib/lp/soyuz/model/packagecloner.py 2010-06-22 12:51:37 +0000
203@@ -305,7 +305,7 @@
204 to be copied.
205 """
206 store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
207- store.execute('''
208+ query = ('''
209 INSERT INTO SourcePackagePublishingHistory (
210 sourcepackagerelease, distroseries, status, component,
211 section, archive, datecreated, datepublished, pocket)
212@@ -322,6 +322,18 @@
213 PackagePublishingStatus.PENDING,
214 PackagePublishingStatus.PUBLISHED,
215 origin.pocket, origin.archive))
216+ if origin.packagesets:
217+ query += '''AND spph.sourcepackagerelease IN
218+ (SELECT spr.id
219+ FROM SourcePackageRelease AS spr,
220+ packagesetsources AS pss,
221+ flatpackagesetinclusion AS fpsi
222+ WHERE spr.sourcepackagename
223+ = pss.sourcepackagename
224+ AND pss.packageset = fpsi.child
225+ AND fpsi.parent in %s)
226+ ''' % sqlvalues([p.id for p in origin.packagesets])
227+ store.execute(query)
228
229 def packageSetDiff(self, origin, destination, logger=None):
230 """Please see `IPackageCloner`."""
231
232=== modified file 'lib/lp/soyuz/scripts/populate_archive.py'
233--- lib/lp/soyuz/scripts/populate_archive.py 2010-06-22 12:51:34 +0000
234+++ lib/lp/soyuz/scripts/populate_archive.py 2010-06-22 12:51:37 +0000
235@@ -53,7 +53,7 @@
236 self, from_archive, from_distribution, from_suite, from_user,
237 component, to_distribution, to_suite, to_archive, to_user, reason,
238 include_binaries, arch_tags, merge_copy_flag,
239- packageset_delta_flag):
240+ packageset_delta_flag, packageset_tags):
241 """Create archive, populate it with packages and builds.
242
243 Please note: if a component was specified for the origin then the
244@@ -79,6 +79,9 @@
245 existing copy archive.
246 :param packageset_delta_flag: only show packages that are fresher or
247 new in the origin archive. Do not copy anything.
248+
249+ :param packageset_tags: list of packagesets to limit the packages
250+ copied to.
251 """
252 # Avoid circular imports.
253 from lp.registry.interfaces.person import IPersonSet
254@@ -108,13 +111,14 @@
255 for proc_family in proc_families:
256 ignore_this = aa_set.new(archive, proc_family)
257
258- def build_location(distro, suite, component):
259+ def build_location(distro, suite, component, packageset_names=None):
260 """Build and return package location."""
261- location = build_package_location(distro, suite=suite)
262+ location = build_package_location(
263+ distro, suite=suite, packageset_names=packageset_names)
264 if component is not None:
265 try:
266 the_component = getUtility(IComponentSet)[component]
267- except NotFoundError, e:
268+ except NotFoundError:
269 raise SoyuzScriptError(
270 "Invalid component name: '%s'" % component)
271 location.component = the_component
272@@ -122,7 +126,9 @@
273
274 archive_set = getUtility(IArchiveSet)
275 # Build the origin package location.
276- the_origin = build_location(from_distribution, from_suite, component)
277+ the_origin = build_location(
278+ from_distribution, from_suite, component,
279+ packageset_names=packageset_tags)
280
281 # Use a non-PPA(!) origin archive if specified and existent.
282 if from_archive is not None and from_user is None:
283@@ -315,7 +321,7 @@
284 opts.from_user, opts.component, opts.to_distribution,
285 opts.to_suite, opts.to_archive, opts.to_user, opts.reason,
286 opts.include_binaries, opts.arch_tags, opts.merge_copy_flag,
287- opts.packageset_delta_flag)
288+ opts.packageset_delta_flag, opts.packageset_tags)
289
290 def add_my_options(self):
291 """Parse command line arguments for copy archive creation/population.
292@@ -379,6 +385,11 @@
293 'Only show packages that are fresher or new in origin '
294 'archive. Destination archive must exist already.'))
295
296+ self.parser.add_option(
297+ "--package-set", dest="packageset_tags", action="append",
298+ help=(
299+ 'Limit to copying packages in the selected packagesets.'))
300+
301 def _createMissingBuilds(
302 self, distroseries, archive, proc_families):
303 """Create builds for all cloned source packages.
304
305=== modified file 'lib/lp/soyuz/scripts/tests/test_populatearchive.py'
306--- lib/lp/soyuz/scripts/tests/test_populatearchive.py 2010-06-22 12:51:34 +0000
307+++ lib/lp/soyuz/scripts/tests/test_populatearchive.py 2010-06-22 12:51:37 +0000
308@@ -206,7 +206,7 @@
309 def getScript(self, test_args=None):
310 """Return an ArchivePopulator instance."""
311 if test_args is None:
312- test_args = []
313+ test_args = []
314 script = ArchivePopulator("test copy archives", test_args=test_args)
315 script.logger = QuietFakeLogger()
316 script.txn = self.layer.txn
317@@ -214,7 +214,7 @@
318
319 def copyArchive(self, distroseries, archive_name, owner,
320 architectures=None, component="main", from_user=None,
321- from_archive=None):
322+ from_archive=None, packageset_names=None):
323 """Run the copy-archive script."""
324 extra_args = [
325 '--from-distribution', distroseries.distribution.name,
326@@ -240,6 +240,12 @@
327 for architecture in architectures:
328 extra_args.extend(['-a', architecture])
329
330+ if packageset_names is None:
331+ packageset_names = []
332+
333+ for packageset_name in packageset_names:
334+ extra_args.extend(['--package-set', packageset_name])
335+
336 script = self.getScript(test_args=extra_args)
337 script.mainTask()
338
339@@ -521,6 +527,86 @@
340 component="main")
341 self.checkBuilds(copy_archive, package_infos)
342
343+ def testCopyArchiveSubsetsBasedOnPackageset(self):
344+ """Test that --package-set limits the sources copied."""
345+ package_infos = [
346+ PackageInfo(
347+ "bzr", "2.1", status=PackagePublishingStatus.PUBLISHED),
348+ PackageInfo(
349+ "apt", "2.2", status=PackagePublishingStatus.PUBLISHED),
350+ ]
351+ owner = self.createTargetOwner()
352+ distroseries = self.createSourceDistribution(package_infos)
353+ packageset_name = u"apt-packageset"
354+ spn = self.factory.getOrMakeSourcePackageName(name="apt")
355+ self.factory.makePackageset(
356+ name=packageset_name, distroseries=distroseries, packages=(spn,))
357+ archive_name = self.getTargetArchiveName(distroseries.distribution)
358+ copy_archive = self.copyArchive(
359+ distroseries, archive_name, owner,
360+ packageset_names=[packageset_name])
361+ self.checkCopiedSources(
362+ copy_archive, distroseries, [package_infos[1]])
363+
364+ def testCopyArchiveUnionsPackagesets(self):
365+ """Test that package sets are unioned when copying archives."""
366+ package_infos = [
367+ PackageInfo(
368+ "bzr", "2.1", status=PackagePublishingStatus.PUBLISHED),
369+ PackageInfo(
370+ "apt", "2.2", status=PackagePublishingStatus.PUBLISHED),
371+ PackageInfo(
372+ "gcc", "4.5", status=PackagePublishingStatus.PUBLISHED),
373+ ]
374+ owner = self.createTargetOwner()
375+ distroseries = self.createSourceDistribution(package_infos)
376+ apt_packageset_name = u"apt-packageset"
377+ apt_spn = self.factory.getOrMakeSourcePackageName(name="apt")
378+ gcc_packageset_name = u"gcc-packageset"
379+ gcc_spn = self.factory.getOrMakeSourcePackageName(name="gcc")
380+ self.factory.makePackageset(
381+ name=apt_packageset_name, distroseries=distroseries,
382+ packages=(apt_spn,))
383+ self.factory.makePackageset(
384+ name=gcc_packageset_name, distroseries=distroseries,
385+ packages=(gcc_spn,))
386+ archive_name = self.getTargetArchiveName(distroseries.distribution)
387+ copy_archive = self.copyArchive(
388+ distroseries, archive_name, owner,
389+ packageset_names=[apt_packageset_name, gcc_packageset_name])
390+ self.checkCopiedSources(
391+ copy_archive, distroseries, package_infos[1:])
392+
393+ def testCopyArchiveRecursivelyCopiesPackagesets(self):
394+ """Test that package set copies include subsets."""
395+ package_infos = [
396+ PackageInfo(
397+ "bzr", "2.1", status=PackagePublishingStatus.PUBLISHED),
398+ PackageInfo(
399+ "apt", "2.2", status=PackagePublishingStatus.PUBLISHED),
400+ PackageInfo(
401+ "gcc", "4.5", status=PackagePublishingStatus.PUBLISHED),
402+ ]
403+ owner = self.createTargetOwner()
404+ distroseries = self.createSourceDistribution(package_infos)
405+ apt_packageset_name = u"apt-packageset"
406+ apt_spn = self.factory.getOrMakeSourcePackageName(name="apt")
407+ gcc_packageset_name = u"gcc-packageset"
408+ gcc_spn = self.factory.getOrMakeSourcePackageName(name="gcc")
409+ apt_packageset = self.factory.makePackageset(
410+ name=apt_packageset_name, distroseries=distroseries,
411+ packages=(apt_spn,))
412+ gcc_packageset = self.factory.makePackageset(
413+ name=gcc_packageset_name, distroseries=distroseries,
414+ packages=(gcc_spn,))
415+ apt_packageset.add((gcc_packageset,))
416+ archive_name = self.getTargetArchiveName(distroseries.distribution)
417+ copy_archive = self.copyArchive(
418+ distroseries, archive_name, owner,
419+ packageset_names=[apt_packageset_name])
420+ self.checkCopiedSources(
421+ copy_archive, distroseries, package_infos[1:])
422+
423 def testCopyFromPPA(self):
424 """Test we can create a copy archive with a PPA as the source."""
425 ppa_owner_name = "ppa-owner"
426@@ -712,6 +798,20 @@
427 exception_type=SoyuzScriptError,
428 exception_text="Invalid user name: '%s'" % invalid_user)
429
430+ def testUnknownPackagesetName(self):
431+ """Try copy archive population with an unknown packageset name.
432+
433+ The caller can request copying specific packagesets. We test
434+ what happens if they request a packageset that doesn't exist.
435+ """
436+ unknown_packageset = "unknown"
437+ extra_args = ['-a', '386', "--package-set", unknown_packageset]
438+ self.runScript(
439+ extra_args=extra_args,
440+ exception_type=PackageLocationError,
441+ exception_text="Could not find packageset No such package set"
442+ " (in the specified distro series): '%s'." % unknown_packageset)
443+
444 def testPackagesetDelta(self):
445 """Try to calculate the delta between two source package sets."""
446 hoary = getUtility(IDistributionSet)['ubuntu']['hoary']

Subscribers

People subscribed via source and target branches

to status/vote changes: