Launchpad itself

Merge lp:~jelmer/launchpad/bug471148 into lp:launchpad/db-devel

  1. bug471148
  2. Merge into db-devel
Proposed by Jelmer Vernooij
Status: Superseded
Proposed branch: lp:~jelmer/launchpad/bug471148
Merge into: lp:launchpad/db-devel
Diff against target: 380 lines (+222/-14)
9 files modified
database/sampledata/current-dev.sql (+5/-4)
database/sampledata/current.sql (+5/-4)
database/schema/patch-9999-24-0.sql (+8/-0)
lib/lp/soyuz/interfaces/processor.py (+24/-1)
lib/lp/soyuz/model/processor.py (+12/-0)
lib/lp/soyuz/model/publishing.py (+25/-5)
lib/lp/soyuz/tests/test_processor.py (+30/-0)
lib/lp/soyuz/tests/test_publishing.py (+80/-0)
lib/lp/testing/factory.py (+33/-0)
To merge this branch: bzr merge lp:~jelmer/launchpad/bug471148
Reviewer Review Type Date Requested Status
Stuart Bishop db Pending
Canonical Launchpad Engineering Pending
Review via email: mp+20070@code.launchpad.net

This proposal has been superseded by a proposal from 2010-02-25.

Commit message

Add a restricted flag to architectures and only allow archives with explicit associations with that architecture to build on that architecture.

To post a comment you must log in.
Revision history for this message
Jelmer Vernooij (jelmer) wrote :

This makes it possible for processor families to be marked restricted. Source packages in archives will only be built using these processor families if they are explicitly permitted, using an entry in ArchiveArch.

This patch also adds some infrastructure to create processor families and processors, which we use for testing.

I reran "make newsampledata" which doesn't appear to have happened in a while, so there are some updates there related to earlier branches as well.

Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

It looks as if this diff got polluted with extraneous changes.

Revision history for this message
Jelmer Vernooij (jelmer) wrote :

Hi Jeroen,

On Wed, 2010-02-24 at 18:18 +0000, Jeroen T. Vermeulen wrote:
> It looks as if this diff got polluted with extraneous changes.
 The sampledata doesn't appear to have been regenerated a couple of times
after some other schema changes landed, and that causes the diff to
database/sampledata/current{,-devel}.sql to be so large - the only real
change I've made to the sample data is to add a ARM processor family
that has the new restricted column set to true. Other than that, this is
a fairly simple branch against db-devel.

If you'd rather have me submit a separate branch that just runs "make
newsampledata" in db-devel so that the diff for this branch isn't
cluttered by it, please let me know.

Cheers,

Jelmer

Revision history for this message
Jelmer Vernooij (jelmer) wrote :

Muharem has now submitted a separate branch which just updates the existing sampledata, that should make the diff for this branch a fair bit smaller.

Revision history for this message
Muharem Hrnjadovic (al-maisan) wrote :

Yep, it's lp:~al-maisan/launchpad/sampledata-update and it was just approved.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'database/sampledata/current-dev.sql'
2--- database/sampledata/current-dev.sql 2010-02-25 08:39:19 +0000
3+++ database/sampledata/current-dev.sql 2010-02-25 16:08:19 +0000
4@@ -8143,10 +8143,11 @@
5
6 ALTER TABLE processorfamily DISABLE TRIGGER ALL;
7
8-INSERT INTO processorfamily (id, name, title, description) VALUES (1, 'x86', 'Intel 386 compatible chips', 'Bring back the 8086!');
9-INSERT INTO processorfamily (id, name, title, description) VALUES (2, 'powerpc', 'PowerPC compatible systems, G3 G4 etc', 'An architecture conceived by Motorola and developed further in cooperation with IBM. Was used very successfully by Apple for their PowerMac range, until 2007.');
10-INSERT INTO processorfamily (id, name, title, description) VALUES (3, 'amd64', 'AMD64 and Intel EM64T and compatible systems', 'A 64-bit extension to the venerable x86 architecture, pioneered by AMD and later adopted by Intel as well.');
11-INSERT INTO processorfamily (id, name, title, description) VALUES (4, 'hppa', 'PA-RISC Processors', 'The HP PA-RISC and compatible processors');
12+INSERT INTO processorfamily (id, name, title, description, restricted) VALUES (1, 'x86', 'Intel 386 compatible chips', 'Bring back the 8086!', false);
13+INSERT INTO processorfamily (id, name, title, description, restricted) VALUES (2, 'powerpc', 'PowerPC compatible systems, G3 G4 etc', 'An architecture conceived by Motorola and developed further in cooperation with IBM. Was used very successfully by Apple for their PowerMac range, until 2007.', false);
14+INSERT INTO processorfamily (id, name, title, description, restricted) VALUES (3, 'amd64', 'AMD64 and Intel EM64T and compatible systems', 'A 64-bit extension to the venerable x86 architecture, pioneered by AMD and later adopted by Intel as well.', false);
15+INSERT INTO processorfamily (id, name, title, description, restricted) VALUES (4, 'hppa', 'PA-RISC Processors', 'The HP PA-RISC and compatible processors', false);
16+INSERT INTO processorfamily (id, name, title, description, restricted) VALUES (5, 'arm', 'ARM Processors', 'The ARM and compatible processors', true);
17
18
19 ALTER TABLE processorfamily ENABLE TRIGGER ALL;
20
21=== modified file 'database/sampledata/current.sql'
22--- database/sampledata/current.sql 2010-02-25 08:39:19 +0000
23+++ database/sampledata/current.sql 2010-02-25 16:08:19 +0000
24@@ -8039,10 +8039,11 @@
25
26 ALTER TABLE processorfamily DISABLE TRIGGER ALL;
27
28-INSERT INTO processorfamily (id, name, title, description) VALUES (1, 'x86', 'Intel 386 compatible chips', 'Bring back the 8086!');
29-INSERT INTO processorfamily (id, name, title, description) VALUES (2, 'powerpc', 'PowerPC compatible systems, G3 G4 etc', 'An architecture conceived by Motorola and developed further in cooperation with IBM. Was used very successfully by Apple for their PowerMac range, until 2007.');
30-INSERT INTO processorfamily (id, name, title, description) VALUES (3, 'amd64', 'AMD64 and Intel EM64T and compatible systems', 'A 64-bit extension to the venerable x86 architecture, pioneered by AMD and later adopted by Intel as well.');
31-INSERT INTO processorfamily (id, name, title, description) VALUES (4, 'hppa', 'PA-RISC Processors', 'The HP PA-RISC and compatible processors');
32+INSERT INTO processorfamily (id, name, title, description, restricted) VALUES (1, 'x86', 'Intel 386 compatible chips', 'Bring back the 8086!', false);
33+INSERT INTO processorfamily (id, name, title, description, restricted) VALUES (2, 'powerpc', 'PowerPC compatible systems, G3 G4 etc', 'An architecture conceived by Motorola and developed further in cooperation with IBM. Was used very successfully by Apple for their PowerMac range, until 2007.', false);
34+INSERT INTO processorfamily (id, name, title, description, restricted) VALUES (3, 'amd64', 'AMD64 and Intel EM64T and compatible systems', 'A 64-bit extension to the venerable x86 architecture, pioneered by AMD and later adopted by Intel as well.', false);
35+INSERT INTO processorfamily (id, name, title, description, restricted) VALUES (4, 'hppa', 'PA-RISC Processors', 'The HP PA-RISC and compatible processors', false);
36+INSERT INTO processorfamily (id, name, title, description, restricted) VALUES (5, 'arm', 'ARM Processors', 'The ARM and compatible processors', true);
37
38
39 ALTER TABLE processorfamily ENABLE TRIGGER ALL;
40
41=== added file 'database/schema/patch-9999-24-0.sql'
42--- database/schema/patch-9999-24-0.sql 1970-01-01 00:00:00 +0000
43+++ database/schema/patch-9999-24-0.sql 2010-02-25 16:08:19 +0000
44@@ -0,0 +1,8 @@
45+-- Copyright 2010 Canonical Ltd. This software is licensed under the
46+-- GNU Affero General Public License version 3 (see the file LICENSE).
47+
48+SET client_min_messages=ERROR;
49+
50+ALTER TABLE processorfamily ADD COLUMN restricted boolean DEFAULT FALSE NOT NULL;
51+
52+INSERT INTO LaunchpadDatabaseRevision VALUES (9999, 24, 0);
53
54=== modified file 'lib/lp/soyuz/interfaces/processor.py'
55--- lib/lp/soyuz/interfaces/processor.py 2009-06-25 04:06:00 +0000
56+++ lib/lp/soyuz/interfaces/processor.py 2010-02-25 16:08:19 +0000
57@@ -13,7 +13,10 @@
58 'IProcessorFamilySet',
59 ]
60
61+from canonical.launchpad import _
62+
63 from zope.interface import Interface, Attribute
64+from zope.schema import Bool
65
66 class IProcessor(Interface):
67 """The SQLObject Processor Interface"""
68@@ -29,11 +32,21 @@
69 name = Attribute("The Processor Family Name")
70 title = Attribute("The Processor Family Title")
71 description = Attribute("The Processor Name Description")
72-
73 processors = Attribute("The Processors in this family.")
74+ restricted = Bool(title=_("Whether this family is restricted."))
75+
76+ def addProcessor(name, title, description):
77+ """Add a new processor to this family.
78+
79+ :param name: Name of the processor
80+ :param title: Title of the processor
81+ :param description: Description of the processor
82+ :return: A `IProcessor`
83+ """
84
85 class IProcessorFamilySet(Interface):
86 """Operations related to ProcessorFamily instances."""
87+
88 def getByName(name):
89 """Return the ProcessorFamily instance with the matching name.
90
91@@ -49,3 +62,13 @@
92
93 :return: A `IProcessorFamily` instance if found, None otherwise.
94 """
95+
96+ def new(name, title, description, restricted):
97+ """Create a new processor family.
98+
99+ :param name: Name of the family.
100+ :param title: Title for the family.
101+ :param description: Extended description of the family
102+ :param restricted: Whether the processor family is restricted
103+ :return: a `IProcessorFamily`.
104+ """
105
106=== modified file 'lib/lp/soyuz/model/processor.py'
107--- lib/lp/soyuz/model/processor.py 2009-06-25 04:06:00 +0000
108+++ lib/lp/soyuz/model/processor.py 2010-02-25 16:08:19 +0000
109@@ -10,6 +10,7 @@
110 from zope.interface import implements
111
112 from sqlobject import StringCol, ForeignKey, SQLMultipleJoin
113+from storm.locals import Bool
114
115 from canonical.database.sqlbase import SQLBase
116
117@@ -39,6 +40,12 @@
118 description = StringCol(dbName='description', notNull=True)
119
120 processors = SQLMultipleJoin('Processor', joinColumn='family')
121+ restricted = Bool(allow_none=False, default=False)
122+
123+ def addProcessor(self, name, title, description):
124+ """See `IProcessorFamily`."""
125+ return Processor(family=self, name=name, title=title,
126+ description=description)
127
128
129 class ProcessorFamilySet:
130@@ -62,3 +69,8 @@
131 # but there is also the possibility that the user specified a name for
132 # a non-existent processor.
133 return rset.one()
134+
135+ def new(self, name, title, description, restricted=False):
136+ """See `IProcessorFamily`."""
137+ return ProcessorFamily(name=name, title=title,
138+ description=description, restricted=restricted)
139
140=== modified file 'lib/lp/soyuz/model/publishing.py'
141--- lib/lp/soyuz/model/publishing.py 2010-02-15 12:59:55 +0000
142+++ lib/lp/soyuz/model/publishing.py 2010-02-25 16:08:19 +0000
143@@ -45,14 +45,15 @@
144 LibraryFileAlias, LibraryFileContent)
145 from lp.soyuz.model.packagediff import PackageDiff
146 from lp.soyuz.interfaces.archive import ArchivePurpose
147+from lp.soyuz.interfaces.archivearch import IArchiveArchSet
148 from lp.soyuz.interfaces.component import IComponentSet
149 from lp.soyuz.interfaces.queue import PackageUploadStatus
150 from lp.soyuz.interfaces.publishing import (
151- active_publishing_status, IArchiveSafePublisher,
152- IBinaryPackageFilePublishing, IBinaryPackagePublishingHistory,
153- IPublishingEdit, IPublishingSet, ISourcePackageFilePublishing,
154- ISourcePackagePublishingHistory, PackagePublishingPriority,
155- PackagePublishingStatus, PoolFileOverwriteError)
156+ active_publishing_status, IBinaryPackageFilePublishing,
157+ IBinaryPackagePublishingHistory, IPublishingSet,
158+ ISourcePackageFilePublishing, ISourcePackagePublishingHistory,
159+ PackagePublishingPriority, PackagePublishingStatus,
160+ PoolFileOverwriteError)
161 from lp.soyuz.interfaces.build import BuildSetStatus, BuildStatus, IBuildSet
162 from lp.soyuz.scripts.changeoverride import ArchiveOverriderError
163 from canonical.launchpad.components.decoratedresultset import (
164@@ -505,6 +506,22 @@
165 the_url = self._proxied_urls((changes_lfa,), self.archive)[0]
166 return the_url
167
168+ def _getAllowedArchitectures(self, available_archs):
169+ """Filter out any restricted architectures not specifically allowed
170+ for an archive.
171+
172+ :param available_archs: Architectures to consider
173+ :return: Sequence of `IDistroArch` instances.
174+ """
175+ associated_proc_families = [
176+ archivearch.processorfamily for archivearch
177+ in getUtility(IArchiveArchSet).getByArchive(self.archive)]
178+ # Return all distroarches with unrestricted processor families or with
179+ # processor families the archive is explicitly associated with.
180+ return [distroarch for distroarch in available_archs
181+ if not distroarch.processorfamily.restricted or
182+ distroarch.processorfamily in associated_proc_families]
183+
184 def createMissingBuilds(self, architectures_available=None,
185 pas_verify=None, logger=None):
186 """See `ISourcePackagePublishingHistory`."""
187@@ -515,6 +532,9 @@
188 architectures_available = list(
189 self.distroseries.enabled_architectures)
190
191+ architectures_available = self._getAllowedArchitectures(
192+ architectures_available)
193+
194 build_architectures = determineArchitecturesToBuild(
195 self, architectures_available, self.distroseries, pas_verify)
196
197
198=== added file 'lib/lp/soyuz/tests/test_processor.py'
199--- lib/lp/soyuz/tests/test_processor.py 1970-01-01 00:00:00 +0000
200+++ lib/lp/soyuz/tests/test_processor.py 2010-02-25 16:08:19 +0000
201@@ -0,0 +1,30 @@
202+# Copyright 2010 Canonical Ltd. This software is licensed under the
203+# GNU Affero General Public License version 3 (see the file LICENSE).
204+
205+"""Test Processor and ProcessorFamily features."""
206+
207+from zope.component import getUtility
208+
209+from canonical.testing import LaunchpadZopelessLayer
210+
211+from lp.soyuz.interfaces.processor import (IProcessor, IProcessorFamily,
212+ IProcessorFamilySet)
213+from lp.testing import TestCaseWithFactory
214+
215+
216+class ProcessorFamilyTests(TestCaseWithFactory):
217+ """Test ProcessorFamily."""
218+
219+ layer = LaunchpadZopelessLayer
220+
221+ def test_create(self):
222+ family = getUtility(IProcessorFamilySet).new("avr", "Atmel AVR",
223+ "The Modified Harvard architecture 8-bit RISC processors.")
224+ self.assertProvides(family, IProcessorFamily)
225+
226+ def test_add_processor(self):
227+ family = getUtility(IProcessorFamilySet).new("avr", "Atmel AVR",
228+ "The Modified Harvard architecture 8-bit RISC processors.")
229+ proc = family.addProcessor("avr2001", "The 2001 AVR", "Fast as light.")
230+ self.assertProvides(proc, IProcessor)
231+ self.assertEquals(family, proc.family)
232
233=== modified file 'lib/lp/soyuz/tests/test_publishing.py'
234--- lib/lp/soyuz/tests/test_publishing.py 2010-02-24 10:18:16 +0000
235+++ lib/lp/soyuz/tests/test_publishing.py 2010-02-25 16:08:19 +0000
236@@ -31,6 +31,7 @@
237 from lp.registry.interfaces.sourcepackage import SourcePackageUrgency
238 from lp.registry.interfaces.sourcepackagename import ISourcePackageNameSet
239 from lp.soyuz.interfaces.archive import ArchivePurpose
240+from lp.soyuz.interfaces.archivearch import IArchiveArchSet
241 from lp.soyuz.interfaces.binarypackagename import IBinaryPackageNameSet
242 from lp.soyuz.interfaces.binarypackagerelease import BinaryPackageFormat
243 from lp.soyuz.interfaces.build import BuildStatus
244@@ -861,6 +862,85 @@
245 binary, binary.distroarchseries.distroseries, 'universe')
246
247
248+class BuildRecordCreationTests(TestNativePublishingBase):
249+ """Test the creation of build records."""
250+
251+ def setUp(self):
252+ super(BuildRecordCreationTests, self).setUp()
253+ self.distro = self.factory.makeDistribution()
254+ self.distroseries = self.factory.makeDistroSeries(
255+ distribution=self.distro, name="crazy")
256+ self.archive = self.factory.makeArchive()
257+ self.avr_family = self.factory.makeProcessorFamily(
258+ name="avr", restricted=True)
259+ self.factory.makeProcessor(self.avr_family, "avr2001")
260+ self.avr_distroarch = self.factory.makeDistroArchSeries(
261+ architecturetag='avr', processorfamily=self.avr_family,
262+ distroseries=self.distroseries, supports_virtualized=True)
263+ self.sparc_family = self.factory.makeProcessorFamily(name="sparc",
264+ restricted=False)
265+ self.factory.makeProcessor(self.sparc_family, "sparc64")
266+ self.sparc_distroarch = self.factory.makeDistroArchSeries(
267+ architecturetag='sparc', processorfamily=self.sparc_family,
268+ distroseries=self.distroseries, supports_virtualized=True)
269+ self.distroseries.nominatedarchindep = self.sparc_distroarch
270+ self.addFakeChroots(self.distroseries)
271+
272+ def getPubSource(self, architecturehintlist):
273+ return super(BuildRecordCreationTests, self).getPubSource(
274+ archive=self.archive, distroseries=self.distroseries,
275+ architecturehintlist=architecturehintlist)
276+
277+ def test__getAllowedArchitectures_restricted(self):
278+ """Test _getAllowedArchitectures doesn't return unrestricted
279+ archs.
280+
281+ For a normal archive, only unrestricted architectures should
282+ be used.
283+ """
284+ available_archs = [self.sparc_distroarch, self.avr_distroarch]
285+ pubrec = self.getPubSource(architecturehintlist='any')
286+ self.assertEquals([self.sparc_distroarch],
287+ pubrec._getAllowedArchitectures(available_archs))
288+
289+ def test__getAllowedArchitectures_restricted_override(self):
290+ """Test _getAllowedArchitectures honors overrides of restricted archs.
291+
292+ Restricted architectures should only be allowed if there is
293+ an explicit ArchiveArch association with the archive.
294+ """
295+ available_archs = [self.sparc_distroarch, self.avr_distroarch]
296+ getUtility(IArchiveArchSet).new(self.archive, self.avr_family)
297+ pubrec = self.getPubSource(architecturehintlist='any')
298+ self.assertEquals([self.sparc_distroarch, self.avr_distroarch],
299+ pubrec._getAllowedArchitectures(available_archs))
300+
301+ def test_createMissingBuilds_restricts_any(self):
302+ pubrec = self.getPubSource(architecturehintlist='any')
303+ builds = pubrec.createMissingBuilds()
304+ self.assertEquals(1, len(builds))
305+ self.assertEquals(self.sparc_distroarch, builds[0].distroarchseries)
306+
307+ def test_createMissingBuilds_restricts_explicitlist(self):
308+ pubrec = self.getPubSource(architecturehintlist='sparc i386 avr')
309+ builds = pubrec.createMissingBuilds()
310+ self.assertEquals(1, len(builds))
311+ self.assertEquals(self.sparc_distroarch, builds[0].distroarchseries)
312+
313+ def test_createMissingBuilds_restricts_all(self):
314+ pubrec = self.getPubSource(architecturehintlist='all')
315+ builds = pubrec.createMissingBuilds()
316+ self.assertEquals(1, len(builds))
317+ self.assertEquals(self.sparc_distroarch, builds[0].distroarchseries)
318+
319+ def test_createMissingBuilds_restrict_override(self):
320+ getUtility(IArchiveArchSet).new(self.archive, self.avr_family)
321+ pubrec = self.getPubSource(architecturehintlist='any')
322+ builds = pubrec.createMissingBuilds()
323+ self.assertEquals(2, len(builds))
324+ self.assertEquals(self.avr_distroarch, builds[0].distroarchseries)
325+ self.assertEquals(self.sparc_distroarch, builds[1].distroarchseries)
326+
327
328 def test_suite():
329 return unittest.TestLoader().loadTestsFromName(__name__)
330
331=== modified file 'lib/lp/testing/factory.py'
332--- lib/lp/testing/factory.py 2010-02-25 07:42:32 +0000
333+++ lib/lp/testing/factory.py 2010-02-25 16:08:19 +0000
334@@ -126,6 +126,7 @@
335 from lp.soyuz.adapters.packagelocation import PackageLocation
336 from lp.soyuz.interfaces.component import IComponentSet
337 from lp.soyuz.interfaces.packageset import IPackagesetSet
338+from lp.soyuz.interfaces.processor import IProcessorFamilySet
339 from lp.soyuz.interfaces.publishing import PackagePublishingStatus
340 from lp.soyuz.interfaces.section import ISectionSet
341 from lp.soyuz.model.buildqueue import BuildQueue
342@@ -613,6 +614,38 @@
343 productseries=productseries,
344 name=name)
345
346+ def makeProcessor(self, family, name, title=None, description=None):
347+ """Create a new processor.
348+
349+ :param family: Family of the processor
350+ :param name: Name of the processor
351+ :param title: Optional title
352+ :param description: Optional description
353+ :return: A `IProcessor`
354+ """
355+ if title is None:
356+ title = "The %s processor" % name
357+ if description is None:
358+ description = "The %s and processor and compatible processors"
359+ return family.addProcessor(name, title, description)
360+
361+ def makeProcessorFamily(self, name, title=None, description=None,
362+ restricted=False):
363+ """Create a new processor family.
364+
365+ :param name: Name of the family (e.g. x86)
366+ :param title: Optional title of the family
367+ :param description: Optional extended description
368+ :param restricted: Whether the processor family is restricted
369+ :return: A `IProcessorFamily`
370+ """
371+ if description is None:
372+ description = "Description of the %s processor family" % name
373+ if title is None:
374+ title = "%s and compatible processors." % name
375+ return getUtility(IProcessorFamilySet).new(name, title, description,
376+ restricted=restricted)
377+
378 def makeProductRelease(self, milestone=None, product=None,
379 productseries=None):
380 if milestone is None:

Subscribers

People subscribed via source and target branches

to status/vote changes: