Launchpad itself

Merge lp:~stevenk/launchpad/db-add-derivedistroseries-api into lp:launchpad/db-devel

  1. db-add-derivedistroseries-api
  2. Merge into db-devel
Proposed by Steve Kowalik
Status: Rejected
Rejected by: Steve Kowalik
Proposed branch: lp:~stevenk/launchpad/db-add-derivedistroseries-api
Merge into: lp:launchpad/db-devel
Diff against target: 805 lines (+449/-26)
12 files modified
lib/lp/registry/interfaces/distroseries.py (+80/-4)
lib/lp/registry/model/distroseries.py (+50/-0)
lib/lp/registry/stories/webservice/xx-derivedistroseries.txt (+68/-0)
lib/lp/registry/tests/test_derivedistroseries.py (+78/-0)
lib/lp/soyuz/configure.zcml (+5/-2)
lib/lp/soyuz/interfaces/distributionjob.py (+1/-1)
lib/lp/soyuz/interfaces/packagecloner.py (+5/-2)
lib/lp/soyuz/model/initialisedistroseriesjob.py (+21/-3)
lib/lp/soyuz/model/packagecloner.py (+46/-8)
lib/lp/soyuz/scripts/initialise_distroseries.py (+17/-3)
lib/lp/soyuz/scripts/tests/test_initialise_distroseries.py (+62/-3)
lib/lp/soyuz/tests/test_initialisedistroseriesjob.py (+16/-0)
To merge this branch: bzr merge lp:~stevenk/launchpad/db-add-derivedistroseries-api
Reviewer Review Type Date Requested Status
Gavin Panella (community) Approve
Review via email: mp+35500@code.launchpad.net

Commit message

Add a new method to IDistroSeries, deriveDistroSeries().

Description of the change

This branch adds a new method to IDistroSeries: deriveDistroSeries().

This method is intended to be called via API scripts for both the distro use-case as well as the Linaro use-case to initialise a new distroseries (which may or may not exist), from an existing distroseries. It will create the new distroseries if it doesn't exist, under the specified distribution, unless it isn't specified, in which case it will use the current distroseries's distribution (.parent in the interface).

After it has verified everything, and created the distroseries if need be, it checks the details submitted via the InitialiseDistroSeries.check() method, and if that doesn't raise an InitislationError exception, an InitialiseDistroSeriesJob is created. The method doesn't return the job id, since there is no way for the user to check the status of the job.

I have written two seperate tests, one that calls the method directly and tests it, and a lighter doctest that checks via the API.

I have discussed this change with both Julian and Michael, and they agreed that it sounded good.

To test: bin/test -vvt test_derivedistroseries -t xx-derivedistroseries.txt

This branch also drive-bys two mis-spellings of distribution, since I noticed them.

To post a comment you must log in.
Revision history for this message
Gavin Panella (allenap) wrote :

Hi Steven,

I don't really understand what a derived distro series is, but it
sounds good, and triggering a job to do it seems neat.

I have quite a few comments, some trivial, but some do need work. I'm
mostly concerned with the way authorization is handled and with test
coverage.

Gavin.

[1]

+ @operation_parameters(
+ name=TextLine(
+ title=_("The name of the distroseries to derive."),
+ required=True),
...

Here, and with the other parameters, consider using copy_field() to
copy the relevant fields from IDistroSeries instead of defining them
again. Some of them provide extra validation, and it ensures that
things remain in sync.

[2]

+ status=TextLine(
+ title=_("Status the derived distroseries to created is."),

s/created/create/

[3]

+ if not user.inTeam('soyuz-team'):
+ raise Unauthorized

Access should be configured in ZCML, see configure.zcml in
lib/lp/registry/.

You could:

- Write tests to ensure that only members of ~soyuz-team can access
  deriveDistroSeries().

- Define a new security adapter that only permits, say, launchpad.Edit
  to ~soyuz-team,

- Define a new interface, IDistroSeriesDerivation, with only
  deriveDistroSeries() in it, and get IDistroSeries to inherit from
  it.

- Add something like the following to the ZCML:

    <require
        permission="launchpad.Edit"
        interface="lp.registry.interfaces.distroseries.IDistroSeriesDerivation"/>

[4]

+ #if self.distribution is ubuntu and not user.in_tech_board
+ #elsif not user a driver

I guess you can drop this.

[5]

+ for param in (displayname, summary, description, version):
+ if not param:

Avoid using implicit truthiness :) Or falsiness. Instead:

    if param is None:
        raise ...

Or, depending on what you need:

    if param is None or len(param) == 0:
        raise ...

[6]

Back to to the parameters:

+ arches=List(
+ title=_("The list of arches to copy to the derived "
+ "distroseries."),
+ required=False),

Is "arches" a standard term? If not, consider expanding this to
"architectures", or whatever it's meant to be, "archbishops" perhaps.

[7]

+ child = getUtility(IDistroSeriesSet).new(

I'm probably missing something, but IDistroSeriesSet does not define a
new() method, and it's not implemented on DistroSeriesSet either. That
makes me suspect that this part of the conditional is not tested.

[8]

+ raise DerivationError(
+ "DistroSeries %s parent series isn't %s" % (
+ child.name, self.name))

Perhaps this exception could explain why this is a requirement too. Or
is it blindingly obvious to someone who expects to use this function.

[9]

+ ids = InitialiseDistroSeries(child)

Can you use a longer and more meaningful variable name?

[10]

+We can't call .deriveDistroSeries() with a distroseries that isn't the

Trailing whitespace.

review: Needs Fixing
Revision history for this message
Gavin Panella (allenap) wrote :

As discussed, [3] is okay to leave as long as bigjools is happy with the approach, though I might talk to someone better versed in Zope to ensure that I haven't missed something.

Revision history for this message
Gavin Panella (allenap) wrote :

Cool, the changes look good.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/lp/registry/interfaces/distroseries.py'
--- lib/lp/registry/interfaces/distroseries.py 2010-09-28 18:11:41 +0000
+++ lib/lp/registry/interfaces/distroseries.py 2010-10-12 08:34:50 +0000
@@ -8,6 +8,7 @@
8__metaclass__ = type8__metaclass__ = type
99
10__all__ = [10__all__ = [
11 'DerivationError',
11 'IDistroSeries',12 'IDistroSeries',
12 'IDistroSeriesEditRestricted',13 'IDistroSeriesEditRestricted',
13 'IDistroSeriesPublic',14 'IDistroSeriesPublic',
@@ -16,20 +17,25 @@
1617
17from lazr.enum import DBEnumeratedType18from lazr.enum import DBEnumeratedType
18from lazr.restful.declarations import (19from lazr.restful.declarations import (
20 call_with,
19 export_as_webservice_entry,21 export_as_webservice_entry,
20 export_factory_operation,22 export_factory_operation,
21 export_read_operation,23 export_read_operation,
24 export_write_operation,
22 exported,25 exported,
23 LAZR_WEBSERVICE_EXPORTED,26 LAZR_WEBSERVICE_EXPORTED,
24 operation_parameters,27 operation_parameters,
25 operation_returns_collection_of,28 operation_returns_collection_of,
26 operation_returns_entry,29 operation_returns_entry,
27 rename_parameters_as,30 rename_parameters_as,
31 REQUEST_USER,
32 webservice_error,
28 )33 )
29from lazr.restful.fields import (34from lazr.restful.fields import (
30 Reference,35 Reference,
31 ReferenceChoice,36 ReferenceChoice,
32 )37 )
38from lazr.restful.interface import copy_field
33from zope.component import getUtility39from zope.component import getUtility
34from zope.interface import (40from zope.interface import (
35 Attribute,41 Attribute,
@@ -39,6 +45,7 @@
39 Bool,45 Bool,
40 Choice,46 Choice,
41 Datetime,47 Datetime,
48 List,
42 Object,49 Object,
43 TextLine,50 TextLine,
44 )51 )
@@ -673,8 +680,8 @@
673 If sourcename is passed, only packages that are built from680 If sourcename is passed, only packages that are built from
674 source packages by that name will be returned.681 source packages by that name will be returned.
675 If archive is passed, restricted the results to the given archive,682 If archive is passed, restricted the results to the given archive,
676 if it is suppressed the results will be restricted to the distribtion683 if it is suppressed the results will be restricted to the
677 'main_archive'.684 distribution 'main_archive'.
678 """685 """
679686
680 def getSourcePackagePublishing(status, pocket, component=None,687 def getSourcePackagePublishing(status, pocket, component=None,
@@ -683,8 +690,8 @@
683690
684 According status and pocket.691 According status and pocket.
685 If archive is passed, restricted the results to the given archive,692 If archive is passed, restricted the results to the given archive,
686 if it is suppressed the results will be restricted to the distribtion693 if it is suppressed the results will be restricted to the
687 'main_archive'.694 distribution 'main_archive'.
688 """695 """
689696
690 def getBinaryPackageCaches(archive=None):697 def getBinaryPackageCaches(archive=None):
@@ -789,6 +796,69 @@
789 :param format: The SourcePackageFormat to check.796 :param format: The SourcePackageFormat to check.
790 """797 """
791798
799 @operation_parameters(
800 name=copy_field(name, required=True),
801 displayname=copy_field(displayname, required=False),
802 title=copy_field(title, required=False),
803 summary=TextLine(
804 title=_("The summary of the distroseries to derive."),
805 required=False),
806 description=copy_field(description, required=False),
807 version=copy_field(version, required=False),
808 distribution=copy_field(distribution, required=False),
809 status=copy_field(status, required=False),
810 architectures=List(
811 title=_("The list of architectures to copy to the derived "
812 "distroseries."),
813 required=False),
814 packagesets=List(
815 title=_("The list of packagesets to copy to the derived "
816 "distroseries"),
817 required=False),
818 rebuild=Bool(
819 title=_("If binaries will be copied to the derived "
820 "distroseries."),
821 required=True),
822 )
823 @call_with(user=REQUEST_USER)
824 @export_write_operation()
825 def deriveDistroSeries(
826 user, name, displayname, title, summary, description, version,
827 distribution, status, architectures, packagesets, rebuild):
828 """Derive a distroseries from this one.
829
830 This method performs checks, can create the new distroseries if
831 necessary, and then creates a job to populate the new
832 distroseries.
833
834 :param name: The name of the new distroseries we will create if it
835 doesn't exist, or the name of the distroseries we will look
836 up, and then initialise.
837 :param displayname: The Display Name for the new distroseries.
838 If the distroseries already exists this parameter is ignored.
839 :param title: The Title for the new distroseries. If the
840 distroseries already exists this parameter is ignored.
841 :param summary: The Summary for the new distroseries. If the
842 distroseries already exists this parameter is ignored.
843 :param description: The Description for the new distroseries. If the
844 distroseries already exists this parameter is ignored.
845 :param version: The version for the new distroseries. If the
846 distroseries already exists this parameter is ignored.
847 :param distribution: The distribution the derived series will
848 belong to. If it isn't specified this distroseries'
849 distribution is used.
850 :param status: The status the new distroseries will be created
851 in. If the distroseries isn't specified, this parameter will
852 be ignored. Defaults to FROZEN.
853 :param architectures: The architectures to copy to the derived
854 series. If not specified, all of the architectures are copied.
855 :param packagesets: The packagesets to copy to the derived series.
856 If not specified, all of the packagesets are copied.
857 :param rebuild: Whether binaries will be copied to the derived
858 series. If it's true, they will not be, and if it's false, they
859 will be.
860 """
861
792862
793class IDistroSeries(IDistroSeriesEditRestricted, IDistroSeriesPublic,863class IDistroSeries(IDistroSeriesEditRestricted, IDistroSeriesPublic,
794 IStructuralSubscriptionTarget):864 IStructuralSubscriptionTarget):
@@ -856,5 +926,11 @@
856 """926 """
857927
858928
929class DerivationError(Exception):
930 """Raised when there is a problem deriving a distroseries."""
931 webservice_error(400) # Bad Request
932 _message_prefix = "Error deriving distro series"
933
934
859# Monkey patch for circular import avoidance done in935# Monkey patch for circular import avoidance done in
860# _schema_circular_imports.py936# _schema_circular_imports.py
861937
=== modified file 'lib/lp/registry/model/distroseries.py'
--- lib/lp/registry/model/distroseries.py 2010-10-03 15:30:06 +0000
+++ lib/lp/registry/model/distroseries.py 2010-10-12 08:34:50 +0000
@@ -35,6 +35,7 @@
35 )35 )
36from zope.component import getUtility36from zope.component import getUtility
37from zope.interface import implements37from zope.interface import implements
38from zope.security.interfaces import Unauthorized
3839
39from canonical.database.constants import (40from canonical.database.constants import (
40 DEFAULT,41 DEFAULT,
@@ -88,6 +89,7 @@
88 )89 )
89from lp.bugs.model.bugtask import BugTask90from lp.bugs.model.bugtask import BugTask
90from lp.registry.interfaces.distroseries import (91from lp.registry.interfaces.distroseries import (
92 DerivationError,
91 IDistroSeries,93 IDistroSeries,
92 IDistroSeriesSet,94 IDistroSeriesSet,
93 )95 )
@@ -128,6 +130,9 @@
128from lp.soyuz.interfaces.binarypackagebuild import IBinaryPackageBuildSet130from lp.soyuz.interfaces.binarypackagebuild import IBinaryPackageBuildSet
129from lp.soyuz.interfaces.binarypackagename import IBinaryPackageName131from lp.soyuz.interfaces.binarypackagename import IBinaryPackageName
130from lp.soyuz.interfaces.buildrecords import IHasBuildRecords132from lp.soyuz.interfaces.buildrecords import IHasBuildRecords
133from lp.soyuz.interfaces.distributionjob import (
134 IInitialiseDistroSeriesJobSource,
135 )
131from lp.soyuz.interfaces.publishing import (136from lp.soyuz.interfaces.publishing import (
132 active_publishing_status,137 active_publishing_status,
133 ICanPublishPackages,138 ICanPublishPackages,
@@ -162,6 +167,10 @@
162 )167 )
163from lp.soyuz.model.section import Section168from lp.soyuz.model.section import Section
164from lp.soyuz.model.sourcepackagerelease import SourcePackageRelease169from lp.soyuz.model.sourcepackagerelease import SourcePackageRelease
170from lp.soyuz.scripts.initialise_distroseries import (
171 InitialisationError,
172 InitialiseDistroSeries,
173 )
165from lp.translations.interfaces.languagepack import LanguagePackType174from lp.translations.interfaces.languagepack import LanguagePackType
166from lp.translations.model.distroseries_translations_copy import (175from lp.translations.model.distroseries_translations_copy import (
167 copy_active_translations,176 copy_active_translations,
@@ -1847,6 +1856,47 @@
1847 ISourcePackageFormatSelectionSet).getBySeriesAndFormat(1856 ISourcePackageFormatSelectionSet).getBySeriesAndFormat(
1848 self, format) is not None1857 self, format) is not None
18491858
1859 def deriveDistroSeries(
1860 self, user, name, distribution=None, displayname=None,
1861 title=None, summary=None, description=None, version=None,
1862 status=SeriesStatus.FROZEN, architectures=(), packagesets=(),
1863 rebuild=False):
1864 """See `IDistroSeries`."""
1865 # XXX StevenK bug=643369 This should be in the security adapter
1866 # This should be allowed if the user is a driver for self.parent
1867 # or the child.parent's drivers.
1868 if not (user.inTeam('soyuz-team') or user.inTeam('admins')):
1869 raise Unauthorized
1870 child = IStore(self).find(DistroSeries, name=name).one()
1871 if child is None:
1872 if distribution is None:
1873 distribution = self.distribution
1874 for param in (
1875 displayname, title, summary, description, version):
1876 if param is None or len(param) == 0:
1877 raise DerivationError(
1878 "Display Name, Title, Summary, Description and"
1879 " Version all need to be set when creating a"
1880 " distroseries")
1881 child = distribution.newSeries(
1882 name=name, displayname=displayname, title=title,
1883 summary=summary, description=description,
1884 version=version, parent_series=self, owner=user)
1885 child.status = status
1886 IStore(self).add(child)
1887 else:
1888 if child.parent_series is not self:
1889 raise DerivationError(
1890 "DistroSeries %s parent series isn't %s" % (
1891 child.name, self.name))
1892 initialise_series = InitialiseDistroSeries(child)
1893 try:
1894 initialise_series.check()
1895 except InitialisationError, e:
1896 raise DerivationError(e)
1897 getUtility(IInitialiseDistroSeriesJobSource).create(
1898 child, architectures, packagesets, rebuild)
1899
18501900
1851class DistroSeriesSet:1901class DistroSeriesSet:
1852 implements(IDistroSeriesSet)1902 implements(IDistroSeriesSet)
18531903
=== added file 'lib/lp/registry/stories/webservice/xx-derivedistroseries.txt'
--- lib/lp/registry/stories/webservice/xx-derivedistroseries.txt 1970-01-01 00:00:00 +0000
+++ lib/lp/registry/stories/webservice/xx-derivedistroseries.txt 2010-10-12 08:34:50 +0000
@@ -0,0 +1,68 @@
1Derive Distributions
2--------------------
3
4Using the DistroSeries.deriveDistroSeries() function, we can call it with the
5parent distroseries. We can call it with the distroseries already created,
6or it can create it for the user.
7
8Set Up
9------
10
11 >>> login('admin@canonical.com')
12 >>> soyuz = factory.makeTeam(name='soyuz-team')
13 >>> parent = factory.makeDistroSeries()
14 >>> child = factory.makeDistroSeries(parent_series=parent)
15 >>> other = factory.makeDistroSeries()
16 >>> logout()
17 >>> from canonical.launchpad.testing.pages import webservice_for_person
18 >>> from canonical.launchpad.webapp.interfaces import OAuthPermission
19 >>> soyuz_webservice = webservice_for_person(
20 ... soyuz.teamowner, permission=OAuthPermission.WRITE_PUBLIC)
21
22Calling
23-------
24
25We can't call .deriveDistroSeries() with a distroseries that isn't the
26child's parent
27
28 >>> series_url = '/%s/%s' % (parent.parent.name, parent.name)
29 >>> other_series_url = '/%s/%s' % (
30 ... other.parent.name, other.name)
31 >>> child_name = child.name
32 >>> series = webservice.get(series_url).jsonBody()
33 >>> other_series = webservice.get(other_series_url).jsonBody()
34 >>> derived = soyuz_webservice.named_post(
35 ... other_series['self_link'], 'deriveDistroSeries', {},
36 ... name=child_name, rebuild=False)
37 >>> print derived
38 HTTP/1.1 400 Bad Request
39 Status: 400 Bad Request
40 ...
41 <BLANKLINE>
42 DistroSeries ... parent series isn't ...
43 <BLANKLINE>
44 ...
45
46If we call it correctly, it works.
47
48 >>> derived = soyuz_webservice.named_post(
49 ... series['self_link'], 'deriveDistroSeries', {},
50 ... name=child_name, rebuild=False)
51 >>> print derived
52 HTTP/1.1 200 Ok
53 Status: 200 Ok
54 ...
55 <BLANKLINE>
56 ...
57
58And we can verify the job exists.
59
60 >>> from zope.component import getUtility
61 >>> from lp.soyuz.interfaces.distributionjob import (
62 ... IInitialiseDistroSeriesJobSource)
63 >>> login('admin@canonical.com')
64 >>> [job] = list(
65 ... getUtility(IInitialiseDistroSeriesJobSource).iterReady())
66 >>> job.distroseries == child
67 True
68
069
=== added file 'lib/lp/registry/tests/test_derivedistroseries.py'
--- lib/lp/registry/tests/test_derivedistroseries.py 1970-01-01 00:00:00 +0000
+++ lib/lp/registry/tests/test_derivedistroseries.py 2010-10-12 08:34:50 +0000
@@ -0,0 +1,78 @@
1# Copyright 2010 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""Test initialising a distroseries using
5IDistroSeries.deriveDistroSeries."""
6
7__metaclass__ = type
8
9from canonical.testing.layers import LaunchpadFunctionalLayer
10from lp.registry.interfaces.distroseries import DerivationError
11from lp.soyuz.interfaces.distributionjob import (
12 IInitialiseDistroSeriesJobSource,
13 )
14from lp.testing import (
15 login,
16 logout,
17 TestCaseWithFactory,
18 )
19from zope.component import getUtility
20from zope.security.interfaces import Unauthorized
21
22
23class TestDeriveDistroSeries(TestCaseWithFactory):
24
25 layer = LaunchpadFunctionalLayer
26
27 def setUp(self):
28 super(TestDeriveDistroSeries, self).setUp()
29 self.soyuz = self.factory.makeTeam(name='soyuz-team')
30 self.parent = self.factory.makeDistroSeries()
31 self.child = self.factory.makeDistroSeries(
32 parent_series=self.parent)
33
34 def test_no_permission_to_call(self):
35 login('admin@canonical.com')
36 person = self.factory.makePerson()
37 logout()
38 self.assertRaises(
39 Unauthorized, self.parent.deriveDistroSeries, person,
40 self.child.name)
41
42 def test_no_distroseries_and_no_arguments(self):
43 """Test that calling deriveDistroSeries() when the distroseries
44 doesn't exist, and not enough arguments are specified that the
45 function errors."""
46 self.assertRaisesWithContent(
47 DerivationError,
48 'Display Name, Title, Summary, Description and Version all '
49 'need to be set when creating a distroseries',
50 self.parent.deriveDistroSeries, self.soyuz.teamowner,
51 'foobuntu')
52
53 def test_parent_is_not_self(self):
54 login('admin@canonical.com')
55 other = self.factory.makeDistroSeries()
56 logout()
57 self.assertRaisesWithContent(
58 DerivationError,
59 "DistroSeries %s parent series isn't %s" % (
60 self.child.name, other.name),
61 other.deriveDistroSeries, self.soyuz.teamowner,
62 self.child.name)
63
64 def test_create_new_distroseries(self):
65 self.parent.deriveDistroSeries(
66 self.soyuz.teamowner, self.child.name)
67 [job] = list(
68 getUtility(IInitialiseDistroSeriesJobSource).iterReady())
69 self.assertEqual(job.distroseries, self.child)
70
71 def test_create_fully_new_distroseries(self):
72 self.parent.deriveDistroSeries(
73 self.soyuz.teamowner, 'foobuntu', displayname='Foobuntu',
74 title='The Foobuntu', summary='Foobuntu',
75 description='Foobuntu is great', version='11.11')
76 [job] = list(
77 getUtility(IInitialiseDistroSeriesJobSource).iterReady())
78 self.assertEqual(job.distroseries.name, 'foobuntu')
079
=== modified file 'lib/lp/soyuz/configure.zcml'
--- lib/lp/soyuz/configure.zcml 2010-10-06 18:53:53 +0000
+++ lib/lp/soyuz/configure.zcml 2010-10-12 08:34:50 +0000
@@ -905,9 +905,12 @@
905 provides="lp.soyuz.interfaces.distributionjob.IInitialiseDistroSeriesJobSource">905 provides="lp.soyuz.interfaces.distributionjob.IInitialiseDistroSeriesJobSource">
906 <allow interface="lp.soyuz.interfaces.distributionjob.IInitialiseDistroSeriesJobSource"/>906 <allow interface="lp.soyuz.interfaces.distributionjob.IInitialiseDistroSeriesJobSource"/>
907 </securedutility>907 </securedutility>
908908 <class class="lp.soyuz.model.distributionjob.DistributionJob">
909 <allow interface="lp.soyuz.interfaces.distributionjob.IDistributionJob" />
910 </class>
909 <class class="lp.soyuz.model.initialisedistroseriesjob.InitialiseDistroSeriesJob">911 <class class="lp.soyuz.model.initialisedistroseriesjob.InitialiseDistroSeriesJob">
910 <allow interface="lp.services.job.interfaces.job.IRunnableJob" />912 <allow interface="lp.soyuz.interfaces.distributionjob.IInitialiseDistroSeriesJob" />
913 <allow interface="lp.soyuz.interfaces.distributionjob.IDistributionJob" />
911 </class>914 </class>
912915
913</configure>916</configure>
914917
=== modified file 'lib/lp/soyuz/interfaces/distributionjob.py'
--- lib/lp/soyuz/interfaces/distributionjob.py 2010-09-16 01:38:42 +0000
+++ lib/lp/soyuz/interfaces/distributionjob.py 2010-10-12 08:34:50 +0000
@@ -58,7 +58,7 @@
58class IInitialiseDistroSeriesJobSource(IJobSource):58class IInitialiseDistroSeriesJobSource(IJobSource):
59 """An interface for acquiring IDistributionJobs."""59 """An interface for acquiring IDistributionJobs."""
6060
61 def create(distroseries):61 def create(distroseries, arches, packagesets, rebuild):
62 """Create a new initialisation job for a distroseries."""62 """Create a new initialisation job for a distroseries."""
6363
6464
6565
=== modified file 'lib/lp/soyuz/interfaces/packagecloner.py'
--- lib/lp/soyuz/interfaces/packagecloner.py 2010-10-01 17:48:37 +0000
+++ lib/lp/soyuz/interfaces/packagecloner.py 2010-10-12 08:34:50 +0000
@@ -19,7 +19,8 @@
1919
20 def clonePackages(20 def clonePackages(
21 origin, destination, distroarchseries_list=None,21 origin, destination, distroarchseries_list=None,
22 proc_familes=None, always_create=False):22 proc_families=None, sourcepackagenames=None,
23 always_create=False):
23 """Copies the source packages from origin to destination as24 """Copies the source packages from origin to destination as
24 well as the binary packages for the DistroArchSeries specified.25 well as the binary packages for the DistroArchSeries specified.
2526
@@ -27,8 +28,10 @@
27 :param destination: the location to which the data is to be copied.28 :param destination: the location to which the data is to be copied.
28 :param distroarchseries_list: the binary packages will be copied29 :param distroarchseries_list: the binary packages will be copied
29 for the distroarchseries pairs specified (if any).30 for the distroarchseries pairs specified (if any).
30 :param proc_familes: the processor families that builds will be31 :param proc_families: the processor families that builds will be
31 created for.32 created for.
33 :param sourcepackagenames: the source packages which are to be
34 copied.
32 :param always_create: if builds should always be created.35 :param always_create: if builds should always be created.
33 """36 """
3437
3538
=== modified file 'lib/lp/soyuz/model/initialisedistroseriesjob.py'
--- lib/lp/soyuz/model/initialisedistroseriesjob.py 2010-09-16 01:38:42 +0000
+++ lib/lp/soyuz/model/initialisedistroseriesjob.py 2010-10-12 08:34:50 +0000
@@ -33,16 +33,34 @@
33 classProvides(IInitialiseDistroSeriesJobSource)33 classProvides(IInitialiseDistroSeriesJobSource)
3434
35 @classmethod35 @classmethod
36 def create(cls, distroseries):36 def create(
37 cls, distroseries, arches=(), packagesets=(), rebuild=False):
37 """See `IInitialiseDistroSeriesJob`."""38 """See `IInitialiseDistroSeriesJob`."""
39 metadata = {
40 'arches': arches, 'packagesets': packagesets,
41 'rebuild': rebuild}
38 job = DistributionJob(42 job = DistributionJob(
39 distroseries.distribution, distroseries, cls.class_job_type,43 distroseries.distribution, distroseries, cls.class_job_type,
40 ())44 metadata)
41 IMasterStore(DistributionJob).add(job)45 IMasterStore(DistributionJob).add(job)
42 return cls(job)46 return cls(job)
4347
48 @property
49 def arches(self):
50 return tuple(self.metadata['arches'])
51
52 @property
53 def packagesets(self):
54 return tuple(self.metadata['packagesets'])
55
56 @property
57 def rebuild(self):
58 return self.metadata['rebuild']
59
44 def run(self):60 def run(self):
45 """See `IRunnableJob`."""61 """See `IRunnableJob`."""
46 ids = InitialiseDistroSeries(self.distroseries)62 ids = InitialiseDistroSeries(
63 self.distroseries, self.arches, self.packagesets,
64 self.rebuild)
47 ids.check()65 ids.check()
48 ids.initialise()66 ids.initialise()
4967
=== modified file 'lib/lp/soyuz/model/packagecloner.py'
--- lib/lp/soyuz/model/packagecloner.py 2010-10-05 05:26:58 +0000
+++ lib/lp/soyuz/model/packagecloner.py 2010-10-12 08:34:50 +0000
@@ -61,7 +61,8 @@
61 implements(IPackageCloner)61 implements(IPackageCloner)
6262
63 def clonePackages(self, origin, destination, distroarchseries_list=None,63 def clonePackages(self, origin, destination, distroarchseries_list=None,
64 proc_families=None, always_create=False):64 proc_families=None, sourcepackagenames=None,
65 always_create=False):
65 """Copies packages from origin to destination package location.66 """Copies packages from origin to destination package location.
6667
67 Binary packages are only copied for the `DistroArchSeries` pairs68 Binary packages are only copied for the `DistroArchSeries` pairs
@@ -77,19 +78,24 @@
77 for the distroarchseries pairs specified (if any).78 for the distroarchseries pairs specified (if any).
78 @param proc_families: the processor families to create builds for.79 @param proc_families: the processor families to create builds for.
79 @type proc_families: Iterable80 @type proc_families: Iterable
81 @param sourcepackagenames: the sourcepackages to copy to the
82 destination
83 @type sourcepackagenames: Iterable
80 @param always_create: if we should create builds for every source84 @param always_create: if we should create builds for every source
81 package copied, useful if no binaries are to be copied.85 package copied, useful if no binaries are to be copied.
82 @type always_create: Boolean86 @type always_create: Boolean
83 """87 """
84 # First clone the source packages.88 # First clone the source packages.
85 self._clone_source_packages(origin, destination)89 self._clone_source_packages(
90 origin, destination, sourcepackagenames)
8691
87 # Are we also supposed to clone binary packages from origin to92 # Are we also supposed to clone binary packages from origin to
88 # destination distroarchseries pairs?93 # destination distroarchseries pairs?
89 if distroarchseries_list is not None:94 if distroarchseries_list is not None:
90 for (origin_das, destination_das) in distroarchseries_list:95 for (origin_das, destination_das) in distroarchseries_list:
91 self._clone_binary_packages(96 self._clone_binary_packages(
92 origin, destination, origin_das, destination_das)97 origin, destination, origin_das, destination_das,
98 sourcepackagenames)
9399
94 if proc_families is None:100 if proc_families is None:
95 proc_families = []101 proc_families = []
@@ -147,8 +153,9 @@
147 # Commit to avoid MemoryError: bug 304459153 # Commit to avoid MemoryError: bug 304459
148 transaction.commit()154 transaction.commit()
149155
150 def _clone_binary_packages(self, origin, destination, origin_das,156 def _clone_binary_packages(
151 destination_das):157 self, origin, destination, origin_das, destination_das,
158 sourcepackagenames=None):
152 """Copy binary publishing data from origin to destination.159 """Copy binary publishing data from origin to destination.
153160
154 @type origin: PackageLocation161 @type origin: PackageLocation
@@ -163,9 +170,12 @@
163 @type destination_das: DistroArchSeries170 @type destination_das: DistroArchSeries
164 @param destination_das: the DistroArchSeries to which to copy171 @param destination_das: the DistroArchSeries to which to copy
165 binary packages172 binary packages
173 @param sourcepackagenames: List of source packages to restrict
174 the copy to
175 @type sourcepackagenames: Iterable
166 """176 """
167 store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)177 store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
168 store.execute('''178 query = '''
169 INSERT INTO BinaryPackagePublishingHistory (179 INSERT INTO BinaryPackagePublishingHistory (
170 binarypackagerelease, distroarchseries, status,180 binarypackagerelease, distroarchseries, status,
171 component, section, priority, archive, datecreated,181 component, section, priority, archive, datecreated,
@@ -183,7 +193,22 @@
183 destination.pocket, origin_das,193 destination.pocket, origin_das,
184 PackagePublishingStatus.PENDING,194 PackagePublishingStatus.PENDING,
185 PackagePublishingStatus.PUBLISHED,195 PackagePublishingStatus.PUBLISHED,
186 origin.pocket, origin.archive))196 origin.pocket, origin.archive)
197
198 if sourcepackagenames and len(sourcepackagenames) > 0:
199 query += '''AND bpph.binarypackagerelease IN (
200 SELECT bpr.id
201 FROM BinaryPackageRelease AS bpr,
202 BinaryPackageBuild as bpb,
203 SourcePackageRelease as spr,
204 SourcePackageName as spn
205 WHERE bpb.id = bpr.build AND
206 bpb.source_package_release = spr.id
207 AND spr.sourcepackagename = spn.id
208 AND spn.name IN %s)''' % sqlvalues(
209 sourcepackagenames)
210
211 store.execute(query)
187212
188 def mergeCopy(self, origin, destination):213 def mergeCopy(self, origin, destination):
189 """Please see `IPackageCloner`."""214 """Please see `IPackageCloner`."""
@@ -378,7 +403,8 @@
378 " AND secsrc.component = %s" % quote(destination.component))403 " AND secsrc.component = %s" % quote(destination.component))
379 store.execute(pop_query)404 store.execute(pop_query)
380405
381 def _clone_source_packages(self, origin, destination):406 def _clone_source_packages(
407 self, origin, destination, sourcepackagenames):
382 """Copy source publishing data from origin to destination.408 """Copy source publishing data from origin to destination.
383409
384 @type origin: PackageLocation410 @type origin: PackageLocation
@@ -387,6 +413,9 @@
387 @type destination: PackageLocation413 @type destination: PackageLocation
388 @param destination: the location to which the data is414 @param destination: the location to which the data is
389 to be copied.415 to be copied.
416 @type sourcepackagenames: Iterable
417 @param sourcepackagenames: List of source packages to restrict
418 the copy to
390 """419 """
391 store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)420 store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
392 query = '''421 query = '''
@@ -407,6 +436,15 @@
407 PackagePublishingStatus.PUBLISHED,436 PackagePublishingStatus.PUBLISHED,
408 origin.pocket, origin.archive)437 origin.pocket, origin.archive)
409438
439 if sourcepackagenames and len(sourcepackagenames) > 0:
440 query += '''AND spph.sourcepackagerelease IN (
441 (SELECT spr.id
442 FROM SourcePackageRelease AS spr,
443 SourcePackageName AS spn
444 WHERE spr.sourcepackagename = spn.id
445 AND spn.name IN %s))''' % sqlvalues(
446 sourcepackagenames)
447
410 if origin.packagesets:448 if origin.packagesets:
411 query += '''AND spph.sourcepackagerelease IN449 query += '''AND spph.sourcepackagerelease IN
412 (SELECT spr.id450 (SELECT spr.id
413451
=== modified file 'lib/lp/soyuz/scripts/initialise_distroseries.py'
--- lib/lp/soyuz/scripts/initialise_distroseries.py 2010-10-06 11:46:51 +0000
+++ lib/lp/soyuz/scripts/initialise_distroseries.py 2010-10-12 08:34:50 +0000
@@ -16,7 +16,6 @@
16from canonical.launchpad.interfaces.lpstorm import IMasterStore16from canonical.launchpad.interfaces.lpstorm import IMasterStore
17from lp.buildmaster.enums import BuildStatus17from lp.buildmaster.enums import BuildStatus
18from lp.registry.interfaces.pocket import PackagePublishingPocket18from lp.registry.interfaces.pocket import PackagePublishingPocket
19from lp.registry.model.distroseries import DistroSeries
20from lp.soyuz.adapters.packagelocation import PackageLocation19from lp.soyuz.adapters.packagelocation import PackageLocation
21from lp.soyuz.enums import (20from lp.soyuz.enums import (
22 ArchivePurpose,21 ArchivePurpose,
@@ -59,10 +58,14 @@
59 in the initialisation of a derivative.58 in the initialisation of a derivative.
60 """59 """
6160
62 def __init__(self, distroseries, arches=(), rebuild=False):61 def __init__(
62 self, distroseries, arches=(), packagesets=(), rebuild=False):
63 # Avoid circular imports
64 from lp.registry.model.distroseries import DistroSeries
63 self.distroseries = distroseries65 self.distroseries = distroseries
64 self.parent = self.distroseries.parent_series66 self.parent = self.distroseries.parent_series
65 self.arches = arches67 self.arches = arches
68 self.packagesets = packagesets
66 self.rebuild = rebuild69 self.rebuild = rebuild
67 self._store = IMasterStore(DistroSeries)70 self._store = IMasterStore(DistroSeries)
6871
@@ -180,6 +183,15 @@
180 """183 """
181 archive_set = getUtility(IArchiveSet)184 archive_set = getUtility(IArchiveSet)
182185
186 spns = []
187 # The overhead from looking up each packageset is mitigated by
188 # this usually running from a job
189 if self.packagesets:
190 for pkgsetname in self.packagesets:
191 pkgset = getUtility(IPackagesetSet).getByName(
192 pkgsetname, distroseries=self.parent)
193 spns += list(pkgset.getSourcesIncluded())
194
183 for archive in self.parent.distribution.all_distro_archives:195 for archive in self.parent.distribution.all_distro_archives:
184 if archive.purpose not in (196 if archive.purpose not in (
185 ArchivePurpose.PRIMARY, ArchivePurpose.DEBUG):197 ArchivePurpose.PRIMARY, ArchivePurpose.DEBUG):
@@ -204,7 +216,7 @@
204 distroarchseries_list = ()216 distroarchseries_list = ()
205 getUtility(IPackageCloner).clonePackages(217 getUtility(IPackageCloner).clonePackages(
206 origin, destination, distroarchseries_list,218 origin, destination, distroarchseries_list,
207 proc_families, self.rebuild)219 proc_families, spns, self.rebuild)
208220
209 def _copy_component_section_and_format_selections(self):221 def _copy_component_section_and_format_selections(self):
210 """Copy the section, component and format selections from the parent222 """Copy the section, component and format selections from the parent
@@ -282,6 +294,8 @@
282 parent_to_child = {}294 parent_to_child = {}
283 # Create the packagesets, and any archivepermissions295 # Create the packagesets, and any archivepermissions
284 for parent_ps in packagesets:296 for parent_ps in packagesets:
297 if self.packagesets and parent_ps.name not in self.packagesets:
298 continue
285 child_ps = getUtility(IPackagesetSet).new(299 child_ps = getUtility(IPackagesetSet).new(
286 parent_ps.name, parent_ps.description,300 parent_ps.name, parent_ps.description,
287 self.distroseries.owner, distroseries=self.distroseries,301 self.distroseries.owner, distroseries=self.distroseries,
288302
=== modified file 'lib/lp/soyuz/scripts/tests/test_initialise_distroseries.py'
--- lib/lp/soyuz/scripts/tests/test_initialise_distroseries.py 2010-10-06 11:46:51 +0000
+++ lib/lp/soyuz/scripts/tests/test_initialise_distroseries.py 2010-10-12 08:34:50 +0000
@@ -19,7 +19,10 @@
19from lp.registry.interfaces.pocket import PackagePublishingPocket19from lp.registry.interfaces.pocket import PackagePublishingPocket
20from lp.soyuz.enums import SourcePackageFormat20from lp.soyuz.enums import SourcePackageFormat
21from lp.soyuz.interfaces.archivepermission import IArchivePermissionSet21from lp.soyuz.interfaces.archivepermission import IArchivePermissionSet
22from lp.soyuz.interfaces.packageset import IPackagesetSet22from lp.soyuz.interfaces.packageset import (
23 IPackagesetSet,
24 NoSuchPackageSet,
25 )
23from lp.soyuz.interfaces.publishing import PackagePublishingStatus26from lp.soyuz.interfaces.publishing import PackagePublishingStatus
24from lp.soyuz.interfaces.sourcepackageformat import (27from lp.soyuz.interfaces.sourcepackageformat import (
25 ISourcePackageFormatSelectionSet,28 ISourcePackageFormatSelectionSet,
@@ -155,9 +158,9 @@
155 child.isSourcePackageFormatPermitted(158 child.isSourcePackageFormatPermitted(
156 SourcePackageFormat.FORMAT_1_0))159 SourcePackageFormat.FORMAT_1_0))
157160
158 def _full_initialise(self, arches=(), rebuild=False):161 def _full_initialise(self, arches=(), packagesets=(), rebuild=False):
159 child = self.factory.makeDistroSeries(parent_series=self.parent)162 child = self.factory.makeDistroSeries(parent_series=self.parent)
160 ids = InitialiseDistroSeries(child, arches, rebuild)163 ids = InitialiseDistroSeries(child, arches, packagesets, rebuild)
161 ids.check()164 ids.check()
162 ids.initialise()165 ids.initialise()
163 return child166 return child
@@ -230,6 +233,33 @@
230 child.main_archive, 'udev', uploader,233 child.main_archive, 'udev', uploader,
231 distroseries=child))234 distroseries=child))
232235
236 def test_copy_limit_packagesets(self):
237 # If a parent series has packagesets, we can decide which ones we
238 # want to copy
239 test1 = getUtility(IPackagesetSet).new(
240 u'test1', u'test 1 packageset', self.parent.owner,
241 distroseries=self.parent)
242 test2 = getUtility(IPackagesetSet).new(
243 u'test2', u'test 2 packageset', self.parent.owner,
244 distroseries=self.parent)
245 packages = ('udev', 'chromium', 'libc6')
246 for pkg in packages:
247 test1.addSources(pkg)
248 child = self._full_initialise(packagesets=('test1',))
249 child_test1 = getUtility(IPackagesetSet).getByName(
250 u'test1', distroseries=child)
251 self.assertEqual(test1.description, child_test1.description)
252 self.assertRaises(
253 NoSuchPackageSet, getUtility(IPackagesetSet).getByName,
254 u'test2', distroseries=child)
255 parent_srcs = test1.getSourcesIncluded(direct_inclusion=True)
256 child_srcs = child_test1.getSourcesIncluded(
257 direct_inclusion=True)
258 self.assertEqual(parent_srcs, child_srcs)
259 child.updatePackageCount()
260 self.assertEqual(child.sourcecount, len(packages))
261 self.assertEqual(child.binarycount, 2) # Chromium is FTBFS
262
233 def test_rebuild_flag(self):263 def test_rebuild_flag(self):
234 # No binaries will get copied if we specify rebuild=True264 # No binaries will get copied if we specify rebuild=True
235 self.parent.updatePackageCount()265 self.parent.updatePackageCount()
@@ -254,6 +284,35 @@
254 self.assertEqual(284 self.assertEqual(
255 das[0].architecturetag, self.parent_das.architecturetag)285 das[0].architecturetag, self.parent_das.architecturetag)
256286
287 def test_limit_packagesets_rebuild_and_one_das(self):
288 # We can limit the source packages copied, and only builds
289 # for the copied source will be created
290 test1 = getUtility(IPackagesetSet).new(
291 u'test1', u'test 1 packageset', self.parent.owner,
292 distroseries=self.parent)
293 test2 = getUtility(IPackagesetSet).new(
294 u'test2', u'test 2 packageset', self.parent.owner,
295 distroseries=self.parent)
296 packages = ('udev', 'chromium')
297 for pkg in packages:
298 test1.addSources(pkg)
299 self.factory.makeDistroArchSeries(distroseries=self.parent)
300 child = self._full_initialise(
301 arches=[self.parent_das.architecturetag],
302 packagesets=('test1',), rebuild=True)
303 child.updatePackageCount()
304 builds = child.getBuildRecords(
305 build_state=BuildStatus.NEEDSBUILD,
306 pocket=PackagePublishingPocket.RELEASE)
307 self.assertEqual(child.sourcecount, len(packages))
308 self.assertEqual(child.binarycount, 0)
309 self.assertEqual(builds.count(), len(packages))
310 das = list(IStore(DistroArchSeries).find(
311 DistroArchSeries, distroseries = child))
312 self.assertEqual(len(das), 1)
313 self.assertEqual(
314 das[0].architecturetag, self.parent_das.architecturetag)
315
257 def test_script(self):316 def test_script(self):
258 # Do an end-to-end test using the command-line tool317 # Do an end-to-end test using the command-line tool
259 uploader = self.factory.makePerson()318 uploader = self.factory.makePerson()
260319
=== modified file 'lib/lp/soyuz/tests/test_initialisedistroseriesjob.py'
--- lib/lp/soyuz/tests/test_initialisedistroseriesjob.py 2010-09-16 01:38:42 +0000
+++ lib/lp/soyuz/tests/test_initialisedistroseriesjob.py 2010-10-12 08:34:50 +0000
@@ -72,3 +72,19 @@
72 # returns an InitialisationError, then it's good.72 # returns an InitialisationError, then it's good.
73 self.assertRaisesWithContent(73 self.assertRaisesWithContent(
74 InitialisationError, "Parent series required.", job.run)74 InitialisationError, "Parent series required.", job.run)
75
76 def test_arguments(self):
77 """Test that InitialiseDistroSeriesJob specified with arguments can
78 be gotten out again."""
79 distroseries = self.factory.makeDistroSeries()
80 arches = (u'i386', u'amd64')
81 packagesets = (u'foo', u'bar', u'baz')
82
83 job = getUtility(IInitialiseDistroSeriesJobSource).create(
84 distroseries, arches, packagesets)
85
86 naked_job = removeSecurityProxy(job)
87 self.assertEqual(naked_job.distroseries, distroseries)
88 self.assertEqual(naked_job.arches, arches)
89 self.assertEqual(naked_job.packagesets, packagesets)
90 self.assertEqual(naked_job.rebuild, False)

Subscribers

People subscribed via source and target branches

to status/vote changes: