Launchpad itself

Merge lp:~stevenk/launchpad/db-add-ifp-job into lp:launchpad/db-devel

  1. db-add-ifp-job
  2. Merge into db-devel
Proposed by Steve Kowalik
Status: Superseded
Proposed branch: lp:~stevenk/launchpad/db-add-ifp-job
Merge into: lp:launchpad/db-devel
Diff against target: 422 lines (+371/-0)
8 files modified
database/schema/comments.sql (+6/-0)
database/schema/patch-2207-99-0.sql (+22/-0)
database/schema/security.cfg (+1/-0)
lib/lp/soyuz/interfaces/initialisedistroseriesjob.py (+56/-0)
lib/lp/soyuz/model/doinitialisedistroseriesjob.py (+55/-0)
lib/lp/soyuz/model/initialisedistroseriesjob.py (+120/-0)
lib/lp/soyuz/tests/test_doinitialisedistroseries.py (+60/-0)
lib/lp/soyuz/tests/test_initialisedistroseriesjob.py (+51/-0)
To merge this branch: bzr merge lp:~stevenk/launchpad/db-add-ifp-job
Reviewer Review Type Date Requested Status
Launchpad code reviewers Pending
Review via email: mp+31349@code.launchpad.net

This proposal has been superseded by a proposal from 2010-08-23.

Description of the change

This branch adds only the plumbing for IDistroSeries.initialiseFromParent() to be run via the job system. There are some other changes required to have this in a useable state, but as it stands right now, the code is testable, and self-contained, so it can land, and I can make the small changes required when the other changes are ready.

The cronscript and other related bits to enable this work will be landed last.

This will require a database review, as well as a new database number.

To post a comment you must log in.
Revision history for this message
James Westby (james-w) wrote :

Hi,

Comparing this with CopyArchiveJob shows plenty of overlap, but also
some differences given the current implementation.

  - initializeFromParent eventually calls packagecloner, which
    copy archives also use, so there is room for consolidation.
  - i-f-p works on multiple archives at once, so ArchiveJob isn't
    a direct fit, but we could create multiple jobs for i-f-p
    (and there are comments in the code about whether copying partner
     is the right thing to do anyway)
  - copy archives always work on a distroseries, so an alternative would
    be to move to DistroSeriesJob instead of ArchiveJob there, and have
    archives moved in to json.
  - i-f-p is really a string of jobs, with the package clone being one of
    them, so perhaps breaking that method up would make sense.

I think what would help is looking at some other jobs, such as a
SyncPackagesJob and deciding what they would use, to decide if ArchiveJob,
DistroSeriesJob, or something else would make the best base.

Thanks,

James

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'database/schema/comments.sql'
2--- database/schema/comments.sql 2010-08-23 04:51:48 +0000
3+++ database/schema/comments.sql 2010-08-23 06:57:47 +0000
4@@ -2263,6 +2263,12 @@
5 COMMENT ON COLUMN HWDMIValue.value IS 'The value';
6 COMMENT ON COLUMN HWDMIValue.handle IS 'The handle to which this key/value pair belongs.';
7
8+-- InitialiseDistroSeriesJob
9+
10+COMMENT ON TABLE InitialiseDistroSeriesJob IS 'Contains references to job to be run to initialise distro series.';
11+COMMENT ON COLUMN InitialiseDistroSeriesJob.distroseries IS 'The distroseries to be initialised.';
12+COMMENT ON COLUMN InitialiseDistroSeriesJob.json_data IS 'A JSON struct containing data for the job.';
13+
14 -- Job
15
16 COMMENT ON TABLE Job IS 'Common info about a job.';
17
18=== added file 'database/schema/patch-2207-99-0.sql'
19--- database/schema/patch-2207-99-0.sql 1970-01-01 00:00:00 +0000
20+++ database/schema/patch-2207-99-0.sql 2010-08-23 06:57:47 +0000
21@@ -0,0 +1,22 @@
22+-- Copyright 2009 Canonical Ltd. This software is licensed under the
23+-- GNU Affero General Public License version 3 (see the file LICENSE).
24+
25+SET client_min_messages=ERROR;
26+
27+-- The `InitialiseDistroSeriesJob` table captures the data required for an ifp job.
28+
29+CREATE TABLE InitialiseDistroSeriesJob (
30+ id serial PRIMARY KEY,
31+ -- FK to the `Job` record with the "generic" data about this archive
32+ -- job.
33+ job integer NOT NULL CONSTRAINT initialisedistroseriesjob__job__fk REFERENCES job,
34+ -- FK to the associated `InitialiseDistroSeries` record.
35+ distroseries integer NOT NULL CONSTRAINT initialisedistroseriesjob__distroseries__fk REFERENCES DistroSeries,
36+ -- JSON data for use by the job
37+ json_data text
38+);
39+
40+ALTER TABLE InitialiseDistroSeriesJob ADD CONSTRAINT initialisedistroseriesjob__job__key UNIQUE (job);
41+
42+INSERT INTO LaunchpadDatabaseRevision VALUES (2207, 99, 0);
43+
44
45=== modified file 'database/schema/security.cfg'
46--- database/schema/security.cfg 2010-08-15 11:29:23 +0000
47+++ database/schema/security.cfg 2010-08-23 06:57:47 +0000
48@@ -159,6 +159,7 @@
49 public.hwtest = SELECT
50 public.hwvendorid = SELECT
51 public.hwvendorname = SELECT
52+public.initialisedistroseriesjob = SELECT, INSERT, UPDATE, DELETE
53 public.job = SELECT, INSERT, UPDATE, DELETE
54 public.karmacache = SELECT
55 public.karmacategory = SELECT
56
57=== added file 'lib/lp/soyuz/interfaces/initialisedistroseriesjob.py'
58--- lib/lp/soyuz/interfaces/initialisedistroseriesjob.py 1970-01-01 00:00:00 +0000
59+++ lib/lp/soyuz/interfaces/initialisedistroseriesjob.py 2010-08-23 06:57:47 +0000
60@@ -0,0 +1,56 @@
61+# Copyright 2010 Canonical Ltd. This software is licensed under the
62+# GNU Affero General Public License version 3 (see the file LICENSE).
63+
64+__metaclass__ = type
65+
66+__all__ = [
67+ "IDoInitialiseDistroSeriesJob",
68+ "IDoInitialiseDistroSeriesJobSource",
69+ "IInitialiseDistroSeriesJob",
70+ "IInitialiseDistroSeriesJobSource",
71+]
72+
73+from lazr.enum import DBEnumeratedType, DBItem
74+from zope.interface import Attribute, Interface
75+from zope.schema import Int, Object
76+
77+from canonical.launchpad import _
78+
79+from lp.services.job.interfaces.job import IJob, IJobSource, IRunnableJob
80+from lp.registry.interfaces.distroseries import IDistroSeries
81+
82+
83+class IInitialiseDistroSeriesJob(Interface):
84+ """A Job that initialises a distro series, based on a parent."""
85+
86+ id = Int(
87+ title=_('DB ID'), required=True, readonly=True,
88+ description=_("The tracking number for this job."))
89+
90+ distroseries = Object(
91+ title=_('The DistroSeries this job is about.'), schema=IDistroSeries,
92+ required=True)
93+
94+ job = Object(
95+ title=_('The common Job attributes'), schema=IJob, required=True)
96+
97+ metadata = Attribute('A dict of data about the job.')
98+
99+ def destroySelf():
100+ """Destroy this object."""
101+
102+
103+class IInitialiseDistroSeriesJobSource(IJobSource):
104+ """An interface for acquiring IInitialiseDistroSeriesJobs."""
105+
106+ def create(distroseries):
107+ """Create a new IInitialiseDistroSeriesJobs for a distroseries."""
108+
109+
110+class IDoInitialiseDistroSeriesJob(IRunnableJob):
111+ """A Job that initialises a distro series, based on a parent."""
112+
113+
114+class IDoInitialiseDistroSeriesJobSource(IInitialiseDistroSeriesJobSource):
115+ """An interface for acquiring IInitialiseDistroSeriesJobs."""
116+
117
118=== added file 'lib/lp/soyuz/model/doinitialisedistroseriesjob.py'
119--- lib/lp/soyuz/model/doinitialisedistroseriesjob.py 1970-01-01 00:00:00 +0000
120+++ lib/lp/soyuz/model/doinitialisedistroseriesjob.py 2010-08-23 06:57:47 +0000
121@@ -0,0 +1,55 @@
122+# Copyright 2010 Canonical Ltd. This software is licensed under the
123+# GNU Affero General Public License version 3 (see the file LICENSE).
124+
125+__metaclass__ = type
126+
127+__all__ = [
128+ "DoInitialiseDistroSeriesJob",
129+]
130+
131+from zope.component import getUtility
132+from zope.interface import classProvides, implements
133+
134+from canonical.launchpad.webapp.interfaces import (
135+ DEFAULT_FLAVOR, IStoreSelector, MAIN_STORE)
136+
137+from lp.services.job.model.job import Job
138+from lp.soyuz.interfaces.initialisedistroseriesjob import (
139+ IDoInitialiseDistroSeriesJob,
140+ IDoInitialiseDistroSeriesJobSource)
141+from lp.soyuz.model.initialisedistroseriesjob import (
142+ InitialiseDistroSeriesJob,
143+ InitialiseDistroSeriesJobDerived)
144+from lp.soyuz.scripts.initialise_distroseries import (
145+ InitialiseDistroSeries)
146+
147+
148+class DoInitialiseDistroSeriesJob(InitialiseDistroSeriesJobDerived):
149+
150+ implements(IDoInitialiseDistroSeriesJob)
151+
152+ classProvides(IDoInitialiseDistroSeriesJobSource)
153+
154+ @classmethod
155+ def create(cls, distroseries):
156+ """See `IDoInitialiseDistroSeriesJob`."""
157+ # If there's already a job, don't create a new one.
158+ store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
159+ existing_job = store.find(
160+ InitialiseDistroSeriesJob,
161+ InitialiseDistroSeriesJob.distroseries == distroseries,
162+ InitialiseDistroSeriesJob.job == Job.id,
163+ Job.id.is_in(Job.ready_jobs)
164+ ).any()
165+
166+ if existing_job is not None:
167+ return cls(existing_job)
168+ else:
169+ return super(
170+ DoInitialiseDistroSeriesJob, cls).create(distroseries)
171+
172+ def run(self):
173+ """See `IRunnableJob`."""
174+ ids = InitialiseDistroSeries(self.distroseries)
175+ ids.check()
176+ ids.initialise()
177
178=== added file 'lib/lp/soyuz/model/initialisedistroseriesjob.py'
179--- lib/lp/soyuz/model/initialisedistroseriesjob.py 1970-01-01 00:00:00 +0000
180+++ lib/lp/soyuz/model/initialisedistroseriesjob.py 2010-08-23 06:57:47 +0000
181@@ -0,0 +1,120 @@
182+# Copyright 2010 Canonical Ltd. This software is licensed under the
183+# GNU Affero General Public License version 3 (see the file LICENSE).
184+
185+__metaclass__ = type
186+
187+__all__ = [
188+ "InitialiseDistroSeriesJob",
189+ "InitialiseDistroSeriesJobDerived",
190+]
191+
192+import simplejson
193+
194+from sqlobject import SQLObjectNotFound
195+from storm.base import Storm
196+from storm.locals import And, Int, Reference, Unicode
197+
198+from zope.component import getUtility
199+from zope.interface import classProvides, implements
200+
201+from canonical.database.enumcol import EnumCol
202+from canonical.launchpad.webapp.interfaces import (
203+ DEFAULT_FLAVOR, IStoreSelector, MAIN_STORE, MASTER_FLAVOR)
204+
205+from lazr.delegates import delegates
206+
207+from lp.registry.model.distroseries import DistroSeries
208+from lp.soyuz.interfaces.initialisedistroseriesjob import (
209+ IInitialiseDistroSeriesJob,
210+ IInitialiseDistroSeriesJobSource)
211+
212+from lp.services.job.model.job import Job
213+from lp.services.job.runner import BaseRunnableJob
214+
215+
216+class InitialiseDistroSeriesJob(Storm):
217+ """Base class for jobs related to InitialiseDistroSeriess."""
218+
219+ implements(IInitialiseDistroSeriesJob)
220+
221+ __storm_table__ = 'InitialiseDistroSeriesJob'
222+
223+ id = Int(primary=True)
224+
225+ job_id = Int(name='job')
226+ job = Reference(job_id, Job.id)
227+
228+ distroseries_id = Int(name='distroseries')
229+ distroseries = Reference(distroseries_id, DistroSeries.id)
230+
231+ _json_data = Unicode('json_data')
232+
233+ def __init__(self, distroseries, metadata):
234+ super(InitialiseDistroSeriesJob, self).__init__()
235+ json_data = simplejson.dumps(metadata)
236+ self.job = Job()
237+ self.distroseries = distroseries
238+ self._json_data = json_data.decode('utf-8')
239+
240+ @property
241+ def metadata(self):
242+ return simplejson.loads(self._json_data)
243+
244+ @classmethod
245+ def get(cls, key):
246+ """Return the instance of this class whose key is supplied."""
247+ store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
248+ instance = store.get(cls, key)
249+ if instance is None:
250+ raise SQLObjectNotFound(
251+ 'No occurence of %s has key %s' % (cls.__name__, key))
252+ return instance
253+
254+
255+class InitialiseDistroSeriesJobDerived(BaseRunnableJob):
256+ """Intermediate class for deriving from InitialiseDistroSeriesJob."""
257+ delegates(IInitialiseDistroSeriesJob)
258+ classProvides(IInitialiseDistroSeriesJobSource)
259+
260+ def __init__(self, job):
261+ self.context = job
262+
263+ @classmethod
264+ def create(cls, distroseries):
265+ """See `IInitialiseDistroSeriesJob`."""
266+ # If there's already a job, don't create a new one.
267+ job = InitialiseDistroSeriesJob(
268+ distroseries, {})
269+ return cls(job)
270+
271+ @classmethod
272+ def get(cls, job_id):
273+ """Get a job by id.
274+
275+ :return: the InitialiseDistroSeriesJob with the specified id, as
276+ the current InitialiseDistroSeriesJobDerived subclass.
277+ :raises: SQLObjectNotFound if there is no job with the specified id.
278+ """
279+ job = InitialiseDistroSeriesJob.get(job_id)
280+ return cls(job)
281+
282+ @classmethod
283+ def iterReady(cls):
284+ """Iterate through all ready InitialiseDistroSeriesJobs."""
285+ store = getUtility(IStoreSelector).get(MAIN_STORE, MASTER_FLAVOR)
286+ jobs = store.find(
287+ InitialiseDistroSeriesJob,
288+ And(InitialiseDistroSeriesJob.job == Job.id,
289+ Job.id.is_in(Job.ready_jobs),
290+ InitialiseDistroSeriesJob.distroseries == DistroSeries.id))
291+ return (cls(job) for job in jobs)
292+
293+ def getOopsVars(self):
294+ """See `IRunnableJob`."""
295+ vars = BaseRunnableJob.getOopsVars(self)
296+ vars.extend([
297+ ('distroseries_id', self.context.distroseries.id),
298+ ('distroseries_job_id', self.context.id),
299+ ])
300+ return vars
301+
302
303=== added file 'lib/lp/soyuz/tests/test_doinitialisedistroseries.py'
304--- lib/lp/soyuz/tests/test_doinitialisedistroseries.py 1970-01-01 00:00:00 +0000
305+++ lib/lp/soyuz/tests/test_doinitialisedistroseries.py 2010-08-23 06:57:47 +0000
306@@ -0,0 +1,60 @@
307+from canonical.testing import LaunchpadZopelessLayer
308+from zope.security.proxy import removeSecurityProxy
309+
310+from lp.soyuz.model.doinitialisedistroseriesjob import (
311+ DoInitialiseDistroSeriesJob)
312+from lp.soyuz.scripts.initialise_distroseries import InitialisationError
313+from lp.testing import TestCaseWithFactory
314+from lp.testing.fakemethod import FakeMethod
315+
316+
317+class DoInitialiseDistroSeriesJobTests(TestCaseWithFactory):
318+ """Test case for DoInitialiseDistroSeriesJob."""
319+
320+ layer = LaunchpadZopelessLayer
321+
322+ def test_getOopsVars(self):
323+ distroseries = self.factory.makeDistroSeries()
324+ job = DoInitialiseDistroSeriesJob.create(distroseries)
325+ vars = job.getOopsVars()
326+ self.assertIn(('distroseries_id', distroseries.id), vars)
327+ self.assertIn(('distroseries_job_id', job.context.id), vars)
328+
329+ def _getJobs(self):
330+ """Return the pending DoInitialiseDistroSeriesJobs as a list."""
331+ return list(DoInitialiseDistroSeriesJob.iterReady())
332+
333+ def _getJobCount(self):
334+ """Return the number of DoInitialiseDistroSeriesJobs in
335+ the queue."""
336+ return len(self._getJobs())
337+
338+ def test_create_only_creates_one(self):
339+ distroseries = self.factory.makeDistroSeries()
340+ # If there's already a DoInitialiseDistroSeriesJob for a DistroSeries,
341+ # DoInitialiseDistroSeriesJob.create() won't create a new one.
342+ job = DoInitialiseDistroSeriesJob.create(distroseries)
343+
344+ # There will now be one job in the queue.
345+ self.assertEqual(1, self._getJobCount())
346+
347+ new_job = DoInitialiseDistroSeriesJob.create(distroseries)
348+
349+ # The two jobs will in fact be the same job.
350+ self.assertEqual(job, new_job)
351+
352+ # And the queue will still have a length of 1.
353+ self.assertEqual(1, self._getJobCount())
354+
355+ def test_run(self):
356+ """Test that DoInitialiseDistroSeriesJob.run() actually
357+ initialises builds and copies from the parent."""
358+ distroseries = self.factory.makeDistroSeries()
359+
360+ job = DoInitialiseDistroSeriesJob.create(distroseries)
361+
362+ # Since our new distroseries doesn't have a parent set, and the first
363+ # thing that run() will execute is checking the distroseries, if it
364+ # returns an InitialisationError, then it's good.
365+ self.assertRaisesWithContent(
366+ InitialisationError, "Parent series required.", job.run)
367
368=== added file 'lib/lp/soyuz/tests/test_initialisedistroseriesjob.py'
369--- lib/lp/soyuz/tests/test_initialisedistroseriesjob.py 1970-01-01 00:00:00 +0000
370+++ lib/lp/soyuz/tests/test_initialisedistroseriesjob.py 2010-08-23 06:57:47 +0000
371@@ -0,0 +1,51 @@
372+# Copyright 2010 Canonical Ltd. This software is licensed under the
373+# GNU Affero General Public License version 3 (see the file LICENSE).
374+
375+import unittest
376+
377+from canonical.testing import LaunchpadZopelessLayer
378+
379+from lp.soyuz.model.initialisedistroseriesjob import (
380+ InitialiseDistroSeriesJob, InitialiseDistroSeriesJobDerived)
381+from lp.testing import TestCaseWithFactory
382+
383+
384+class InitialiseDistroSeriesJobTestCase(TestCaseWithFactory):
385+ """Test case for basic InitialiseDistroSeriesJob gubbins."""
386+
387+ layer = LaunchpadZopelessLayer
388+
389+ def test_instantiate(self):
390+ distroseries = self.factory.makeDistroSeries()
391+
392+ metadata = ('some', 'arbitrary', 'metadata')
393+ distroseries_job = InitialiseDistroSeriesJob(
394+ distroseries, metadata)
395+
396+ self.assertEqual(distroseries, distroseries_job.distroseries)
397+
398+ # When we actually access the InitialiseDistroSeriesJob's
399+ # metadata it gets deserialized from JSON, so the representation
400+ # returned by distroseries_job.metadata will be different from what
401+ # we originally passed in.
402+ metadata_expected = [u'some', u'arbitrary', u'metadata']
403+ self.assertEqual(metadata_expected, distroseries_job.metadata)
404+
405+
406+class InitialiseDistroSeriesJobDerivedTestCase(TestCaseWithFactory):
407+ """Test case for the InitialiseDistroSeriesJobDerived class."""
408+
409+ layer = LaunchpadZopelessLayer
410+
411+ def test_create_explodes(self):
412+ # InitialiseDistroSeriesJobDerived.create() will blow up
413+ # because it needs to be subclassed to work properly.
414+ distroseries = self.factory.makeDistroSeries()
415+ self.assertRaises(
416+ AttributeError, InitialiseDistroSeriesJobDerived.create,
417+ distroseries)
418+
419+
420+def test_suite():
421+ return unittest.TestLoader().loadTestsFromName(__name__)
422+

Subscribers

People subscribed via source and target branches

to status/vote changes: