libdep-service

Merge lp:~jml/libdep-service/celery-task into lp:libdep-service

  1. celery-task
  2. Merge into trunk
Proposed by Jonathan Lange
Status: Merged
Approved by: James Westby
Approved revision: 80
Merged at revision: 74
Proposed branch: lp:~jml/libdep-service/celery-task
Merge into: lp:libdep-service
Diff against target: 260 lines (+153/-18)
8 files modified
django_project/dev.cfg (+16/-0)
django_project/main.cfg (+1/-0)
djlibdep/api.py (+16/-0)
djlibdep/aptfile.py (+0/-18)
djlibdep/tasks.py (+50/-0)
djlibdep/tests/__init__.py (+1/-0)
djlibdep/tests/test_api.py (+10/-0)
djlibdep/tests/test_tasks.py (+59/-0)
To merge this branch: bzr merge lp:~jml/libdep-service/celery-task
Reviewer Review Type Date Requested Status
James Westby (community) Approve
Review via email: mp+131968@code.launchpad.net

Commit message

Add celery tasks for updating the database using Contents / apt-file

Description of the change

Adds kombu as an installed application, haven't added it as a dependency.

The change to insert_packages to protect against duplicate insertions means that we're going to be storing the whole thing in memory. I haven't tried catching the IntegrityError in this branch. My memory from past efforts shows that this doesn't work, as by the time you catch it the transaction has already been voided.

I just used subtask directly. I could also have used TaskSets, but don't really see what we'd win there. To my annoyance, celery duplicates something of Twisted but makes itself look simpler by merely not documenting how it actually works.

http://docs.celeryproject.org/en/2.5/userguide/tasksets.html

Tests are fairly straightforward.

Thanks,
jml

To post a comment you must log in.
Revision history for this message
James Westby (james-w) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'django_project/dev.cfg'
2--- django_project/dev.cfg 2012-06-28 11:49:52 +0000
3+++ django_project/dev.cfg 2012-10-30 10:29:21 +0000
4@@ -1,5 +1,21 @@
5 [django]
6 databases = databases
7+installed_apps =
8+ django.contrib.auth
9+ django.contrib.contenttypes
10+ django.contrib.sessions
11+ django.contrib.sites
12+ django.contrib.messages
13+ django.contrib.staticfiles
14+ django.contrib.admin
15+ djlibdep
16+ djcelery
17+ south
18+ django_configglue
19+ preflight
20+ django_openid_auth
21+ pgtools
22+ kombu.transport.django
23
24 [django_file_logging_handler]
25 level = INFO
26
27=== modified file 'django_project/main.cfg'
28--- django_project/main.cfg 2012-10-23 21:25:16 +0000
29+++ django_project/main.cfg 2012-10-30 10:29:21 +0000
30@@ -35,6 +35,7 @@
31 login_redirect_url = /
32 template_context_processors = django.contrib.auth.context_processors.auth
33 django.core.context_processors.request
34+test_runner = djcelery.contrib.test_runner.CeleryTestSuiteRunner
35
36
37 [django_logging]
38
39=== modified file 'djlibdep/api.py'
40--- djlibdep/api.py 2012-10-29 11:50:29 +0000
41+++ djlibdep/api.py 2012-10-30 10:29:21 +0000
42@@ -39,15 +39,31 @@
43 :param libs: An iterable of ``(package, library, architecture)``.
44 """
45 # XXX: Can change this to insert multiple rows at once.
46+ seen = set()
47 for package, library, architecture in libs:
48+ if (package, library) in seen:
49+ continue
50 db.insert_new_library(package, library, package, architecture)
51+ seen.add((package, library))
52
53
54 def update_packages(db, architecture, libs):
55+ """Update the library to package mappings for a single architecture.
56+
57+ Note that this runs one **long** transaction, deleting any previous
58+ mappings for the architecture and creating entirely new ones.
59+
60+ :param db: A ``PackageDatabase`` object.
61+ :param architecture: The architecture to update.
62+ :param libs: An iterable of ``(package, library)``, where ``package``
63+ provides ``library``.
64+ """
65+ # XXX: distro: Change this to be for architecture **and** distro.
66 db._store.execute(
67 'DELETE FROM libdep WHERE architecture = ?', (unicode(architecture),))
68 insert_packages(
69 db, ((package, library, architecture) for (package, library) in libs))
70+ db._store.commit()
71
72
73 def get_summary():
74
75=== modified file 'djlibdep/aptfile.py'
76--- djlibdep/aptfile.py 2012-10-29 11:50:29 +0000
77+++ djlibdep/aptfile.py 2012-10-30 10:29:21 +0000
78@@ -21,13 +21,6 @@
79
80 from fixtures import TempDir
81
82-from .api import update_packages
83-
84-
85-SUPPORTED_ARCHITECTURES = ['i386', 'amd64']
86-DEFAULT_ARCHIVE = 'http://archive.ubuntu.com/ubuntu'
87-DEFAULT_SUITE = 'oneiric'
88-
89
90 def get_contents_url(archive, suite, architecture):
91 """Get the URL for a contents file.
92@@ -157,14 +150,3 @@
93 with gzip.open(contents_path, 'r') as contents:
94 for package, library in iter_libraries_in_contents(contents):
95 yield package, library
96-
97-
98-def update_from_contents(db, archive, suite, architecture):
99- """Update the database with Contents from an archive."""
100- update_packages(
101- db, architecture, iter_library_packages(archive, suite, architecture))
102-
103-
104-def update_database(db):
105- for architecture in SUPPORTED_ARCHITECTURES:
106- update_from_contents(db, DEFAULT_ARCHIVE, DEFAULT_SUITE, architecture)
107
108=== added file 'djlibdep/tasks.py'
109--- djlibdep/tasks.py 1970-01-01 00:00:00 +0000
110+++ djlibdep/tasks.py 2012-10-30 10:29:21 +0000
111@@ -0,0 +1,50 @@
112+# Copyright (C) 2012 Canonical Ltd.
113+#
114+# This program is free software: you can redistribute it and/or modify
115+# it under the terms of the GNU Affero General Public License as published by
116+# the Free Software Foundation, version 3 of the License.
117+#
118+# This program is distributed in the hope that it will be useful,
119+# but WITHOUT ANY WARRANTY; without even the implied warranty of
120+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
121+# GNU Affero General Public License for more details.
122+#
123+# You should have received a copy of the GNU Affero General Public License
124+# along with this program. If not, see <http://www.gnu.org/licenses/>.
125+
126+
127+from celery.task import (
128+ subtask,
129+ task,
130+ )
131+
132+from devportalbinary.database import get_dependency_database
133+
134+from .api import update_packages
135+from .aptfile import iter_library_packages
136+
137+
138+DEFAULT_ARCHIVE = 'http://archive.ubuntu.com/ubuntu'
139+
140+# XXX: distro: Only one suite in the database for now. Ought to have more.
141+DEFAULT_SUITE = 'oneiric'
142+
143+SUPPORTED_ARCHITECTURES = set(['i386', 'amd64'])
144+
145+
146+@task()
147+def update_architecture(architecture):
148+ # XXX: distro: Actually want this to be an update per Contents file.
149+ # Thing is, we don't have distro (aka suite) records in the database, so
150+ # we currently only operate for one distro. When we expand, change this
151+ # to have multiple parameters.
152+ db = get_dependency_database()
153+ update_packages(
154+ db, architecture,
155+ iter_library_packages(DEFAULT_ARCHIVE, DEFAULT_SUITE, architecture))
156+
157+
158+@task
159+def update_all():
160+ for architecture in SUPPORTED_ARCHITECTURES:
161+ subtask(update_architecture).delay(architecture)
162
163=== added file 'djlibdep/tests/Contents.gz'
164Binary files djlibdep/tests/Contents.gz 1970-01-01 00:00:00 +0000 and djlibdep/tests/Contents.gz 2012-10-30 10:29:21 +0000 differ
165=== modified file 'djlibdep/tests/__init__.py'
166--- djlibdep/tests/__init__.py 2012-10-24 14:25:42 +0000
167+++ djlibdep/tests/__init__.py 2012-10-30 10:29:21 +0000
168@@ -24,6 +24,7 @@
169 'interface',
170 'pep8',
171 'preflight',
172+ 'tasks',
173 'test_double',
174 'test_double_impl',
175 'views',
176
177=== modified file 'djlibdep/tests/test_api.py'
178--- djlibdep/tests/test_api.py 2012-10-29 11:30:41 +0000
179+++ djlibdep/tests/test_api.py 2012-10-30 10:29:21 +0000
180@@ -91,6 +91,16 @@
181 self.assertEqual(
182 {'i386': {'libfoo.so.1': ['libbar', 'libfoo']}}, result)
183
184+ def test_duplicate_entries(self):
185+ db = self.get_package_db()
186+ insert_packages(
187+ db, [('libfoo', 'libfoo.so.1', 'i386'),
188+ ('libfoo', 'libfoo.so.1', 'i386'),
189+ ])
190+ result = get_binaries_for_libraries(db, ['libfoo.so.1'], ['i386'])
191+ self.assertEqual(
192+ {'i386': {'libfoo.so.1': ['libfoo']}}, result)
193+
194
195 class TestDatabaseUpdate(TestCase, WithDatabase):
196
197
198=== added file 'djlibdep/tests/test_tasks.py'
199--- djlibdep/tests/test_tasks.py 1970-01-01 00:00:00 +0000
200+++ djlibdep/tests/test_tasks.py 2012-10-30 10:29:21 +0000
201@@ -0,0 +1,59 @@
202+# Copyright (C) 2012 Canonical Ltd.
203+#
204+# This program is free software: you can redistribute it and/or modify
205+# it under the terms of the GNU Affero General Public License as published by
206+# the Free Software Foundation, version 3 of the License.
207+#
208+# This program is distributed in the hope that it will be useful,
209+# but WITHOUT ANY WARRANTY; without even the implied warranty of
210+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
211+# GNU Affero General Public License for more details.
212+#
213+# You should have received a copy of the GNU Affero General Public License
214+# along with this program. If not, see <http://www.gnu.org/licenses/>.
215+
216+import os
217+
218+from devportalbinary.testing import DatabaseFixture
219+from fixtures import MonkeyPatch
220+from testtools import TestCase
221+from testtools.matchers import GreaterThan
222+
223+
224+from ..api import (
225+ get_architecture_counts,
226+ get_total_rows,
227+ )
228+from ..tasks import (
229+ update_all,
230+ update_architecture,
231+ )
232+
233+
234+SAMPLE_CONTENTS = os.path.join(os.path.dirname(__file__), 'Contents.gz')
235+
236+
237+class TestUpdateArchitecture(TestCase):
238+
239+ def download_file(self, url, path, buffer_size=None):
240+ os.link(SAMPLE_CONTENTS, path)
241+
242+ def test_updates_database(self):
243+ db = self.useFixture(DatabaseFixture()).db
244+ self.useFixture(
245+ MonkeyPatch('djlibdep.aptfile.download_file', self.download_file))
246+ update_architecture.delay('i386')
247+ # XXX: Not a great assertion, but it's a start. We want to be sure
248+ # that something is in the database.
249+ total_rows = get_total_rows(db)
250+ self.assertThat(total_rows, GreaterThan(0))
251+ self.assertEqual({'i386': total_rows}, get_architecture_counts(db))
252+
253+ def test_update_all(self):
254+ db = self.useFixture(DatabaseFixture()).db
255+ self.useFixture(
256+ MonkeyPatch('djlibdep.aptfile.download_file', self.download_file))
257+ update_all.delay()
258+ counts = get_architecture_counts(db)
259+ self.assertEqual(counts['i386'], counts['amd64'])
260+ self.assertThat(counts['amd64'], GreaterThan(0))

Subscribers

People subscribed via source and target branches