pkgme service

Merge lp:~jml/pkgme-service/add-download-file into lp:pkgme-service

  1. add-download-file
  2. Merge into trunk
Proposed by Jonathan Lange
Status: Merged
Approved by: James Westby
Approved revision: 141
Merged at revision: 141
Proposed branch: lp:~jml/pkgme-service/add-download-file
Merge into: lp:pkgme-service
Diff against target: 261 lines (+188/-3)
3 files modified
src/djpkgme/osutils.py (+80/-0)
src/djpkgme/tasks.py (+1/-3)
src/djpkgme/tests/test_osutils.py (+107/-0)
To merge this branch: bzr merge lp:~jml/pkgme-service/add-download-file
Reviewer Review Type Date Requested Status
James Westby (community) Approve
Review via email: mp+134127@code.launchpad.net

Commit message

Add download_file from pkgme-devportal

Description of the change

I wrote a useful message here before, but the Internet ate it.

pkgme-service uses download_file, but pkgme-devportal does not. This branch
copies download_file into pkgme-service so it can be removed from devportalbinary.

To post a comment you must log in.
Revision history for this message
James Westby (james-w) :
review: Approve
Revision history for this message
ISD Branch Mangler (isd-branches-mangler) wrote :
Download full text (6.1 KiB)

The attempt to merge lp:~jml/pkgme-service/add-download-file into lp:pkgme-service failed. Below is the output from the failed tests.

python2.7 bootstrap.py --distribute --version 1.5.1 \
    --download-base=download-cache/dist --eggs=eggs \
    --setup-source distribute_setup.py
touch --no-create bin/buildout
./bin/buildout
Develop: '/tmp/tmpHkM6lC/.'
Updating scripts.
Updating filetemplates.
touch --no-create bin/py
./bin/py django_project/manage.py test djpkgme
Creating test database for alias 'default'...
Destroying test database for alias 'default'...

Tree is up to date at revision 44 of branch bzr+ssh://bazaar.launchpad.net/+branch/ca-download-cache
..............................E...............................
======================================================================
ERROR: test_gtk (djpkgme.tests.test_integration.TestEndToEnd)
djpkgme.tests.test_integration.TestEndToEnd.test_gtk
----------------------------------------------------------------------
_StringException: Empty attachments:
  twisted-log

celery-log: {{{
[2012-11-13 22:31:00,675: WARNING/MainProcess] -------------- <email address hidden> v2.5.0
---- **** -----
--- * *** * -- [Configuration]
-- * - **** --- . broker: djkombu.transport.DatabaseTransport:////
- ** ---------- . loader: djcelery.loaders.DjangoLoader
- ** ---------- . logfile: <email address hidden>@WARNING
- ** ---------- . concurrency: 1
- ** ---------- . events: ON
- *** --- * --- . beat: OFF
-- ******* ----
--- ***** ----- [Queues]
 -------------- . celery: exchange:celery (direct) binding:celery
[2012-11-13 22:31:00,687: WARNING/MainProcess] <email address hidden> has started.
[2012-11-13 22:31:06,258: INFO/PoolWorker-1] ['tar', '--force-local', '--no-same-owner', '--no-same-permissions', '-xf', u'/tmp/tmp7kivhZ/download/package-name4.tar.gz', '-C', '/tmp/tmp7kivhZ/working'] finished in 0.445s
[2012-11-13 22:31:07,815: INFO/PoolWorker-1] ['/tmp/eggs/pkgme_devportal-0.4.10-py2.7.egg/devportalbinary/backends/python-stub/want'] finished in 1.555s
[2012-11-13 22:31:09,035: INFO/PoolWorker-1] ['/tmp/eggs/pkgme_devportal-0.4.10-py2.7.egg/devportalbinary/backends/diff/want'] finished in 1.220s
[2012-11-13 22:31:09,937: INFO/PoolWorker-1] ['/tmp/eggs/pkgme_devportal-0.4.10-py2.7.egg/devportalbinary/backends/pdf/want'] finished in 0.900s
[2012-11-13 22:31:11,504: INFO/PoolWorker-1] ['/tmp/eggs/pkgme_devportal-0.4.10-py2.7.egg/devportalbinary/backends/deb-bin/want'] finished in 1.567s
[2012-11-13 22:31:13,235: INFO/PoolWorker-1] ['/tmp/eggs/pkgme_devportal-0.4.10-py2.7.egg/devportalbinary/backends/exe/want'] finished in 1.730s
[2012-11-13 22:31:14,021: INFO/PoolWorker-1] ['/tmp/eggs/pkgme_devportal-0.4.10-py2.7.egg/devportalbinary/backends/deb-src/want'] finished in 0.785s
[2012-11-13 22:31:15,127: INFO/PoolWorker-1] ['/tmp/eggs/pkgme_devportal-0.4.10-py2.7.egg/devportalbinary/backends/text/want'] finished in 1.106s
[2012-11-13 22:31:17,969: INFO/PoolWorker-1] ['/tmp/eggs/pkgme_devportal-0.4.10-py2.7.egg/devportalbinary/backends/image/want'] finished in 2.840s
[2012-11-13 ...

Read more...

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'src/djpkgme/osutils.py'
2--- src/djpkgme/osutils.py 2012-09-20 16:11:44 +0000
3+++ src/djpkgme/osutils.py 2012-11-13 15:21:25 +0000
4@@ -10,8 +10,16 @@
5 import os
6 import shutil
7 import tarfile
8+import tempfile
9+from urllib2 import (
10+ HTTPError,
11+ Request,
12+ URLError,
13+ urlopen,
14+ )
15 import zipfile
16
17+from bzrlib import urlutils
18 from pkgme.run_script import (
19 run_subprocess,
20 ScriptFailed,
21@@ -130,3 +138,75 @@
22 finally:
23 t.close()
24 return destination
25+
26+
27+def _open_file_for_writing(url, directory=None, name=None, working_dir=None):
28+ """Open a file for writing.
29+
30+ If neither 'directory' nor 'name' are specified then make a secure
31+ tempfile with 'mkstemp'. If directory is specified, then opens a file in
32+ that directory, either with 'name' or the trailing segment of the URL. If
33+ directory is not specified and name is specified, then opens the file in a
34+ temporary directory.
35+
36+ If 'working_dir' is specified, then all temporary files or directories
37+ will be created inside that directory.
38+ """
39+ if directory is None:
40+ if name is None:
41+ fd, path = tempfile.mkstemp(dir=working_dir)
42+ else:
43+ return _open_file_for_writing(
44+ url, tempfile.mkdtemp(dir=working_dir), name, working_dir)
45+ else:
46+ if name is None:
47+ name = urlutils.unescape(urlutils.basename(url))
48+ path = os.path.join(directory, name)
49+ fd = os.open(path, os.O_RDWR | os.O_CREAT | os.O_EXCL)
50+ return fd, path
51+
52+
53+def _wrap_urlopen(request):
54+ """urlopen with better error messages."""
55+ url = request.get_full_url()
56+ try:
57+ return urlopen(request)
58+ except HTTPError, original:
59+ raise HTTPError(
60+ original.url,
61+ original.code,
62+ '%s: <%s>' % (original.msg, url),
63+ original.hdrs,
64+ original.fp,
65+ )
66+ except URLError, original:
67+ if original.args:
68+ message = '%s: %s' % (str(original.args[0]), url)
69+ else:
70+ message = url
71+ e = URLError(message)
72+ e.original = original
73+ raise e
74+
75+
76+def download_file(url, directory=None, name=None, working_dir=None,
77+ bufsize=4 * 2 ** 10, headers=None):
78+ """Download 'url' into 'directory'."""
79+ # XXX: download_file is no longer used within pkgme-devportal
80+ request = Request(url)
81+ if headers is not None:
82+ for h in headers:
83+ request.add_header(h, headers[h])
84+ download = _wrap_urlopen(request)
85+ try:
86+ fd, path = _open_file_for_writing(url, directory, name, working_dir)
87+ try:
88+ while True:
89+ data = download.read(bufsize)
90+ if not data:
91+ return path
92+ os.write(fd, data)
93+ finally:
94+ os.close(fd)
95+ finally:
96+ download.close()
97
98=== modified file 'src/djpkgme/tasks.py'
99--- src/djpkgme/tasks.py 2012-09-18 12:34:40 +0000
100+++ src/djpkgme/tasks.py 2012-11-13 15:21:25 +0000
101@@ -12,9 +12,6 @@
102 from oops_dictconfig import config_from_dict
103
104 from devportalbinary.metadata import MetadataBackend
105-from devportalbinary.database import (
106- download_file,
107- )
108 from fixtures import (
109 TempDir,
110 )
111@@ -33,6 +30,7 @@
112 submit_pkgme_info,
113 )
114 from .osutils import (
115+ download_file,
116 make_tarball,
117 prepare_file,
118 )
119
120=== modified file 'src/djpkgme/tests/test_osutils.py'
121--- src/djpkgme/tests/test_osutils.py 2012-09-20 16:55:57 +0000
122+++ src/djpkgme/tests/test_osutils.py 2012-11-13 15:21:25 +0000
123@@ -1,19 +1,27 @@
124 import errno
125 import os
126+from StringIO import StringIO
127 import subprocess
128 import tarfile
129+from urllib2 import (
130+ HTTPError,
131+ URLError,
132+ )
133 import zipfile
134
135 from fixtures import TempDir
136 from testtools import TestCase
137 from testtools.matchers import (
138 DirExists,
139+ Equals,
140 FileContains,
141 HasPermissions,
142+ Matcher,
143 Not,
144 TarballContains,
145 )
146 from treeshape import (
147+ CONTENT,
148 from_rough_spec,
149 HasFileTree,
150 FileTree,
151@@ -21,7 +29,9 @@
152 )
153
154 from ..osutils import (
155+ _open_file_for_writing,
156 DEFAULT_WORKING_DIRECTORY_NAME,
157+ download_file,
158 make_tarball,
159 prepare_file,
160 try_extract_with_tar_command,
161@@ -223,3 +233,100 @@
162 self.assertEqual(
163 destination,
164 os.path.join(destination_dir, '%s-2.tar.gz' % (package_name,)))
165+
166+
167+class IsChildPath(Matcher):
168+
169+ def __init__(self, parent_path):
170+ super(IsChildPath, self).__init__()
171+ self._parent_path = parent_path
172+
173+ def match(self, child_path):
174+ parent_segments = os.path.abspath(self._parent_path).split(os.sep)
175+ child_segments = os.path.abspath(child_path).split(os.sep)
176+ return Equals(parent_segments).match(
177+ child_segments[:len(parent_segments)])
178+
179+
180+class TestOpenForWriting(TestCase):
181+
182+ def assertIsWriteable(self, fd, path):
183+ contents = self.getUniqueString()
184+ os.write(fd, contents)
185+ os.close(fd)
186+ self.assertThat(path, FileContains(contents))
187+
188+ def test_default(self):
189+ fd, path = _open_file_for_writing('http://example.org/foo')
190+ self.addCleanup(os.unlink, path)
191+ self.assertIsWriteable(fd, path)
192+
193+ def test_directory_provided(self):
194+ temp_path = self.useFixture(TempDir()).path
195+ fd, path = _open_file_for_writing(
196+ 'http://example.org/foo', temp_path)
197+ self.assertIsWriteable(fd, path)
198+ self.assertEqual(os.path.join(temp_path, 'foo'), path)
199+
200+ def test_name_provided(self):
201+ fd, path = _open_file_for_writing(
202+ 'http://example.org/foo', name='bar')
203+ self.addCleanup(os.unlink, path)
204+ self.assertIsWriteable(fd, path)
205+ self.assertEqual('bar', os.path.basename(path))
206+
207+ def test_directory_and_name_provided(self):
208+ temp_path = self.useFixture(TempDir()).path
209+ fd, path = _open_file_for_writing(
210+ 'http://example.org/foo', temp_path, name='bar')
211+ self.assertIsWriteable(fd, path)
212+ self.assertEqual(os.path.join(temp_path, 'bar'), path)
213+
214+ def test_working_directory_no_name(self):
215+ working_dir = self.useFixture(TempDir()).path
216+ fd, path = _open_file_for_writing(
217+ 'http://example.org/foo', working_dir=working_dir)
218+ self.assertIsWriteable(fd, path)
219+ self.assertThat(path, IsChildPath(working_dir))
220+
221+ def test_working_directory_name(self):
222+ working_dir = self.useFixture(TempDir()).path
223+ fd, path = _open_file_for_writing(
224+ 'http://example.org/foo', name='bar', working_dir=working_dir)
225+ self.assertIsWriteable(fd, path)
226+ self.assertEqual('bar', os.path.basename(path))
227+ self.assertThat(path, IsChildPath(working_dir))
228+
229+
230+class TestDownloadFile(TestCase):
231+
232+ def test_saves_file(self):
233+ from djpkgme import osutils
234+ contents = self.getUniqueString()
235+ def urlopen(url):
236+ return StringIO(contents)
237+ self.patch(osutils, 'urlopen', urlopen)
238+ path = download_file('http://example.org/foo')
239+ self.assertThat(path, FileContains(contents))
240+
241+ def test_url_error_includes_url(self):
242+ from djpkgme import osutils
243+ def urlopen(url):
244+ raise URLError("whatever")
245+ self.patch(osutils, 'urlopen', urlopen)
246+ url = 'http://example.org/foo'
247+ e = self.assertRaises(URLError, download_file, url)
248+ self.assertEqual("<urlopen error whatever: %s>" % (url,), str(e))
249+
250+ def test_http_error_includes_url(self):
251+ from djpkgme import osutils
252+ url = 'http://example.org/foo'
253+ # HTTPError must get a valid fp, otherwise it won't store the URL.
254+ tree = self.useFixture(FileTree({'thing.txt': {CONTENT: 'whatever'}}))
255+ fp = open(os.path.join(tree.path, 'thing.txt'), 'r')
256+ def urlopen(url):
257+ raise HTTPError(url, 404, 'whatever dude', {}, fp)
258+ self.patch(osutils, 'urlopen', urlopen)
259+ e = self.assertRaises(HTTPError, download_file, url)
260+ self.assertEqual(
261+ "HTTP Error 404: whatever dude: <%s>" % (url,), str(e))

Subscribers

People subscribed via source and target branches