Launchpad itself

Merge lp:~jelmer/launchpad/sync-tbz2 into lp:launchpad/db-devel

  1. sync-tbz2
  2. Merge into db-devel
Proposed by Jelmer Vernooij
Status: Merged
Approved by: Jelmer Vernooij
Approved revision: not available
Merged at revision: not available
Proposed branch: lp:~jelmer/launchpad/sync-tbz2
Merge into: lp:launchpad/db-devel
Diff against target: 916 lines (+303/-297)
7 files modified
lib/canonical/launchpad/scripts/logger.py (+1/-1)
lib/lp/soyuz/scripts/ftpmaster.py (+27/-27)
lib/lp/soyuz/scripts/tests/sync_source_home/Debian_incoming_main_Sources (+28/-0)
lib/lp/soyuz/scripts/tests/sync_source_home/sample1_1.0-1.dsc (+37/-0)
lib/lp/soyuz/scripts/tests/test_sync_source.py (+105/-28)
scripts/ftpmaster-tools/dak_utils.py (+3/-153)
scripts/ftpmaster-tools/sync-source.py (+102/-88)
To merge this branch: bzr merge lp:~jelmer/launchpad/sync-tbz2
Reviewer Review Type Date Requested Status
Guilherme Salgado (community) code Approve
Review via email: mp+18615@code.launchpad.net

Commit message

Fixes support for syncing packages in the the version 3 source format.

To post a comment you must log in.
Revision history for this message
Jelmer Vernooij (jelmer) wrote :

Fixes support for syncing packages in the the version 3 source format (bug 225151).

This patch adds support for .orig.tar.bz2 files as well as multiple component orig tarballs in the script that syncs source packages from Debian. These new file formats are used by the new Debian source package format, v3.0.

This branch also refactors sync-source.py to use code from lp.soyuz.scripts.ftpmaster (well unit tested) rather its local copy (no tests). It also removes duplicate code for parsing dsc files from lp.soyuz.scripts.dak_utils in favor of the Dsc parser in debian_bundle.deb822.

Revision history for this message
Guilherme Salgado (salgado) wrote :
Download full text (10.4 KiB)

Hi Jelmer,

I have just a few comments/questions below, but this looks good. And
thanks a lot for getting rid of the untested/duplicated code.

 review approve code

On Thu, 2010-02-04 at 16:24 +0000, Jelmer Vernooij wrote:

> === modified file 'lib/canonical/launchpad/scripts/logger.py'
> --- lib/canonical/launchpad/scripts/logger.py 2009-10-16 18:07:01 +0000
> +++ lib/canonical/launchpad/scripts/logger.py 2010-02-04 16:24:23 +0000
[...]
> @@ -847,19 +854,14 @@
> Return the fetched filename if it was present in Librarian or None
> if it wasn't.
> """
> - # XXX cprov 2007-01-10 bug=78683: Looking for files within ubuntu
> - # only. It doesn't affect the usual sync-source procedure. However
> - # it needs to be revisited for derivation, we probably need
> - # to pass the target distribution in order to make proper lookups.
> - ubuntu = getUtility(IDistributionSet)['ubuntu']

Does this mean the bug above will be fixed too?

> try:
> - libraryfilealias = ubuntu.getFileByName(
> + libraryfilealias = self.todistro.getFileByName(
> filename, source=True, binary=False)
> except NotFoundError:
> return None
>
> - self.debug(
> - "\t%s: already in distro - downloading from librarian" %
> + self.logger.info(
> + "%s: already in distro - downloading from librarian" %
> filename)
>
> output_file = open(filename, 'w')
> @@ -871,23 +873,23 @@
> """Try to fetch files from Librarian.
>
> It raises SyncSourceError if anything else then an
> - 'orig.tar.gz' was found in Librarian.
> - Return a boolean indicating whether or not the 'orig.tar.gz' is
> - required in the upload.
> + orig tarball was found in Librarian.
> + Return the names of the files retrieved from the librarian.
> """
> - orig_filename = None
> + retrieved = []
> for filename in self.files.keys():
> if not self.fetchFileFromLibrarian(filename):
> continue
> + file_type = determine_source_file_type(filename)
> # set the return code if an orig was, in fact,
> # fetched from Librarian
> - if filename.endswith("orig.tar.gz"):
> - orig_filename = filename
> - else:
> + if not file_type in (SourcePackageFileType.ORIG_TARBALL,
> + SourcePackageFileType.COMPONENT_ORIG_TARBALL):

The indentation above is a bit odd; the two elements of the tuple should
be lined up.

> raise SyncSourceError(
> - 'Oops, only orig.tar.gz can be retrieved from librarian')
> + 'Oops, only orig tarball can be retrieved from librarian.')
> + retrieved.append(filename)
>
> - return orig_filename
> + return retrieved
>
> def fetchSyncFiles(self):
> """Fetch files from the original sync source.

>
> === modified file 'scripts/ftpmaster-tools/sync-source.py'
> --- scripts/ftpmaster-tools/sync-source.py 2010-02...

review: Approve (code)
Revision history for this message
Jelmer Vernooij (jelmer) wrote :

Hi Guilherme,

Thanks for the review.

On Fri, 2010-02-05 at 17:48 +0000, Guilherme Salgado wrote:
> On Thu, 2010-02-04 at 16:24 +0000, Jelmer Vernooij wrote:
> > === modified file 'lib/canonical/launchpad/scripts/logger.py'
> > --- lib/canonical/launchpad/scripts/logger.py 2009-10-16 18:07:01 +0000
> > +++ lib/canonical/launchpad/scripts/logger.py 2010-02-04 16:24:23 +0000
> [...]
> > @@ -847,19 +854,14 @@
> > Return the fetched filename if it was present in Librarian or None
> > if it wasn't.
> > """
> > - # XXX cprov 2007-01-10 bug=78683: Looking for files within ubuntu
> > - # only. It doesn't affect the usual sync-source procedure. However
> > - # it needs to be revisited for derivation, we probably need
> > - # to pass the target distribution in order to make proper lookups.
> > - ubuntu = getUtility(IDistributionSet)['ubuntu']
> Does this mean the bug above will be fixed too?
 Partially - bug 78683 is about sync-source only working with Ubuntu, and
this change gets us further in that direction but there are still some
places where we hardcode 'ubuntu' in scripts/ftp-master/sync-source.py.

I've addressed the style issue you've raised and I'll ec2 land after I
finish testing on dogfood.

Cheers,

Jelmer

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/canonical/launchpad/scripts/logger.py'
2--- lib/canonical/launchpad/scripts/logger.py 2009-10-16 18:07:01 +0000
3+++ lib/canonical/launchpad/scripts/logger.py 2010-02-11 16:55:27 +0000
4@@ -5,7 +5,7 @@
5
6 """Logging setup for scripts.
7
8-Don't import from this module. Import it from canonical.scripts.
9+Don't import from this module. Import it from canonical.launchpad.scripts.
10
11 Parts of this may be moved into canonical.launchpad somewhere if it is
12 to be used for non-script stuff.
13
14=== modified file 'lib/lp/soyuz/scripts/ftpmaster.py'
15--- lib/lp/soyuz/scripts/ftpmaster.py 2009-12-13 11:55:40 +0000
16+++ lib/lp/soyuz/scripts/ftpmaster.py 2010-02-11 16:55:27 +0000
17@@ -36,11 +36,14 @@
18 from canonical.launchpad.interfaces.launchpad import ILaunchpadCelebrities
19 from canonical.launchpad.interfaces.librarian import ILibraryFileAliasSet
20 from canonical.launchpad.webapp.interfaces import NotFoundError
21+from lp.archiveuploader.utils import (
22+ determine_source_file_type)
23 from lp.registry.interfaces.distribution import IDistributionSet
24 from lp.registry.interfaces.series import SeriesStatus
25 from lp.registry.interfaces.person import IPersonSet
26 from lp.registry.interfaces.pocket import (
27 PackagePublishingPocket, pocketsuffix)
28+from lp.registry.interfaces.sourcepackage import SourcePackageFileType
29 from lp.soyuz.interfaces.binarypackagename import IBinaryPackageNameSet
30 from lp.soyuz.interfaces.binarypackagerelease import IBinaryPackageReleaseSet
31 from lp.soyuz.interfaces.publishing import PackagePublishingStatus
32@@ -803,12 +806,14 @@
33 pocket_chroot.chroot.open()
34 copy_and_close(pocket_chroot.chroot, local_file)
35
36+
37 class SyncSourceError(Exception):
38 """Raised when an critical error occurs inside SyncSource.
39
40 The entire procedure should be aborted in order to avoid unknown problems.
41 """
42
43+
44 class SyncSource:
45 """Sync Source procedure helper class.
46
47@@ -818,20 +823,22 @@
48 'aptMD5Sum' is provided as a classmethod during the integration time.
49 """
50
51- def __init__(self, files, origin, debug, downloader):
52+ def __init__(self, files, origin, logger, downloader, todistro):
53 """Store local context.
54
55 files: a dictionary where the keys are the filename and the
56 value another dictionary with the file informations.
57 origin: a dictionary similar to 'files' but where the values
58 contain information for download files to be synchronized
59- debug: a debug function, 'debug(message)'
60+ logger: a logger
61 downloader: a callable that fetchs URLs, 'downloader(url, destination)'
62+ todistro: target distribution object
63 """
64 self.files = files
65 self.origin = origin
66- self.debug = debug
67+ self.logger = logger
68 self.downloader = downloader
69+ self.todistro = todistro
70
71 @classmethod
72 def generateMD5Sum(self, filename):
73@@ -847,19 +854,14 @@
74 Return the fetched filename if it was present in Librarian or None
75 if it wasn't.
76 """
77- # XXX cprov 2007-01-10 bug=78683: Looking for files within ubuntu
78- # only. It doesn't affect the usual sync-source procedure. However
79- # it needs to be revisited for derivation, we probably need
80- # to pass the target distribution in order to make proper lookups.
81- ubuntu = getUtility(IDistributionSet)['ubuntu']
82 try:
83- libraryfilealias = ubuntu.getFileByName(
84+ libraryfilealias = self.todistro.getFileByName(
85 filename, source=True, binary=False)
86 except NotFoundError:
87 return None
88
89- self.debug(
90- "\t%s: already in distro - downloading from librarian" %
91+ self.logger.info(
92+ "%s: already in distro - downloading from librarian" %
93 filename)
94
95 output_file = open(filename, 'w')
96@@ -871,23 +873,23 @@
97 """Try to fetch files from Librarian.
98
99 It raises SyncSourceError if anything else then an
100- 'orig.tar.gz' was found in Librarian.
101- Return a boolean indicating whether or not the 'orig.tar.gz' is
102- required in the upload.
103+ orig tarball was found in Librarian.
104+ Return the names of the files retrieved from the librarian.
105 """
106- orig_filename = None
107+ retrieved = []
108 for filename in self.files.keys():
109 if not self.fetchFileFromLibrarian(filename):
110 continue
111+ file_type = determine_source_file_type(filename)
112 # set the return code if an orig was, in fact,
113 # fetched from Librarian
114- if filename.endswith("orig.tar.gz"):
115- orig_filename = filename
116- else:
117+ if not file_type in (SourcePackageFileType.ORIG_TARBALL,
118+ SourcePackageFileType.COMPONENT_ORIG_TARBALL):
119 raise SyncSourceError(
120- 'Oops, only orig.tar.gz can be retrieved from librarian')
121+ 'Oops, only orig tarball can be retrieved from librarian.')
122+ retrieved.append(filename)
123
124- return orig_filename
125+ return retrieved
126
127 def fetchSyncFiles(self):
128 """Fetch files from the original sync source.
129@@ -896,21 +898,19 @@
130 """
131 dsc_filename = None
132 for filename in self.files.keys():
133+ file_type = determine_source_file_type(filename)
134+ if file_type == SourcePackageFileType.DSC:
135+ dsc_filename = filename
136 if os.path.exists(filename):
137+ self.logger.info(" - <%s: cached>" % (filename))
138 continue
139- self.debug(
140+ self.logger.info(
141 " - <%s: downloading from %s>" %
142 (filename, self.origin["url"]))
143 download_f = ("%s%s" % (self.origin["url"],
144 self.files[filename]["remote filename"]))
145 sys.stdout.flush()
146 self.downloader(download_f, filename)
147- # only set the dsc_filename if the DSC was really downloaded.
148- # this loop usually includes the other files for the upload,
149- # DIFF and ORIG.
150- if filename.endswith(".dsc"):
151- dsc_filename = filename
152-
153 return dsc_filename
154
155 def checkDownloadedFiles(self):
156
157=== modified file 'lib/lp/soyuz/scripts/tests/sync_source_home/Debian_incoming_main_Sources'
158--- lib/lp/soyuz/scripts/tests/sync_source_home/Debian_incoming_main_Sources 2008-02-14 12:38:50 +0000
159+++ lib/lp/soyuz/scripts/tests/sync_source_home/Debian_incoming_main_Sources 2010-02-11 16:55:27 +0000
160@@ -10,3 +10,31 @@
161 fc1464e5985b962a042d5354452f361d 164 bar_1.0.orig.tar.gz
162 a259bf88aedcdeded4e5d94ad4af3b4a 610 bar_1.0-1.diff.gz
163
164+Format: 3.0 (quilt)
165+Package: sample1
166+Binary: sample1
167+Architecture: all
168+Version: 1.0-1
169+Maintainer: Raphael Hertzog <hertzog@debian.org>
170+Standards-Version: 3.8.1
171+Checksums-Sha1:
172+ 724e3cdad5f4af13383e5d71f525465add1ce897 1741 sample1_1.0-1.dsc
173+ 296cb89762668772c3ae4799bd9ce7973f4154b2 171 sample1_1.0.orig-component1.tar.bz2
174+ ccb198be27c065880696cbeb6fb5aebf190c8544 157 sample1_1.0.orig-component2.tar.lzma
175+ ea3c0c6ee23410a9c454d9b35681a80460343456 201 sample1_1.0.orig-component3.tar.gz
176+ c62b2496d3ff0c64dcf6488c95e1d5cbc9f5f30d 201 sample1_1.0.orig.tar.gz
177+ b3c92ffaabc2e3e95e5f74e1de94351fddcb065c 1327 sample1_1.0-1.debian.tar.gz
178+Checksums-Sha256:
179+ d5390b56d3d5ea301bb565487eded8a6739d29ea5d680389f35b87b30f846412 1741 sample1_1.0-1.dsc
180+ a0871b73922a228ef8c1f4c1a7e1aef94d8a59d919218242655b7faae352d5c9 171 sample1_1.0.orig-component1.tar.bz2
181+ 39963ed90d70e918a6cae8d73c81f42dcb7f0dc283a4715f032a1e28b98d1a25 157 sample1_1.0.orig-component2.tar.lzma
182+ dc174acae810faf5f9a679e042f480905622fe4a854510dad6e200dbe6381c8d 201 sample1_1.0.orig-component3.tar.gz
183+ 9c0f0b42f6283614b204a47600cb9b0d8b19dfa4cb43c69e6a0a40f3d1064e0e 201 sample1_1.0.orig.tar.gz
184+ 6864ceeb315ad7b6a59636a0251525cbd8e9b76c6e934f3746271d4ebce5da3c 1327 sample1_1.0-1.debian.tar.gz
185+Files:
186+ aea108574786aa670dc5ef26dc225be2 1741 sample1_1.0-1.dsc
187+ 8a90ee77cf3c68c32bdb5c55d40d06f4 171 sample1_1.0.orig-component1.tar.bz2
188+ 3535d4766df47e531372802a0c2ddb82 157 sample1_1.0.orig-component2.tar.lzma
189+ 673b42066e8e858208b5d0c7a006c5fb 201 sample1_1.0.orig-component3.tar.gz
190+ 9047a93bf12a7c749045e9bc9ff9ee99 201 sample1_1.0.orig.tar.gz
191+ 9404bc99cdb06876a988d4c2262c073e 1327 sample1_1.0-1.debian.tar.gz
192
193=== added file 'lib/lp/soyuz/scripts/tests/sync_source_home/sample1_1.0-1.debian.tar.gz'
194Binary files lib/lp/soyuz/scripts/tests/sync_source_home/sample1_1.0-1.debian.tar.gz 1970-01-01 00:00:00 +0000 and lib/lp/soyuz/scripts/tests/sync_source_home/sample1_1.0-1.debian.tar.gz 2010-02-11 16:55:27 +0000 differ
195=== added file 'lib/lp/soyuz/scripts/tests/sync_source_home/sample1_1.0-1.dsc'
196--- lib/lp/soyuz/scripts/tests/sync_source_home/sample1_1.0-1.dsc 1970-01-01 00:00:00 +0000
197+++ lib/lp/soyuz/scripts/tests/sync_source_home/sample1_1.0-1.dsc 2010-02-11 16:55:27 +0000
198@@ -0,0 +1,37 @@
199+-----BEGIN PGP SIGNED MESSAGE-----
200+Hash: SHA1
201+
202+Format: 3.0 (quilt)
203+Source: sample1
204+Binary: sample1
205+Architecture: all
206+Version: 1.0-1
207+Maintainer: Raphael Hertzog <hertzog@debian.org>
208+Standards-Version: 3.8.1
209+Build-Depends: quilt, debhelper (>= 7)
210+Checksums-Sha1:
211+ 296cb89762668772c3ae4799bd9ce7973f4154b2 171 sample1_1.0.orig-component1.tar.bz2
212+ ccb198be27c065880696cbeb6fb5aebf190c8544 157 sample1_1.0.orig-component2.tar.lzma
213+ ea3c0c6ee23410a9c454d9b35681a80460343456 201 sample1_1.0.orig-component3.tar.gz
214+ c62b2496d3ff0c64dcf6488c95e1d5cbc9f5f30d 201 sample1_1.0.orig.tar.gz
215+ b3c92ffaabc2e3e95e5f74e1de94351fddcb065c 1327 sample1_1.0-1.debian.tar.gz
216+Checksums-Sha256:
217+ a0871b73922a228ef8c1f4c1a7e1aef94d8a59d919218242655b7faae352d5c9 171 sample1_1.0.orig-component1.tar.bz2
218+ 39963ed90d70e918a6cae8d73c81f42dcb7f0dc283a4715f032a1e28b98d1a25 157 sample1_1.0.orig-component2.tar.lzma
219+ dc174acae810faf5f9a679e042f480905622fe4a854510dad6e200dbe6381c8d 201 sample1_1.0.orig-component3.tar.gz
220+ 9c0f0b42f6283614b204a47600cb9b0d8b19dfa4cb43c69e6a0a40f3d1064e0e 201 sample1_1.0.orig.tar.gz
221+ 6864ceeb315ad7b6a59636a0251525cbd8e9b76c6e934f3746271d4ebce5da3c 1327 sample1_1.0-1.debian.tar.gz
222+Files:
223+ 8a90ee77cf3c68c32bdb5c55d40d06f4 171 sample1_1.0.orig-component1.tar.bz2
224+ 3535d4766df47e531372802a0c2ddb82 157 sample1_1.0.orig-component2.tar.lzma
225+ 673b42066e8e858208b5d0c7a006c5fb 201 sample1_1.0.orig-component3.tar.gz
226+ 9047a93bf12a7c749045e9bc9ff9ee99 201 sample1_1.0.orig.tar.gz
227+ 9404bc99cdb06876a988d4c2262c073e 1327 sample1_1.0-1.debian.tar.gz
228+
229+-----BEGIN PGP SIGNATURE-----
230+Version: GnuPG v1.4.10 (GNU/Linux)
231+
232+iEYEARECAAYFAktq7/kACgkQPa9Uoh7vUnahEQCgmRTupXtrHegzFOb5jQMOCSXS
233+/p4AoIroJtmjVryFpWqAABZTFGjeWo85
234+=vwh8
235+-----END PGP SIGNATURE-----
236
237=== added file 'lib/lp/soyuz/scripts/tests/sync_source_home/sample1_1.0.orig-component1.tar.bz2'
238Binary files lib/lp/soyuz/scripts/tests/sync_source_home/sample1_1.0.orig-component1.tar.bz2 1970-01-01 00:00:00 +0000 and lib/lp/soyuz/scripts/tests/sync_source_home/sample1_1.0.orig-component1.tar.bz2 2010-02-11 16:55:27 +0000 differ
239=== added file 'lib/lp/soyuz/scripts/tests/sync_source_home/sample1_1.0.orig-component2.tar.lzma'
240Binary files lib/lp/soyuz/scripts/tests/sync_source_home/sample1_1.0.orig-component2.tar.lzma 1970-01-01 00:00:00 +0000 and lib/lp/soyuz/scripts/tests/sync_source_home/sample1_1.0.orig-component2.tar.lzma 2010-02-11 16:55:27 +0000 differ
241=== added file 'lib/lp/soyuz/scripts/tests/sync_source_home/sample1_1.0.orig-component3.tar.gz'
242Binary files lib/lp/soyuz/scripts/tests/sync_source_home/sample1_1.0.orig-component3.tar.gz 1970-01-01 00:00:00 +0000 and lib/lp/soyuz/scripts/tests/sync_source_home/sample1_1.0.orig-component3.tar.gz 2010-02-11 16:55:27 +0000 differ
243=== added file 'lib/lp/soyuz/scripts/tests/sync_source_home/sample1_1.0.orig.tar.gz'
244Binary files lib/lp/soyuz/scripts/tests/sync_source_home/sample1_1.0.orig.tar.gz 1970-01-01 00:00:00 +0000 and lib/lp/soyuz/scripts/tests/sync_source_home/sample1_1.0.orig.tar.gz 2010-02-11 16:55:27 +0000 differ
245=== modified file 'lib/lp/soyuz/scripts/tests/test_sync_source.py'
246--- lib/lp/soyuz/scripts/tests/test_sync_source.py 2009-06-25 04:06:00 +0000
247+++ lib/lp/soyuz/scripts/tests/test_sync_source.py 2010-02-11 16:55:27 +0000
248@@ -1,7 +1,7 @@
249 # Copyright 2009 Canonical Ltd. This software is licensed under the
250 # GNU Affero General Public License version 3 (see the file LICENSE).
251
252-"""ChrootManager facilities tests."""
253+"""SyncSource facilities tests."""
254
255 __metaclass__ = type
256
257@@ -12,11 +12,15 @@
258 import tempfile
259 from unittest import TestCase, TestLoader
260
261-from lp.archiveuploader.tagfiles import parse_tagfile
262+from zope.component import getUtility
263+
264 from canonical.config import config
265+from canonical.launchpad.scripts import BufferLogger
266 from canonical.librarian.ftests.harness import (
267 fillLibrarianFile, cleanupLibrarianFiles)
268 from canonical.testing import LaunchpadZopelessLayer
269+from lp.archiveuploader.tagfiles import parse_tagfile
270+from lp.registry.interfaces.distribution import IDistributionSet
271 from lp.soyuz.scripts.ftpmaster import (
272 SyncSource, SyncSourceError)
273
274@@ -37,7 +41,7 @@
275 self._home = os.path.abspath('')
276 self._jail = tempfile.mkdtemp()
277 os.chdir(self._jail)
278- self.messages = []
279+ self.logger = BufferLogger()
280 self.downloads = []
281
282 def tearDown(self):
283@@ -54,9 +58,9 @@
284 """Return a list of files present in jail."""
285 return os.listdir(self._jail)
286
287- def local_debug(self, message):
288- """Store debug messages for future inspection."""
289- self.messages.append(message)
290+ def get_messages(self):
291+ """Retrieve the messages sent using the logger."""
292+ return self.logger.buffer.getvalue().splitlines()
293
294 def local_downloader(self, url, filename):
295 """Store download requests for future inspections."""
296@@ -72,8 +76,9 @@
297 them later.
298 """
299 sync_source = SyncSource(
300- files=files, origin=origin, debug=self.local_debug,
301- downloader=self.local_downloader)
302+ files=files, origin=origin, logger=self.logger,
303+ downloader=self.local_downloader,
304+ todistro=getUtility(IDistributionSet)['ubuntu'])
305 return sync_source
306
307 def testInstantiate(self):
308@@ -86,8 +91,8 @@
309 self.assertEqual(sync_source.files, files)
310 self.assertEqual(sync_source.origin, origin)
311
312- sync_source.debug('opa')
313- self.assertEqual(self.messages, ['opa'])
314+ sync_source.logger.debug('opa')
315+ self.assertEqual(self.get_messages(), ['DEBUG: opa'])
316
317 sync_source.downloader('somewhere', 'foo')
318 self.assertEqual(self.downloads, [('somewhere', 'foo')])
319@@ -159,26 +164,26 @@
320 test_file is skipped.
321 """
322 files = {
323- 'foo.diff.gz': {'remote filename': 'xxx'},
324- 'foo.dsc': {'remote filename': 'yyy'},
325- 'foo.orig.gz': {'remote filename': 'zzz'},
326+ 'foo_0.1.diff.gz': {'remote filename': 'xxx'},
327+ 'foo_0.1.dsc': {'remote filename': 'yyy'},
328+ 'foo_0.1.orig.gz': {'remote filename': 'zzz'},
329 }
330 origin = {'url': 'http://somewhere/'}
331
332 sync_source = self._getSyncSource(files, origin)
333
334- test_file = open('foo.diff.gz', 'w')
335+ test_file = open('foo_0.1.diff.gz', 'w')
336 test_file.write('nahhh')
337 test_file.close()
338
339 dsc_filename = sync_source.fetchSyncFiles()
340
341- self.assertEqual(dsc_filename, 'foo.dsc')
342+ self.assertEqual(dsc_filename, 'foo_0.1.dsc')
343
344 self.assertEqual(
345 self.downloads,
346- [('http://somewhere/yyy', 'foo.dsc'),
347- ('http://somewhere/zzz', 'foo.orig.gz')])
348+ [('http://somewhere/zzz', 'foo_0.1.orig.gz'),
349+ ('http://somewhere/yyy', 'foo_0.1.dsc')])
350
351 for filename in files.keys():
352 self.assertTrue(os.path.exists(filename))
353@@ -196,13 +201,13 @@
354 origin = {}
355 sync_source = self._getSyncSource(files, origin)
356
357- orig_filename = sync_source.fetchLibrarianFiles()
358+ librarian_files = sync_source.fetchLibrarianFiles()
359
360- self.assertEqual(orig_filename, 'netapplet_1.0.0.orig.tar.gz')
361+ self.assertEqual(librarian_files, ['netapplet_1.0.0.orig.tar.gz'])
362 self.assertEqual(self._listFiles(), ['netapplet_1.0.0.orig.tar.gz'])
363 self.assertEqual(
364- self.messages,
365- ['\tnetapplet_1.0.0.orig.tar.gz: already in distro '
366+ self.get_messages(),
367+ ['INFO: netapplet_1.0.0.orig.tar.gz: already in distro '
368 '- downloading from librarian'])
369
370 def testFetchLibrarianFilesGotDuplicatedDSC(self):
371@@ -224,8 +229,8 @@
372 sync_source.fetchLibrarianFiles)
373
374 self.assertEqual(
375- self.messages,
376- ['\tfoobar-1.0.dsc: already in distro '
377+ self.get_messages(),
378+ ['INFO: foobar-1.0.dsc: already in distro '
379 '- downloading from librarian'])
380 self.assertEqual(self._listFiles(), ['foobar-1.0.dsc'])
381
382@@ -261,7 +266,7 @@
383 stdout, stderr = process.communicate()
384 return (process.returncode, stdout, stderr)
385
386- def testSyncSourceRun(self):
387+ def testSyncSourceRunV1(self):
388 """Try a simple sync-source.py run.
389
390 It will run in a special tree prepared to cope with sync-source
391@@ -288,15 +293,17 @@
392 self.assertEqual(
393 err.splitlines(),
394 ['W: Could not find blacklist file on '
395- '/srv/launchpad.net/dak/sync-blacklist.txt'])
396+ '/srv/launchpad.net/dak/sync-blacklist.txt',
397+ 'INFO - <bar_1.0-1.diff.gz: cached>',
398+ 'INFO - <bar_1.0.orig.tar.gz: cached>',
399+ 'INFO - <bar_1.0-1.dsc: cached>',
400+ ])
401 self.assertEqual(
402 out.splitlines(),
403 ['Getting binaries for hoary...',
404 '[Updating] bar (0 [Ubuntu] < 1.0-1 [Debian])',
405 ' * Trying to add bar...',
406- ' - <bar_1.0-1.diff.gz: cached>',
407- ' - <bar_1.0.orig.tar.gz: cached>',
408- ' - <bar_1.0-1.dsc: cached>'])
409+ ])
410
411 expected_changesfile = 'bar_1.0-1_source.changes'
412 self.assertTrue(
413@@ -331,6 +338,76 @@
414
415 os.unlink(expected_changesfile)
416
417+ def testSyncSourceRunV3(self):
418+ """Try a simple sync-source.py run with a version 3 source format
419+ package.
420+
421+ It will run in a special tree prepared to cope with sync-source
422+ requirements (see `setUp`). It contains a usable archive index
423+ named as '$distribution_$suite_$component_Sources' and the
424+ 'etherwake' source files.
425+
426+ Check that:
427+ * return code is ZERO,
428+ * check standard error and standard output,
429+ * check if the expected changesfile was generated,
430+ * parse and inspect the changesfile using the archiveuploader
431+ component (the same approach adopted by Soyuz).
432+ * delete the changesfile.
433+ """
434+ returncode, out, err = self.runSyncSource(
435+ extra_args=['-b', 'cprov', '-D', 'debian', '-C', 'main',
436+ '-S', 'incoming', 'sample1'])
437+
438+ self.assertEqual(
439+ 0, returncode, "\nScript Failed:%s\nStdout:\n%s\nStderr\n%s\n"
440+ % (returncode, out, err))
441+
442+ self.assertEqual(
443+ err.splitlines(),
444+ ['W: Could not find blacklist file on '
445+ '/srv/launchpad.net/dak/sync-blacklist.txt',
446+ 'INFO - <sample1_1.0.orig-component3.tar.gz: cached>',
447+ 'INFO - <sample1_1.0-1.dsc: cached>',
448+ 'INFO - <sample1_1.0-1.debian.tar.gz: cached>',
449+ 'INFO - <sample1_1.0.orig-component1.tar.bz2: cached>',
450+ 'INFO - <sample1_1.0.orig-component2.tar.lzma: cached>',
451+ 'INFO - <sample1_1.0.orig.tar.gz: cached>'])
452+ self.assertEqual(
453+ out.splitlines(),
454+ ['Getting binaries for hoary...',
455+ '[Updating] sample1 (0 [Ubuntu] < 1.0-1 [Debian])',
456+ ' * Trying to add sample1...',
457+ ])
458+
459+ expected_changesfile = 'sample1_1.0-1_source.changes'
460+ self.assertTrue(
461+ os.path.exists(expected_changesfile),
462+ "Couldn't find %s." % expected_changesfile)
463+
464+ # Parse the generated unsigned changesfile.
465+ parsed_changes = parse_tagfile(
466+ expected_changesfile, allow_unsigned=True)
467+
468+ # It refers to the right source/version.
469+ self.assertEqual(parsed_changes['source'], 'sample1')
470+ self.assertEqual(parsed_changes['version'], '1.0-1')
471+
472+ # It includes the correct 'origin' and 'target' information.
473+ self.assertEqual(parsed_changes['origin'], 'Debian/incoming')
474+ self.assertEqual(parsed_changes['distribution'], 'hoary')
475+
476+ # And finally, 'maintainer' role was preserved and 'changed-by'
477+ # role was assigned as specified in the sync-source command-line.
478+ self.assertEqual(
479+ parsed_changes['maintainer'],
480+ 'Raphael Hertzog <hertzog@debian.org>')
481+ self.assertEqual(
482+ parsed_changes['changed-by'],
483+ 'Celso Providelo <celso.providelo@canonical.com>')
484+
485+ os.unlink(expected_changesfile)
486+
487
488 def test_suite():
489 return TestLoader().loadTestsFromName(__name__)
490
491=== modified file 'scripts/ftpmaster-tools/dak_utils.py'
492--- scripts/ftpmaster-tools/dak_utils.py 2009-11-18 02:58:23 +0000
493+++ scripts/ftpmaster-tools/dak_utils.py 2010-02-11 16:55:27 +0000
494@@ -34,156 +34,6 @@
495 re_no_epoch = re.compile(r"^\d+\:")
496 re_extract_src_version = re.compile(r"(\S+)\s*\((.*)\)")
497
498-changes_parse_error_exc = "Can't parse line in .changes file"
499-invalid_dsc_format_exc = "Invalid .dsc file"
500-nk_format_exc = "Unknown Format: in .changes file";
501-no_files_exc = "No Files: field in .dsc or .changes file.";
502-
503-################################################################################
504-
505-def parse_changes(filename, signing_rules=0):
506- """Parses a changes file and returns a dictionary where each field is a
507-key. The mandatory first argument is the filename of the .changes
508-file.
509-
510-signing_rules is an optional argument:
511-
512- o If signing_rules == -1, no signature is required.
513- o If signing_rules == 0 (the default), a signature is required.
514- o If signing_rules == 1, it turns on the same strict format checking
515- as dpkg-source.
516-
517-The rules for (signing_rules == 1)-mode are:
518-
519- o The PGP header consists of "-----BEGIN PGP SIGNED MESSAGE-----"
520- followed by any PGP header data and must end with a blank line.
521-
522- o The data section must end with a blank line and must be followed by
523- "-----BEGIN PGP SIGNATURE-----".
524-"""
525-
526- error = ""
527- changes = {}
528-
529- changes_in = open(filename)
530- lines = changes_in.readlines()
531-
532- if not lines:
533- raise changes_parse_error_exc, "[Empty changes file]"
534-
535- # Reindex by line number so we can easily verify the format of
536- # .dsc files...
537- index = 0
538- indexed_lines = {}
539- for line in lines:
540- index += 1
541- indexed_lines[index] = line[:-1]
542-
543- inside_signature = 0
544-
545- num_of_lines = len(indexed_lines.keys())
546- index = 0
547- first = -1
548- while index < num_of_lines:
549- index += 1
550- line = indexed_lines[index]
551- if line == "":
552- if signing_rules == 1:
553- index += 1
554- if index > num_of_lines:
555- raise invalid_dsc_format_exc, index
556- line = indexed_lines[index]
557- if not line.startswith("-----BEGIN PGP SIGNATURE"):
558- raise invalid_dsc_format_exc, index
559- inside_signature = 0
560- break
561- else:
562- continue
563- if line.startswith("-----BEGIN PGP SIGNATURE"):
564- break
565- if line.startswith("-----BEGIN PGP SIGNED MESSAGE"):
566- inside_signature = 1
567- if signing_rules == 1:
568- while index < num_of_lines and line != "":
569- index += 1
570- line = indexed_lines[index]
571- continue
572- # If we're not inside the signed data, don't process anything
573- if signing_rules >= 0 and not inside_signature:
574- continue
575- slf = re_single_line_field.match(line)
576- if slf:
577- field = slf.groups()[0].lower()
578- changes[field] = slf.groups()[1]
579- first = 1
580- continue
581- if line == " .":
582- changes[field] += '\n'
583- continue
584- mlf = re_multi_line_field.match(line)
585- if mlf:
586- if first == -1:
587- raise changes_parse_error_exc, "'%s'\n [Multi-line field continuing on from nothing?]" % (line)
588- if first == 1 and changes[field] != "":
589- changes[field] += '\n'
590- first = 0
591- changes[field] += mlf.groups()[0] + '\n'
592- continue
593- error += line
594-
595- if signing_rules == 1 and inside_signature:
596- raise invalid_dsc_format_exc, index
597-
598- changes_in.close()
599- changes["filecontents"] = "".join(lines)
600-
601- if error:
602- raise changes_parse_error_exc, error
603-
604- return changes
605-
606-################################################################################
607-
608-def build_file_list(changes, is_a_dsc=0):
609- files = {};
610-
611- # Make sure we have a Files: field to parse...
612- if not changes.has_key("files"):
613- raise no_files_exc;
614-
615- # Make sure we recognise the format of the Files: field
616- format = changes.get("format", "");
617- if format != "":
618- format = float(format.split()[0]);
619- if not is_a_dsc and (format < 1.5 or format > 2.0):
620- raise nk_format_exc, format;
621-
622- # Parse each entry/line:
623- for i in changes["files"].split('\n'):
624- if not i:
625- break;
626- s = i.split();
627- section = priority = "";
628- try:
629- if is_a_dsc:
630- (md5, size, name) = s;
631- else:
632- (md5, size, section, priority, name) = s;
633- except ValueError:
634- raise changes_parse_error_exc, i;
635-
636- if section == "":
637- section = "-";
638- if priority == "":
639- priority = "-";
640-
641- (section, component) = extract_component_from_section(section);
642-
643- files[name] = Dict(md5sum=md5, size=size, section=section,
644- priority=priority, component=component);
645-
646- return files
647-
648 ################################################################################
649
650 def fubar(msg, exit_code=1):
651@@ -309,9 +159,9 @@
652 ################################################################################
653
654 def join_with_commas_and(list):
655- if len(list) == 0: return "nothing";
656- if len(list) == 1: return list[0];
657- return ", ".join(list[:-1]) + " and " + list[-1];
658+ if len(list) == 0: return "nothing";
659+ if len(list) == 1: return list[0];
660+ return ", ".join(list[:-1]) + " and " + list[-1];
661
662 ################################################################################
663
664
665=== modified file 'scripts/ftpmaster-tools/sync-source.py'
666--- scripts/ftpmaster-tools/sync-source.py 2010-02-02 22:34:39 +0000
667+++ scripts/ftpmaster-tools/sync-source.py 2010-02-11 16:55:27 +0000
668@@ -18,6 +18,7 @@
669
670 import apt_pkg
671 import commands
672+from debian_bundle.deb822 import Dsc
673 import errno
674 import optparse
675 import os
676@@ -25,7 +26,6 @@
677 import shutil
678 import stat
679 import string
680-import sys
681 import tempfile
682 import time
683 import urllib
684@@ -44,6 +44,7 @@
685 from canonical.librarian.client import LibrarianClient
686 from canonical.lp import initZopeless
687 from lp.registry.interfaces.pocket import PackagePublishingPocket
688+from lp.soyuz.scripts.ftpmaster import SyncSource, SyncSourceError
689
690
691 reject_message = ""
692@@ -130,7 +131,7 @@
693
694 def generate_changes(dsc, dsc_files, suite, changelog, urgency, closes,
695 lp_closes, section, priority, description,
696- have_orig_tar_gz, requested_by, origin):
697+ files_from_librarian, requested_by, origin):
698 """Generate a .changes as a string"""
699
700 # XXX cprov 2007-07-03:
701@@ -164,7 +165,7 @@
702 changes += changelog
703 changes += "Files: \n"
704 for filename in dsc_files:
705- if filename.endswith(".orig.tar.gz") and have_orig_tar_gz:
706+ if filename in files_from_librarian:
707 continue
708 changes += " %s %s %s %s %s\n" % (dsc_files[filename]["md5sum"],
709 dsc_files[filename]["size"],
710@@ -361,13 +362,91 @@
711 source, source_component, binary, current_version,
712 current_component)
713
714+def split_gpg_and_payload(sequence):
715+ """Return a (gpg_pre, payload, gpg_post) tuple
716+
717+ Each element of the returned tuple is a list of lines (with trailing
718+ whitespace stripped).
719+ """
720+ # XXX JRV 20100211: Copied from deb822.py in python-debian. When
721+ # Launchpad switches to Lucid this copy should be removed.
722+ # bug=520508
723+
724+ gpg_pre_lines = []
725+ lines = []
726+ gpg_post_lines = []
727+ state = 'SAFE'
728+ gpgre = re.compile(r'^-----(?P<action>BEGIN|END) PGP (?P<what>[^-]+)-----$')
729+ blank_line = re.compile('^$')
730+ first_line = True
731+
732+ for line in sequence:
733+ line = line.strip('\r\n')
734+
735+ # skip initial blank lines, if any
736+ if first_line:
737+ if blank_line.match(line):
738+ continue
739+ else:
740+ first_line = False
741+
742+ m = gpgre.match(line)
743+
744+ if not m:
745+ if state == 'SAFE':
746+ if not blank_line.match(line):
747+ lines.append(line)
748+ else:
749+ if not gpg_pre_lines:
750+ # There's no gpg signature, so we should stop at
751+ # this blank line
752+ break
753+ elif state == 'SIGNED MESSAGE':
754+ if blank_line.match(line):
755+ state = 'SAFE'
756+ else:
757+ gpg_pre_lines.append(line)
758+ elif state == 'SIGNATURE':
759+ gpg_post_lines.append(line)
760+ else:
761+ if m.group('action') == 'BEGIN':
762+ state = m.group('what')
763+ elif m.group('action') == 'END':
764+ gpg_post_lines.append(line)
765+ break
766+ if not blank_line.match(line):
767+ if not lines:
768+ gpg_pre_lines.append(line)
769+ else:
770+ gpg_post_lines.append(line)
771+
772+ if len(lines):
773+ return (gpg_pre_lines, lines, gpg_post_lines)
774+ else:
775+ raise EOFError('only blank lines found in input')
776+
777
778 def import_dsc(dsc_filename, suite, previous_version, signing_rules,
779- have_orig_tar_gz, requested_by, origin, current_sources,
780+ files_from_librarian, requested_by, origin, current_sources,
781 current_binaries):
782- dsc = dak_utils.parse_changes(dsc_filename, signing_rules)
783- dsc_files = dak_utils.build_file_list(dsc, is_a_dsc=1)\
784-
785+ dsc_file = open(dsc_filename, 'r')
786+ dsc = Dsc(dsc_file)
787+
788+ if signing_rules.startswith("must be signed"):
789+ dsc_file.seek(0)
790+ # XXX JRV 20100211: When Launchpad starts depending on Lucid,
791+ # use dsc.split_gpg_and_payload() instead.
792+ # bug=520508
793+ (gpg_pre, payload, gpg_post) = split_gpg_and_payload(dsc_file)
794+ if gpg_pre == [] and gpg_post == []:
795+ dak_utils.fubar("signature required for %s but not present"
796+ % dsc_filename)
797+ if signing_rules == "must be signed and valid":
798+ if (gpg_pre[0] != "-----BEGIN PGP SIGNED MESSAGE-----" or
799+ gpg_post[0] != "-----BEGIN PGP SIGNATURE-----"):
800+ dak_utils.fubar("signature for %s invalid %r %r" % (dsc_filename, gpg_pre, gpg_post))
801+
802+ dsc_files = dict((entry['name'], entry) for entry in dsc['files'])
803 check_dsc(dsc, current_sources, current_binaries)
804
805 # Add the .dsc itself to dsc_files so it's listed in the Files: field
806@@ -400,7 +479,7 @@
807
808 changes = generate_changes(
809 dsc, dsc_files, suite, changelog, urgency, closes, lp_closes,
810- section, priority, description, have_orig_tar_gz, requested_by,
811+ section, priority, description, files_from_librarian, requested_by,
812 origin)
813
814 # XXX cprov 2007-07-03: Soyuz wants an unsigned changes
815@@ -565,86 +644,21 @@
816 if not Sources.has_key(pkg):
817 dak_utils.fubar("%s doesn't exist in the Sources file." % (pkg))
818
819- have_orig_tar_gz = False
820-
821- # Fetch the source
822- files = Sources[pkg]["files"]
823- for filename in files:
824- # First see if we can find the source in the librarian
825- archive_ids = [a.id for a in Options.todistro.all_distro_archives]
826- query = """
827- SELECT DISTINCT ON (LibraryFileContent.sha1, LibraryFileContent.filesize)
828- LibraryFileAlias.id
829- FROM SourcePackageFilePublishing, LibraryFileAlias, LibraryFileContent
830- WHERE
831- LibraryFileAlias.id = SourcePackageFilePublishing.libraryfilealias AND
832- LibraryFileContent.id = LibraryFileAlias.content AND
833- SourcePackageFilePublishing.libraryfilealiasfilename = %s AND
834- SourcePackageFilePublishing.archive IN %s
835- """ % sqlvalues(filename, archive_ids)
836- cur = cursor()
837- cur.execute(query)
838- results = cur.fetchall()
839- if results:
840- if not filename.endswith("orig.tar.gz"):
841- dak_utils.fubar(
842- "%s (from %s) is in the DB but isn't an orig.tar.gz. "
843- "(Probably published in an older release)"
844- % (filename, pkg))
845- if len(results) > 1:
846- dak_utils.fubar(
847- "%s (from %s) returns multiple IDs (%s) for "
848- "orig.tar.gz. Help?" % (filename, pkg, results))
849- have_orig_tar_gz = filename
850- print (" - <%s: already in distro - downloading from librarian>"
851- % (filename))
852- output_file = open(filename, 'w')
853- librarian_input = Library.getFileByAlias(results[0][0])
854- output_file.write(librarian_input.read())
855- output_file.close()
856- continue
857-
858- # Download the file
859- download_f = (
860- "%s%s" % (origin["url"], files[filename]["remote filename"]))
861- if not os.path.exists(filename):
862- print " - <%s: downloading from %s>" % (filename, origin["url"])
863- sys.stdout.flush()
864- urllib.urlretrieve(download_f, filename)
865- else:
866- print " - <%s: cached>" % (filename)
867-
868- # Check md5sum and size match Source
869- actual_md5sum = md5sum_file(filename)
870- expected_md5sum = files[filename]["md5sum"]
871- if actual_md5sum != expected_md5sum:
872- dak_utils.fubar(
873- "%s: md5sum check failed (%s [actual] vs. %s [expected])."
874- % (filename, actual_md5sum, expected_md5sum))
875- actual_size = os.stat(filename)[stat.ST_SIZE]
876- expected_size = int(files[filename]["size"])
877- if actual_size != expected_size:
878- dak_utils.fubar(
879- "%s: size mismatch (%s [actual] vs. %s [expected])."
880- % (filename, actual_size, expected_size))
881-
882- # Remember the name of the .dsc file
883- if filename.endswith(".dsc"):
884- dsc_filename = os.path.abspath(filename)
885-
886- if origin["dsc"] == "must be signed and valid":
887- signing_rules = 1
888- elif origin["dsc"] == "must be signed":
889- signing_rules = 0
890- else:
891- signing_rules = -1
892-
893- import_dsc(dsc_filename, suite, previous_version, signing_rules,
894- have_orig_tar_gz, requested_by, origin, current_sources,
895- current_binaries)
896-
897- if have_orig_tar_gz:
898- os.unlink(have_orig_tar_gz)
899+ syncsource = SyncSource(Sources[pkg]["files"], origin, Log,
900+ urllib.urlretrieve, Options.todistro)
901+ try:
902+ files_from_librarian = syncsource.fetchLibrarianFiles()
903+ dsc_filename = syncsource.fetchSyncFiles()
904+ syncsource.checkDownloadedFiles()
905+ except SyncSourceError, e:
906+ dak_utils.fubar("Fetching files failed: %s" % (str(e),))
907+
908+ if dsc_filename is None:
909+ dak_utils.fubar("No dsc filename in %r" % Sources[pkg]["files"].keys())
910+
911+ import_dsc(os.path.abspath(dsc_filename), suite, previous_version,
912+ origin["dsc"], files_from_librarian, requested_by, origin,
913+ current_sources, current_binaries)
914
915
916 def do_diff(Sources, Suite, origin, arguments, current_binaries):

Subscribers

People subscribed via source and target branches

to status/vote changes: