Launchpad itself

Merge lp:~wgrant/launchpad/distroseries-source-format-selection into lp:launchpad/db-devel

  1. distroseries-source-format-selection
  2. Merge into db-devel
Proposed by William Grant
Status: Merged
Merge reported by: William Grant
Merged at revision: not available
Proposed branch: lp:~wgrant/launchpad/distroseries-source-format-selection
Merge into: lp:launchpad/db-devel
Diff against target: 1205 lines (+727/-96)
20 files modified
lib/lp/archiveuploader/dscfile.py (+144/-58)
lib/lp/archiveuploader/nascentuploadfile.py (+8/-4)
lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_1.0-bzip2/bar_1.0-1.dsc (+10/-0)
lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_1.0-bzip2/bar_1.0-1_source.changes (+20/-0)
lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_3.0-quilt/bar_1.0-1.dsc (+9/-3)
lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_3.0-quilt/bar_1.0-1_source.changes (+12/-6)
lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig/bar_1.0-2.dsc (+22/-0)
lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig/bar_1.0-2_source.changes (+25/-0)
lib/lp/archiveuploader/tests/data/suite/bar_1.0_3.0-native/bar_1.0.dsc (+13/-0)
lib/lp/archiveuploader/tests/data/suite/bar_1.0_3.0-native/bar_1.0_source.changes (+25/-0)
lib/lp/archiveuploader/tests/nascentuploadfile.txt (+172/-0)
lib/lp/archiveuploader/tests/test_ppauploadprocessor.py (+47/-0)
lib/lp/archiveuploader/tests/test_uploadprocessor.py (+119/-0)
lib/lp/archiveuploader/tests/test_utils.py (+46/-12)
lib/lp/archiveuploader/uploadpolicy.py (+4/-0)
lib/lp/archiveuploader/utils.py (+13/-6)
lib/lp/registry/interfaces/sourcepackage.py (+19/-1)
lib/lp/soyuz/interfaces/files.py (+6/-1)
lib/lp/soyuz/model/files.py (+12/-2)
lib/lp/soyuz/model/queue.py (+1/-3)
To merge this branch: bzr merge lp:~wgrant/launchpad/distroseries-source-format-selection
Reviewer Review Type Date Requested Status
Julian Edwards (community) code Approve
Review via email: mp+14977@code.launchpad.net
To post a comment you must log in.
Revision history for this message
William Grant (wgrant) wrote :

This is hopefully the last of my Debian source format 3.0 branches. It is a pretty simple extension and testing of existing code.

The two new formats are:

 - 3.0 (native): Like 1.0's native variant, but with bz2 compression permitted on the tarball.
 - 3.0 (quilt): Replaces 1.0's non-native variant. Requires an orig.tar.(gz|bz2), a debian.tar.(gz|bz2) (replacing 1.0's diff.gz), and at most one orig-$COMPONENT.tar.(gz|bz2) for each COMPONENT.

determine_source_file_type is extended to identify bz2 compression, debian tarballs, and component orig tarballs. All checks to allow duplicated ORIG_TARBALLS are extended to also permit COMPONENT_ORIG_TARBALLs, as they should be treated identically.

There are pretty simple additional checks in NascentUpload. Additional file type counters are added, as is one for bz2 compression. Compression and new file checks are added to the 1.0 section, as are fairly obvious new 3.0 (quilt) and 3.0 (native) sections.

A few new test source packages are required. I also convinced the 'absolutely-anything' upload policy to permit PPA uploads, since I needed to test with an unsigned PPA upload. I haven't tested all filetype-related rejection cases (the old code did not either), since that would require constructing several more bad source packages.

Revision history for this message
Julian Edwards (julian-edwards) wrote :
Download full text (41.8 KiB)

Thanks for making this change William. I've commented inline.

> === modified file 'lib/lp/archiveuploader/dscfile.py'
> --- lib/lp/archiveuploader/dscfile.py 2009-11-16 22:06:14 +0000
> +++ lib/lp/archiveuploader/dscfile.py 2009-11-18 03:10:29 +0000
> @@ -32,7 +32,8 @@
> from lp.archiveuploader.utils import (
> prefix_multi_line_string, safe_fix_maintainer, ParseMaintError,
> re_valid_pkg_name, re_valid_version, re_issource,
> - determine_source_file_type)
> + re_is_component_orig_tar_ext, determine_source_file_type,
> + get_source_file_extension)

Can you put the imports in alphabetical order please.

> from canonical.encoding import guess as guess_encoding
> from lp.registry.interfaces.person import IPersonSet, PersonCreationRationale
> from lp.registry.interfaces.sourcepackage import SourcePackageFileType
> @@ -347,8 +348,9 @@
> distribution=self.policy.distro,
> purpose=ArchivePurpose.PARTNER)]
> elif (self.policy.archive.purpose == ArchivePurpose.PPA and
> - determine_source_file_type(filename) ==
> - SourcePackageFileType.ORIG_TARBALL):
> + determine_source_file_type(filename) in (
> + SourcePackageFileType.ORIG_TARBALL,
> + SourcePackageFileType.COMPONENT_ORIG_TARBALL)):
> archives = [self.policy.archive, self.policy.distro.main_archive]
> else:
> archives = [self.policy.archive]
> @@ -374,9 +376,13 @@
> """
>
> diff_count = 0
> + debian_tar_count = 0
> orig_tar_count = 0
> + component_orig_tar_counts = {}
> native_tar_count = 0
>
> + bzip2_count = 0
> +
> files_missing = False
> for sub_dsc_file in self.files:
> filetype = determine_source_file_type(sub_dsc_file.filename)
> @@ -385,12 +391,23 @@
> diff_count += 1
> elif filetype == SourcePackageFileType.ORIG_TARBALL:
> orig_tar_count += 1
> + elif filetype == SourcePackageFileType.COMPONENT_ORIG_TARBALL:
> + component = re_is_component_orig_tar_ext.match(
> + get_source_file_extension(sub_dsc_file.filename)).group(1)
> + if component not in component_orig_tar_counts:
> + component_orig_tar_counts[component] = 0
> + component_orig_tar_counts[component] += 1
> + elif filetype == SourcePackageFileType.DEBIAN_TARBALL:
> + debian_tar_count += 1
> elif filetype == SourcePackageFileType.NATIVE_TARBALL:
> native_tar_count += 1

This big list of "elif" statements is looking a bit unwieldy, especially with all the new conditions. Unless you can suggest anything better, how about we do something like:

            def orig_tarball_handler:
                orig_tar_count += 1

            def ....

            filetype_to_action_map = {
                SourcePackageFileType.ORIG_TARBALL : orig_tarball_handler,
                ...
                }

            # Or you could use lambda funcs there --^ it's probably tidier.

            file_type...

review: Needs Fixing (code)
Revision history for this message
William Grant (wgrant) wrote :
Download full text (9.4 KiB)

On Tue, 2009-11-24 at 23:20 +0000, Julian Edwards wrote:
> Review: Needs Fixing code
> Thanks for making this change William. I've commented inline.

Thanks for the review, Julian. I've fixed the main issues -- an
intermediate diff is attached.

> > === modified file 'lib/lp/archiveuploader/dscfile.py'
> > --- lib/lp/archiveuploader/dscfile.py 2009-11-16 22:06:14 +0000
> > +++ lib/lp/archiveuploader/dscfile.py 2009-11-18 03:10:29 +0000
> > @@ -32,7 +32,8 @@
> > from lp.archiveuploader.utils import (
> > prefix_multi_line_string, safe_fix_maintainer, ParseMaintError,
> > re_valid_pkg_name, re_valid_version, re_issource,
> > - determine_source_file_type)
> > + re_is_component_orig_tar_ext, determine_source_file_type,
> > + get_source_file_extension)
>
> Can you put the imports in alphabetical order please.

Done.

> > from canonical.encoding import guess as guess_encoding
> > from lp.registry.interfaces.person import IPersonSet, PersonCreationRationale
> > from lp.registry.interfaces.sourcepackage import SourcePackageFileType
> > @@ -347,8 +348,9 @@
> > distribution=self.policy.distro,
> > purpose=ArchivePurpose.PARTNER)]
> > elif (self.policy.archive.purpose == ArchivePurpose.PPA and
> > - determine_source_file_type(filename) ==
> > - SourcePackageFileType.ORIG_TARBALL):
> > + determine_source_file_type(filename) in (
> > + SourcePackageFileType.ORIG_TARBALL,
> > + SourcePackageFileType.COMPONENT_ORIG_TARBALL)):
> > archives = [self.policy.archive, self.policy.distro.main_archive]
> > else:
> > archives = [self.policy.archive]
> > @@ -374,9 +376,13 @@
> > """
> >
> > diff_count = 0
> > + debian_tar_count = 0
> > orig_tar_count = 0
> > + component_orig_tar_counts = {}
> > native_tar_count = 0
> >
> > + bzip2_count = 0
> > +
> > files_missing = False
> > for sub_dsc_file in self.files:
> > filetype = determine_source_file_type(sub_dsc_file.filename)
> > @@ -385,12 +391,23 @@
> > diff_count += 1
> > elif filetype == SourcePackageFileType.ORIG_TARBALL:
> > orig_tar_count += 1
> > + elif filetype == SourcePackageFileType.COMPONENT_ORIG_TARBALL:
> > + component = re_is_component_orig_tar_ext.match(
> > + get_source_file_extension(sub_dsc_file.filename)).group(1)
> > + if component not in component_orig_tar_counts:
> > + component_orig_tar_counts[component] = 0
> > + component_orig_tar_counts[component] += 1
> > + elif filetype == SourcePackageFileType.DEBIAN_TARBALL:
> > + debian_tar_count += 1
> > elif filetype == SourcePackageFileType.NATIVE_TARBALL:
> > native_tar_count += 1
>
> This big list of "elif" statements is looking a bit unwieldy, especially with all the new conditions. Unless you can suggest anything better, how about we do something like:
> [snip]

While your suggested solution would...

Read more...

8656..8661.diff
1=== modified file 'lib/lp/archiveuploader/dscfile.py'
2--- lib/lp/archiveuploader/dscfile.py 2009-11-18 02:58:23 +0000
3+++ lib/lp/archiveuploader/dscfile.py 2009-11-25 09:40:30 +0000
4@@ -30,10 +30,9 @@
5 from lp.archiveuploader.tagfiles import (
6 parse_tagfile, TagFileParseError)
7 from lp.archiveuploader.utils import (
8- prefix_multi_line_string, safe_fix_maintainer, ParseMaintError,
9- re_valid_pkg_name, re_valid_version, re_issource,
10- re_is_component_orig_tar_ext, determine_source_file_type,
11- get_source_file_extension)
12+ determine_source_file_type, get_source_file_extension,
13+ ParseMaintError, prefix_multi_line_string, re_is_component_orig_tar_ext,
14+ re_issource, re_valid_pkg_name, re_valid_version, safe_fix_maintainer)
15 from canonical.encoding import guess as guess_encoding
16 from lp.registry.interfaces.person import IPersonSet, PersonCreationRationale
17 from lp.registry.interfaces.sourcepackage import SourcePackageFileType
18@@ -161,6 +160,9 @@
19
20 Can raise UploadError.
21 """
22+ # Avoid circular imports.
23+ from lp.archiveuploader.nascentupload import EarlyReturnUploadError
24+
25 SourceUploadFile.__init__(
26 self, filepath, digest, size, component_and_section, priority,
27 package, version, changes, policy, logger)
28@@ -187,6 +189,10 @@
29 if 'format' not in self._dict:
30 self._dict['format'] = "1.0"
31
32+ if self.format is None:
33+ raise EarlyReturnUploadError(
34+ "Unsupported source format: %s" % self._dict['format'])
35+
36 if self.policy.unsigned_dsc_ok:
37 self.logger.debug("DSC file can be unsigned.")
38 else:
39@@ -209,7 +215,11 @@
40 @property
41 def format(self):
42 """Return the DSC format."""
43- return self._dict['format']
44+ try:
45+ return SourcePackageFormat.getTermByToken(
46+ self._dict['format']).value
47+ except LookupError:
48+ return None
49
50 @property
51 def architecture(self):
52@@ -232,8 +242,6 @@
53 This method is an error generator, i.e, it returns an iterator over all
54 exceptions that are generated while processing DSC file checks.
55 """
56- # Avoid circular imports.
57- from lp.archiveuploader.nascentupload import EarlyReturnUploadError
58
59 for error in SourceUploadFile.verify(self):
60 yield error
61@@ -272,14 +280,8 @@
62 yield UploadError(
63 "%s: invalid version %s" % (self.filename, self.dsc_version))
64
65- try:
66- format_term = SourcePackageFormat.getTermByToken(self.format)
67- except LookupError:
68- raise EarlyReturnUploadError(
69- "Unsupported source format: %s" % self.format)
70-
71 if not self.policy.distroseries.isSourcePackageFormatPermitted(
72- format_term.value):
73+ self.format):
74 yield UploadError(
75 "%s: format '%s' is not permitted in %s." %
76 (self.filename, self.format, self.policy.distroseries.name))
77@@ -375,35 +377,32 @@
78 and checksum.
79 """
80
81- diff_count = 0
82- debian_tar_count = 0
83- orig_tar_count = 0
84+ file_type_counts = {
85+ SourcePackageFileType.DIFF: 0,
86+ SourcePackageFileType.ORIG_TARBALL: 0,
87+ SourcePackageFileType.DEBIAN_TARBALL: 0,
88+ SourcePackageFileType.NATIVE_TARBALL: 0,
89+ }
90 component_orig_tar_counts = {}
91- native_tar_count = 0
92-
93 bzip2_count = 0
94-
95 files_missing = False
96+
97 for sub_dsc_file in self.files:
98- filetype = determine_source_file_type(sub_dsc_file.filename)
99-
100- if filetype == SourcePackageFileType.DIFF:
101- diff_count += 1
102- elif filetype == SourcePackageFileType.ORIG_TARBALL:
103- orig_tar_count += 1
104- elif filetype == SourcePackageFileType.COMPONENT_ORIG_TARBALL:
105+ file_type = determine_source_file_type(sub_dsc_file.filename)
106+
107+ if file_type is None:
108+ yield UploadError('Unknown file: ' + sub_dsc_file.filename)
109+ continue
110+
111+ if file_type == SourcePackageFileType.COMPONENT_ORIG_TARBALL:
112+ # Split the count by component name.
113 component = re_is_component_orig_tar_ext.match(
114 get_source_file_extension(sub_dsc_file.filename)).group(1)
115 if component not in component_orig_tar_counts:
116 component_orig_tar_counts[component] = 0
117 component_orig_tar_counts[component] += 1
118- elif filetype == SourcePackageFileType.DEBIAN_TARBALL:
119- debian_tar_count += 1
120- elif filetype == SourcePackageFileType.NATIVE_TARBALL:
121- native_tar_count += 1
122 else:
123- yield UploadError('Unknown file: ' + sub_dsc_file.filename)
124- continue
125+ file_type_counts[file_type] += 1
126
127 if sub_dsc_file.filename.endswith('.bz2'):
128 bzip2_count += 1
129@@ -452,99 +451,10 @@
130 yield error
131 files_missing = True
132
133- # Reject if we have more than one file of any type.
134- if orig_tar_count > 1:
135- yield UploadError(
136- "%s: has more than one orig.tar.*."
137- % self.filename)
138- if debian_tar_count > 1:
139- yield UploadError(
140- "%s: has more than one debian.tar.*."
141- % self.filename)
142- if native_tar_count > 1:
143- yield UploadError(
144- "%s: has more than one tar.*."
145- % self.filename)
146- if diff_count > 1:
147- yield UploadError(
148- "%s: has more than one diff.gz."
149- % self.filename)
150-
151- if ((orig_tar_count == 0 and native_tar_count == 0) or
152- (orig_tar_count > 0 and native_tar_count > 0)):
153- yield UploadError(
154- "%s: must have exactly one tar.* or orig.tar.*."
155- % self.filename)
156-
157- if native_tar_count > 0 and debian_tar_count > 0:
158- yield UploadError(
159- "%s: must have no more than one tar.* or debian.tar.*."
160- % self.filename)
161-
162- # Format 1.0 must be native (exactly one tar.gz), or
163- # have an orig.tar.gz and a diff.gz. It cannot have
164- # compression types other than 'gz'.
165- if self.format == '1.0':
166- if bzip2_count > 0:
167- yield UploadError(
168- "%s: is format 1.0 but uses bzip2 compression."
169- % self.filename)
170-
171- if ((diff_count == 0 and native_tar_count == 0) or
172- (diff_count > 0 and native_tar_count > 0)):
173- yield UploadError(
174- "%s: must have exactly one diff.gz or tar.gz."
175- % self.filename)
176-
177- if debian_tar_count > 0:
178- yield UploadError(
179- "%s: is format 1.0 but has debian.tar.*."
180- % self.filename)
181-
182- if len(component_orig_tar_counts) > 0:
183- yield UploadError(
184- "%s: is format 1.0 but has orig-COMPONENT.tar.*."
185- % self.filename)
186- # Format 3.0 (native) must have exactly one tar.*.
187- # gz and bz2 are valid compression types.
188- elif self.format == '3.0 (native)':
189- if native_tar_count == 0:
190- yield UploadError(
191- "%s: must have exactly one tar.*."
192- % self.filename)
193-
194- if diff_count > 0:
195- yield UploadError(
196- "%s: is format 3.0 but has diff.gz."
197- % self.filename)
198-
199- if len(component_orig_tar_counts) > 0:
200- yield UploadError(
201- "%s: is native but has orig-COMPONENT.tar.*."
202- % self.filename)
203- elif self.format == '3.0 (quilt)':
204- if orig_tar_count == 0:
205- yield UploadError(
206- "%s: must have exactly one orig.tar.*."
207- % self.filename)
208-
209- if debian_tar_count == 0:
210- yield UploadError(
211- "%s: must have exactly one debian.tar.*."
212- % self.filename)
213-
214- if diff_count > 0:
215- yield UploadError(
216- "%s: is format 3.0 but has diff.gz."
217- % self.filename)
218-
219- for component in component_orig_tar_counts:
220- if component_orig_tar_counts[component] > 1:
221- yield UploadError(
222- "%s: has more than one orig-%s.tar.*."
223- % (self.filename, component))
224- else:
225- raise AssertionError("Unknown source format.")
226+ for error in format_to_file_checker_map[self.format](
227+ self.filename, file_type_counts, component_orig_tar_counts,
228+ bzip2_count):
229+ yield error
230
231 if files_missing:
232 yield UploadError(
233@@ -727,3 +637,78 @@
234 yield error
235
236
237+def check_format_1_0_files(filename, file_type_counts, component_counts,
238+ bzip2_count):
239+ """Check that the given counts of each file type suit format 1.0.
240+
241+ A 1.0 source must be native (with only one tar.gz), or have an orig.tar.gz
242+ and a diff.gz. It cannot use bzip2 compression.
243+ """
244+ if bzip2_count > 0:
245+ yield UploadError(
246+ "%s: is format 1.0 but uses bzip2 compression."
247+ % filename)
248+
249+ if file_type_counts not in ({
250+ SourcePackageFileType.NATIVE_TARBALL: 1,
251+ SourcePackageFileType.ORIG_TARBALL: 0,
252+ SourcePackageFileType.DEBIAN_TARBALL: 0,
253+ SourcePackageFileType.DIFF: 0,
254+ }, {
255+ SourcePackageFileType.ORIG_TARBALL: 1,
256+ SourcePackageFileType.DIFF: 1,
257+ SourcePackageFileType.NATIVE_TARBALL: 0,
258+ SourcePackageFileType.DEBIAN_TARBALL: 0,
259+ }) or len(component_counts) > 0:
260+ yield UploadError(
261+ "%s: must have exactly one tar.gz, or an orig.tar.gz and diff.gz"
262+ % filename)
263+
264+
265+def check_format_3_0_native_files(filename, file_type_counts,
266+ component_counts, bzip2_count):
267+ """Check that the given counts of each file type suit format 3.0 (native).
268+
269+ A 3.0 (native) source must have only one tar.*. Both gzip and bzip2
270+ compression are permissible.
271+ """
272+
273+ if file_type_counts != {
274+ SourcePackageFileType.NATIVE_TARBALL: 1,
275+ SourcePackageFileType.ORIG_TARBALL: 0,
276+ SourcePackageFileType.DEBIAN_TARBALL: 0,
277+ SourcePackageFileType.DIFF: 0,
278+ } or len(component_counts) > 0:
279+ yield UploadError("%s: must have only a tar.*." % filename)
280+
281+
282+def check_format_3_0_quilt_files(filename, file_type_counts,
283+ component_counts, bzip2_count):
284+ """Check that the given counts of each file type suit format 3.0 (native).
285+
286+ A 3.0 (quilt) source must have exactly one orig.tar.*, one debian.tar.*,
287+ and at most one orig-COMPONENT.tar.* for each COMPONENT. Both gzip and
288+ bzip2 compression are permissible.
289+ """
290+ if file_type_counts != {
291+ SourcePackageFileType.ORIG_TARBALL: 1,
292+ SourcePackageFileType.DEBIAN_TARBALL: 1,
293+ SourcePackageFileType.NATIVE_TARBALL: 0,
294+ SourcePackageFileType.DIFF: 0,
295+ }:
296+ yield UploadError(
297+ "%s: must have only an orig.tar.*, a debian.tar.*, and "
298+ "optionally orig-*.tar.*" % filename)
299+
300+ for component in component_counts:
301+ if component_counts[component] > 1:
302+ yield UploadError(
303+ "%s: has more than one orig-%s.tar.*."
304+ % (filename, component))
305+
306+
307+format_to_file_checker_map = {
308+ SourcePackageFormat.FORMAT_1_0: check_format_1_0_files,
309+ SourcePackageFormat.FORMAT_3_0_NATIVE: check_format_3_0_native_files,
310+ SourcePackageFormat.FORMAT_3_0_QUILT: check_format_3_0_quilt_files,
311+ }
Revision history for this message
Julian Edwards (julian-edwards) wrote :
Download full text (5.6 KiB)

> While your suggested solution would not work (at least not without the
> 'nonlocal' keyword), I believe I have found something better. I keep one
> dict mapping SPFTs to counts, and another mapping component names to
> counts. This means there's only one special case (for
> COMPONENT_ORIG_TARBALL), and the code for each format becomes much
> cleaner too.

Ah, this is 100% better than my suggestion, nice one!

> I've split it, and completely changed the verification style. The
> failure messages are much less verbose, but one would have to manually
> edit the .dsc to get any of them anyway. I feel the reduction in code
> makes things much better.

Actually it is not only better, it's easier to test because there are
fewer cases!

> This does make testing much easier.
>
> However, there are now just a few code paths, but a lot of potential
> failure cases. How completely should I test?

We always test 100% code coverage at least and any peculiar corner cases
that are obvious.

I'd be happy here to have 2 tests per format type; one that tests it accepts
correctly and one that tests it fails correctly, since that's pretty much
all the code is doing now.

> > Also, can you please change it to compare against the format enum instead
> > of strings. Then the assertion error below will make more sense.
> >
> > > else:
> > > raise AssertionError("Unknown source format.")
>
> It will now die with a KeyError when looking up the file checker, which
> seems as reasonable as the old explicit AssertionError.

I don't really like that, it's not as explicit as the old error, particularly
because the error will appear in the rejection email (unlikely as it is).

Can you catch the KeyError and re-raise it as AssertionError with some good
text. This will ensure if/when we add more format types, it will be easier to
track down why the failure occurred.

> The two uses are on different types: this is a SourceUploadFile, while
> the other is a SourcePackageReleaseFile. Should I instead create a
> utility function alongside SPFT, perhaps?

I would make a mix-in class that both can inherit from and put the property
on that.

> @@ -375,35 +377,32 @@
> and checksum.
> """
>
> - diff_count = 0
> - debian_tar_count = 0
> - orig_tar_count = 0
> + file_type_counts = {
> + SourcePackageFileType.DIFF: 0,
> + SourcePackageFileType.ORIG_TARBALL: 0,
> + SourcePackageFileType.DEBIAN_TARBALL: 0,
> + SourcePackageFileType.NATIVE_TARBALL: 0,
> + }
> component_orig_tar_counts = {}
> - native_tar_count = 0
> -
> bzip2_count = 0
> -

*Loads* better!

> files_missing = False
> +
> for sub_dsc_file in self.files:
> - filetype = determine_source_file_type(sub_dsc_file.filename)
> -
> - if filetype == SourcePackageFileType.DIFF:
> - diff_count += 1
> - elif filetype == SourcePackageFileType.ORIG_TARBALL:
> - orig_tar_count += 1
> - elif filetype == SourcePackageFileType.COMPONENT_ORIG_TARBALL:
> + file_type = determine_source_file_t...

Read more...

review: Needs Fixing (code)
Revision history for this message
William Grant (wgrant) wrote :
Download full text (6.2 KiB)

On Wed, 2009-11-25 at 23:03 +0000, Julian Edwards wrote:
> Review: Needs Fixing code
> > While your suggested solution would not work (at least not without the
> > 'nonlocal' keyword), I believe I have found something better. I keep one
> > dict mapping SPFTs to counts, and another mapping component names to
> > counts. This means there's only one special case (for
> > COMPONENT_ORIG_TARBALL), and the code for each format becomes much
> > cleaner too.
>
> Ah, this is 100% better than my suggestion, nice one!
>
> > I've split it, and completely changed the verification style. The
> > failure messages are much less verbose, but one would have to manually
> > edit the .dsc to get any of them anyway. I feel the reduction in code
> > makes things much better.
>
> Actually it is not only better, it's easier to test because there are
> fewer cases!
>
> > This does make testing much easier.
> >
> > However, there are now just a few code paths, but a lot of potential
> > failure cases. How completely should I test?
>
> We always test 100% code coverage at least and any peculiar corner cases
> that are obvious.
>
> I'd be happy here to have 2 tests per format type; one that tests it accepts
> correctly and one that tests it fails correctly, since that's pretty much
> all the code is doing now.

I've added tests in nascentuploadfile.txt.

> > > Also, can you please change it to compare against the format enum instead
> > > of strings. Then the assertion error below will make more sense.
> > >
> > > > else:
> > > > raise AssertionError("Unknown source format.")
> >
> > It will now die with a KeyError when looking up the file checker, which
> > seems as reasonable as the old explicit AssertionError.
>
> I don't really like that, it's not as explicit as the old error, particularly
> because the error will appear in the rejection email (unlikely as it is).
>
> Can you catch the KeyError and re-raise it as AssertionError with some good
> text. This will ensure if/when we add more format types, it will be easier to
> track down why the failure occurred.

Done.

> > The two uses are on different types: this is a SourceUploadFile, while
> > the other is a SourcePackageReleaseFile. Should I instead create a
> > utility function alongside SPFT, perhaps?
>
> I would make a mix-in class that both can inherit from and put the property
> on that.

As you suggested on IRC, I've put it in
lp.soyuz.model.files.SourceFileMixin.

> > @@ -375,35 +377,32 @@
> > and checksum.
> > """
> >
> > - diff_count = 0
> > - debian_tar_count = 0
> > - orig_tar_count = 0
> > + file_type_counts = {
> > + SourcePackageFileType.DIFF: 0,
> > + SourcePackageFileType.ORIG_TARBALL: 0,
> > + SourcePackageFileType.DEBIAN_TARBALL: 0,
> > + SourcePackageFileType.NATIVE_TARBALL: 0,
> > + }
> > component_orig_tar_counts = {}
> > - native_tar_count = 0
> > -
> > bzip2_count = 0
> > -
>
> *Loads* better!
>
> > files_missing = False
> > +
> > for sub_dsc_file in self.files:
> > - filetype = determine_source_fil...

Read more...

8661..8665.diff
1=== modified file 'lib/lp/archiveuploader/dscfile.py'
2--- lib/lp/archiveuploader/dscfile.py 2009-11-25 09:40:38 +0000
3+++ lib/lp/archiveuploader/dscfile.py 2009-11-26 00:42:45 +0000
4@@ -451,7 +451,13 @@
5 yield error
6 files_missing = True
7
8- for error in format_to_file_checker_map[self.format](
9+ try:
10+ file_checker = format_to_file_checker_map[self.format]
11+ except KeyError:
12+ raise AssertionError(
13+ "No file checker for source format %s." % self.format)
14+
15+ for error in file_checker(
16 self.filename, file_type_counts, component_orig_tar_counts,
17 bzip2_count):
18 yield error
19@@ -649,17 +655,23 @@
20 "%s: is format 1.0 but uses bzip2 compression."
21 % filename)
22
23- if file_type_counts not in ({
24- SourcePackageFileType.NATIVE_TARBALL: 1,
25- SourcePackageFileType.ORIG_TARBALL: 0,
26- SourcePackageFileType.DEBIAN_TARBALL: 0,
27- SourcePackageFileType.DIFF: 0,
28- }, {
29- SourcePackageFileType.ORIG_TARBALL: 1,
30- SourcePackageFileType.DIFF: 1,
31- SourcePackageFileType.NATIVE_TARBALL: 0,
32- SourcePackageFileType.DEBIAN_TARBALL: 0,
33- }) or len(component_counts) > 0:
34+ valid_file_type_counts = [
35+ {
36+ SourcePackageFileType.NATIVE_TARBALL: 1,
37+ SourcePackageFileType.ORIG_TARBALL: 0,
38+ SourcePackageFileType.DEBIAN_TARBALL: 0,
39+ SourcePackageFileType.DIFF: 0,
40+ },
41+ {
42+ SourcePackageFileType.ORIG_TARBALL: 1,
43+ SourcePackageFileType.DIFF: 1,
44+ SourcePackageFileType.NATIVE_TARBALL: 0,
45+ SourcePackageFileType.DEBIAN_TARBALL: 0,
46+ },
47+ ]
48+
49+ if (file_type_counts not in valid_file_type_counts or
50+ len(component_counts) > 0):
51 yield UploadError(
52 "%s: must have exactly one tar.gz, or an orig.tar.gz and diff.gz"
53 % filename)
54@@ -673,12 +685,17 @@
55 compression are permissible.
56 """
57
58- if file_type_counts != {
59- SourcePackageFileType.NATIVE_TARBALL: 1,
60- SourcePackageFileType.ORIG_TARBALL: 0,
61- SourcePackageFileType.DEBIAN_TARBALL: 0,
62- SourcePackageFileType.DIFF: 0,
63- } or len(component_counts) > 0:
64+ valid_file_type_counts = [
65+ {
66+ SourcePackageFileType.NATIVE_TARBALL: 1,
67+ SourcePackageFileType.ORIG_TARBALL: 0,
68+ SourcePackageFileType.DEBIAN_TARBALL: 0,
69+ SourcePackageFileType.DIFF: 0,
70+ },
71+ ]
72+
73+ if (file_type_counts not in valid_file_type_counts or
74+ len(component_counts) > 0):
75 yield UploadError("%s: must have only a tar.*." % filename)
76
77
78@@ -690,12 +707,17 @@
79 and at most one orig-COMPONENT.tar.* for each COMPONENT. Both gzip and
80 bzip2 compression are permissible.
81 """
82- if file_type_counts != {
83- SourcePackageFileType.ORIG_TARBALL: 1,
84- SourcePackageFileType.DEBIAN_TARBALL: 1,
85- SourcePackageFileType.NATIVE_TARBALL: 0,
86- SourcePackageFileType.DIFF: 0,
87- }:
88+
89+ valid_file_type_counts = [
90+ {
91+ SourcePackageFileType.ORIG_TARBALL: 1,
92+ SourcePackageFileType.DEBIAN_TARBALL: 1,
93+ SourcePackageFileType.NATIVE_TARBALL: 0,
94+ SourcePackageFileType.DIFF: 0,
95+ },
96+ ]
97+
98+ if file_type_counts not in valid_file_type_counts:
99 yield UploadError(
100 "%s: must have only an orig.tar.*, a debian.tar.*, and "
101 "optionally orig-*.tar.*" % filename)
102
103=== modified file 'lib/lp/archiveuploader/nascentuploadfile.py'
104--- lib/lp/archiveuploader/nascentuploadfile.py 2009-11-14 02:55:15 +0000
105+++ lib/lp/archiveuploader/nascentuploadfile.py 2009-11-25 23:38:51 +0000
106@@ -35,7 +35,6 @@
107 re_no_epoch, re_no_revision, re_valid_version, re_valid_pkg_name,
108 re_extract_src_version, determine_source_file_type)
109 from canonical.encoding import guess as guess_encoding
110-from lp.registry.interfaces.sourcepackage import SourcePackageFileType
111 from lp.soyuz.interfaces.binarypackagename import (
112 IBinaryPackageNameSet)
113 from lp.soyuz.interfaces.binarypackagerelease import (
114@@ -49,6 +48,7 @@
115 from lp.soyuz.interfaces.publishing import (
116 PackagePublishingPriority)
117 from lp.soyuz.interfaces.section import ISectionSet
118+from lp.soyuz.model.files import SourceFileMixin
119 from canonical.librarian.utils import filechunks
120
121
122@@ -332,7 +332,7 @@
123 return getUtility(ISectionSet)[self.section_name]
124
125
126-class SourceUploadFile(PackageUploadFile):
127+class SourceUploadFile(SourceFileMixin, PackageUploadFile):
128 """Files mentioned in changesfile as source (orig, diff, tar).
129
130 This class only check consistency on information contained in
131@@ -340,6 +340,11 @@
132 Further checks on file contents and package consistency are done
133 in DSCFile.
134 """
135+
136+ @property
137+ def filetype(self):
138+ return determine_source_file_type(self.filename)
139+
140 def verify(self):
141 """Verify the uploaded source file.
142
143@@ -352,9 +357,7 @@
144 "Architecture field." % (self.filename))
145
146 version_chopped = re_no_epoch.sub('', self.version)
147- if determine_source_file_type(self.filename) in (
148- SourcePackageFileType.ORIG_TARBALL,
149- SourcePackageFileType.COMPONENT_ORIG_TARBALL):
150+ if self.is_orig:
151 version_chopped = re_no_revision.sub('', version_chopped)
152
153 source_match = re_issource.match(self.filename)
154
155=== modified file 'lib/lp/archiveuploader/tests/nascentuploadfile.txt'
156--- lib/lp/archiveuploader/tests/nascentuploadfile.txt 2009-10-28 05:24:11 +0000
157+++ lib/lp/archiveuploader/tests/nascentuploadfile.txt 2009-11-26 00:36:18 +0000
158@@ -569,6 +569,115 @@
159 ['File ed_0.2-20.dsc mentioned in the changes has a size mismatch. 578 != 500']
160
161
162+=== Format file type verification ===
163+
164+DSCFile performs additional verification on the types of the referenced
165+files, confirming that they are suitable for the source package's
166+format. There is an error generator to verify each format.
167+
168+ >>> from lp.archiveuploader.dscfile import (check_format_1_0_files,
169+ ... check_format_3_0_native_files, check_format_3_0_quilt_files)
170+ >>> from lp.registry.interfaces.sourcepackage import SourcePackageFileType
171+
172+==== 1.0 ====
173+
174+A 1.0 source can contain either a tar.gz or an orig.tar.gz and diff.gz.
175+
176+ >>> list(check_format_1_0_files('foo_1.dsc', {
177+ ... SourcePackageFileType.ORIG_TARBALL: 1,
178+ ... SourcePackageFileType.DIFF: 1,
179+ ... SourcePackageFileType.DEBIAN_TARBALL: 0,
180+ ... SourcePackageFileType.NATIVE_TARBALL: 0,
181+ ... }, {}, 0))
182+ []
183+
184+ >>> list(check_format_1_0_files('foo_1.dsc', {
185+ ... SourcePackageFileType.NATIVE_TARBALL: 1,
186+ ... SourcePackageFileType.ORIG_TARBALL: 0,
187+ ... SourcePackageFileType.DIFF: 0,
188+ ... SourcePackageFileType.DEBIAN_TARBALL: 0,
189+ ... }, {}, 0))
190+ []
191+
192+But if we have some other combination, or bzip2 compression, errors
193+will be generated.
194+
195+ >>> list(check_format_1_0_files('foo_1.dsc', {
196+ ... SourcePackageFileType.NATIVE_TARBALL: 1,
197+ ... SourcePackageFileType.ORIG_TARBALL: 1,
198+ ... SourcePackageFileType.DIFF: 1,
199+ ... SourcePackageFileType.DEBIAN_TARBALL: 0,
200+ ... }, {}, 1))
201+ [UploadError('foo_1.dsc: is format 1.0 but uses bzip2 compression.',), UploadError('foo_1.dsc: must have exactly one tar.gz, or an orig.tar.gz and diff.gz',)]
202+
203+The files are also bad if there are any components:
204+
205+ >>> list(check_format_1_0_files('foo_1.dsc', {
206+ ... SourcePackageFileType.ORIG_TARBALL: 1,
207+ ... SourcePackageFileType.DIFF: 1,
208+ ... SourcePackageFileType.DEBIAN_TARBALL: 0,
209+ ... SourcePackageFileType.NATIVE_TARBALL: 0,
210+ ... }, {'foo': 1}, 0))
211+ [UploadError('foo_1.dsc: must have exactly one tar.gz, or an orig.tar.gz and diff.gz',)]
212+
213+==== 3.0 (native) ====
214+
215+A 3.0 (native) source must contain just a tar.(gz|bz2).
216+
217+ >>> list(check_format_3_0_native_files('foo_1.dsc', {
218+ ... SourcePackageFileType.NATIVE_TARBALL: 1,
219+ ... SourcePackageFileType.ORIG_TARBALL: 0,
220+ ... SourcePackageFileType.DIFF: 0,
221+ ... SourcePackageFileType.DEBIAN_TARBALL: 0,
222+ ... }, {}, 1))
223+ []
224+
225+ >>> list(check_format_3_0_native_files('foo_1.dsc', {
226+ ... SourcePackageFileType.NATIVE_TARBALL: 1,
227+ ... SourcePackageFileType.ORIG_TARBALL: 1,
228+ ... SourcePackageFileType.DIFF: 0,
229+ ... SourcePackageFileType.DEBIAN_TARBALL: 0,
230+ ... }, {}, 1))
231+ [UploadError('foo_1.dsc: must have only a tar.*.',)]
232+
233+ >>> list(check_format_3_0_native_files('foo_1.dsc', {
234+ ... SourcePackageFileType.NATIVE_TARBALL: 1,
235+ ... SourcePackageFileType.ORIG_TARBALL: 0,
236+ ... SourcePackageFileType.DIFF: 0,
237+ ... SourcePackageFileType.DEBIAN_TARBALL: 0,
238+ ... }, {'foo': 1}, 0))
239+ [UploadError('foo_1.dsc: must have only a tar.*.',)]
240+
241+==== 3.0 (quilt) ====
242+
243+A 3.0 (quilt) source must have an orig.tar.*, a debian.tar.*, and at
244+most one orig-COMPONENT.tar.* for each COMPONENT.
245+
246+ >>> list(check_format_3_0_quilt_files('foo_1.dsc', {
247+ ... SourcePackageFileType.ORIG_TARBALL: 1,
248+ ... SourcePackageFileType.DEBIAN_TARBALL: 1,
249+ ... SourcePackageFileType.NATIVE_TARBALL: 0,
250+ ... SourcePackageFileType.DIFF: 0,
251+ ... }, {'foo': 1}, 1))
252+ []
253+
254+ >>> list(check_format_3_0_quilt_files('foo_1.dsc', {
255+ ... SourcePackageFileType.NATIVE_TARBALL: 1,
256+ ... SourcePackageFileType.ORIG_TARBALL: 1,
257+ ... SourcePackageFileType.DIFF: 0,
258+ ... SourcePackageFileType.DEBIAN_TARBALL: 1,
259+ ... }, {}, 1))
260+ [UploadError('foo_1.dsc: must have only an orig.tar.*, a debian.tar.*, and optionally orig-*.tar.*',)]
261+
262+ >>> list(check_format_3_0_quilt_files('foo_1.dsc', {
263+ ... SourcePackageFileType.ORIG_TARBALL: 1,
264+ ... SourcePackageFileType.DEBIAN_TARBALL: 1,
265+ ... SourcePackageFileType.NATIVE_TARBALL: 0,
266+ ... SourcePackageFileType.DIFF: 0,
267+ ... }, {'foo': 2}, 0))
268+ [UploadError('foo_1.dsc: has more than one orig-foo.tar.*.',)]
269+
270+
271 === Sub-DSC files or DSCUploadedFiles ===
272
273 Sub-DSCFiles are DSCUploadedFile objects.
274
275=== modified file 'lib/lp/soyuz/interfaces/files.py'
276--- lib/lp/soyuz/interfaces/files.py 2009-06-25 04:06:00 +0000
277+++ lib/lp/soyuz/interfaces/files.py 2009-11-25 23:23:30 +0000
278@@ -14,7 +14,7 @@
279 'ISourcePackageReleaseFileSet',
280 ]
281
282-from zope.schema import Int
283+from zope.schema import Bool, Int
284 from zope.interface import Interface
285 from canonical.launchpad import _
286
287@@ -68,6 +68,11 @@
288 title=_('The type of this file'), required=True, readonly=False,
289 )
290
291+ is_orig = Bool(
292+ title=_('Whether this file is an original tarball'),
293+ required=True, readonly=False,
294+ )
295+
296
297 class ISourcePackageReleaseFileSet(Interface):
298 """The set of all `SourcePackageRelease`s."""
299
300=== modified file 'lib/lp/soyuz/model/files.py'
301--- lib/lp/soyuz/model/files.py 2009-06-25 04:06:00 +0000
302+++ lib/lp/soyuz/model/files.py 2009-11-25 23:38:57 +0000
303@@ -60,7 +60,18 @@
304 "binarypackagerelease.binarypackagename"])
305
306
307-class SourcePackageReleaseFile(SQLBase):
308+class SourceFileMixin:
309+ """Mix-in class for common functionality between source file classes."""
310+
311+ @property
312+ def is_orig(self):
313+ return self.filetype in (
314+ SourcePackageFileType.ORIG_TARBALL,
315+ SourcePackageFileType.COMPONENT_ORIG_TARBALL
316+ )
317+
318+
319+class SourcePackageReleaseFile(SourceFileMixin, SQLBase):
320 """See ISourcePackageFile"""
321
322 implements(ISourcePackageReleaseFile)
323@@ -90,4 +101,3 @@
324 prejoins=["libraryfile", "libraryfile.content",
325 "sourcepackagerelease",
326 "sourcepackagerelease.sourcepackagename"])
327-
328
329=== modified file 'lib/lp/soyuz/model/queue.py'
330--- lib/lp/soyuz/model/queue.py 2009-11-14 02:55:15 +0000
331+++ lib/lp/soyuz/model/queue.py 2009-11-25 23:36:55 +0000
332@@ -61,7 +61,6 @@
333 NonBuildableSourceUploadError, QueueBuildAcceptError,
334 QueueInconsistentStateError, QueueSourceAcceptError,
335 QueueStateWriteProtectedError)
336-from lp.registry.interfaces.sourcepackage import SourcePackageFileType
337 from canonical.launchpad.mail import (
338 format_address, signed_message_from_string, sendmail)
339 from lp.soyuz.scripts.processaccepted import (
340@@ -1468,9 +1467,7 @@
341 published_sha1 = published_file.content.sha1
342
343 # Multiple orig(s) with the same content are fine.
344- if source_file.filetype in (
345- SourcePackageFileType.ORIG_TARBALL,
346- SourcePackageFileType.COMPONENT_ORIG_TARBALL):
347+ if source_file.is_orig:
348 if proposed_sha1 == published_sha1:
349 continue
350 raise QueueInconsistentStateError(
Revision history for this message
Julian Edwards (julian-edwards) wrote :

Looks good! Sorry for the delay in replying. I'll catch up with you about this my Monday morning.

review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/lp/archiveuploader/dscfile.py'
2--- lib/lp/archiveuploader/dscfile.py 2009-11-16 22:06:14 +0000
3+++ lib/lp/archiveuploader/dscfile.py 2009-12-12 05:58:17 +0000
4@@ -30,9 +30,9 @@
5 from lp.archiveuploader.tagfiles import (
6 parse_tagfile, TagFileParseError)
7 from lp.archiveuploader.utils import (
8- prefix_multi_line_string, safe_fix_maintainer, ParseMaintError,
9- re_valid_pkg_name, re_valid_version, re_issource,
10- determine_source_file_type)
11+ determine_source_file_type, get_source_file_extension,
12+ ParseMaintError, prefix_multi_line_string, re_is_component_orig_tar_ext,
13+ re_issource, re_valid_pkg_name, re_valid_version, safe_fix_maintainer)
14 from canonical.encoding import guess as guess_encoding
15 from lp.registry.interfaces.person import IPersonSet, PersonCreationRationale
16 from lp.registry.interfaces.sourcepackage import SourcePackageFileType
17@@ -160,6 +160,9 @@
18
19 Can raise UploadError.
20 """
21+ # Avoid circular imports.
22+ from lp.archiveuploader.nascentupload import EarlyReturnUploadError
23+
24 SourceUploadFile.__init__(
25 self, filepath, digest, size, component_and_section, priority,
26 package, version, changes, policy, logger)
27@@ -186,6 +189,10 @@
28 if 'format' not in self._dict:
29 self._dict['format'] = "1.0"
30
31+ if self.format is None:
32+ raise EarlyReturnUploadError(
33+ "Unsupported source format: %s" % self._dict['format'])
34+
35 if self.policy.unsigned_dsc_ok:
36 self.logger.debug("DSC file can be unsigned.")
37 else:
38@@ -208,7 +215,11 @@
39 @property
40 def format(self):
41 """Return the DSC format."""
42- return self._dict['format']
43+ try:
44+ return SourcePackageFormat.getTermByToken(
45+ self._dict['format']).value
46+ except LookupError:
47+ return None
48
49 @property
50 def architecture(self):
51@@ -231,8 +242,6 @@
52 This method is an error generator, i.e, it returns an iterator over all
53 exceptions that are generated while processing DSC file checks.
54 """
55- # Avoid circular imports.
56- from lp.archiveuploader.nascentupload import EarlyReturnUploadError
57
58 for error in SourceUploadFile.verify(self):
59 yield error
60@@ -271,14 +280,8 @@
61 yield UploadError(
62 "%s: invalid version %s" % (self.filename, self.dsc_version))
63
64- try:
65- format_term = SourcePackageFormat.getTermByToken(self.format)
66- except LookupError:
67- raise EarlyReturnUploadError(
68- "Unsupported source format: %s" % self.format)
69-
70 if not self.policy.distroseries.isSourcePackageFormatPermitted(
71- format_term.value):
72+ self.format):
73 yield UploadError(
74 "%s: format '%s' is not permitted in %s." %
75 (self.filename, self.format, self.policy.distroseries.name))
76@@ -347,8 +350,9 @@
77 distribution=self.policy.distro,
78 purpose=ArchivePurpose.PARTNER)]
79 elif (self.policy.archive.purpose == ArchivePurpose.PPA and
80- determine_source_file_type(filename) ==
81- SourcePackageFileType.ORIG_TARBALL):
82+ determine_source_file_type(filename) in (
83+ SourcePackageFileType.ORIG_TARBALL,
84+ SourcePackageFileType.COMPONENT_ORIG_TARBALL)):
85 archives = [self.policy.archive, self.policy.distro.main_archive]
86 else:
87 archives = [self.policy.archive]
88@@ -373,24 +377,36 @@
89 and checksum.
90 """
91
92- diff_count = 0
93- orig_tar_count = 0
94- native_tar_count = 0
95-
96+ file_type_counts = {
97+ SourcePackageFileType.DIFF: 0,
98+ SourcePackageFileType.ORIG_TARBALL: 0,
99+ SourcePackageFileType.DEBIAN_TARBALL: 0,
100+ SourcePackageFileType.NATIVE_TARBALL: 0,
101+ }
102+ component_orig_tar_counts = {}
103+ bzip2_count = 0
104 files_missing = False
105+
106 for sub_dsc_file in self.files:
107- filetype = determine_source_file_type(sub_dsc_file.filename)
108+ file_type = determine_source_file_type(sub_dsc_file.filename)
109
110- if filetype == SourcePackageFileType.DIFF:
111- diff_count += 1
112- elif filetype == SourcePackageFileType.ORIG_TARBALL:
113- orig_tar_count += 1
114- elif filetype == SourcePackageFileType.NATIVE_TARBALL:
115- native_tar_count += 1
116- else:
117+ if file_type is None:
118 yield UploadError('Unknown file: ' + sub_dsc_file.filename)
119 continue
120
121+ if file_type == SourcePackageFileType.COMPONENT_ORIG_TARBALL:
122+ # Split the count by component name.
123+ component = re_is_component_orig_tar_ext.match(
124+ get_source_file_extension(sub_dsc_file.filename)).group(1)
125+ if component not in component_orig_tar_counts:
126+ component_orig_tar_counts[component] = 0
127+ component_orig_tar_counts[component] += 1
128+ else:
129+ file_type_counts[file_type] += 1
130+
131+ if sub_dsc_file.filename.endswith('.bz2'):
132+ bzip2_count += 1
133+
134 try:
135 library_file, file_archive = self._getFileByName(
136 sub_dsc_file.filename)
137@@ -435,37 +451,16 @@
138 yield error
139 files_missing = True
140
141- # Reject if we have more than one file of any type.
142- if orig_tar_count > 1:
143- yield UploadError(
144- "%s: has more than one orig.tar.*."
145- % self.filename)
146- if native_tar_count > 1:
147- yield UploadError(
148- "%s: has more than one tar.*."
149- % self.filename)
150- if diff_count > 1:
151- yield UploadError(
152- "%s: has more than one diff.gz."
153- % self.filename)
154-
155- if ((orig_tar_count == 0 and native_tar_count == 0) or
156- (orig_tar_count > 0 and native_tar_count > 0)):
157- yield UploadError(
158- "%s: must have exactly one tar.* or orig.tar.*."
159- % self.filename)
160-
161- # Format 1.0 must be native (exactly one tar.gz), or
162- # have an orig.tar.gz and a diff.gz. It cannot have
163- # compression types other than 'gz'.
164- if self.format == '1.0':
165- if ((diff_count == 0 and native_tar_count == 0) or
166- (diff_count > 0 and native_tar_count > 0)):
167- yield UploadError(
168- "%s: must have exactly one diff.gz or tar.gz."
169- % self.filename)
170- else:
171- raise AssertionError("Unknown source format.")
172+ try:
173+ file_checker = format_to_file_checker_map[self.format]
174+ except KeyError:
175+ raise AssertionError(
176+ "No file checker for source format %s." % self.format)
177+
178+ for error in file_checker(
179+ self.filename, file_type_counts, component_orig_tar_counts,
180+ bzip2_count):
181+ yield error
182
183 if files_missing:
184 yield UploadError(
185@@ -648,3 +643,94 @@
186 yield error
187
188
189+def check_format_1_0_files(filename, file_type_counts, component_counts,
190+ bzip2_count):
191+ """Check that the given counts of each file type suit format 1.0.
192+
193+ A 1.0 source must be native (with only one tar.gz), or have an orig.tar.gz
194+ and a diff.gz. It cannot use bzip2 compression.
195+ """
196+ if bzip2_count > 0:
197+ yield UploadError(
198+ "%s: is format 1.0 but uses bzip2 compression."
199+ % filename)
200+
201+ valid_file_type_counts = [
202+ {
203+ SourcePackageFileType.NATIVE_TARBALL: 1,
204+ SourcePackageFileType.ORIG_TARBALL: 0,
205+ SourcePackageFileType.DEBIAN_TARBALL: 0,
206+ SourcePackageFileType.DIFF: 0,
207+ },
208+ {
209+ SourcePackageFileType.ORIG_TARBALL: 1,
210+ SourcePackageFileType.DIFF: 1,
211+ SourcePackageFileType.NATIVE_TARBALL: 0,
212+ SourcePackageFileType.DEBIAN_TARBALL: 0,
213+ },
214+ ]
215+
216+ if (file_type_counts not in valid_file_type_counts or
217+ len(component_counts) > 0):
218+ yield UploadError(
219+ "%s: must have exactly one tar.gz, or an orig.tar.gz and diff.gz"
220+ % filename)
221+
222+
223+def check_format_3_0_native_files(filename, file_type_counts,
224+ component_counts, bzip2_count):
225+ """Check that the given counts of each file type suit format 3.0 (native).
226+
227+ A 3.0 (native) source must have only one tar.*. Both gzip and bzip2
228+ compression are permissible.
229+ """
230+
231+ valid_file_type_counts = [
232+ {
233+ SourcePackageFileType.NATIVE_TARBALL: 1,
234+ SourcePackageFileType.ORIG_TARBALL: 0,
235+ SourcePackageFileType.DEBIAN_TARBALL: 0,
236+ SourcePackageFileType.DIFF: 0,
237+ },
238+ ]
239+
240+ if (file_type_counts not in valid_file_type_counts or
241+ len(component_counts) > 0):
242+ yield UploadError("%s: must have only a tar.*." % filename)
243+
244+
245+def check_format_3_0_quilt_files(filename, file_type_counts,
246+ component_counts, bzip2_count):
247+ """Check that the given counts of each file type suit format 3.0 (native).
248+
249+ A 3.0 (quilt) source must have exactly one orig.tar.*, one debian.tar.*,
250+ and at most one orig-COMPONENT.tar.* for each COMPONENT. Both gzip and
251+ bzip2 compression are permissible.
252+ """
253+
254+ valid_file_type_counts = [
255+ {
256+ SourcePackageFileType.ORIG_TARBALL: 1,
257+ SourcePackageFileType.DEBIAN_TARBALL: 1,
258+ SourcePackageFileType.NATIVE_TARBALL: 0,
259+ SourcePackageFileType.DIFF: 0,
260+ },
261+ ]
262+
263+ if file_type_counts not in valid_file_type_counts:
264+ yield UploadError(
265+ "%s: must have only an orig.tar.*, a debian.tar.*, and "
266+ "optionally orig-*.tar.*" % filename)
267+
268+ for component in component_counts:
269+ if component_counts[component] > 1:
270+ yield UploadError(
271+ "%s: has more than one orig-%s.tar.*."
272+ % (filename, component))
273+
274+
275+format_to_file_checker_map = {
276+ SourcePackageFormat.FORMAT_1_0: check_format_1_0_files,
277+ SourcePackageFormat.FORMAT_3_0_NATIVE: check_format_3_0_native_files,
278+ SourcePackageFormat.FORMAT_3_0_QUILT: check_format_3_0_quilt_files,
279+ }
280
281=== modified file 'lib/lp/archiveuploader/nascentuploadfile.py'
282--- lib/lp/archiveuploader/nascentuploadfile.py 2009-12-07 10:53:13 +0000
283+++ lib/lp/archiveuploader/nascentuploadfile.py 2009-12-12 05:58:17 +0000
284@@ -35,7 +35,6 @@
285 re_no_epoch, re_no_revision, re_valid_version, re_valid_pkg_name,
286 re_extract_src_version, determine_source_file_type)
287 from canonical.encoding import guess as guess_encoding
288-from lp.registry.interfaces.sourcepackage import SourcePackageFileType
289 from lp.soyuz.interfaces.binarypackagename import (
290 IBinaryPackageNameSet)
291 from lp.soyuz.interfaces.binarypackagerelease import (
292@@ -49,6 +48,7 @@
293 from lp.soyuz.interfaces.publishing import (
294 PackagePublishingPriority)
295 from lp.soyuz.interfaces.section import ISectionSet
296+from lp.soyuz.model.files import SourceFileMixin
297 from canonical.librarian.utils import filechunks
298
299
300@@ -332,7 +332,7 @@
301 return getUtility(ISectionSet)[self.section_name]
302
303
304-class SourceUploadFile(PackageUploadFile):
305+class SourceUploadFile(SourceFileMixin, PackageUploadFile):
306 """Files mentioned in changesfile as source (orig, diff, tar).
307
308 This class only check consistency on information contained in
309@@ -340,6 +340,11 @@
310 Further checks on file contents and package consistency are done
311 in DSCFile.
312 """
313+
314+ @property
315+ def filetype(self):
316+ return determine_source_file_type(self.filename)
317+
318 def verify(self):
319 """Verify the uploaded source file.
320
321@@ -352,8 +357,7 @@
322 "Architecture field." % (self.filename))
323
324 version_chopped = re_no_epoch.sub('', self.version)
325- if determine_source_file_type(self.filename) == (
326- SourcePackageFileType.ORIG_TARBALL):
327+ if self.is_orig:
328 version_chopped = re_no_revision.sub('', version_chopped)
329
330 source_match = re_issource.match(self.filename)
331
332=== added directory 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_1.0-bzip2'
333=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_1.0-bzip2/bar_1.0-1.diff.gz'
334Binary files lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_1.0-bzip2/bar_1.0-1.diff.gz 1970-01-01 00:00:00 +0000 and lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_1.0-bzip2/bar_1.0-1.diff.gz 2009-12-12 05:58:17 +0000 differ
335=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_1.0-bzip2/bar_1.0-1.dsc'
336--- lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_1.0-bzip2/bar_1.0-1.dsc 1970-01-01 00:00:00 +0000
337+++ lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_1.0-bzip2/bar_1.0-1.dsc 2009-12-12 05:58:17 +0000
338@@ -0,0 +1,10 @@
339+Format: 1.0
340+Source: bar
341+Version: 1.0-1
342+Binary: bar
343+Maintainer: Launchpad team <launchpad@lists.canonical.com>
344+Architecture: any
345+Standards-Version: 3.6.2
346+Files:
347+ f1b3ab12b2d0886bc5d87737bb933b83 170 bar_1.0.orig.tar.bz2
348+ 1e35b810764f140af9616de8274e6e73 537 bar_1.0-1.diff.gz
349
350=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_1.0-bzip2/bar_1.0-1_source.changes'
351--- lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_1.0-bzip2/bar_1.0-1_source.changes 1970-01-01 00:00:00 +0000
352+++ lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_1.0-bzip2/bar_1.0-1_source.changes 2009-12-12 05:58:17 +0000
353@@ -0,0 +1,20 @@
354+Format: 1.7
355+Date: Thu, 16 Feb 2006 15:34:09 +0000
356+Source: bar
357+Binary: bar
358+Architecture: source
359+Version: 1.0-1
360+Distribution: breezy
361+Urgency: low
362+Maintainer: Launchpad team <launchpad@lists.canonical.com>
363+Changed-By: Daniel Silverstone <daniel.silverstone@canonical.com>
364+Description:
365+ bar - Stuff for testing
366+Changes:
367+ bar (1.0-1) breezy; urgency=low
368+ .
369+ * Initial version
370+Files:
371+ 8e34d4e135decd09145c08a78a2a61ff 276 devel optional bar_1.0-1.dsc
372+ f1b3ab12b2d0886bc5d87737bb933b83 170 devel optional bar_1.0.orig.tar.bz2
373+ 1e35b810764f140af9616de8274e6e73 537 devel optional bar_1.0-1.diff.gz
374
375=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_1.0-bzip2/bar_1.0.orig.tar.bz2'
376Binary files lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_1.0-bzip2/bar_1.0.orig.tar.bz2 1970-01-01 00:00:00 +0000 and lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_1.0-bzip2/bar_1.0.orig.tar.bz2 2009-12-12 05:58:17 +0000 differ
377=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_3.0-quilt/bar_1.0-1.debian.tar.bz2'
378Binary files lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_3.0-quilt/bar_1.0-1.debian.tar.bz2 1970-01-01 00:00:00 +0000 and lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_3.0-quilt/bar_1.0-1.debian.tar.bz2 2009-12-12 05:58:17 +0000 differ
379=== removed file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_3.0-quilt/bar_1.0-1.debian.tar.gz'
380Binary files lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_3.0-quilt/bar_1.0-1.debian.tar.gz 2009-11-12 11:54:56 +0000 and lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_3.0-quilt/bar_1.0-1.debian.tar.gz 1970-01-01 00:00:00 +0000 differ
381=== modified file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_3.0-quilt/bar_1.0-1.dsc'
382--- lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_3.0-quilt/bar_1.0-1.dsc 2009-11-12 11:54:56 +0000
383+++ lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_3.0-quilt/bar_1.0-1.dsc 2009-12-12 05:58:17 +0000
384@@ -6,11 +6,17 @@
385 Maintainer: Launchpad team <launchpad@lists.canonical.com>
386 Standards-Version: 3.6.2
387 Checksums-Sha1:
388+ e7a106587e0f11220e27ccc5db7527cae05e4054 178 bar_1.0.orig-comp1.tar.gz
389+ a645cc0856469f549f11bb9bdc4ca6dda4eedd4b 156 bar_1.0.orig-comp2.tar.bz2
390 73a04163fee97fd2257ab266bd48f1d3d528e012 164 bar_1.0.orig.tar.gz
391- abce262314a7c0ca00e43598f21b41a3e6ff6b21 688 bar_1.0-1.debian.tar.gz
392+ 727de2395d3a3e35c375159db01072935fea02f8 737 bar_1.0-1.debian.tar.bz2
393 Checksums-Sha256:
394+ aff0bbc01c5883ac76e89cba7a9b6cd79b521183f0de86c93e93ab8804f5a256 178 bar_1.0.orig-comp1.tar.gz
395+ 955131b3a5bd881c008ff822c0f9b7544350647fe1f355f65a138e0e5e5e8d71 156 bar_1.0.orig-comp2.tar.bz2
396 f1ecff929899b567f45d6734b69d59a4f3c04dabce3cc8e6ed6d64073eda360e 164 bar_1.0.orig.tar.gz
397- ffdcce60fca14618f68483ca77a206f332a3773dc7ece1c3e6de55c0118c69c6 688 bar_1.0-1.debian.tar.gz
398+ 3d2137f9f80f4b6c80f73acb8b3ac7986c962c4268d8948f54823d27148a6116 737 bar_1.0-1.debian.tar.bz2
399 Files:
400+ eed105761436486f7eebaa8c017bc59a 178 bar_1.0.orig-comp1.tar.gz
401+ 0a30b50fa846e75261808ee1f41d1cbe 156 bar_1.0.orig-comp2.tar.bz2
402 fc1464e5985b962a042d5354452f361d 164 bar_1.0.orig.tar.gz
403- 056db4dfe7de8322296b6d417592ee01 688 bar_1.0-1.debian.tar.gz
404+ e68110184d4d9e7afd81520cc490b396 737 bar_1.0-1.debian.tar.bz2
405
406=== modified file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_3.0-quilt/bar_1.0-1_source.changes'
407--- lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_3.0-quilt/bar_1.0-1_source.changes 2009-11-12 11:54:56 +0000
408+++ lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_3.0-quilt/bar_1.0-1_source.changes 2009-12-12 05:58:17 +0000
409@@ -15,14 +15,20 @@
410 .
411 * Initial version
412 Checksums-Sha1:
413- bc97e185cf31af33bf8d109044ce51f32d09c229 645 bar_1.0-1.dsc
414+ 762b018e756c1d9be905d40291bc8e975c57a8f0 1115 bar_1.0-1.dsc
415+ e7a106587e0f11220e27ccc5db7527cae05e4054 178 bar_1.0.orig-comp1.tar.gz
416+ a645cc0856469f549f11bb9bdc4ca6dda4eedd4b 156 bar_1.0.orig-comp2.tar.bz2
417 73a04163fee97fd2257ab266bd48f1d3d528e012 164 bar_1.0.orig.tar.gz
418- abce262314a7c0ca00e43598f21b41a3e6ff6b21 688 bar_1.0-1.debian.tar.gz
419+ 727de2395d3a3e35c375159db01072935fea02f8 737 bar_1.0-1.debian.tar.bz2
420 Checksums-Sha256:
421- ae0fb16941a95518332a8ee962d00d55963b491c2df94b3f230a65d2bdbeedf8 645 bar_1.0-1.dsc
422+ 49f2656a0e221b0f07dc208f583e2523b09bc30104855372534b5562478efdb5 1115 bar_1.0-1.dsc
423+ aff0bbc01c5883ac76e89cba7a9b6cd79b521183f0de86c93e93ab8804f5a256 178 bar_1.0.orig-comp1.tar.gz
424+ 955131b3a5bd881c008ff822c0f9b7544350647fe1f355f65a138e0e5e5e8d71 156 bar_1.0.orig-comp2.tar.bz2
425 f1ecff929899b567f45d6734b69d59a4f3c04dabce3cc8e6ed6d64073eda360e 164 bar_1.0.orig.tar.gz
426- ffdcce60fca14618f68483ca77a206f332a3773dc7ece1c3e6de55c0118c69c6 688 bar_1.0-1.debian.tar.gz
427+ 3d2137f9f80f4b6c80f73acb8b3ac7986c962c4268d8948f54823d27148a6116 737 bar_1.0-1.debian.tar.bz2
428 Files:
429- c320d2827f08f09ec2e1bbbac635225c 645 devel optional bar_1.0-1.dsc
430+ 662977e76501561c902c34bf94a2054d 1115 devel optional bar_1.0-1.dsc
431+ eed105761436486f7eebaa8c017bc59a 178 devel optional bar_1.0.orig-comp1.tar.gz
432+ 0a30b50fa846e75261808ee1f41d1cbe 156 devel optional bar_1.0.orig-comp2.tar.bz2
433 fc1464e5985b962a042d5354452f361d 164 devel optional bar_1.0.orig.tar.gz
434- 056db4dfe7de8322296b6d417592ee01 688 devel optional bar_1.0-1.debian.tar.gz
435+ e68110184d4d9e7afd81520cc490b396 737 devel optional bar_1.0-1.debian.tar.bz2
436
437=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_3.0-quilt/bar_1.0.orig-comp1.tar.gz'
438Binary files lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_3.0-quilt/bar_1.0.orig-comp1.tar.gz 1970-01-01 00:00:00 +0000 and lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_3.0-quilt/bar_1.0.orig-comp1.tar.gz 2009-12-12 05:58:17 +0000 differ
439=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_3.0-quilt/bar_1.0.orig-comp2.tar.bz2'
440Binary files lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_3.0-quilt/bar_1.0.orig-comp2.tar.bz2 1970-01-01 00:00:00 +0000 and lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_3.0-quilt/bar_1.0.orig-comp2.tar.bz2 2009-12-12 05:58:17 +0000 differ
441=== added directory 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig'
442=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig/bar_1.0-2.debian.tar.bz2'
443Binary files lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig/bar_1.0-2.debian.tar.bz2 1970-01-01 00:00:00 +0000 and lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig/bar_1.0-2.debian.tar.bz2 2009-12-12 05:58:17 +0000 differ
444=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig/bar_1.0-2.dsc'
445--- lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig/bar_1.0-2.dsc 1970-01-01 00:00:00 +0000
446+++ lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig/bar_1.0-2.dsc 2009-12-12 05:58:17 +0000
447@@ -0,0 +1,22 @@
448+Format: 3.0 (quilt)
449+Source: bar
450+Binary: bar
451+Architecture: any
452+Version: 1.0-2
453+Maintainer: Launchpad team <launchpad@lists.canonical.com>
454+Standards-Version: 3.6.2
455+Checksums-Sha1:
456+ e7a106587e0f11220e27ccc5db7527cae05e4054 178 bar_1.0.orig-comp1.tar.gz
457+ a645cc0856469f549f11bb9bdc4ca6dda4eedd4b 156 bar_1.0.orig-comp2.tar.bz2
458+ 73a04163fee97fd2257ab266bd48f1d3d528e012 164 bar_1.0.orig.tar.gz
459+ 62776c07df0412798956af8b625a9f1744c8f0fb 807 bar_1.0-2.debian.tar.bz2
460+Checksums-Sha256:
461+ aff0bbc01c5883ac76e89cba7a9b6cd79b521183f0de86c93e93ab8804f5a256 178 bar_1.0.orig-comp1.tar.gz
462+ 955131b3a5bd881c008ff822c0f9b7544350647fe1f355f65a138e0e5e5e8d71 156 bar_1.0.orig-comp2.tar.bz2
463+ f1ecff929899b567f45d6734b69d59a4f3c04dabce3cc8e6ed6d64073eda360e 164 bar_1.0.orig.tar.gz
464+ a2a4a82e6f6b212e7de11df327fea18bf2b8e846d221e417b0a479d0f8bf9d79 807 bar_1.0-2.debian.tar.bz2
465+Files:
466+ eed105761436486f7eebaa8c017bc59a 178 bar_1.0.orig-comp1.tar.gz
467+ 0a30b50fa846e75261808ee1f41d1cbe 156 bar_1.0.orig-comp2.tar.bz2
468+ fc1464e5985b962a042d5354452f361d 164 bar_1.0.orig.tar.gz
469+ 8b17b1469943483df4d5093955553992 807 bar_1.0-2.debian.tar.bz2
470
471=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig/bar_1.0-2_source.changes'
472--- lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig/bar_1.0-2_source.changes 1970-01-01 00:00:00 +0000
473+++ lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig/bar_1.0-2_source.changes 2009-12-12 05:58:17 +0000
474@@ -0,0 +1,25 @@
475+Format: 1.8
476+Date: Sun, 01 Nov 2009 10:49:22 +1100
477+Source: bar
478+Binary: bar
479+Architecture: source
480+Version: 1.0-2
481+Distribution: breezy
482+Urgency: low
483+Maintainer: Launchpad team <launchpad@lists.canonical.com>
484+Changed-By: William Grant <me@williamgrant.id.au>
485+Description:
486+ bar - Stuff for testing
487+Changes:
488+ bar (1.0-2) breezy; urgency=low
489+ .
490+ * Another version
491+Checksums-Sha1:
492+ d9ccd09135965a759bd741fba1f5223a0314aac8 1115 bar_1.0-2.dsc
493+ 62776c07df0412798956af8b625a9f1744c8f0fb 807 bar_1.0-2.debian.tar.bz2
494+Checksums-Sha256:
495+ 53ec1fb4c1671e203c433db570a667750bf4a503e4556ac2c9600389e119744a 1115 bar_1.0-2.dsc
496+ a2a4a82e6f6b212e7de11df327fea18bf2b8e846d221e417b0a479d0f8bf9d79 807 bar_1.0-2.debian.tar.bz2
497+Files:
498+ 014288c2cdeca7a9a5e3201aca044c3b 1115 devel optional bar_1.0-2.dsc
499+ 8b17b1469943483df4d5093955553992 807 devel optional bar_1.0-2.debian.tar.bz2
500
501=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig/bar_1.0.orig-comp1.tar.gz'
502Binary files lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig/bar_1.0.orig-comp1.tar.gz 1970-01-01 00:00:00 +0000 and lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig/bar_1.0.orig-comp1.tar.gz 2009-12-12 05:58:17 +0000 differ
503=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig/bar_1.0.orig-comp2.tar.bz2'
504Binary files lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig/bar_1.0.orig-comp2.tar.bz2 1970-01-01 00:00:00 +0000 and lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig/bar_1.0.orig-comp2.tar.bz2 2009-12-12 05:58:17 +0000 differ
505=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig/bar_1.0.orig.tar.gz'
506Binary files lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig/bar_1.0.orig.tar.gz 1970-01-01 00:00:00 +0000 and lib/lp/archiveuploader/tests/data/suite/bar_1.0-2_3.0-quilt_without_orig/bar_1.0.orig.tar.gz 2009-12-12 05:58:17 +0000 differ
507=== added directory 'lib/lp/archiveuploader/tests/data/suite/bar_1.0_3.0-native'
508=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0_3.0-native/bar_1.0.dsc'
509--- lib/lp/archiveuploader/tests/data/suite/bar_1.0_3.0-native/bar_1.0.dsc 1970-01-01 00:00:00 +0000
510+++ lib/lp/archiveuploader/tests/data/suite/bar_1.0_3.0-native/bar_1.0.dsc 2009-12-12 05:58:17 +0000
511@@ -0,0 +1,13 @@
512+Format: 3.0 (native)
513+Source: bar
514+Binary: bar
515+Architecture: any
516+Version: 1.0
517+Maintainer: Launchpad team <launchpad@lists.canonical.com>
518+Standards-Version: 3.6.2
519+Checksums-Sha1:
520+ f931b8f8fe90b365ad19d84387712e64784cd5ef 776 bar_1.0.tar.bz2
521+Checksums-Sha256:
522+ 1193877b5fc00c410550ad01ec2ed6f97f98a3c8dd2de3753a49251734c842ab 776 bar_1.0.tar.bz2
523+Files:
524+ f9c5c454799806682e832bb4a73d8c5c 776 bar_1.0.tar.bz2
525
526=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0_3.0-native/bar_1.0.tar.bz2'
527Binary files lib/lp/archiveuploader/tests/data/suite/bar_1.0_3.0-native/bar_1.0.tar.bz2 1970-01-01 00:00:00 +0000 and lib/lp/archiveuploader/tests/data/suite/bar_1.0_3.0-native/bar_1.0.tar.bz2 2009-12-12 05:58:17 +0000 differ
528=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0_3.0-native/bar_1.0_source.changes'
529--- lib/lp/archiveuploader/tests/data/suite/bar_1.0_3.0-native/bar_1.0_source.changes 1970-01-01 00:00:00 +0000
530+++ lib/lp/archiveuploader/tests/data/suite/bar_1.0_3.0-native/bar_1.0_source.changes 2009-12-12 05:58:17 +0000
531@@ -0,0 +1,25 @@
532+Format: 1.8
533+Date: Thu, 16 Feb 2006 15:34:09 +0000
534+Source: bar
535+Binary: bar
536+Architecture: source
537+Version: 1.0
538+Distribution: breezy
539+Urgency: low
540+Maintainer: Launchpad team <launchpad@lists.canonical.com>
541+Changed-By: Daniel Silverstone <daniel.silverstone@canonical.com>
542+Description:
543+ bar - Stuff for testing
544+Changes:
545+ bar (1.0) breezy; urgency=low
546+ .
547+ * Initial version
548+Checksums-Sha1:
549+ 3662dbba7c9f5afcdb6aa240a579ffbe1307f64d 406 bar_1.0.dsc
550+ f931b8f8fe90b365ad19d84387712e64784cd5ef 776 bar_1.0.tar.bz2
551+Checksums-Sha256:
552+ b874abb035c04a228272a45625bfd8259998addcbdfd8df0c5c61024b9d40857 406 bar_1.0.dsc
553+ 1193877b5fc00c410550ad01ec2ed6f97f98a3c8dd2de3753a49251734c842ab 776 bar_1.0.tar.bz2
554+Files:
555+ 0fbea3d7be741c94c5f964ee023ef164 406 devel optional bar_1.0.dsc
556+ f9c5c454799806682e832bb4a73d8c5c 776 devel optional bar_1.0.tar.bz2
557
558=== modified file 'lib/lp/archiveuploader/tests/nascentuploadfile.txt'
559--- lib/lp/archiveuploader/tests/nascentuploadfile.txt 2009-12-03 15:51:44 +0000
560+++ lib/lp/archiveuploader/tests/nascentuploadfile.txt 2009-12-12 05:58:17 +0000
561@@ -216,6 +216,16 @@
562 >>> src_match.group(3)
563 'tar.gz'
564
565+ >>> src_match = re_issource.match('foo_1.0.tar.bz2')
566+ >>> src_match.group(0)
567+ 'foo_1.0.tar.bz2'
568+ >>> src_match.group(1)
569+ 'foo'
570+ >>> src_match.group(2)
571+ '1.0'
572+ >>> src_match.group(3)
573+ 'tar.bz2'
574+
575 >>> src_match = re_issource.match('foo_1.0.diff.gz')
576 >>> src_match.group(0)
577 'foo_1.0.diff.gz'
578@@ -236,6 +246,56 @@
579 >>> src_match.group(3)
580 'dsc'
581
582+ >>> src_match = re_issource.match('foo_1.0.debian.tar.gz')
583+ >>> src_match.group(0)
584+ 'foo_1.0.debian.tar.gz'
585+ >>> src_match.group(1)
586+ 'foo'
587+ >>> src_match.group(2)
588+ '1.0'
589+ >>> src_match.group(3)
590+ 'debian.tar.gz'
591+
592+ >>> src_match = re_issource.match('foo_1.0.debian.tar.bz2')
593+ >>> src_match.group(0)
594+ 'foo_1.0.debian.tar.bz2'
595+ >>> src_match.group(1)
596+ 'foo'
597+ >>> src_match.group(2)
598+ '1.0'
599+ >>> src_match.group(3)
600+ 'debian.tar.bz2'
601+
602+ >>> src_match = re_issource.match('foo_1.0.orig-foo.tar.gz')
603+ >>> src_match.group(0)
604+ 'foo_1.0.orig-foo.tar.gz'
605+ >>> src_match.group(1)
606+ 'foo'
607+ >>> src_match.group(2)
608+ '1.0'
609+ >>> src_match.group(3)
610+ 'orig-foo.tar.gz'
611+
612+ >>> src_match = re_issource.match('foo_1.0.orig-bar.tar.bz2')
613+ >>> src_match.group(0)
614+ 'foo_1.0.orig-bar.tar.bz2'
615+ >>> src_match.group(1)
616+ 'foo'
617+ >>> src_match.group(2)
618+ '1.0'
619+ >>> src_match.group(3)
620+ 'orig-bar.tar.bz2'
621+
622+ >>> src_match = re_issource.match('foo_1.0.porig-bar.tar.bz2')
623+ >>> src_match.group(0)
624+ 'foo_1.0.porig-bar.tar.bz2'
625+ >>> src_match.group(1)
626+ 'foo'
627+ >>> src_match.group(2)
628+ '1.0.porig-bar'
629+ >>> src_match.group(3)
630+ 'tar.bz2'
631+
632 And finally some failures:
633
634 >>> re_isadeb.match('foo-bar_1.0_i386.bed') is None
635@@ -244,6 +304,9 @@
636 >>> re_issource.match('foo_1.0.c') is None
637 True
638
639+ >>> re_issource.match('foo_1.0.diff.bz2') is None
640+ True
641+
642 However a custom upload is essencially a tarball, so it also matches
643 the is_source regexp:
644
645@@ -506,6 +569,115 @@
646 ['File ed_0.2-20.dsc mentioned in the changes has a size mismatch. 578 != 500']
647
648
649+=== Format file type verification ===
650+
651+DSCFile performs additional verification on the types of the referenced
652+files, confirming that they are suitable for the source package's
653+format. There is an error generator to verify each format.
654+
655+ >>> from lp.archiveuploader.dscfile import (check_format_1_0_files,
656+ ... check_format_3_0_native_files, check_format_3_0_quilt_files)
657+ >>> from lp.registry.interfaces.sourcepackage import SourcePackageFileType
658+
659+==== 1.0 ====
660+
661+A 1.0 source can contain either a tar.gz or an orig.tar.gz and diff.gz.
662+
663+ >>> list(check_format_1_0_files('foo_1.dsc', {
664+ ... SourcePackageFileType.ORIG_TARBALL: 1,
665+ ... SourcePackageFileType.DIFF: 1,
666+ ... SourcePackageFileType.DEBIAN_TARBALL: 0,
667+ ... SourcePackageFileType.NATIVE_TARBALL: 0,
668+ ... }, {}, 0))
669+ []
670+
671+ >>> list(check_format_1_0_files('foo_1.dsc', {
672+ ... SourcePackageFileType.NATIVE_TARBALL: 1,
673+ ... SourcePackageFileType.ORIG_TARBALL: 0,
674+ ... SourcePackageFileType.DIFF: 0,
675+ ... SourcePackageFileType.DEBIAN_TARBALL: 0,
676+ ... }, {}, 0))
677+ []
678+
679+But if we have some other combination, or bzip2 compression, errors
680+will be generated.
681+
682+ >>> list(check_format_1_0_files('foo_1.dsc', {
683+ ... SourcePackageFileType.NATIVE_TARBALL: 1,
684+ ... SourcePackageFileType.ORIG_TARBALL: 1,
685+ ... SourcePackageFileType.DIFF: 1,
686+ ... SourcePackageFileType.DEBIAN_TARBALL: 0,
687+ ... }, {}, 1))
688+ [UploadError('foo_1.dsc: is format 1.0 but uses bzip2 compression.',), UploadError('foo_1.dsc: must have exactly one tar.gz, or an orig.tar.gz and diff.gz',)]
689+
690+The files are also bad if there are any components:
691+
692+ >>> list(check_format_1_0_files('foo_1.dsc', {
693+ ... SourcePackageFileType.ORIG_TARBALL: 1,
694+ ... SourcePackageFileType.DIFF: 1,
695+ ... SourcePackageFileType.DEBIAN_TARBALL: 0,
696+ ... SourcePackageFileType.NATIVE_TARBALL: 0,
697+ ... }, {'foo': 1}, 0))
698+ [UploadError('foo_1.dsc: must have exactly one tar.gz, or an orig.tar.gz and diff.gz',)]
699+
700+==== 3.0 (native) ====
701+
702+A 3.0 (native) source must contain just a tar.(gz|bz2).
703+
704+ >>> list(check_format_3_0_native_files('foo_1.dsc', {
705+ ... SourcePackageFileType.NATIVE_TARBALL: 1,
706+ ... SourcePackageFileType.ORIG_TARBALL: 0,
707+ ... SourcePackageFileType.DIFF: 0,
708+ ... SourcePackageFileType.DEBIAN_TARBALL: 0,
709+ ... }, {}, 1))
710+ []
711+
712+ >>> list(check_format_3_0_native_files('foo_1.dsc', {
713+ ... SourcePackageFileType.NATIVE_TARBALL: 1,
714+ ... SourcePackageFileType.ORIG_TARBALL: 1,
715+ ... SourcePackageFileType.DIFF: 0,
716+ ... SourcePackageFileType.DEBIAN_TARBALL: 0,
717+ ... }, {}, 1))
718+ [UploadError('foo_1.dsc: must have only a tar.*.',)]
719+
720+ >>> list(check_format_3_0_native_files('foo_1.dsc', {
721+ ... SourcePackageFileType.NATIVE_TARBALL: 1,
722+ ... SourcePackageFileType.ORIG_TARBALL: 0,
723+ ... SourcePackageFileType.DIFF: 0,
724+ ... SourcePackageFileType.DEBIAN_TARBALL: 0,
725+ ... }, {'foo': 1}, 0))
726+ [UploadError('foo_1.dsc: must have only a tar.*.',)]
727+
728+==== 3.0 (quilt) ====
729+
730+A 3.0 (quilt) source must have an orig.tar.*, a debian.tar.*, and at
731+most one orig-COMPONENT.tar.* for each COMPONENT.
732+
733+ >>> list(check_format_3_0_quilt_files('foo_1.dsc', {
734+ ... SourcePackageFileType.ORIG_TARBALL: 1,
735+ ... SourcePackageFileType.DEBIAN_TARBALL: 1,
736+ ... SourcePackageFileType.NATIVE_TARBALL: 0,
737+ ... SourcePackageFileType.DIFF: 0,
738+ ... }, {'foo': 1}, 1))
739+ []
740+
741+ >>> list(check_format_3_0_quilt_files('foo_1.dsc', {
742+ ... SourcePackageFileType.NATIVE_TARBALL: 1,
743+ ... SourcePackageFileType.ORIG_TARBALL: 1,
744+ ... SourcePackageFileType.DIFF: 0,
745+ ... SourcePackageFileType.DEBIAN_TARBALL: 1,
746+ ... }, {}, 1))
747+ [UploadError('foo_1.dsc: must have only an orig.tar.*, a debian.tar.*, and optionally orig-*.tar.*',)]
748+
749+ >>> list(check_format_3_0_quilt_files('foo_1.dsc', {
750+ ... SourcePackageFileType.ORIG_TARBALL: 1,
751+ ... SourcePackageFileType.DEBIAN_TARBALL: 1,
752+ ... SourcePackageFileType.NATIVE_TARBALL: 0,
753+ ... SourcePackageFileType.DIFF: 0,
754+ ... }, {'foo': 2}, 0))
755+ [UploadError('foo_1.dsc: has more than one orig-foo.tar.*.',)]
756+
757+
758 === Sub-DSC files or DSCUploadedFiles ===
759
760 Sub-DSCFiles are DSCUploadedFile objects.
761
762=== modified file 'lib/lp/archiveuploader/tests/test_ppauploadprocessor.py'
763--- lib/lp/archiveuploader/tests/test_ppauploadprocessor.py 2009-11-08 02:26:46 +0000
764+++ lib/lp/archiveuploader/tests/test_ppauploadprocessor.py 2009-12-12 05:58:17 +0000
765@@ -32,6 +32,8 @@
766 from lp.registry.interfaces.pocket import PackagePublishingPocket
767 from lp.soyuz.interfaces.publishing import PackagePublishingStatus
768 from lp.soyuz.interfaces.queue import NonBuildableSourceUploadError
769+from lp.soyuz.interfaces.sourcepackageformat import (
770+ ISourcePackageFormatSelectionSet, SourcePackageFormat)
771 from canonical.launchpad.interfaces import (
772 ILaunchpadCelebrities, ILibraryFileAliasSet, NotFoundError)
773 from canonical.launchpad.testing.fakepackager import FakePackager
774@@ -1161,6 +1163,51 @@
775 self.uploadprocessor.last_processed_upload.queue_root.status,
776 PackageUploadStatus.DONE)
777
778+ def test30QuiltMultipleReusedOrigs(self):
779+ """Official orig*.tar.* can be reused for PPA uploads.
780+
781+ The 3.0 (quilt) format supports multiple original tarballs. In a
782+ PPA upload, any number of these can be reused from the primary
783+ archive.
784+ """
785+ # We need to accept unsigned .changes and .dscs, and 3.0 (quilt)
786+ # sources.
787+ self.options.context = 'absolutely-anything'
788+ getUtility(ISourcePackageFormatSelectionSet).add(
789+ self.breezy, SourcePackageFormat.FORMAT_3_0_QUILT)
790+
791+ # First upload a complete 3.0 (quilt) source to the primary
792+ # archive.
793+ upload_dir = self.queueUpload("bar_1.0-1_3.0-quilt")
794+ self.processUpload(self.uploadprocessor, upload_dir)
795+
796+ self.assertEqual(
797+ self.uploadprocessor.last_processed_upload.queue_root.status,
798+ PackageUploadStatus.NEW)
799+
800+ [queue_item] = self.breezy.getQueueItems(
801+ status=PackageUploadStatus.NEW, name="bar",
802+ version="1.0-1", exact_match=True)
803+ queue_item.setAccepted()
804+ queue_item.realiseUpload()
805+ self.layer.commit()
806+ unused = stub.test_emails.pop()
807+
808+ # Now upload a 3.0 (quilt) source with missing orig*.tar.* to a
809+ # PPA. All of the missing files will be retrieved from the
810+ # primary archive.
811+ upload_dir = self.queueUpload(
812+ "bar_1.0-2_3.0-quilt_without_orig", "~name16/ubuntu")
813+ self.assertEquals(
814+ self.processUpload(self.uploadprocessor, upload_dir),
815+ ['accepted'])
816+
817+ queue_item = self.uploadprocessor.last_processed_upload.queue_root
818+
819+ self.assertEqual(queue_item.status, PackageUploadStatus.DONE)
820+ self.assertEqual(
821+ queue_item.sources[0].sourcepackagerelease.files.count(), 5)
822+
823
824 class TestPPAUploadProcessorQuotaChecks(TestPPAUploadProcessorBase):
825 """Functional test for uploadprocessor.py quota checks in PPA."""
826
827=== modified file 'lib/lp/archiveuploader/tests/test_uploadprocessor.py'
828--- lib/lp/archiveuploader/tests/test_uploadprocessor.py 2009-11-16 22:06:14 +0000
829+++ lib/lp/archiveuploader/tests/test_uploadprocessor.py 2009-12-12 05:58:17 +0000
830@@ -40,6 +40,7 @@
831 from lp.registry.interfaces.distribution import IDistributionSet
832 from lp.registry.interfaces.distroseries import DistroSeriesStatus
833 from lp.registry.interfaces.pocket import PackagePublishingPocket
834+from lp.registry.interfaces.sourcepackage import SourcePackageFileType
835 from lp.soyuz.interfaces.archive import ArchivePurpose, IArchiveSet
836 from lp.soyuz.interfaces.queue import PackageUploadStatus
837 from lp.soyuz.interfaces.publishing import PackagePublishingStatus
838@@ -1501,6 +1502,124 @@
839 "breezy." in raw_msg,
840 "Source was not rejected properly:\n%s" % raw_msg)
841
842+ def test30QuiltUpload(self):
843+ """Ensure that 3.0 (quilt) uploads work properly. """
844+ self.setupBreezy(
845+ permitted_formats=[SourcePackageFormat.FORMAT_3_0_QUILT])
846+ self.layer.txn.commit()
847+ self.options.context = 'absolutely-anything'
848+ uploadprocessor = UploadProcessor(
849+ self.options, self.layer.txn, self.log)
850+
851+ # Upload the source.
852+ upload_dir = self.queueUpload("bar_1.0-1_3.0-quilt")
853+ self.processUpload(uploadprocessor, upload_dir)
854+ # Make sure it went ok:
855+ from_addr, to_addrs, raw_msg = stub.test_emails.pop()
856+ self.assertTrue(
857+ "rejected" not in raw_msg,
858+ "Failed to upload bar source:\n%s" % raw_msg)
859+ spph = self._publishPackage("bar", "1.0-1")
860+
861+ self.assertEquals(
862+ sorted((sprf.libraryfile.filename, sprf.filetype)
863+ for sprf in spph.sourcepackagerelease.files),
864+ [('bar_1.0-1.debian.tar.bz2',
865+ SourcePackageFileType.DEBIAN_TARBALL),
866+ ('bar_1.0-1.dsc',
867+ SourcePackageFileType.DSC),
868+ ('bar_1.0.orig-comp1.tar.gz',
869+ SourcePackageFileType.COMPONENT_ORIG_TARBALL),
870+ ('bar_1.0.orig-comp2.tar.bz2',
871+ SourcePackageFileType.COMPONENT_ORIG_TARBALL),
872+ ('bar_1.0.orig.tar.gz',
873+ SourcePackageFileType.ORIG_TARBALL)])
874+
875+ def test30QuiltUploadWithSameComponentOrig(self):
876+ """Ensure that 3.0 (quilt) uploads with shared component origs work.
877+ """
878+ self.setupBreezy(
879+ permitted_formats=[SourcePackageFormat.FORMAT_3_0_QUILT])
880+ self.layer.txn.commit()
881+ self.options.context = 'absolutely-anything'
882+ uploadprocessor = UploadProcessor(
883+ self.options, self.layer.txn, self.log)
884+
885+ # Upload the first source.
886+ upload_dir = self.queueUpload("bar_1.0-1_3.0-quilt")
887+ self.processUpload(uploadprocessor, upload_dir)
888+ # Make sure it went ok:
889+ from_addr, to_addrs, raw_msg = stub.test_emails.pop()
890+ self.assertTrue(
891+ "rejected" not in raw_msg,
892+ "Failed to upload bar source:\n%s" % raw_msg)
893+ spph = self._publishPackage("bar", "1.0-1")
894+
895+ # Upload another source sharing the same (component) orig.
896+ upload_dir = self.queueUpload("bar_1.0-2_3.0-quilt_without_orig")
897+ self.assertEquals(
898+ self.processUpload(uploadprocessor, upload_dir), ['accepted'])
899+
900+ queue_item = uploadprocessor.last_processed_upload.queue_root
901+ self.assertEquals(
902+ sorted((sprf.libraryfile.filename, sprf.filetype) for sprf
903+ in queue_item.sources[0].sourcepackagerelease.files),
904+ [('bar_1.0-2.debian.tar.bz2',
905+ SourcePackageFileType.DEBIAN_TARBALL),
906+ ('bar_1.0-2.dsc',
907+ SourcePackageFileType.DSC),
908+ ('bar_1.0.orig-comp1.tar.gz',
909+ SourcePackageFileType.COMPONENT_ORIG_TARBALL),
910+ ('bar_1.0.orig-comp2.tar.bz2',
911+ SourcePackageFileType.COMPONENT_ORIG_TARBALL),
912+ ('bar_1.0.orig.tar.gz',
913+ SourcePackageFileType.ORIG_TARBALL)])
914+
915+ def test30NativeUpload(self):
916+ """Ensure that 3.0 (native) uploads work properly. """
917+ self.setupBreezy(
918+ permitted_formats=[SourcePackageFormat.FORMAT_3_0_NATIVE])
919+ self.layer.txn.commit()
920+ self.options.context = 'absolutely-anything'
921+ uploadprocessor = UploadProcessor(
922+ self.options, self.layer.txn, self.log)
923+
924+ # Upload the source.
925+ upload_dir = self.queueUpload("bar_1.0_3.0-native")
926+ self.processUpload(uploadprocessor, upload_dir)
927+ # Make sure it went ok:
928+ from_addr, to_addrs, raw_msg = stub.test_emails.pop()
929+ self.assertTrue(
930+ "rejected" not in raw_msg,
931+ "Failed to upload bar source:\n%s" % raw_msg)
932+ spph = self._publishPackage("bar", "1.0")
933+
934+ self.assertEquals(
935+ sorted((sprf.libraryfile.filename, sprf.filetype)
936+ for sprf in spph.sourcepackagerelease.files),
937+ [('bar_1.0.dsc',
938+ SourcePackageFileType.DSC),
939+ ('bar_1.0.tar.bz2',
940+ SourcePackageFileType.NATIVE_TARBALL)])
941+
942+ def test10Bzip2UploadIsRejected(self):
943+ """Ensure that 1.0 sources with bzip2 compression are rejected."""
944+ self.setupBreezy()
945+ self.layer.txn.commit()
946+ self.options.context = 'absolutely-anything'
947+ uploadprocessor = UploadProcessor(
948+ self.options, self.layer.txn, self.log)
949+
950+ # Upload the source.
951+ upload_dir = self.queueUpload("bar_1.0-1_1.0-bzip2")
952+ self.processUpload(uploadprocessor, upload_dir)
953+ # Make sure it was rejected.
954+ from_addr, to_addrs, raw_msg = stub.test_emails.pop()
955+ self.assertTrue(
956+ "bar_1.0-1.dsc: is format 1.0 but uses bzip2 compression."
957+ in raw_msg,
958+ "Source was not rejected properly:\n%s" % raw_msg)
959+
960
961 def test_suite():
962 return unittest.TestLoader().loadTestsFromName(__name__)
963
964=== modified file 'lib/lp/archiveuploader/tests/test_utils.py'
965--- lib/lp/archiveuploader/tests/test_utils.py 2009-11-17 21:38:28 +0000
966+++ lib/lp/archiveuploader/tests/test_utils.py 2009-12-12 05:58:17 +0000
967@@ -23,18 +23,52 @@
968 """lp.archiveuploader.utils.determine_source_file_type should work."""
969 from lp.archiveuploader.utils import determine_source_file_type
970
971- self.assertEquals(
972- SourcePackageFileType.DSC,
973- determine_source_file_type('foo_1.0-1.dsc'))
974- self.assertEquals(
975- SourcePackageFileType.DIFF,
976- determine_source_file_type('foo_1.0-1.diff.gz'))
977- self.assertEquals(
978- SourcePackageFileType.ORIG_TARBALL,
979- determine_source_file_type('foo_1.0.orig.tar.gz'))
980- self.assertEquals(
981- SourcePackageFileType.NATIVE_TARBALL,
982- determine_source_file_type('foo_1.0.tar.gz'))
983+ # .dsc -> DSC
984+ self.assertEquals(
985+ determine_source_file_type('foo_1.0-1.dsc'),
986+ SourcePackageFileType.DSC)
987+
988+ # .diff.gz -> DIFF
989+ self.assertEquals(
990+ determine_source_file_type('foo_1.0-1.diff.gz'),
991+ SourcePackageFileType.DIFF)
992+
993+ # DIFFs can only be gzipped.
994+ self.assertEquals(
995+ determine_source_file_type('foo_1.0.diff.bz2'), None)
996+
997+ # Plain original tarballs can be gzipped or bzip2ed.
998+ self.assertEquals(
999+ determine_source_file_type('foo_1.0.orig.tar.gz'),
1000+ SourcePackageFileType.ORIG_TARBALL)
1001+ self.assertEquals(
1002+ determine_source_file_type('foo_1.0.orig.tar.bz2'),
1003+ SourcePackageFileType.ORIG_TARBALL)
1004+
1005+ # Component original tarballs too.
1006+ self.assertEquals(
1007+ determine_source_file_type('foo_1.0.orig-foo.tar.gz'),
1008+ SourcePackageFileType.COMPONENT_ORIG_TARBALL)
1009+ self.assertEquals(
1010+ determine_source_file_type('foo_1.0.orig-bar.tar.bz2'),
1011+ SourcePackageFileType.COMPONENT_ORIG_TARBALL)
1012+
1013+ # And Debian tarballs...
1014+ self.assertEquals(
1015+ determine_source_file_type('foo_1.0-1.debian.tar.gz'),
1016+ SourcePackageFileType.DEBIAN_TARBALL)
1017+ self.assertEquals(
1018+ determine_source_file_type('foo_1.0-2.debian.tar.bz2'),
1019+ SourcePackageFileType.DEBIAN_TARBALL)
1020+
1021+ # And even native tarballs!
1022+ self.assertEquals(
1023+ determine_source_file_type('foo_1.0.tar.gz'),
1024+ SourcePackageFileType.NATIVE_TARBALL)
1025+ self.assertEquals(
1026+ determine_source_file_type('foo_1.0.tar.bz2'),
1027+ SourcePackageFileType.NATIVE_TARBALL)
1028+
1029 self.assertEquals(None, determine_source_file_type('foo_1.0'))
1030 self.assertEquals(None, determine_source_file_type('foo_1.0.blah.gz'))
1031
1032
1033=== modified file 'lib/lp/archiveuploader/uploadpolicy.py'
1034--- lib/lp/archiveuploader/uploadpolicy.py 2009-11-05 21:57:35 +0000
1035+++ lib/lp/archiveuploader/uploadpolicy.py 2009-12-12 05:58:17 +0000
1036@@ -373,6 +373,10 @@
1037 """Nothing, let it go."""
1038 pass
1039
1040+ def rejectPPAUploads(self, upload):
1041+ """We allow PPA uploads."""
1042+ return False
1043+
1044 AbstractUploadPolicy._registerPolicy(AnythingGoesUploadPolicy)
1045
1046
1047
1048=== modified file 'lib/lp/archiveuploader/utils.py'
1049--- lib/lp/archiveuploader/utils.py 2009-11-11 21:50:56 +0000
1050+++ lib/lp/archiveuploader/utils.py 2009-12-12 05:58:17 +0000
1051@@ -9,6 +9,7 @@
1052 're_taint_free',
1053 're_isadeb',
1054 're_issource',
1055+ 're_is_component_orig_tar_ext',
1056 're_no_epoch',
1057 're_no_revision',
1058 're_valid_version',
1059@@ -34,13 +35,15 @@
1060
1061 re_isadeb = re.compile(r"(.+?)_(.+?)_(.+)\.(u?d?deb)$")
1062
1063-source_file_exts = ['orig.tar.gz', 'diff.gz', 'tar.gz', 'dsc']
1064+source_file_exts = [
1065+ 'orig(?:-.+)?\.tar\.(?:gz|bz2)', 'diff.gz',
1066+ '(?:debian\.)?tar\.(?:gz|bz2)', 'dsc']
1067 re_issource = re.compile(
1068- r"(.+)_(.+?)\.(%s)" % "|".join(
1069- re.escape(ext) for ext in source_file_exts))
1070-
1071-re_is_orig_tar_ext = re.compile(r"^orig.tar.gz$")
1072-re_is_native_tar_ext = re.compile(r"^tar.gz$")
1073+ r"(.+)_(.+?)\.(%s)" % "|".join(ext for ext in source_file_exts))
1074+re_is_component_orig_tar_ext = re.compile(r"^orig-(.+).tar.(?:gz|bz2)$")
1075+re_is_orig_tar_ext = re.compile(r"^orig.tar.(?:gz|bz2)$")
1076+re_is_debian_tar_ext = re.compile(r"^debian.tar.(?:gz|bz2)$")
1077+re_is_native_tar_ext = re.compile(r"^tar.(?:gz|bz2)$")
1078
1079 re_no_epoch = re.compile(r"^\d+\:")
1080 re_no_revision = re.compile(r"-[^-]+$")
1081@@ -75,6 +78,10 @@
1082 return SourcePackageFileType.DIFF
1083 elif re_is_orig_tar_ext.match(extension):
1084 return SourcePackageFileType.ORIG_TARBALL
1085+ elif re_is_component_orig_tar_ext.match(extension):
1086+ return SourcePackageFileType.COMPONENT_ORIG_TARBALL
1087+ elif re_is_debian_tar_ext.match(extension):
1088+ return SourcePackageFileType.DEBIAN_TARBALL
1089 elif re_is_native_tar_ext.match(extension):
1090 return SourcePackageFileType.NATIVE_TARBALL
1091 else:
1092
1093=== modified file 'lib/lp/registry/interfaces/sourcepackage.py'
1094--- lib/lp/registry/interfaces/sourcepackage.py 2009-11-21 08:56:44 +0000
1095+++ lib/lp/registry/interfaces/sourcepackage.py 2009-12-12 05:58:17 +0000
1096@@ -310,7 +310,9 @@
1097 This is an Ubuntu "diff" file, containing changes that need to be
1098 made to upstream code for the packaging on Ubuntu. Typically this
1099 diff creates additional directories with patches and documentation
1100- used to build the binary packages for Ubuntu. """)
1101+ used to build the binary packages for Ubuntu.
1102+
1103+ This is only part of the 1.0 source package format.""")
1104
1105 NATIVE_TARBALL = DBItem(6, """
1106 Native Tarball
1107@@ -318,6 +320,22 @@
1108 This is a tarball, usually of a mixture of Ubuntu and upstream code,
1109 used in the build process for this source package. """)
1110
1111+ DEBIAN_TARBALL = DBItem(7, """
1112+ Debian Tarball
1113+
1114+ This file is an Ubuntu "orig" file, typically an upstream tarball or
1115+ other lightly-modified upstreamish thing.
1116+
1117+ This is only part of the 3.0 (quilt) source package format.""")
1118+
1119+ COMPONENT_ORIG_TARBALL = DBItem(8, """
1120+ Component Orig Tarball
1121+
1122+ This file is an Ubuntu component "orig" file, typically an upstream
1123+ tarball containing a component of the source package.
1124+
1125+ This is only part of the 3.0 (quilt) source package format.""")
1126+
1127
1128 class SourcePackageType(DBEnumeratedType):
1129 """Source Package Format
1130
1131=== modified file 'lib/lp/soyuz/interfaces/files.py'
1132--- lib/lp/soyuz/interfaces/files.py 2009-06-25 04:06:00 +0000
1133+++ lib/lp/soyuz/interfaces/files.py 2009-12-12 05:58:17 +0000
1134@@ -14,7 +14,7 @@
1135 'ISourcePackageReleaseFileSet',
1136 ]
1137
1138-from zope.schema import Int
1139+from zope.schema import Bool, Int
1140 from zope.interface import Interface
1141 from canonical.launchpad import _
1142
1143@@ -68,6 +68,11 @@
1144 title=_('The type of this file'), required=True, readonly=False,
1145 )
1146
1147+ is_orig = Bool(
1148+ title=_('Whether this file is an original tarball'),
1149+ required=True, readonly=False,
1150+ )
1151+
1152
1153 class ISourcePackageReleaseFileSet(Interface):
1154 """The set of all `SourcePackageRelease`s."""
1155
1156=== modified file 'lib/lp/soyuz/model/files.py'
1157--- lib/lp/soyuz/model/files.py 2009-06-25 04:06:00 +0000
1158+++ lib/lp/soyuz/model/files.py 2009-12-12 05:58:17 +0000
1159@@ -60,7 +60,18 @@
1160 "binarypackagerelease.binarypackagename"])
1161
1162
1163-class SourcePackageReleaseFile(SQLBase):
1164+class SourceFileMixin:
1165+ """Mix-in class for common functionality between source file classes."""
1166+
1167+ @property
1168+ def is_orig(self):
1169+ return self.filetype in (
1170+ SourcePackageFileType.ORIG_TARBALL,
1171+ SourcePackageFileType.COMPONENT_ORIG_TARBALL
1172+ )
1173+
1174+
1175+class SourcePackageReleaseFile(SourceFileMixin, SQLBase):
1176 """See ISourcePackageFile"""
1177
1178 implements(ISourcePackageReleaseFile)
1179@@ -90,4 +101,3 @@
1180 prejoins=["libraryfile", "libraryfile.content",
1181 "sourcepackagerelease",
1182 "sourcepackagerelease.sourcepackagename"])
1183-
1184
1185=== modified file 'lib/lp/soyuz/model/queue.py'
1186--- lib/lp/soyuz/model/queue.py 2009-12-07 10:04:25 +0000
1187+++ lib/lp/soyuz/model/queue.py 2009-12-12 05:58:17 +0000
1188@@ -61,7 +61,6 @@
1189 NonBuildableSourceUploadError, QueueBuildAcceptError,
1190 QueueInconsistentStateError, QueueSourceAcceptError,
1191 QueueStateWriteProtectedError)
1192-from lp.registry.interfaces.sourcepackage import SourcePackageFileType
1193 from canonical.launchpad.mail import (
1194 format_address, signed_message_from_string, sendmail)
1195 from lp.soyuz.scripts.processaccepted import (
1196@@ -1471,8 +1470,7 @@
1197 published_sha1 = published_file.content.sha1
1198
1199 # Multiple orig(s) with the same content are fine.
1200- if source_file.filetype == (
1201- SourcePackageFileType.ORIG_TARBALL):
1202+ if source_file.is_orig:
1203 if proposed_sha1 == published_sha1:
1204 continue
1205 raise QueueInconsistentStateError(

Subscribers

People subscribed via source and target branches

to status/vote changes: