Launchpad itself

Merge lp:~wgrant/launchpad/publisher-release-cleanup into lp:launchpad

  1. publisher-release-cleanup
  2. Merge into devel
Proposed by William Grant
Status: Merged
Approved by: Abel Deuring
Approved revision: no longer in the source branch.
Merged at revision: 11730
Proposed branch: lp:~wgrant/launchpad/publisher-release-cleanup
Merge into: lp:launchpad
Prerequisite: lp:~wgrant/launchpad/better-publisher-index-tests
Diff against target: 509 lines (+95/-202)
3 files modified
lib/lp/archivepublisher/ftparchive.py (+12/-71)
lib/lp/archivepublisher/publishing.py (+76/-82)
lib/lp/archivepublisher/tests/test_publisher.py (+7/-49)
To merge this branch: bzr merge lp:~wgrant/launchpad/publisher-release-cleanup
Reviewer Review Type Date Requested Status
Abel Deuring (community) code Approve
Review via email: mp+38530@code.launchpad.net

Commit message

Replace the messy FTPArchiveHandler.release_files_needed dict with a simple Publisher.release_files_needed set of suites.

Description of the change

= Summary =
lp.archivepublisher has two index generation mechanisms: native DB publication for partner, debug and PPA archives; and apt-ftparchive publication for primary and copy archives. The apt-ftparchive handler is deprecated and to be phased out, but its release_files_needed attribute is still used by the native publisher (bug #655690).

== Proposed fix ==
Move release_files_needed from FTPArchiveHandler to Publisher, and only instantiate FTPArchiveHandler when it's actually needed.

== Implementation details ==
While moving release_files_needed I realised that it wasn't very sensible: it consisted of a map from suite to component to a list of architectures. While the set of suites changes depending on what needs regeneration, the component and architecture lists must contain the full set each time -- calculating them in a roundabout manner and storing them in the dict is pointless.

So I changed release_files_needed into a set of suites, and adjusted its users to retrieve the component and architecture lists directly from the publisher configuration. This mainly involved the removal of hacks from _writeDistroArchSeries, which I've split and renamed.

I also removed some duplicated code from lib/lp/archivepublisher/ftparchive.py. It was XXX'd by kiko four years ago as probably being redundant, and investigation confirms that proposition.

== Tests ==
test_publisher.py has had complex release_files_needed verification removed. The new simpler data structure removes what they are testing, and their role is fulfilled by testAllIndicesArePublished as added in the prereq.

== Demo and Q/A ==
This is just a refactoring. It's fine as long as the tests work.

= Launchpad lint =
There is plenty of inherited lint that could be fixed, but this branch is heavy enough, and there are five following it fixing most of it.

To post a comment you must log in.
Revision history for this message
Abel Deuring (adeuring) :
review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/lp/archivepublisher/ftparchive.py'
--- lib/lp/archivepublisher/ftparchive.py 2010-10-06 13:50:50 +0000
+++ lib/lp/archivepublisher/ftparchive.py 2010-10-15 11:41:18 +0000
@@ -157,7 +157,6 @@
157 else:157 else:
158 self.distroseries.append(distroseries)158 self.distroseries.append(distroseries)
159 self.publisher = publisher159 self.publisher = publisher
160 self.release_files_needed = {}
161160
162 # We need somewhere to note down where the debian-installer161 # We need somewhere to note down where the debian-installer
163 # components came from. in _di_release_components we store162 # components came from. in _di_release_components we store
@@ -199,13 +198,9 @@
199 We do this to have Packages or Sources for them even if we lack198 We do this to have Packages or Sources for them even if we lack
200 anything in them currently.199 anything in them currently.
201 """200 """
202 # XXX: kiko 2006-08-24: suffix is completely unnecessary here. Just
203 # iterate over the pockets, and do the suffix check inside
204 # createEmptyPocketRequest; that would also allow us to replace
205 # the == "" check we do there by a RELEASE match
206 for distroseries in self.distroseries:201 for distroseries in self.distroseries:
207 components = self._config.componentsForSeries(distroseries.name)202 components = self._config.componentsForSeries(distroseries.name)
208 for pocket, suffix in pocketsuffix.items():203 for pocket in PackagePublishingPocket.items:
209 if not fullpublish:204 if not fullpublish:
210 if not self.publisher.isDirty(distroseries, pocket):205 if not self.publisher.isDirty(distroseries, pocket):
211 continue206 continue
@@ -213,29 +208,15 @@
213 if not self.publisher.isAllowed(distroseries, pocket):208 if not self.publisher.isAllowed(distroseries, pocket):
214 continue209 continue
215210
211 self.publisher.release_files_needed.add(
212 (distroseries.name, pocket))
213
216 for comp in components:214 for comp in components:
217 self.createEmptyPocketRequest(distroseries, suffix, comp)215 self.createEmptyPocketRequest(distroseries, pocket, comp)
218216
219 def requestReleaseFile(self, suite_name, component_name, arch_name):217 def createEmptyPocketRequest(self, distroseries, pocket, comp):
220 """Request Release file generation for given context.
221
222 'suite_name', 'component_name' and 'arch_name' will be organised as
223 a dictionary (self.release_files_needed) keyed by 'suite_name' which
224 value will be another dictionary keyed by 'component_name' and
225 containing a set of 'arch_name's as value.
226 """
227 suite_special = self.release_files_needed.setdefault(
228 suite_name, {})
229 suite_component_special = suite_special.setdefault(
230 component_name, set())
231 suite_component_special.add(arch_name)
232
233 def createEmptyPocketRequest(self, distroseries, suffix, comp):
234 """Creates empty files for a release pocket and distroseries"""218 """Creates empty files for a release pocket and distroseries"""
235 full_distroseries_name = distroseries.name + suffix219 if pocket == PackagePublishingPocket.RELEASE:
236 arch_tags = self._config.archTagsForSeries(distroseries.name)
237
238 if suffix == "":
239 # organize distroseries and component pair as220 # organize distroseries and component pair as
240 # debian-installer -> distroseries_component221 # debian-installer -> distroseries_component
241 # internal map. Only the main pocket actually222 # internal map. Only the main pocket actually
@@ -246,6 +227,8 @@
246 ".".join(["override", distroseries.name, comp,227 ".".join(["override", distroseries.name, comp,
247 "debian-installer"]))228 "debian-installer"]))
248229
230 full_distroseries_name = distroseries.name + pocketsuffix[pocket]
231
249 # Touch the source file lists and override files232 # Touch the source file lists and override files
250 f_touch(self._config.overrideroot,233 f_touch(self._config.overrideroot,
251 ".".join(["override", full_distroseries_name, comp]))234 ".".join(["override", full_distroseries_name, comp]))
@@ -254,20 +237,10 @@
254 f_touch(self._config.overrideroot,237 f_touch(self._config.overrideroot,
255 ".".join(["override", full_distroseries_name, comp, "src"]))238 ".".join(["override", full_distroseries_name, comp, "src"]))
256239
257 dr_comps = self.release_files_needed.setdefault(
258 full_distroseries_name, {})
259
260 f_touch(self._config.overrideroot,240 f_touch(self._config.overrideroot,
261 "_".join([full_distroseries_name, comp, "source"]))241 "_".join([full_distroseries_name, comp, "source"]))
262 dr_comps.setdefault(comp, set()).add("source")242
263243 for arch in self._config.archTagsForSeries(distroseries.name):
264 for arch in arch_tags:
265 # organize dr/comp/arch into temporary binary
266 # archive map for the architecture in question.
267 dr_special = self.release_files_needed.setdefault(
268 full_distroseries_name, {})
269 dr_special.setdefault(comp, set()).add("binary-"+arch)
270
271 # Touch more file lists for the archs.244 # Touch more file lists for the archs.
272 f_touch(self._config.overrideroot,245 f_touch(self._config.overrideroot,
273 "_".join([full_distroseries_name, comp, "binary-"+arch]))246 "_".join([full_distroseries_name, comp, "binary-"+arch]))
@@ -714,9 +687,6 @@
714687
715 Also outputs a debian-installer file list if necessary.688 Also outputs a debian-installer file list if necessary.
716 """689 """
717 self.release_files_needed.setdefault(
718 dr_pocketed, {}).setdefault(component, set()).add(arch)
719
720 files = []690 files = []
721 di_files = []691 di_files = []
722 f_path = os.path.join(self._config.overrideroot,692 f_path = os.path.join(self._config.overrideroot,
@@ -807,37 +777,8 @@
807 """Generates the config stanza for an individual pocket."""777 """Generates the config stanza for an individual pocket."""
808 dr_pocketed = distroseries_name + pocketsuffix[pocket]778 dr_pocketed = distroseries_name + pocketsuffix[pocket]
809779
810 # XXX kiko 2006-08-24: I have no idea what the code below is meant
811 # to do -- it appears to be a rehash of createEmptyPocketRequests.
812 archs = self._config.archTagsForSeries(distroseries_name)780 archs = self._config.archTagsForSeries(distroseries_name)
813 comps = self._config.componentsForSeries(distroseries_name)781 comps = self._config.componentsForSeries(distroseries_name)
814 for comp in comps:
815 comp_path = os.path.join(self._config.overrideroot,
816 "_".join([dr_pocketed, comp, "source"]))
817 if not os.path.exists(comp_path):
818 # Create empty files so that even if we don't output
819 # anything here apt-ftparchive will DTRT
820 f_touch(comp_path)
821 f_touch(self._config.overrideroot,
822 ".".join(["override", dr_pocketed, comp]))
823 f_touch(self._config.overrideroot,
824 ".".join(["override", dr_pocketed, comp, "src"]))
825
826 if len(comps) == 0:
827 self.log.debug("Did not find any components to create config "
828 "for %s" % dr_pocketed)
829 return
830
831 # Second up, pare archs down as appropriate
832 for arch in archs:
833 # XXX: kiko 2006-08-24: why is it comps[0] here?
834 arch_path = os.path.join(self._config.overrideroot,
835 "_".join([dr_pocketed, comps[0], "binary-"+arch]))
836 if not os.path.exists(arch_path):
837 # Create an empty file if we don't have one so that
838 # apt-ftparchive will dtrt.
839 f_touch(arch_path)
840 # XXX kiko 2006-08-24: End uncomprehensible code.
841782
842 self.log.debug("Generating apt config for %s" % dr_pocketed)783 self.log.debug("Generating apt config for %s" % dr_pocketed)
843 apt_config.write(STANZA_TEMPLATE % {784 apt_config.write(STANZA_TEMPLATE % {
844785
=== modified file 'lib/lp/archivepublisher/publishing.py'
--- lib/lp/archivepublisher/publishing.py 2010-10-06 13:49:38 +0000
+++ lib/lp/archivepublisher/publishing.py 2010-10-15 11:41:18 +0000
@@ -58,13 +58,19 @@
58 Over time this method needs to be removed and replaced by having58 Over time this method needs to be removed and replaced by having
59 component ordering codified in the database.59 component ordering codified in the database.
60 """60 """
61 ret = []61 remaining = list(components)
62 ordered = []
62 for comp in HARDCODED_COMPONENT_ORDER:63 for comp in HARDCODED_COMPONENT_ORDER:
63 if comp in components:64 if comp in remaining:
64 ret.append(comp)65 ordered.append(comp)
65 components.remove(comp)66 remaining.remove(comp)
66 ret.extend(components)67 ordered.extend(remaining)
67 return ret68 return ordered
69
70
71def get_suffixed_indices(path):
72 """Return a set of paths to compressed copies of the given index."""
73 return set([path + suffix for suffix in ('', '.gz', '.bz2')])
6874
6975
70def _getDiskPool(pubconf, log):76def _getDiskPool(pubconf, log):
@@ -147,21 +153,19 @@
147 else:153 else:
148 self._library = library154 self._library = library
149155
150 # Grab a reference to an apt_handler as we use it later to
151 # probe which components need releases files generated.
152 self.apt_handler = FTPArchiveHandler(self.log, self._config,
153 self._diskpool, self.distro,
154 self)
155 # Track which distroseries pockets have been dirtied by a156 # Track which distroseries pockets have been dirtied by a
156 # change, and therefore need domination/apt-ftparchive work.157 # change, and therefore need domination/apt-ftparchive work.
157 # This is a set of tuples in the form (distroseries.name, pocket)158 # This is a set of tuples in the form (distroseries.name, pocket)
158 self.dirty_pockets = set()159 self.dirty_pockets = set()
159160
161 # Track which pockets need release files. This will contain more
162 # than dirty_pockets in the case of a careful index run.
163 # This is a set of tuples in the form (distroseries.name, pocket)
164 self.release_files_needed = set()
165
160 def isDirty(self, distroseries, pocket):166 def isDirty(self, distroseries, pocket):
161 """True if a publication has happened in this release and pocket."""167 """True if a publication has happened in this release and pocket."""
162 if not (distroseries.name, pocket) in self.dirty_pockets:168 return (distroseries.name, pocket) in self.dirty_pockets
163 return False
164 return True
165169
166 def markPocketDirty(self, distroseries, pocket):170 def markPocketDirty(self, distroseries, pocket):
167 """Mark a pocket dirty only if it's allowed."""171 """Mark a pocket dirty only if it's allowed."""
@@ -176,10 +180,8 @@
176180
177 Otherwise, return False.181 Otherwise, return False.
178 """182 """
179 if (self.allowed_suites and183 return (not self.allowed_suites or
180 (distroseries.name, pocket) not in self.allowed_suites):184 (distroseries.name, pocket) in self.allowed_suites)
181 return False
182 return True
183185
184 def A_publish(self, force_publishing):186 def A_publish(self, force_publishing):
185 """First step in publishing: actual package publishing.187 """First step in publishing: actual package publishing.
@@ -281,7 +283,10 @@
281 def C_doFTPArchive(self, is_careful):283 def C_doFTPArchive(self, is_careful):
282 """Does the ftp-archive step: generates Sources and Packages."""284 """Does the ftp-archive step: generates Sources and Packages."""
283 self.log.debug("* Step C: Set apt-ftparchive up and run it")285 self.log.debug("* Step C: Set apt-ftparchive up and run it")
284 self.apt_handler.run(is_careful)286 apt_handler = FTPArchiveHandler(self.log, self._config,
287 self._diskpool, self.distro,
288 self)
289 apt_handler.run(is_careful)
285290
286 def C_writeIndexes(self, is_careful):291 def C_writeIndexes(self, is_careful):
287 """Write Index files (Packages & Sources) using LP information.292 """Write Index files (Packages & Sources) using LP information.
@@ -297,6 +302,9 @@
297 (distroseries.name, pocket.name))302 (distroseries.name, pocket.name))
298 continue303 continue
299 self.checkDirtySuiteBeforePublishing(distroseries, pocket)304 self.checkDirtySuiteBeforePublishing(distroseries, pocket)
305
306 self.release_files_needed.add((distroseries.name, pocket))
307
300 # Retrieve components from the publisher config because308 # Retrieve components from the publisher config because
301 # it gets overridden in getPubConfig to set the309 # it gets overridden in getPubConfig to set the
302 # correct components for the archive being used.310 # correct components for the archive being used.
@@ -323,7 +331,7 @@
323 (distroseries.name, pocket.name))331 (distroseries.name, pocket.name))
324 continue332 continue
325 self.checkDirtySuiteBeforePublishing(distroseries, pocket)333 self.checkDirtySuiteBeforePublishing(distroseries, pocket)
326 self._writeDistroSeries(distroseries, pocket)334 self._writeSuite(distroseries, pocket)
327335
328 def _writeComponentIndexes(self, distroseries, pocket, component):336 def _writeComponentIndexes(self, distroseries, pocket, component):
329 """Write Index files for single distroseries + pocket + component.337 """Write Index files for single distroseries + pocket + component.
@@ -358,11 +366,6 @@
358366
359 arch_path = 'binary-%s' % arch.architecturetag367 arch_path = 'binary-%s' % arch.architecturetag
360368
361 # XXX wgrant 2010-10-06 bug=655690: Using FTPArchiveHandler
362 # for NMAF is wrong.
363 self.apt_handler.requestReleaseFile(
364 suite_name, component.name, arch_path)
365
366 self.log.debug("Generating Packages for %s" % arch_path)369 self.log.debug("Generating Packages for %s" % arch_path)
367370
368 package_index_root = os.path.join(371 package_index_root = os.path.join(
@@ -394,11 +397,6 @@
394 package_index.close()397 package_index.close()
395 di_index.close()398 di_index.close()
396399
397 # XXX wgrant 2010-10-06 bug=655690: Using FTPArchiveHandler
398 # is wrong here too.
399 self.apt_handler.requestReleaseFile(
400 suite_name, component.name, 'source')
401
402 def cannotModifySuite(self, distroseries, pocket):400 def cannotModifySuite(self, distroseries, pocket):
403 """Return True if the distroseries is stable and pocket is release."""401 """Return True if the distroseries is stable and pocket is release."""
404 return (not distroseries.isUnstable() and402 return (not distroseries.isUnstable() and
@@ -447,34 +445,28 @@
447 return self.distro.displayname445 return self.distro.displayname
448 return "LP-PPA-%s" % get_ppa_reference(self.archive)446 return "LP-PPA-%s" % get_ppa_reference(self.archive)
449447
450 def _writeDistroSeries(self, distroseries, pocket):448 def _writeSuite(self, distroseries, pocket):
451 """Write out the Release files for the provided distroseries."""449 """Write out the Release files for the provided suite."""
452 # XXX: kiko 2006-08-24: Untested method.450 # XXX: kiko 2006-08-24: Untested method.
453451
454 # As we generate file lists for apt-ftparchive we record which452 # As we generate file lists for apt-ftparchive we record which
455 # distroseriess and so on we need to generate Release files for.453 # distroseriess and so on we need to generate Release files for.
456 # We store this in release_files_needed and consume the information454 # We store this in release_files_needed and consume the information
457 # when writeReleaseFiles is called.455 # when writeReleaseFiles is called.
458 full_name = distroseries.name + pocketsuffix[pocket]456 if (distroseries.name, pocket) not in self.release_files_needed:
459 release_files_needed = self.apt_handler.release_files_needed
460 if full_name not in release_files_needed:
461 # If we don't need to generate a release for this release457 # If we don't need to generate a release for this release
462 # and pocket, don't!458 # and pocket, don't!
463 return459 return
464460
465 all_components = set()461 all_components = self._config.componentsForSeries(distroseries.name)
466 all_architectures = set()462 all_architectures = self._config.archTagsForSeries(distroseries.name)
467 all_files = set()463 all_files = set()
468 release_files_needed_items = release_files_needed[full_name].items()464 for component in all_components:
469 for component, architectures in release_files_needed_items:465 self._writeSuiteSource(
470 all_components.add(component)466 distroseries, pocket, component, all_files)
471 for architecture in architectures:467 for architecture in all_architectures:
472 # XXX malcc 2006-09-20: We don't like the way we build this468 self._writeSuiteArch(
473 # all_architectures list. Make this better code.
474 clean_architecture = self._writeDistroArchSeries(
475 distroseries, pocket, component, architecture, all_files)469 distroseries, pocket, component, architecture, all_files)
476 if clean_architecture != "source":
477 all_architectures.add(clean_architecture)
478470
479 drsummary = "%s %s " % (self.distro.displayname,471 drsummary = "%s %s " % (self.distro.displayname,
480 distroseries.displayname)472 distroseries.displayname)
@@ -483,6 +475,7 @@
483 else:475 else:
484 drsummary += pocket.name.capitalize()476 drsummary += pocket.name.capitalize()
485477
478 full_name = distroseries.getSuite(pocket)
486 release_file = Release()479 release_file = Release()
487 release_file["Origin"] = self._getOrigin()480 release_file["Origin"] = self._getOrigin()
488 release_file["Label"] = self._getLabel()481 release_file["Label"] = self._getLabel()
@@ -530,60 +523,61 @@
530 archive_signer = IArchiveSigningKey(self.archive)523 archive_signer = IArchiveSigningKey(self.archive)
531 archive_signer.signRepository(full_name)524 archive_signer.signRepository(full_name)
532525
533 def _writeDistroArchSeries(self, distroseries, pocket, component,526 def _writeSuiteArchOrSource(self, distroseries, pocket, component,
534 architecture, all_files):527 file_stub, arch_name, arch_path,
535 """Write out a Release file for a DAR."""528 all_series_files):
529 """Write out a Release file for an architecture or source."""
536 # XXX kiko 2006-08-24: Untested method.530 # XXX kiko 2006-08-24: Untested method.
537531
538 full_name = distroseries.name + pocketsuffix[pocket]532 suite = distroseries.getSuite(pocket)
539 index_suffixes = ('', '.gz', '.bz2')
540
541 self.log.debug("Writing Release file for %s/%s/%s" % (533 self.log.debug("Writing Release file for %s/%s/%s" % (
542 full_name, component, architecture))534 suite, component, arch_path))
543
544 if architecture != "source":
545 # Strip "binary-" off the front of the architecture
546 clean_architecture = architecture[7:]
547 file_stub = "Packages"
548
549 # Only the primary and PPA archives have debian-installer.
550 if self.archive.purpose != ArchivePurpose.PARTNER:
551 # Set up the debian-installer paths for main_archive.
552 # d-i paths are nested inside the component.
553 di_path = os.path.join(
554 component, "debian-installer", architecture)
555 di_file_stub = os.path.join(di_path, file_stub)
556 for suffix in index_suffixes:
557 all_files.add(di_file_stub + suffix)
558 else:
559 file_stub = "Sources"
560 clean_architecture = architecture
561535
562 # Now, grab the actual (non-di) files inside each of536 # Now, grab the actual (non-di) files inside each of
563 # the suite's architectures537 # the suite's architectures
564 file_stub = os.path.join(component, architecture, file_stub)538 file_stub = os.path.join(component, arch_path, file_stub)
565539
566 for suffix in index_suffixes:540 all_series_files.update(get_suffixed_indices(file_stub))
567 all_files.add(file_stub + suffix)541 all_series_files.add(os.path.join(component, arch_path, "Release"))
568
569 all_files.add(os.path.join(component, architecture, "Release"))
570542
571 release_file = Release()543 release_file = Release()
572 release_file["Archive"] = full_name544 release_file["Archive"] = suite
573 release_file["Version"] = distroseries.version545 release_file["Version"] = distroseries.version
574 release_file["Component"] = component546 release_file["Component"] = component
575 release_file["Origin"] = self._getOrigin()547 release_file["Origin"] = self._getOrigin()
576 release_file["Label"] = self._getLabel()548 release_file["Label"] = self._getLabel()
577 release_file["Architecture"] = clean_architecture549 release_file["Architecture"] = arch_name
578550
579 f = open(os.path.join(self._config.distsroot, full_name,551 f = open(os.path.join(self._config.distsroot, suite,
580 component, architecture, "Release"), "w")552 component, arch_path, "Release"), "w")
581 try:553 try:
582 release_file.dump(f, "utf-8")554 release_file.dump(f, "utf-8")
583 finally:555 finally:
584 f.close()556 f.close()
585557
586 return clean_architecture558 def _writeSuiteSource(self, distroseries, pocket, component,
559 all_series_files):
560 """Write out a Release file for a suite's sources."""
561 self._writeSuiteArchOrSource(
562 distroseries, pocket, component, 'Sources', 'source', 'source',
563 all_series_files)
564
565 def _writeSuiteArch(self, distroseries, pocket, component,
566 arch_name, all_series_files):
567 """Write out a Release file for an architecture in a suite."""
568 file_stub = 'Packages'
569 arch_path = 'binary-' + arch_name
570 # Only the primary and PPA archives have debian-installer.
571 if self.archive.purpose != ArchivePurpose.PARTNER:
572 # Set up the debian-installer paths for main_archive.
573 # d-i paths are nested inside the component.
574 di_path = os.path.join(
575 component, "debian-installer", arch_path)
576 di_file_stub = os.path.join(di_path, file_stub)
577 all_series_files.update(get_suffixed_indices(di_file_stub))
578 self._writeSuiteArchOrSource(
579 distroseries, pocket, component, 'Packages', arch_name, arch_path,
580 all_series_files)
587581
588 def _readIndexFileContents(self, distroseries_name, file_name):582 def _readIndexFileContents(self, distroseries_name, file_name):
589 """Read an index files' contents.583 """Read an index files' contents.
590584
=== modified file 'lib/lp/archivepublisher/tests/test_publisher.py'
--- lib/lp/archivepublisher/tests/test_publisher.py 2010-10-15 11:41:16 +0000
+++ lib/lp/archivepublisher/tests/test_publisher.py 2010-10-15 11:41:18 +0000
@@ -639,13 +639,10 @@
639 ''],639 ''],
640 index_contents)640 index_contents)
641641
642 # Check if apt_handler.release_files_needed has the right requests.
643 # 'source' & 'binary-i386' Release files should be regenerated
644 # for all breezy-autotest components. Note that 'debian-installer'
645 # indexes do not need Release files.
646
647 # We always regenerate all Releases file for a given suite.642 # We always regenerate all Releases file for a given suite.
648 self.checkAllRequestedReleaseFiles(archive_publisher)643 self.assertTrue(
644 ('breezy-autotest', PackagePublishingPocket.RELEASE) in
645 archive_publisher.release_files_needed)
649646
650 # remove PPA root647 # remove PPA root
651 shutil.rmtree(config.personalpackagearchive.root)648 shutil.rmtree(config.personalpackagearchive.root)
@@ -769,45 +766,6 @@
769 # are marked as dirty.766 # are marked as dirty.
770 self.checkDirtyPockets(publisher, expected=allowed_suites)767 self.checkDirtyPockets(publisher, expected=allowed_suites)
771768
772 def assertReleaseFileRequested(self, publisher, suite_name,
773 component_name, archs):
774 """Assert the given context will have it's Release file regenerated.
775
776 Check if a request for the given context is correctly stored in:
777 publisher.apt_handler.release_files_needed
778 """
779 suite = publisher.apt_handler.release_files_needed.get(suite_name)
780 self.assertTrue(
781 suite is not None, 'Suite %s not requested' % suite_name)
782 self.assertTrue(
783 component_name in suite,
784 'Component %s/%s not requested' % (suite_name, component_name))
785 self.assertEquals(archs, suite[component_name])
786
787 def checkAllRequestedReleaseFiles(self, publisher,
788 architecturetags=('hppa', 'i386')):
789 """Check if all expected Release files are going to be regenerated.
790
791 'source', 'binary-i386' and 'binary-hppa' Release Files should be
792 requested for regeneration in all breezy-autotest components.
793 """
794 available_components = sorted([
795 c.name for c in self.breezy_autotest.components])
796 self.assertEqual(available_components,
797 ['main', 'multiverse', 'restricted', 'universe'])
798
799 available_archs = ['binary-%s' % arch for arch in architecturetags]
800
801 # XXX cprov 20071210: Include the artificial component 'source' as a
802 # location to check for generated indexes. Ideally we should
803 # encapsulate this task in publishing.py and this common method
804 # in tests as well.
805 all_archs = set(available_archs)
806 all_archs.add('source')
807 for component in available_components:
808 self.assertReleaseFileRequested(
809 publisher, 'breezy-autotest', component, all_archs)
810
811 def _getReleaseFileOrigin(self, contents):769 def _getReleaseFileOrigin(self, contents):
812 origin_header = 'Origin: '770 origin_header = 'Origin: '
813 [origin_line] = [771 [origin_line] = [
@@ -831,8 +789,9 @@
831 publisher.A_publish(False)789 publisher.A_publish(False)
832 publisher.C_doFTPArchive(False)790 publisher.C_doFTPArchive(False)
833791
834 # We always regenerate all Releases file for a given suite.792 self.assertTrue(
835 self.checkAllRequestedReleaseFiles(publisher)793 ('breezy-autotest', PackagePublishingPocket.RELEASE) in
794 publisher.release_files_needed)
836795
837 publisher.D_writeReleaseFiles(False)796 publisher.D_writeReleaseFiles(False)
838797
@@ -1109,8 +1068,7 @@
1109 """1068 """
11101069
1111 self.assertTrue(1070 self.assertTrue(
1112 series.getSuite(pocket) in1071 (series.name, pocket) in publisher.release_files_needed)
1113 publisher.apt_handler.release_files_needed)
11141072
1115 arch_template = os.path.join(1073 arch_template = os.path.join(
1116 publisher._config.distsroot, series.getSuite(pocket), '%s/%s')1074 publisher._config.distsroot, series.getSuite(pocket), '%s/%s')