Launchpad itself

Merge lp:~edwin-grubbs/launchpad/bug-534462-project-index-timeout into lp:launchpad

  1. bug-534462-project-index-timeout
  2. Merge into devel
Proposed by Edwin Grubbs
Status: Merged
Approved by: Brad Crittenden
Approved revision: no longer in the source branch.
Merged at revision: not available
Proposed branch: lp:~edwin-grubbs/launchpad/bug-534462-project-index-timeout
Merge into: lp:launchpad
Diff against target: 246 lines (+72/-35)
7 files modified
lib/lp/registry/browser/product.py (+3/-4)
lib/lp/registry/doc/distribution.txt (+17/-0)
lib/lp/registry/interfaces/distribution.py (+8/-3)
lib/lp/registry/model/distribution.py (+23/-7)
lib/lp/registry/model/distributionsourcepackage.py (+14/-8)
lib/lp/registry/stories/product/xx-product-index.txt (+1/-1)
lib/lp/registry/templates/product-portlet-packages.pt (+6/-12)
To merge this branch: bzr merge lp:~edwin-grubbs/launchpad/bug-534462-project-index-timeout
Reviewer Review Type Date Requested Status
Brad Crittenden (community) code Approve
Review via email: mp+20980@code.launchpad.net

Commit message

Fixed timeout on gcc project index page due to the Packages in Ubuntu portlet.

Description of the change

Summary
-------

Fixed timeout issue caused by the "Packages in Ubuntu" portlet
on the project index page. I also made DistroSourcePackage.upstream_product
more efficient, even though that is no longer needed by the view,
since searchSourcePackages() can now do the filtering for us.

Implementation details
----------------------

Added has_packaging parameter to searchSourcePackages() so we
do not have to check whether the package has an upstream_product.
    lib/lp/registry/browser/product.py
    lib/lp/registry/doc/distribution.txt
    lib/lp/registry/interfaces/distribution.py
    lib/lp/registry/model/distribution.py

Optimized DistroSourcePackage.upstream_product:
    lib/lp/registry/model/distributionsourcepackage.py

The portlet would either have the title "Packages in Ubuntu" or
"Packages in distributions" depending on the content, which didn't
really make sense, so now there is a single title.
    lib/lp/registry/stories/product/xx-product-index.txt
    lib/lp/registry/templates/product-portlet-packages.pt

Tests
-----

./bin/test -vv -t 'doc/distribution.txt|xx-product-index.txt|distribution-sourcepackage.txt|xx-distribution-source-package.txt'

Demo and Q/A
------------

On launchpad.dev:
* Open http://launchpad.dev/firefox/+packages
  * Remove all the linked packages.
* Open http://launchpad.dev/firefox/+packages
  * The "Packages in Ubuntu" portlet should show
    the mozilla-firefox sourcepackage and a button
    to link to it. (If it doesn't show up, try
    running cronscripts/update-pkgcache.py)

On edge:
* Open https://edge.launchpad.net/gcc
  * This should not timeout.

To post a comment you must log in.
Revision history for this message
Brad Crittenden (bac) :
review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/lp/registry/browser/product.py'
--- lib/lp/registry/browser/product.py 2010-03-08 01:51:58 +0000
+++ lib/lp/registry/browser/product.py 2010-03-09 17:46:07 +0000
@@ -982,14 +982,13 @@
982 """See `LaunchpadFormView`."""982 """See `LaunchpadFormView`."""
983 super(ProductPackagesPortletView, self).setUpFields()983 super(ProductPackagesPortletView, self).setUpFields()
984 ubuntu = getUtility(ILaunchpadCelebrities).ubuntu984 ubuntu = getUtility(ILaunchpadCelebrities).ubuntu
985 source_packages = ubuntu.searchSourcePackages(self.context.name)985 source_packages = ubuntu.searchSourcePackages(
986 self.context.name, has_packaging=False)
986 # Based upon the matches, create a new vocabulary with987 # Based upon the matches, create a new vocabulary with
987 # term descriptions that include a link to the source package.988 # term descriptions that include a link to the source package.
988 self.suggestions = []989 self.suggestions = []
989 vocab_terms = []990 vocab_terms = []
990 for package in source_packages:991 for package in source_packages[:20]:
991 if package.upstream_product is not None:
992 continue
993 self.suggestions.append(package)992 self.suggestions.append(package)
994 item_url = canonical_url(package)993 item_url = canonical_url(package)
995 description = """<a href="%s">%s</a>""" % (994 description = """<a href="%s">%s</a>""" % (
996995
=== modified file 'lib/lp/registry/doc/distribution.txt'
--- lib/lp/registry/doc/distribution.txt 2010-02-15 12:59:55 +0000
+++ lib/lp/registry/doc/distribution.txt 2010-03-09 17:46:07 +0000
@@ -202,6 +202,23 @@
202 DistributionSourcePackage: foobar202 DistributionSourcePackage: foobar
203 DistributionSourcePackage: commercialpackage203 DistributionSourcePackage: commercialpackage
204204
205searchSourcePackages() also has a has_packaging parameter that
206it just passes on to searchSourcePackageCaches(), and it restricts
207the results based on whether the source package has an entry
208in the Packaging table linking it to an upstream project.
209
210 >>> packages = ubuntu.searchSourcePackages('a', has_packaging=True)
211 >>> for dsp in packages:
212 ... print "%s: %s" % (dsp.__class__.__name__, dsp.name)
213 DistributionSourcePackage: mozilla-firefox
214 DistributionSourcePackage: netapplet
215 DistributionSourcePackage: alsa-utils
216 >>> packages = ubuntu.searchSourcePackages('a', has_packaging=False)
217 >>> for dsp in packages:
218 ... print "%s: %s" % (dsp.__class__.__name__, dsp.name)
219 DistributionSourcePackage: foobar
220 DistributionSourcePackage: commercialpackage
221
205222
206=== Searching for binary packages ===223=== Searching for binary packages ===
207224
208225
=== modified file 'lib/lp/registry/interfaces/distribution.py'
--- lib/lp/registry/interfaces/distribution.py 2010-03-05 04:12:27 +0000
+++ lib/lp/registry/interfaces/distribution.py 2010-03-09 17:46:07 +0000
@@ -219,7 +219,8 @@
219 # properties219 # properties
220 currentseries = exported(220 currentseries = exported(
221 Reference(221 Reference(
222 Interface, # Really IDistroSeries, see _schema_circular_imports.py.222 # Really IDistroSeries, see _schema_circular_imports.py.
223 Interface,
223 title=_("Current series"),224 title=_("Current series"),
224 description=_(225 description=_(
225 "The current development series of this distribution. "226 "The current development series of this distribution. "
@@ -405,17 +406,21 @@
405 # _schema_circular_imports.py.406 # _schema_circular_imports.py.
406 @operation_returns_collection_of(Interface)407 @operation_returns_collection_of(Interface)
407 @export_read_operation()408 @export_read_operation()
408 def searchSourcePackages(text):409 def searchSourcePackages(text, has_packaging=None):
409 """Search for source packages that correspond to the given text.410 """Search for source packages that correspond to the given text.
410411
411 This method just decorates the result of searchSourcePackageCaches()412 This method just decorates the result of searchSourcePackageCaches()
412 to return DistributionSourcePackages.413 to return DistributionSourcePackages.
413 """414 """
414415
415 def searchSourcePackageCaches(text):416 def searchSourcePackageCaches(text, has_packaging=None):
416 """Search for source packages that correspond to the given text.417 """Search for source packages that correspond to the given text.
417418
418 :param text: The text that will be matched.419 :param text: The text that will be matched.
420 :param has_packaging: If True, it will filter out
421 packages with no packaging (i.e. no link to the upstream
422 project). False will do the reverse filtering, and None
423 will do no filtering on this field.
419 :return: A result set containing424 :return: A result set containing
420 (DistributionSourcePackageCache, SourcePackageName, rank) tuples425 (DistributionSourcePackageCache, SourcePackageName, rank) tuples
421 ordered by rank.426 ordered by rank.
422427
=== modified file 'lib/lp/registry/model/distribution.py'
--- lib/lp/registry/model/distribution.py 2010-03-06 21:38:23 +0000
+++ lib/lp/registry/model/distribution.py 2010-03-09 17:46:07 +0000
@@ -927,7 +927,8 @@
927 cache.binpkgdescriptions = ' '.join(sorted(binpkgdescriptions))927 cache.binpkgdescriptions = ' '.join(sorted(binpkgdescriptions))
928 cache.changelog = ' '.join(sorted(sprchangelog))928 cache.changelog = ' '.join(sorted(sprchangelog))
929929
930 def searchSourcePackageCaches(self, text):930 def searchSourcePackageCaches(
931 self, text, has_packaging=None):
931 """See `IDistribution`."""932 """See `IDistribution`."""
932 # The query below tries exact matching on the source package933 # The query below tries exact matching on the source package
933 # name as well; this is because source package names are934 # name as well; this is because source package names are
@@ -948,25 +949,40 @@
948 )949 )
949 ]950 ]
950951
952
953 packaging_query = """
954 SELECT 1
955 FROM Packaging
956 WHERE Packaging.sourcepackagename = SourcePackageName.id
957 """
958 has_packaging_condition = ''
959 if has_packaging is True:
960 has_packaging_condition = 'AND EXISTS (%s)' % packaging_query
961 elif has_packaging is False:
962 has_packaging_condition = 'AND NOT EXISTS (%s)' % packaging_query
963
951 # Note: When attempting to convert the query below into straight964 # Note: When attempting to convert the query below into straight
952 # Storm expressions, a 'tuple index out-of-range' error was always965 # Storm expressions, a 'tuple index out-of-range' error was always
953 # raised.966 # raised.
954 dsp_caches_with_ranks = store.using(*origin).find(967 condition = """
955 find_spec,968 distribution = %s AND
956 """distribution = %s AND
957 archive IN %s AND969 archive IN %s AND
958 (fti @@ ftq(%s) OR970 (fti @@ ftq(%s) OR
959 DistributionSourcePackageCache.name ILIKE '%%' || %s || '%%')971 DistributionSourcePackageCache.name ILIKE '%%' || %s || '%%')
972 %s
960 """ % (quote(self), quote(self.all_distro_archive_ids),973 """ % (quote(self), quote(self.all_distro_archive_ids),
961 quote(text), quote_like(text))974 quote(text), quote_like(text), has_packaging_condition)
975 dsp_caches_with_ranks = store.using(*origin).find(
976 find_spec, condition
962 ).order_by('rank DESC')977 ).order_by('rank DESC')
963978
964 return dsp_caches_with_ranks979 return dsp_caches_with_ranks
965980
966 def searchSourcePackages(self, text):981 def searchSourcePackages(self, text, has_packaging=None):
967 """See `IDistribution`."""982 """See `IDistribution`."""
968983
969 dsp_caches_with_ranks = self.searchSourcePackageCaches(text)984 dsp_caches_with_ranks = self.searchSourcePackageCaches(
985 text, has_packaging=has_packaging)
970986
971 # Create a function that will decorate the resulting987 # Create a function that will decorate the resulting
972 # DistributionSourcePackageCaches, converting988 # DistributionSourcePackageCaches, converting
973989
=== modified file 'lib/lp/registry/model/distributionsourcepackage.py'
--- lib/lp/registry/model/distributionsourcepackage.py 2010-03-08 11:22:05 +0000
+++ lib/lp/registry/model/distributionsourcepackage.py 2010-03-09 17:46:07 +0000
@@ -22,6 +22,8 @@
2222
23from canonical.database.sqlbase import sqlvalues23from canonical.database.sqlbase import sqlvalues
24from canonical.launchpad.database.emailaddress import EmailAddress24from canonical.launchpad.database.emailaddress import EmailAddress
25from lp.registry.model.distroseries import DistroSeries
26from lp.registry.model.packaging import Packaging
25from lp.registry.model.structuralsubscription import (27from lp.registry.model.structuralsubscription import (
26 StructuralSubscriptionTargetMixin)28 StructuralSubscriptionTargetMixin)
27from canonical.launchpad.interfaces.lpstorm import IStore29from canonical.launchpad.interfaces.lpstorm import IStore
@@ -315,12 +317,18 @@
315317
316 @property318 @property
317 def upstream_product(self):319 def upstream_product(self):
318 for distroseries in self.distribution.series:320 store = Store.of(self.sourcepackagename)
319 source_package = distroseries.getSourcePackage(321 condition = And(
320 self.sourcepackagename)322 Packaging.sourcepackagename == self.sourcepackagename,
321 if source_package.direct_packaging is not None:323 Packaging.distroseriesID == DistroSeries.id,
322 return source_package.direct_packaging.productseries.product324 DistroSeries.distribution == self.distribution
323 return None325 )
326 result = store.find(Packaging, condition)
327 result.order_by("debversion_sort_key(version) DESC")
328 if result.count() == 0:
329 return None
330 else:
331 return result[0].productseries.product
324332
325 # XXX kiko 2006-08-16: Bad method name, no need to be a property.333 # XXX kiko 2006-08-16: Bad method name, no need to be a property.
326 @property334 @property
@@ -353,8 +361,6 @@
353361
354 def getReleasesAndPublishingHistory(self):362 def getReleasesAndPublishingHistory(self):
355 """See `IDistributionSourcePackage`."""363 """See `IDistributionSourcePackage`."""
356 # Local import of DistroSeries to avoid import loop.
357 from lp.registry.model.distroseries import DistroSeries
358 store = Store.of(self.distribution)364 store = Store.of(self.distribution)
359 result = store.find(365 result = store.find(
360 (SourcePackageRelease, SourcePackagePublishingHistory),366 (SourcePackageRelease, SourcePackagePublishingHistory),
361367
=== modified file 'lib/lp/registry/stories/product/xx-product-index.txt'
--- lib/lp/registry/stories/product/xx-product-index.txt 2010-03-01 21:48:15 +0000
+++ lib/lp/registry/stories/product/xx-product-index.txt 2010-03-09 17:46:07 +0000
@@ -359,7 +359,7 @@
359 >>> print extract_text(359 >>> print extract_text(
360 ... find_tag_by_id(user_browser.contents, 'portlet-packages'))360 ... find_tag_by_id(user_browser.contents, 'portlet-packages'))
361 All packages361 All packages
362 Packages in distributions362 Packages in Ubuntu
363 “mozilla-firefox” source package in Hoary363 “mozilla-firefox” source package in Hoary
364 “mozilla-firefox” source package in Warty Version 0.9 uploaded on...364 “mozilla-firefox” source package in Warty Version 0.9 uploaded on...
365365
366366
=== modified file 'lib/lp/registry/templates/product-portlet-packages.pt'
--- lib/lp/registry/templates/product-portlet-packages.pt 2010-03-01 21:48:15 +0000
+++ lib/lp/registry/templates/product-portlet-packages.pt 2010-03-09 17:46:07 +0000
@@ -6,14 +6,14 @@
6 define="packages context/sourcepackages">6 define="packages context/sourcepackages">
77
8<div class="portlet" id="portlet-packages">8<div class="portlet" id="portlet-packages">
9 <h2>
10 <span class="see-all"><a
11 tal:attributes="href context/menu:overview/packages/fmt:url">
12 All packages</a></span>
13 Packages in Ubuntu
14 </h2>
915
10 <tal:has_packages condition="packages">16 <tal:has_packages condition="packages">
11 <h2>
12 <span class="see-all"><a
13 tal:attributes="href context/menu:overview/packages/fmt:url">All
14 packages</a></span>
15 Packages in distributions
16 </h2>
1717
18 <ul>18 <ul>
19 <tal:pair tal:repeat="package packages">19 <tal:pair tal:repeat="package packages">
@@ -35,12 +35,6 @@
35 </tal:has_packages>35 </tal:has_packages>
3636
37 <tal:has_no_packages condition="not:packages">37 <tal:has_no_packages condition="not:packages">
38 <h2>
39 <span class="see-all"><a
40 tal:attributes="href context/menu:overview/packages/fmt:url">All
41 packages</a></span>
42 Packages in Ubuntu
43 </h2>
4438
45 <p>39 <p>
46 Launchpad doesn't know which Ubuntu packages this project40 Launchpad doesn't know which Ubuntu packages this project