On Sun, Nov 15, 2009 at 2:35 PM, Michael Nelson
<email address hidden> wrote:
> Michael Nelson has proposed merging lp:~michael.nelson/launchpad/443353-api-builds-from-private into lp:launchpad.
>
> Requested reviews:
> Canonical Launchpad Engineering (launchpad)
> Related bugs:
> #443353 API does not include Builds for sources that were sync'd from private PPAs
> https://bugs.launchpad.net/bugs/443353
>
>
> Overview
> ========
>
> This branch fixes bug 443353 by ensuring that getBuildsForSources() will include builds that were originally built in a different archive context, but have since had binaries copied into the source archive context.
>
> Issues
> ======
>
> There are two main issues with this branch IMO.
>
> 1. Ensuring that the result of the union was ordered correctly is hackish, and dependent on a storm implementation detail (that columns in SQL queries for a certain table are ordered alphabetically).
>
> 2. Testing the correct ordering in the doctest isn't so readable. It could be better to print out the results instead, but on the other hand, I didn't want the test to be dependent on database ids. Previously the test simply ensured the last item was the one expected - I've updated this to instead assure that the complete result is sorted as expected.
>
> Any suggestions welcome!
>
Hi Michael,
Thanks for fixing this -- on a Sunday even!
I'm generally OK with this branch, but would like another opportunity
to have a look at it and maybe to talk with you face-to-face.
jml
> Testing/QA
> ==========
>
> To test, run:
> bin/test -vvt doc/publishing.txt
>
> To QA:
>
> Visit:
> https://edge.launchpad.net/ubuntu/+source/openjdk-6
>
> and expand the intrepid 6b12-0ubuntu6.6 security release. Currently this only displays two builds, where as it should display all the builds listed at:
>
> https://edge.launchpad.net/ubuntu/+source/openjdk-6/6b12-0ubuntu6.6
>
>
>
>
> --
> https://code.launchpad.net/~michael.nelson/launchpad/443353-api-builds-from-private/+merge/14896
> Your team Launchpad code reviewers from Canonical is subscribed to branch lp:launchpad.
>
> === modified file 'lib/lp/soyuz/doc/publishing.txt'
> --- lib/lp/soyuz/doc/publishing.txt 2009-11-05 10:51:36 +0000
> +++ lib/lp/soyuz/doc/publishing.txt 2009-11-15 20:35:29 +0000
> @@ -1055,22 +1055,53 @@
> each build found.
>
> >>> cprov_builds.count()
> - 7
> + 8
>
> The `ResultSet` is ordered by ascending
> `SourcePackagePublishingHistory.id` and ascending
> `DistroArchseries.architecturetag` in this order.
>
> - >>> source_pub, build, arch = cprov_builds.last()
> -
> - >>> print source_pub.displayname
> - foo 666 in breezy-autotest
> -
> - >>> print build.title
> - i386 build of foo 666 in ubuntutest breezy-autotest RELEASE
> -
> - >>> print arch.displayname
> - ubuntutest Breezy Badger Autotest i386
> + # The easiest thing we can do here (without printing ids)
> + # is to show that sorting a list of the resulting ids+tags does not
> + # modify the list.
> + >>> from copy import copy
> + >>> ids_and_tags = [(pub.id, arch.architecturetag)
> + ... for pub, build, arch in cprov_builds]
> + >>> ids_and_tags_sorted = copy(ids_and_tags)
> + >>> ids_and_tags_sorted.sort()
> + >>> ids_and_tags == ids_and_tags_sorted
> + True
> +
You don't have to use copy() and .sort(), you can use sorted()
instead. That would be slightly nicer.
Nicer still would be an assertSorted() method called from a unit test :P
> +If a source package is copied from another archive (including the
> +binaries), then these builds will also be included in the result (even
> +though they were build in a different archive context).
> +
Saying something about _why_ would greatly improve this as a document.
Can you please add something like that?
> + # Create a new PPA and publish a source with some builds
> + # and binaries.
> + >>> other_ppa = factory.makeArchive()
> + >>> binaries = test_publisher.getPubBinaries(archive=other_ppa)
> + >>> build = binaries[0].binarypackagerelease.build
> + >>> source_pub = build.sourcepackagerelease.publishings[0]
> +
Can you please add some comments here about how the context of the
build is _not_ Celso's PPA? This can be easily lost in skimming the
docs.
> + # Copy the source into Celso's PPA, ensuring that the binaries
> + # are alse published there.
> + >>> source_pub_cprov = source_pub.copyTo(
> + ... source_pub.distroseries, source_pub.pocket,
> + ... cprov.archive)
> + >>> binaries_cprov = test_publisher.publishBinaryInArchive(
> + ... binaries[0].binarypackagerelease, cprov.archive)
> +
> +Now we will see an extra source in Celso's PPA as well as an extra
> +build - even though the build's context is not Celso's PPA.
> +
> + >>> cprov_sources_new = cprov.archive.getPublishedSources()
> + >>> cprov_sources_new.count()
> + 9
> +
> + >>> cprov_builds_new = publishing_set.getBuildsForSources(
> + ... cprov_sources_new)
> + >>> cprov_builds_new.count()
> + 9
>
Maybe it's worth adding a note in the text saying "(It was 8 befgore
we started)".
> Next we'll create two sources with two builds each (the SoyuzTestPublisher
> default) and show that the number of unpublished builds for these sources
>
> === modified file 'lib/lp/soyuz/model/publishing.py'
> --- lib/lp/soyuz/model/publishing.py 2009-10-29 11:55:35 +0000
> +++ lib/lp/soyuz/model/publishing.py 2009-11-15 20:35:29 +0000
> @@ -40,6 +40,7 @@
> from canonical.database.enumcol import EnumCol
> from lp.registry.interfaces.pocket import PackagePublishingPocket
> from lp.soyuz.model.binarypackagename import BinaryPackageName
> +from lp.soyuz.model.binarypackagerelease import BinaryPackageRelease
> from lp.soyuz.model.files import (
> BinaryPackageFile, SourcePackageReleaseFile)
> from canonical.launchpad.database.librarian import (
> @@ -1233,20 +1234,59 @@
> Build.buildstate.is_in(build_states))
>
> store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
> - result_set = store.find(
> - (SourcePackagePublishingHistory, Build, DistroArchSeries),
> + find_spec = (
> + SourcePackagePublishingHistory, Build, DistroArchSeries)
> +
> + # We'll be looking for builds in the same distroseries as the
> + # SPPH for the same release.
> + builds_for_distroseries_expr = (
> Build.distroarchseriesID == DistroArchSeries.id,
> + SourcePackagePublishingHistory.distroseriesID ==
> + DistroArchSeries.distroseriesID,
> + SourcePackagePublishingHistory.sourcepackagereleaseID ==
> + Build.sourcepackagereleaseID,
> + In(SourcePackagePublishingHistory.id, source_publication_ids)
> + )
> +
I think it's better style to close parens on the same line. PEP 8
agrees with me, IIRC.
> + # First, we'll find the builds that were built in the same
> + # archive context as the published sources.
> + builds_in_same_archive = store.find(
> + find_spec,
> + builds_for_distroseries_expr,
> SourcePackagePublishingHistory.archiveID == Build.archiveID,
> - SourcePackagePublishingHistory.distroseriesID ==
> - DistroArchSeries.distroseriesID,
> - SourcePackagePublishingHistory.sourcepackagereleaseID ==
> - Build.sourcepackagereleaseID,
> - In(SourcePackagePublishingHistory.id, source_publication_ids),
> - *extra_exprs)
> -
> + *extra_exprs)
> +
> + # Next get all the builds that have a binary published in the
> + # same archive... even though the build was not built in
> + # the same context archive.
> + builds_copied_into_archive = store.find(
> + find_spec,
> + builds_for_distroseries_expr,
> + SourcePackagePublishingHistory.archiveID != Build.archiveID,
> + BinaryPackagePublishingHistory.archive == Build.archiveID,
> + BinaryPackagePublishingHistory.binarypackagerelease ==
> + BinaryPackageRelease.id,
> + BinaryPackageRelease.build == Build.id,
> + *extra_exprs)
> +
> + result_set = builds_copied_into_archive.union(
> + builds_in_same_archive).config(distinct=True)
> +
> + # XXX 2009-11-15 Michael Nelson bug=366043. It is not possible
> + # to sort by the name `SourcePackagePublishingHistory.id` as after
> + # the union there are no tables. Nor can we sort by ambiguous `id`
> + # as in this case there are 3 id columns in the result. Specifying
> + # the column index is the only option that I can find. So we're
> + # relying on an implementation detail of Storm that it lists
> + # columns in alphabetical order in sql queries.
I don't get what you mean by "ambiguous `id`". Surely the ID columns
can be referred to by their table name as before?
It might be a good idea to grab Jamu and see if he can solve your problem.
> + sql_columns = SourcePackagePublishingHistory._storm_columns.values()
> + sql_column_names = [column.name for column in sql_columns]
> + sql_column_names.sort()
> +
That's pretty terrible, but if that's what you've got to do, then I'm OK.
You can use sorted() here to make things a little better.
> + # SQL order by uses 1-based column numbers.
> + source_pub_id_col_number = sql_column_names.index('id') + 1
Given that all of this is a workaround for a storm bug, maybe it's a
good idea to put it into a small helper function?
On Sun, Nov 15, 2009 at 2:35 PM, Michael Nelson /bugs.launchpad .net/bugs/ 443353 rces() will include builds that were originally built in a different archive context, but have since had binaries copied into the source archive context.
<email address hidden> wrote:
> Michael Nelson has proposed merging lp:~michael.nelson/launchpad/443353-api-builds-from-private into lp:launchpad.
>
> Requested reviews:
> Canonical Launchpad Engineering (launchpad)
> Related bugs:
> #443353 API does not include Builds for sources that were sync'd from private PPAs
> https:/
>
>
> Overview
> ========
>
> This branch fixes bug 443353 by ensuring that getBuildsForSou
>
> Issues
> ======
>
> There are two main issues with this branch IMO.
>
> 1. Ensuring that the result of the union was ordered correctly is hackish, and dependent on a storm implementation detail (that columns in SQL queries for a certain table are ordered alphabetically).
>
> 2. Testing the correct ordering in the doctest isn't so readable. It could be better to print out the results instead, but on the other hand, I didn't want the test to be dependent on database ids. Previously the test simply ensured the last item was the one expected - I've updated this to instead assure that the complete result is sorted as expected.
>
> Any suggestions welcome!
>
Hi Michael,
Thanks for fixing this -- on a Sunday even!
I'm generally OK with this branch, but would like another opportunity
to have a look at it and maybe to talk with you face-to-face.
jml
> Testing/QA /edge.launchpad .net/ubuntu/ +source/ openjdk- 6 /edge.launchpad .net/ubuntu/ +source/ openjdk- 6/6b12- 0ubuntu6. 6 /code.launchpad .net/~michael. nelson/ launchpad/ 443353- api-builds- from-private/ +merge/ 14896 soyuz/doc/ publishing. txt' soyuz/doc/ publishing. txt 2009-11-05 10:51:36 +0000 soyuz/doc/ publishing. txt 2009-11-15 20:35:29 +0000 count() ublishingHistor y.id` and ascending es.architecture tag` in this order. pub.displayname retag) tags_sorted. sort()
> ==========
>
> To test, run:
> bin/test -vvt doc/publishing.txt
>
> To QA:
>
> Visit:
> https:/
>
> and expand the intrepid 6b12-0ubuntu6.6 security release. Currently this only displays two builds, where as it should display all the builds listed at:
>
> https:/
>
>
>
>
> --
> https:/
> Your team Launchpad code reviewers from Canonical is subscribed to branch lp:launchpad.
>
> === modified file 'lib/lp/
> --- lib/lp/
> +++ lib/lp/
> @@ -1055,22 +1055,53 @@
> each build found.
>
> >>> cprov_builds.
> - 7
> + 8
>
> The `ResultSet` is ordered by ascending
> `SourcePackageP
> `DistroArchseri
>
> - >>> source_pub, build, arch = cprov_builds.last()
> -
> - >>> print source_
> - foo 666 in breezy-autotest
> -
> - >>> print build.title
> - i386 build of foo 666 in ubuntutest breezy-autotest RELEASE
> -
> - >>> print arch.displayname
> - ubuntutest Breezy Badger Autotest i386
> + # The easiest thing we can do here (without printing ids)
> + # is to show that sorting a list of the resulting ids+tags does not
> + # modify the list.
> + >>> from copy import copy
> + >>> ids_and_tags = [(pub.id, arch.architectu
> + ... for pub, build, arch in cprov_builds]
> + >>> ids_and_tags_sorted = copy(ids_and_tags)
> + >>> ids_and_
> + >>> ids_and_tags == ids_and_tags_sorted
> + True
> +
You don't have to use copy() and .sort(), you can use sorted()
instead. That would be slightly nicer.
Nicer still would be an assertSorted() method called from a unit test :P
> +If a source package is copied from another archive (including the
> +binaries), then these builds will also be included in the result (even
> +though they were build in a different archive context).
> +
Saying something about _why_ would greatly improve this as a document.
Can you please add something like that?
> + # Create a new PPA and publish a source with some builds makeArchive( ) getPubBinaries( archive= other_ppa) 0].binarypackag erelease. build kagerelease. publishings[ 0]
> + # and binaries.
> + >>> other_ppa = factory.
> + >>> binaries = test_publisher.
> + >>> build = binaries[
> + >>> source_pub = build.sourcepac
> +
Can you please add some comments here about how the context of the
build is _not_ Celso's PPA? This can be easily lost in skimming the
docs.
> + # Copy the source into Celso's PPA, ensuring that the binaries pub.distroserie s, source_pub.pocket, publishBinaryIn Archive( 0].binarypackag erelease, cprov.archive) getPublishedSou rces() new.count( ) set.getBuildsFo rSources( new.count( )
> + # are alse published there.
> + >>> source_pub_cprov = source_pub.copyTo(
> + ... source_
> + ... cprov.archive)
> + >>> binaries_cprov = test_publisher.
> + ... binaries[
> +
> +Now we will see an extra source in Celso's PPA as well as an extra
> +build - even though the build's context is not Celso's PPA.
> +
> + >>> cprov_sources_new = cprov.archive.
> + >>> cprov_sources_
> + 9
> +
> + >>> cprov_builds_new = publishing_
> + ... cprov_sources_new)
> + >>> cprov_builds_
> + 9
>
Maybe it's worth adding a note in the text saying "(It was 8 befgore
we started)".
> Next we'll create two sources with two builds each (the SoyuzTestPublisher soyuz/model/ publishing. py' soyuz/model/ publishing. py 2009-10-29 11:55:35 +0000 soyuz/model/ publishing. py 2009-11-15 20:35:29 +0000 database. enumcol import EnumCol interfaces. pocket import PackagePublishi ngPocket model.binarypac kagename import BinaryPackageName model.binarypac kagerelease import BinaryPackageRe lease model.files import ( leaseFile) launchpad. database. librarian import ( e.is_in( build_states) ) IStoreSelector) .get(MAIN_ STORE, DEFAULT_FLAVOR) ublishingHistor y, Build, DistroArchSeries), blishingHistory , Build, DistroArchSeries) for_distroserie s_expr = ( hseriesID == DistroArchSerie s.id, blishingHistory .distroseriesID == s.distroseriesI D, blishingHistory .sourcepackager eleaseID == kagereleaseID, ePublishingHist ory.id, source_ publication_ ids)
> default) and show that the number of unpublished builds for these sources
>
> === modified file 'lib/lp/
> --- lib/lp/
> +++ lib/lp/
> @@ -40,6 +40,7 @@
> from canonical.
> from lp.registry.
> from lp.soyuz.
> +from lp.soyuz.
> from lp.soyuz.
> BinaryPackageFile, SourcePackageRe
> from canonical.
> @@ -1233,20 +1234,59 @@
> Build.buildstat
>
> store = getUtility(
> - result_set = store.find(
> - (SourcePackageP
> + find_spec = (
> + SourcePackagePu
> +
> + # We'll be looking for builds in the same distroseries as the
> + # SPPH for the same release.
> + builds_
> Build.distroarc
> + SourcePackagePu
> + DistroArchSerie
> + SourcePackagePu
> + Build.sourcepac
> + In(SourcePackag
> + )
> +
I think it's better style to close parens on the same line. PEP 8
agrees with me, IIRC.
> + # First, we'll find the builds that were built in the same in_same_ archive = store.find( for_distroserie s_expr, blishingHistory .archiveID == Build.archiveID, blishingHistory .distroseriesID == s.distroseriesI D, blishingHistory .sourcepackager eleaseID == kagereleaseID, ePublishingHist ory.id, source_ publication_ ids), copied_ into_archive = store.find( for_distroserie s_expr, blishingHistory .archiveID != Build.archiveID, blishingHistory .archive == Build.archiveID, blishingHistory .binarypackager elease == lease.id, lease.build == Build.id, copied_ into_archive. union( in_same_ archive) .config( distinct= True) ublishingHistor y.id` as after
> + # archive context as the published sources.
> + builds_
> + find_spec,
> + builds_
> SourcePackagePu
> - SourcePackagePu
> - DistroArchSerie
> - SourcePackagePu
> - Build.sourcepac
> - In(SourcePackag
> - *extra_exprs)
> -
> + *extra_exprs)
> +
> + # Next get all the builds that have a binary published in the
> + # same archive... even though the build was not built in
> + # the same context archive.
> + builds_
> + find_spec,
> + builds_
> + SourcePackagePu
> + BinaryPackagePu
> + BinaryPackagePu
> + BinaryPackageRe
> + BinaryPackageRe
> + *extra_exprs)
> +
> + result_set = builds_
> + builds_
> +
> + # XXX 2009-11-15 Michael Nelson bug=366043. It is not possible
> + # to sort by the name `SourcePackageP
> + # the union there are no tables. Nor can we sort by ambiguous `id`
> + # as in this case there are 3 id columns in the result. Specifying
> + # the column index is the only option that I can find. So we're
> + # relying on an implementation detail of Storm that it lists
> + # columns in alphabetical order in sql queries.
I don't get what you mean by "ambiguous `id`". Surely the ID columns
can be referred to by their table name as before?
It might be a good idea to grab Jamu and see if he can solve your problem.
> + sql_columns = SourcePackagePu blishingHistory ._storm_ columns. values( ) names.sort( )
> + sql_column_names = [column.name for column in sql_columns]
> + sql_column_
> +
That's pretty terrible, but if that's what you've got to do, then I'm OK.
You can use sorted() here to make things a little better.
> + # SQL order by uses 1-based column numbers. pub_id_ col_number = sql_column_ names.index( 'id') + 1
> + source_
Given that all of this is a workaround for a storm bug, maybe it's a
good idea to put it into a small helper function?
> result_ set.order_ by( blishingHistory .id, s.architecturet ag) pub_id_ col_number, DistroArchSerie s.architecturet ag)
> - SourcePackagePu
> - DistroArchSerie
> + source_
>
> return result_set
>