Launchpad itself

Merge lp:~salgado/launchpad/remove-security-upload-policy into lp:launchpad

  1. remove-security-upload-policy
  2. Merge into devel
Proposed by Guilherme Salgado
Status: Merged
Approved by: Michael Nelson
Approved revision: no longer in the source branch.
Merged at revision: 11450
Proposed branch: lp:~salgado/launchpad/remove-security-upload-policy
Merge into: lp:launchpad
Diff against target: 979 lines (+144/-722)
6 files modified
lib/lp/archiveuploader/tests/nascentupload-announcements.txt (+8/-269)
lib/lp/archiveuploader/tests/nascentupload-security-uploads.txt (+0/-144)
lib/lp/archiveuploader/tests/test_buildduploads.py (+130/-4)
lib/lp/archiveuploader/tests/test_securityuploads.py (+0/-263)
lib/lp/archiveuploader/uploadpolicy.py (+1/-24)
lib/lp/soyuz/doc/soyuz-set-of-uploads.txt (+5/-18)
To merge this branch: bzr merge lp:~salgado/launchpad/remove-security-upload-policy
Reviewer Review Type Date Requested Status
Michael Nelson (community) code Approve
Jeroen T. Vermeulen (community) Approve
Review via email: mp+33581@code.launchpad.net

Description of the change

Remove SecurityUploadPolicy together with its tests as it's no longer used.

To post a comment you must log in.
Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

Wow, that's a very red diff. Death to doctests!

Not much here to review, really. I'll have to trust that you're not bluffing on how an email to Daniel Silverstone turned into one to Celso Providelo in lines 286—307 of the diff. But then again, doctests have a tendency to invite bluffing and I can't help feeling it's their own fault.

review: Approve
Revision history for this message
Guilherme Salgado (salgado) wrote :

The change from an email do Daniel to one from Celso is in a test that
gets a random (in fact, the last one) upload and prints some details
about it. It changed because I removed the code that did an upload
right before that test.

I got back from ec2 test this morning and soyuz-set-of-uploads.txt is
failing because it uses the security policy in some parts, so I'm having
yet more fun with doctests. I'll let you know once I figure out a way to
fix it -- using one of the test-specific policies that accept anything
is not working.

Revision history for this message
Guilherme Salgado (salgado) wrote :

I've managed to get that test passing again, as well as fixing another one that I broke because it was importing its base class from a file that had been removed.

In the latter case I just had to copy the class from the removed file back.

For the former I removed a chunk of the test but to make sure we don't lose coverage I'll write a unit test on my vostok-upload-policy, which adds a test_uploadpolicy.py file. The rest should be self-explanatory.

Revision history for this message
Michael Nelson (michael.nelson) wrote :

Approving changes in r11428.

14:40 < salgado> noodles775, and I've added a new unit test (in the other MP) to exercise the same code that was exercised by the doctest chunk I've removed
14:40 < noodles775> salgado: The doctest snippet that you removed in r11428 (reject instead of fail for non-existent pocket), you mention that you'll cover it with a unit test in the following branch...
14:40 < noodles775> lol
14:40 < noodles775> but
14:40 < salgado> :)
14:41 < salgado> http://bazaar.launchpad.net/~salgado/launchpad/vostok-upload-policy/revision/11432
14:41 < noodles775> as far as I can see, the following branch tests that an assertion is raised, but not that a rejection is sent
14:41 * noodles775 double checks, guessing that he missed it.
14:41 < salgado> that's correct
14:42 < noodles775> Is there a test elsewhere that ensures when the assertion is raised an email is sent?
14:42 < salgado> I thought just checking for the exception was good enough, as we have plenty of tests showing a rejection is sent when there's an exception

review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/lp/archiveuploader/tests/nascentupload-announcements.txt'
--- lib/lp/archiveuploader/tests/nascentupload-announcements.txt 2010-08-20 12:11:30 +0000
+++ lib/lp/archiveuploader/tests/nascentupload-announcements.txt 2010-08-25 19:32:53 +0000
@@ -17,13 +17,6 @@
17 * AUTO-APPROVED to BACKPORTS (via sync): submitter set receives an17 * AUTO-APPROVED to BACKPORTS (via sync): submitter set receives an
18 'acceptance' warning ('announcement' is skipped).18 'acceptance' warning ('announcement' is skipped).
1919
20 * AUTO-APPROVED sources to SECURITY (via security): submitter set
21 receives an 'acceptance' warning and the target distroseries
22 changeslist address receives an 'announcement' message.
23
24 * AUTO-APPROVED binaries to SECURITY (via security): submitter set
25 receives only an 'acceptance' warning ('announcement' is skipped).
26
27 * NEW, AUTO-APPROVED or UNAPPROVED source uploads targeted to section20 * NEW, AUTO-APPROVED or UNAPPROVED source uploads targeted to section
28 'translations' (all policies, all pockets) do not generate any21 'translations' (all policies, all pockets) do not generate any
29 messages. Remembering that NEW and UNAPPROVED messages are also22 messages. Remembering that NEW and UNAPPROVED messages are also
@@ -530,196 +523,6 @@
530 >>> import os523 >>> import os
531 >>> os.remove(os.path.join(datadir('suite/bar_1.0-4'), 'bar_1.0.orig.tar.gz'))524 >>> os.remove(os.path.join(datadir('suite/bar_1.0-4'), 'bar_1.0.orig.tar.gz'))
532525
533AUTO-APPROVED source upload to SECURITY pocket via 'security' policy:
534
535 >>> security_policy = getPolicy(
536 ... name='security', distro='ubuntu', distroseries=None)
537 >>> security_policy.setDistroSeriesAndPocket('hoary-security')
538
539 >>> bar_src = NascentUpload.from_changesfile_path(
540 ... datadir('suite/bar_1.0-2/bar_1.0-2_source.changes'),
541 ... security_policy, mock_logger_quiet)
542 >>> bar_src.process()
543
544 >>> bar_src.logger = mock_logger
545 >>> result = bar_src.do_accept()
546 DEBUG: Creating queue entry
547 ...
548 DEBUG: Sent a mail:
549 DEBUG: Subject: [ubuntu/hoary-security] bar 1.0-2 (Accepted)
550 DEBUG: Recipients: Daniel Silverstone <daniel.silverstone@canonical.com>
551 DEBUG: Body:
552 DEBUG: bar (1.0-2) breezy; urgency=low
553 DEBUG:
554 DEBUG: * A second upload to ensure that binary overrides of _all work
555 DEBUG:
556 DEBUG: * Also closes Launchpad bug #6
557 DEBUG:
558 DEBUG:
559 DEBUG: Date: Thu, 30 Mar 2006 01:36:14 +0100
560 DEBUG: Changed-By: Daniel Silverstone <daniel.silverstone@canonical.com>
561 DEBUG: Maintainer: Launchpad team <launchpad@lists.canonical.com>
562 DEBUG: http://launchpad.dev/ubuntu/hoary/+source/bar/1.0-2
563 DEBUG:
564 DEBUG: ==
565 DEBUG:
566 DEBUG: Announcing to hoary-announce@lists.ubuntu.com
567 DEBUG:
568 DEBUG: Thank you for your contribution to Ubuntu Linux.
569 DEBUG:
570 DEBUG: --
571 DEBUG: You are receiving this email because you are the uploader, maintainer or
572 DEBUG: signer of the above package.
573 DEBUG: Sent a mail:
574 DEBUG: Subject: [ubuntu/hoary-security] bar 1.0-2 (Accepted)
575 DEBUG: Recipients: hoary-announce@lists.ubuntu.com
576 DEBUG: Body:
577 DEBUG: bar (1.0-2) breezy; urgency=low
578 DEBUG:
579 DEBUG: * A second upload to ensure that binary overrides of _all work
580 DEBUG:
581 DEBUG: * Also closes Launchpad bug #6
582 DEBUG:
583 DEBUG:
584 DEBUG: Date: Thu, 30 Mar 2006 01:36:14 +0100
585 DEBUG: Changed-By: Daniel Silverstone <daniel.silverstone@canonical.com>
586 DEBUG: Maintainer: Launchpad team <launchpad@lists.canonical.com>
587 DEBUG: http://launchpad.dev/ubuntu/hoary/+source/bar/1.0-2
588
589 >>> import operator
590 >>> msgs = pop_notifications(sort_key=operator.itemgetter('To'))
591 >>> len(msgs)
592 2
593
594 >>> [message['To'] for message in msgs]
595 ['Daniel Silverstone <daniel.silverstone@canonical.com>',
596 'hoary-announce@lists.ubuntu.com']
597
598 >>> [message['Subject'] for message in msgs]
599 ['[ubuntu/hoary-security] bar 1.0-2 (Accepted)',
600 '[ubuntu/hoary-security] bar 1.0-2 (Accepted)']
601
602 >>> [message['X-Katie'] for message in msgs]
603 ['Launchpad actually', 'Launchpad actually']
604
605The upload notification message has an attachment with the original changes
606file.
607
608 >>> announcement = msgs[0]
609 >>> attachment = announcement.get_payload()[1]
610 >>> attachment['Content-Disposition']
611 'attachment; filename="changesfile"'
612
613Here it is:
614
615 >>> print attachment.as_string() # doctest: -NORMALIZE_WHITESPACE
616 Content-Type: text/plain; charset="utf-8"
617 MIME-Version: 1.0
618 Content-Transfer-Encoding: quoted-printable
619 Content-Disposition: attachment; filename="changesfile"
620 <BLANKLINE>
621 -----BEGIN PGP SIGNED MESSAGE-----
622 Hash: SHA1
623 <BLANKLINE>
624 Format: 1.7
625 Date: Thu, 30 Mar 2006 01:36:14 +0100
626 Source: bar
627 Binary: bar
628 Architecture: source
629 Version: 1.0-2
630 Distribution: breezy
631 Urgency: low
632 Maintainer: Launchpad team <launchpad@lists.canonical.com>
633 Changed-By: Daniel Silverstone <daniel.silverstone@canonical.com>
634 Launchpad-bugs-fixed: 6
635 Description: =
636 <BLANKLINE>
637 bar - Stuff for testing
638 Changes: =
639 <BLANKLINE>
640 bar (1.0-2) breezy; urgency=3Dlow
641 .
642 * A second upload to ensure that binary overrides of _all work
643 . =
644 <BLANKLINE>
645 * Also closes Launchpad bug #6
646 .
647 Files: =
648 <BLANKLINE>
649 bbaf6fbf41cdbbdd422b0382076f615a 512 devel optional bar_1.0-2.dsc
650 ac6b4efe44e31f47ec9f0d0fac6935f4 622 devel optional bar_1.0-2.diff.gz
651 <BLANKLINE>
652 -----BEGIN PGP SIGNATURE-----
653 Version: GnuPG v1.4.6 (GNU/Linux)
654 <BLANKLINE>
655 iD8DBQFGe+Yjjn63CGxkqMURAuPGAJ9ub5UHjrzKnEGmUK1oCoRuOrdligCePKxt
656 QRCBMda2V9lNtxldkGRtc88=3D
657 =3DgtPz
658 -----END PGP SIGNATURE-----
659 <BLANKLINE>
660
661The upload announcement email also has the attachment with the original
662changes file.
663
664 >>> announcement = msgs[1]
665 >>> attachment = announcement.get_payload()[1]
666 >>> attachment['Content-Disposition']
667 'attachment; filename="changesfile"'
668
669It has the same contents as the attachment of the upload notification
670email.
671
672 >>> print attachment.as_string() # doctest: -NORMALIZE_WHITESPACE
673 Content-Type: text/plain; charset="utf-8"
674 MIME-Version: 1.0
675 Content-Transfer-Encoding: quoted-printable
676 Content-Disposition: attachment; filename="changesfile"
677 <BLANKLINE>
678 -----BEGIN PGP SIGNED MESSAGE-----
679 Hash: SHA1
680 <BLANKLINE>
681 Format: 1.7
682 Date: Thu, 30 Mar 2006 01:36:14 +0100
683 Source: bar
684 Binary: bar
685 Architecture: source
686 Version: 1.0-2
687 Distribution: breezy
688 Urgency: low
689 Maintainer: Launchpad team <launchpad@lists.canonical.com>
690 Changed-By: Daniel Silverstone <daniel.silverstone@canonical.com>
691 Launchpad-bugs-fixed: 6
692 Description: =
693 <BLANKLINE>
694 bar - Stuff for testing
695 Changes: =
696 <BLANKLINE>
697 bar (1.0-2) breezy; urgency=3Dlow
698 .
699 * A second upload to ensure that binary overrides of _all work
700 . =
701 <BLANKLINE>
702 * Also closes Launchpad bug #6
703 .
704 Files: =
705 <BLANKLINE>
706 bbaf6fbf41cdbbdd422b0382076f615a 512 devel optional bar_1.0-2.dsc
707 ac6b4efe44e31f47ec9f0d0fac6935f4 622 devel optional bar_1.0-2.diff.gz
708 <BLANKLINE>
709 -----BEGIN PGP SIGNATURE-----
710 Version: GnuPG v1.4.6 (GNU/Linux)
711 <BLANKLINE>
712 iD8DBQFGe+Yjjn63CGxkqMURAuPGAJ9ub5UHjrzKnEGmUK1oCoRuOrdligCePKxt
713 QRCBMda2V9lNtxldkGRtc88=3D
714 =3DgtPz
715 -----END PGP SIGNATURE-----
716 <BLANKLINE>
717
718Remove orig.tar.gz pumped from librarian to disk during the upload
719checks:
720
721 >>> os.remove(os.path.join(datadir('suite/bar_1.0-2'), 'bar_1.0.orig.tar.gz'))
722
723DEBIAN SYNC upload of a source via the 'sync' policy.526DEBIAN SYNC upload of a source via the 'sync' policy.
724These uploads do not generate any announcement emails for auto-accepted527These uploads do not generate any announcement emails for auto-accepted
725packages, just the upload notification.528packages, just the upload notification.
@@ -762,6 +565,7 @@
762565
763Two emails generated:566Two emails generated:
764567
568 >>> import operator
765 >>> msgs = pop_notifications(sort_key=operator.itemgetter('To'))569 >>> msgs = pop_notifications(sort_key=operator.itemgetter('To'))
766 >>> len(msgs)570 >>> len(msgs)
767 2571 2
@@ -782,58 +586,6 @@
782 >>> os.remove(os.path.join(datadir('suite/bar_1.0-6'),586 >>> os.remove(os.path.join(datadir('suite/bar_1.0-6'),
783 ... 'bar_1.0.orig.tar.gz'))587 ... 'bar_1.0.orig.tar.gz'))
784588
785
786AUTO-APPROVED binary upload to SECURITY pocket via 'security' policy:
787
788 >>> security_policy = getPolicy(
789 ... name='security', distro='ubuntu', distroseries=None)
790 >>> security_policy.setDistroSeriesAndPocket('hoary-security')
791
792 >>> bar_bin = NascentUpload.from_changesfile_path(
793 ... datadir('suite/bar_1.0-2_binary/bar_1.0-2_i386.changes'),
794 ... security_policy, mock_logger_quiet)
795 >>> bar_bin.process()
796
797 >>> bar_bin.logger = mock_logger
798 >>> result = bar_bin.do_accept()
799 DEBUG: Creating queue entry
800 ...
801 DEBUG: Sent a mail:
802 DEBUG: Subject: [ubuntu/hoary-security] bar 1.0-2 (Accepted)
803 DEBUG: Recipients: Daniel Silverstone <daniel.silverstone@canonical.com>
804 DEBUG: Body:
805 DEBUG: bar (1.0-2) breezy; urgency=low
806 DEBUG:
807 DEBUG: * A second upload to ensure that binary overrides of _all work
808 DEBUG:
809 DEBUG: Date: Thu, 30 Mar 2006 01:36:14 +0100
810 DEBUG: Changed-By: Daniel Silverstone <daniel.silverstone@canonical.com>
811 DEBUG: Maintainer: Launchpad team <launchpad@lists.canonical.com>
812 DEBUG: http://launchpad.dev/ubuntu/hoary/+source/bar/1.0-2
813 DEBUG:
814 DEBUG: ==
815 DEBUG:
816 DEBUG: Announcing to hoary-announce@lists.ubuntu.com
817 DEBUG:
818 DEBUG: Thank you for your contribution to Ubuntu Linux.
819 DEBUG:
820 DEBUG: --
821 DEBUG: You are receiving this email because you are the uploader, maintainer or
822 DEBUG: signer of the above package.
823
824One email generated:
825
826 >>> [notification] = pop_notifications()
827
828 >>> notification['X-Katie']
829 'Launchpad actually'
830
831 >>> notification['To']
832 'Daniel Silverstone <daniel.silverstone@canonical.com>'
833
834 >>> notification['Subject']
835 '[ubuntu/hoary-security] bar 1.0-2 (Accepted)'
836
837Dry run uploads should not generate any emails. Call do_accept with589Dry run uploads should not generate any emails. Call do_accept with
838notify=False:590notify=False:
839591
@@ -873,27 +625,14 @@
873 DEBUG: Building recipients list.625 DEBUG: Building recipients list.
874 ...626 ...
875 INFO: Would have sent a mail:627 INFO: Would have sent a mail:
876 INFO: Subject: [ubuntu/hoary-security] bar 1.0-2 (Accepted)628 INFO: Subject: [ubuntu/hoary] bar 1.0-6 (Accepted)
877 INFO: Sender: Root <root@localhost>629 ...
878 INFO: Recipients: Daniel Silverstone <daniel.silverstone@canonical.com>630 INFO: Recipients: Celso Providelo <celso.providelo@canonical.com>
879 INFO: Bcc: Root <root@localhost>631 ...
880 INFO: Body:632 INFO: bar (1.0-6) breezy; urgency=low
881 INFO: bar (1.0-2) breezy; urgency=low633 ...
882 INFO:
883 INFO: * A second upload to ensure that binary overrides of _all work
884 INFO:
885 INFO: Date: Thu, 30 Mar 2006 01:36:14 +0100
886 INFO: Changed-By: Daniel Silverstone <daniel.silverstone@canonical.com>
887 INFO: Maintainer: Launchpad team <launchpad@lists.canonical.com>
888 INFO: http://launchpad.dev/ubuntu/hoary/+source/bar/1.0-2
889 INFO:
890 INFO: ==
891 INFO:
892 INFO: No announcement sent634 INFO: No announcement sent
893 INFO:635 ...
894 INFO: Thank you for your contribution to Ubuntu Linux.
895 INFO:
896 INFO: --
897 INFO: You are receiving this email because you are the uploader, maintainer or636 INFO: You are receiving this email because you are the uploader, maintainer or
898 INFO: signer of the above package.637 INFO: signer of the above package.
899638
900639
=== removed file 'lib/lp/archiveuploader/tests/nascentupload-security-uploads.txt'
--- lib/lp/archiveuploader/tests/nascentupload-security-uploads.txt 2010-08-04 00:16:44 +0000
+++ lib/lp/archiveuploader/tests/nascentupload-security-uploads.txt 1970-01-01 00:00:00 +0000
@@ -1,144 +0,0 @@
1= Fully Transactional Security Uploads =
2
3In order to allow security uploads to be transactional, i.e., either
4get published entirely, source and binaries, or get fully rejected; we
5have to allow the security uploads to produce a changesfile that
6includes the source and all binaries for the task in question.
7
8This is an extension of the already implemented mixed_mode upload,
9which previously allow the user to upload the source and one
10respective binary.
11
12While we don't have Security-in-Soyuz (s-i-s) implemented, this will
13be a more secure way to perform security uploads of binaires built in
14dak, since it guarantee that the interactions will be atomic.
15
16We need to be logged into the security framework in order to get any further
17
18 >>> login('foo.bar@canonical.com')
19
20A NascentUpload is a collection of files in a directory. They
21represent what may turn out to be an acceptable upload to a launchpad
22managed archive.
23
24 >>> from lp.archiveuploader.nascentupload import NascentUpload
25 >>> from lp.archiveuploader.tests import (
26 ... datadir, getPolicy, mock_logger, mock_logger_quiet)
27
28 >>> security_policy = getPolicy(name='security', distro='ubuntu')
29
30We are going to use 'warty/powerpc' distroarchseries and its
31respective processorfamily and processor. Let's create them
32on-the-fly:
33
34 >>> from canonical.testing.layers import LaunchpadZopelessLayer
35 >>> LaunchpadZopelessLayer.switchDbUser('launchpad')
36 >>> from canonical.launchpad.interfaces import IDistributionSet
37 >>> from canonical.launchpad.database import (
38 ... Processor, ProcessorFamily)
39
40 >>> powerpc_family = ProcessorFamily.selectOneBy(name='powerpc')
41 >>> powerpc_proc = Processor(
42 ... family=powerpc_family, name='G4', title='foo', description='nahh')
43
44 >>> ubuntu = getUtility(IDistributionSet)['ubuntu']
45 >>> warty = ubuntu['warty']
46 >>> warty_powerpc = warty.newArch('powerpc', powerpc_family, True, warty.owner)
47
48 >>> import transaction
49 >>> transaction.commit()
50 >>> LaunchpadZopelessLayer.switchDbUser('uploader')
51
52
53== Mixed Security Upload ==
54
55The upload in question contains a source and its 2 builds for i386 and
56powerpc:
57
58 >>> foo_mixed_upload = NascentUpload.from_changesfile_path(
59 ... datadir('suite/foo_1.0-1_multi_binary/foo_1.0-1_multi.changes'),
60 ... security_policy, mock_logger_quiet)
61 >>> foo_mixed_upload.process()
62
63Inspecting the files processed:
64
65 >>> for file in foo_mixed_upload.changes.files:
66 ... print file.filename
67 foo_1.0-1.dsc
68 foo_1.0.orig.tar.gz
69 foo_1.0-1.diff.gz
70 foo_1.0-1_i386.deb
71 foo_1.0-1_powerpc.deb
72
73Perform acceptance, creating the respective PackageUpload item and children.
74Please note how the package name appears only one time in the subject line
75(although the upload has one source and two builds associated with it).
76
77 >>> foo_mixed_upload.logger = mock_logger
78 >>> success = foo_mixed_upload.do_accept()
79 DEBUG: ...
80 DEBUG: Subject: [ubuntu/warty-security] foo 1.0-1 (New)
81 ...
82
83 >>> foo_mixed_queue = foo_mixed_upload.queue_root
84 >>> foo_mixed_queue.status.name
85 'NEW'
86
87Ensure we have only one source attaches to the PackageUpload record:
88
89 >>> foo_mixed_queue.sources.count()
90 1
91
92And it is the right one:
93
94 >>> for source in foo_mixed_queue.sources:
95 ... source.sourcepackagerelease.name
96 u'foo'
97
98Ensure we have the two expected builds attached to the PackageUpload record:
99
100 >>> foo_mixed_queue.builds.count()
101 2
102
103And they are the correct ones:
104
105 >>> for build in foo_mixed_queue.builds:
106 ... build.build.title
107 u'i386 build of foo 1.0-1 in ubuntu warty SECURITY'
108 u'powerpc build of foo 1.0-1 in ubuntu warty SECURITY'
109
110Including the uploaded binaries
111
112 >>> for build in foo_mixed_queue.builds:
113 ... [(bin.name, bin.version) for bin in build.build.binarypackages]
114 [(u'foo', u'1.0-1')]
115 [(u'foo', u'1.0-1')]
116
117
118== Detecting Inconsistencies ==
119
120NascentUpload code will be able to detect inconsistencies in a
121security upload, for example, detecting that the source and the
122binaries sent do not match.
123
124 >>> bar_mixed_upload = NascentUpload.from_changesfile_path(
125 ... datadir('suite/foo_1.0-1_broken_binary/bar_1.0-1_multi.changes'),
126 ... security_policy, mock_logger_quiet)
127 >>> bar_mixed_upload.process()
128
129Inspecting the files processed:
130
131 >>> for file in bar_mixed_upload.changes.files:
132 ... print file.filename
133 bar_1.0-1.dsc
134 bar_1.0.orig.tar.gz
135 bar_1.0-1.diff.gz
136 foo_1.0-1_i386.deb
137 foo_1.0-1_powerpc.deb
138
139 >>> bar_mixed_upload.is_rejected
140 True
141
142 >>> print bar_mixed_upload.rejection_message
143 foo_1.0-1_i386.deb: control file lists name as 'foo', which isn't in changes file.
144 foo_1.0-1_powerpc.deb: control file lists name as 'foo', which isn't in changes file.
1450
=== modified file 'lib/lp/archiveuploader/tests/test_buildduploads.py'
--- lib/lp/archiveuploader/tests/test_buildduploads.py 2010-08-20 20:31:18 +0000
+++ lib/lp/archiveuploader/tests/test_buildduploads.py 2010-08-25 19:32:53 +0000
@@ -5,13 +5,139 @@
55
6__metaclass__ = type6__metaclass__ = type
77
8import os
9
10from zope.component import getUtility
11
8from canonical.database.constants import UTC_NOW12from canonical.database.constants import UTC_NOW
9from canonical.launchpad.ftests import import_public_test_keys13from canonical.launchpad.ftests import import_public_test_keys
10from canonical.launchpad.interfaces import PackagePublishingStatus14from canonical.launchpad.interfaces import (
11from lp.archiveuploader.tests.test_securityuploads import (15 PackagePublishingStatus,
12 TestStagedBinaryUploadBase,16 PackageUploadStatus,
13 )17 )
18
19from lp.archiveuploader.tests.test_uploadprocessor import (
20 TestUploadProcessorBase,
21 )
22from lp.registry.interfaces.distribution import IDistributionSet
14from lp.registry.interfaces.pocket import PackagePublishingPocket23from lp.registry.interfaces.pocket import PackagePublishingPocket
24from lp.soyuz.model.binarypackagebuild import BinaryPackageBuild
25
26
27class TestStagedBinaryUploadBase(TestUploadProcessorBase):
28 name = 'baz'
29 version = '1.0-1'
30 distribution_name = None
31 distroseries_name = None
32 pocket = None
33 policy = 'buildd'
34 no_mails = True
35
36 @property
37 def distribution(self):
38 return getUtility(IDistributionSet)[self.distribution_name]
39
40 @property
41 def distroseries(self):
42 return self.distribution[self.distroseries_name]
43
44 @property
45 def package_name(self):
46 return "%s_%s" % (self.name, self.version)
47
48 @property
49 def source_dir(self):
50 return self.package_name
51
52 @property
53 def source_changesfile(self):
54 return "%s_source.changes" % self.package_name
55
56 @property
57 def binary_dir(self):
58 return "%s_binary" % self.package_name
59
60 def getBinaryChangesfileFor(self, archtag):
61 return "%s_%s.changes" % (self.package_name, archtag)
62
63 def setUp(self):
64 """Setup environment for staged binaries upload via security policy.
65
66 1. Setup queue directory and other basic attributes
67 2. Override policy options to get security policy and not send emails
68 3. Setup a common UploadProcessor with the overridden options
69 4. Store number of build present before issuing any upload
70 5. Upload the source package via security policy
71 6. Clean log messages.
72 7. Commit transaction, so the upload source can be seen.
73 """
74 super(TestStagedBinaryUploadBase, self).setUp()
75 self.options.context = self.policy
76 self.options.nomails = self.no_mails
77 # Set up the uploadprocessor with appropriate options and logger
78 self.uploadprocessor = self.getUploadProcessor(self.layer.txn)
79 self.builds_before_upload = BinaryPackageBuild.select().count()
80 self.source_queue = None
81 self._uploadSource()
82 self.log.lines = []
83 self.layer.txn.commit()
84
85 def assertBuildsCreated(self, amount):
86 """Assert that a given 'amount' of build records was created."""
87 builds_count = BinaryPackageBuild.select().count()
88 self.assertEqual(
89 self.builds_before_upload + amount, builds_count)
90
91 def _prepareUpload(self, upload_dir):
92 """Place a copy of the upload directory into incoming queue."""
93 os.system("cp -a %s %s" %
94 (os.path.join(self.test_files_dir, upload_dir),
95 os.path.join(self.queue_folder, "incoming")))
96
97 def _uploadSource(self):
98 """Upload and Accept (if necessary) the base source."""
99 self._prepareUpload(self.source_dir)
100 self.uploadprocessor.processChangesFile(
101 os.path.join(self.queue_folder, "incoming", self.source_dir),
102 self.source_changesfile)
103 queue_item = self.uploadprocessor.last_processed_upload.queue_root
104 self.assertTrue(
105 queue_item is not None,
106 "Source Upload Failed\nGot: %s" % "\n".join(self.log.lines))
107 acceptable_statuses = [
108 PackageUploadStatus.NEW,
109 PackageUploadStatus.UNAPPROVED,
110 ]
111 if queue_item.status in acceptable_statuses:
112 queue_item.setAccepted()
113 # Store source queue item for future use.
114 self.source_queue = queue_item
115
116 def _uploadBinary(self, archtag):
117 """Upload the base binary.
118
119 Ensure it got processed and has a respective queue record.
120 Return the IBuild attached to upload.
121 """
122 self._prepareUpload(self.binary_dir)
123 self.uploadprocessor.processChangesFile(
124 os.path.join(self.queue_folder, "incoming", self.binary_dir),
125 self.getBinaryChangesfileFor(archtag))
126 queue_item = self.uploadprocessor.last_processed_upload.queue_root
127 self.assertTrue(
128 queue_item is not None,
129 "Binary Upload Failed\nGot: %s" % "\n".join(self.log.lines))
130 self.assertEqual(queue_item.builds.count(), 1)
131 return queue_item.builds[0].build
132
133 def _createBuild(self, archtag):
134 """Create a build record attached to the base source."""
135 spr = self.source_queue.sources[0].sourcepackagerelease
136 build = spr.createBuild(
137 distro_arch_series=self.distroseries[archtag],
138 pocket=self.pocket, archive=self.distroseries.main_archive)
139 self.layer.txn.commit()
140 return build
15141
16142
17class TestBuilddUploads(TestStagedBinaryUploadBase):143class TestBuilddUploads(TestStagedBinaryUploadBase):
18144
=== removed file 'lib/lp/archiveuploader/tests/test_securityuploads.py'
--- lib/lp/archiveuploader/tests/test_securityuploads.py 2010-08-20 20:31:18 +0000
+++ lib/lp/archiveuploader/tests/test_securityuploads.py 1970-01-01 00:00:00 +0000
@@ -1,263 +0,0 @@
1# Copyright 2009-2010 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""Test security uploads use-cases."""
5
6__metaclass__ = type
7
8import os
9
10from zope.component import getUtility
11
12from canonical.launchpad.interfaces import (
13 IDistributionSet,
14 PackageUploadStatus,
15 )
16from lp.archiveuploader.tests.test_uploadprocessor import (
17 TestUploadProcessorBase,
18 )
19from lp.registry.interfaces.pocket import PackagePublishingPocket
20from lp.soyuz.model.binarypackagebuild import BinaryPackageBuild
21from lp.soyuz.model.processor import ProcessorFamily
22
23
24class TestStagedBinaryUploadBase(TestUploadProcessorBase):
25 name = 'baz'
26 version = '1.0-1'
27 distribution_name = None
28 distroseries_name = None
29 pocket = None
30 policy = 'buildd'
31 no_mails = True
32
33 @property
34 def distribution(self):
35 return getUtility(IDistributionSet)[self.distribution_name]
36
37 @property
38 def distroseries(self):
39 return self.distribution[self.distroseries_name]
40
41 @property
42 def package_name(self):
43 return "%s_%s" % (self.name, self.version)
44
45 @property
46 def source_dir(self):
47 return self.package_name
48
49 @property
50 def source_changesfile(self):
51 return "%s_source.changes" % self.package_name
52
53 @property
54 def binary_dir(self):
55 return "%s_binary" % self.package_name
56
57 def getBinaryChangesfileFor(self, archtag):
58 return "%s_%s.changes" % (self.package_name, archtag)
59
60 def setUp(self):
61 """Setup environment for staged binaries upload via security policy.
62
63 1. Setup queue directory and other basic attributes
64 2. Override policy options to get security policy and not send emails
65 3. Setup a common UploadProcessor with the overridden options
66 4. Store number of build present before issuing any upload
67 5. Upload the source package via security policy
68 6. Clean log messages.
69 7. Commit transaction, so the upload source can be seen.
70 """
71 super(TestStagedBinaryUploadBase, self).setUp()
72 self.options.context = self.policy
73 self.options.nomails = self.no_mails
74 # Set up the uploadprocessor with appropriate options and logger
75 self.uploadprocessor = self.getUploadProcessor(self.layer.txn)
76 self.builds_before_upload = BinaryPackageBuild.select().count()
77 self.source_queue = None
78 self._uploadSource()
79 self.log.lines = []
80 self.layer.txn.commit()
81
82 def assertBuildsCreated(self, amount):
83 """Assert that a given 'amount' of build records was created."""
84 builds_count = BinaryPackageBuild.select().count()
85 self.assertEqual(
86 self.builds_before_upload + amount, builds_count)
87
88 def _prepareUpload(self, upload_dir):
89 """Place a copy of the upload directory into incoming queue."""
90 os.system("cp -a %s %s" %
91 (os.path.join(self.test_files_dir, upload_dir),
92 os.path.join(self.queue_folder, "incoming")))
93
94 def _uploadSource(self):
95 """Upload and Accept (if necessary) the base source."""
96 self._prepareUpload(self.source_dir)
97 self.uploadprocessor.processChangesFile(
98 os.path.join(self.queue_folder, "incoming", self.source_dir),
99 self.source_changesfile)
100 queue_item = self.uploadprocessor.last_processed_upload.queue_root
101 self.assertTrue(
102 queue_item is not None,
103 "Source Upload Failed\nGot: %s" % "\n".join(self.log.lines))
104 acceptable_statuses = [
105 PackageUploadStatus.NEW,
106 PackageUploadStatus.UNAPPROVED,
107 ]
108 if queue_item.status in acceptable_statuses:
109 queue_item.setAccepted()
110 # Store source queue item for future use.
111 self.source_queue = queue_item
112
113 def _uploadBinary(self, archtag):
114 """Upload the base binary.
115
116 Ensure it got processed and has a respective queue record.
117 Return the IBuild attached to upload.
118 """
119 self._prepareUpload(self.binary_dir)
120 self.uploadprocessor.processChangesFile(
121 os.path.join(self.queue_folder, "incoming", self.binary_dir),
122 self.getBinaryChangesfileFor(archtag))
123 queue_item = self.uploadprocessor.last_processed_upload.queue_root
124 self.assertTrue(
125 queue_item is not None,
126 "Binary Upload Failed\nGot: %s" % "\n".join(self.log.lines))
127 self.assertEqual(queue_item.builds.count(), 1)
128 return queue_item.builds[0].build
129
130 def _createBuild(self, archtag):
131 """Create a build record attached to the base source."""
132 spr = self.source_queue.sources[0].sourcepackagerelease
133 build = spr.createBuild(
134 distro_arch_series=self.distroseries[archtag],
135 pocket=self.pocket, archive=self.distroseries.main_archive)
136 self.layer.txn.commit()
137 return build
138
139
140class TestStagedSecurityUploads(TestStagedBinaryUploadBase):
141 """Test how security uploads behave inside Soyuz.
142
143 Security uploads still coming from dak system, we have special upload
144 policy which allows source and binary uploads.
145
146 An upload of a source and its binaries does not necessary need
147 to happen in the same batch, and Soyuz is prepared to cope with it.
148
149 The only mandatory condition is to process the sources first.
150
151 This class will start to tests all known/possible cases using a test
152 (empty) upload and its binary.
153
154 * 'lib/lp/archivepublisher/tests/data/suite/baz_1.0-1/'
155 * 'lib/lp/archivepublisher/tests/data/suite/baz_1.0-1_binary/'
156 """
157 name = 'baz'
158 version = '1.0-1'
159 distribution_name = 'ubuntu'
160 distroseries_name = 'warty'
161 pocket = PackagePublishingPocket.SECURITY
162 policy = 'security'
163 no_mails = True
164
165 def setUp(self):
166 """Setup base class and create the required new distroarchseries."""
167 super(TestStagedSecurityUploads, self).setUp()
168 distribution = getUtility(IDistributionSet).getByName(
169 self.distribution_name)
170 distroseries = distribution[self.distroseries.name]
171 proc_family = ProcessorFamily.selectOneBy(name='amd64')
172 distroseries.newArch(
173 'amd64', proc_family, True, distribution.owner)
174
175 def testBuildCreation(self):
176 """Check if the builds get created for a binary security uploads.
177
178 That is the usual case, security binary uploads come after the
179 not published (accepted) source but in the same batch.
180
181 NascentUpload should create appropriate builds attached to the
182 correct source for the incoming binaries.
183 """
184 build_used = self._uploadBinary('i386')
185
186 self.assertBuildsCreated(1)
187 self.assertEqual(
188 u'i386 build of baz 1.0-1 in ubuntu warty SECURITY',
189 build_used.title)
190 self.assertEqual('FULLYBUILT', build_used.status.name)
191
192 build_used = self._uploadBinary('amd64')
193
194 self.assertBuildsCreated(2)
195 self.assertEqual(
196 u'amd64 build of baz 1.0-1 in ubuntu warty SECURITY',
197 build_used.title)
198
199 self.assertEqual('FULLYBUILT', build_used.status.name)
200
201 def testBuildLookup(self):
202 """Check if an available build gets used when it is appropriate.
203
204 It happens when the security source upload got already published
205 when the binary uploads arrive.
206 The queue-build has already created build records for it and
207 NascentUpload should identify this condition and used them instead
208 of creating new ones.
209 Also verify that builds for another architecture does not got
210 erroneously attached.
211 """
212 build_right_candidate = self._createBuild('i386')
213 build_wrong_candidate = self._createBuild('hppa')
214 build_used = self._uploadBinary('i386')
215
216 self.assertEqual(build_right_candidate.id, build_used.id)
217 self.assertNotEqual(build_wrong_candidate.id, build_used.id)
218 self.assertBuildsCreated(2)
219 self.assertEqual(
220 u'i386 build of baz 1.0-1 in ubuntu warty SECURITY',
221 build_used.title)
222 self.assertEqual('FULLYBUILT', build_used.status.name)
223
224 def testCorrectBuildPassedViaCommandLine(self):
225 """Check if command-line build argument gets attached correctly.
226
227 It's also possible to pass an specific buildid via the command-line
228 to be attached to the current upload.
229
230 This is only used in 'buildd' policy and does not produce very useful
231 results in 'security', however we want to check if it, at least,
232 does not 'break the system' entirely.
233 """
234 build_candidate = self._createBuild('i386')
235 self.options.buildid = str(build_candidate.id)
236 self.uploadprocessor = self.getUploadProcessor(self.layer.txn)
237
238 build_used = self._uploadBinary('i386')
239
240 self.assertEqual(build_candidate.id, build_used.id)
241 self.assertBuildsCreated(1)
242 self.assertEqual(
243 u'i386 build of baz 1.0-1 in ubuntu warty SECURITY',
244 build_used.title)
245
246 self.assertEqual('FULLYBUILT', build_used.status.name)
247
248 def testWrongBuildPassedViaCommandLine(self):
249 """Check if a misapplied passed buildid is correctly identified.
250
251 When we identify misapplied build, either by getting it from command
252 line or by a failure in lookup methods the upload is rejected before
253 anything wrong gets into the DB.
254 """
255 build_candidate = self._createBuild('hppa')
256 self.options.buildid = str(build_candidate.id)
257 self.uploadprocessor = self.getUploadProcessor(self.layer.txn)
258
259 self.assertRaises(AssertionError, self._uploadBinary, 'i386')
260
261 self.assertLogContains(
262 "UploadError: Attempt to upload binaries specifying build %d, "
263 "where they don't fit.\n" % (build_candidate.id, ))
2640
=== modified file 'lib/lp/archiveuploader/uploadpolicy.py'
--- lib/lp/archiveuploader/uploadpolicy.py 2010-08-20 20:31:18 +0000
+++ lib/lp/archiveuploader/uploadpolicy.py 2010-08-25 19:32:53 +0000
@@ -331,28 +331,6 @@
331 pass331 pass
332332
333333
334class SecurityUploadPolicy(AbstractUploadPolicy):
335 """The security-upload policy.
336
337 It allows unsigned changes and binary uploads.
338 """
339
340 name = 'security'
341
342 def __init__(self):
343 AbstractUploadPolicy.__init__(self)
344 self.unsigned_dsc_ok = True
345 self.unsigned_changes_ok = True
346 self.can_upload_mixed = True
347 self.can_upload_binaries = True
348
349 def policySpecificChecks(self, upload):
350 """Deny uploads to any pocket other than the security pocket."""
351 if self.pocket != PackagePublishingPocket.SECURITY:
352 upload.reject(
353 "Not permitted to do security upload to non SECURITY pocket")
354
355
356def findPolicyByName(policy_name):334def findPolicyByName(policy_name):
357 """Return a new policy instance for the given policy name."""335 """Return a new policy instance for the given policy name."""
358 return getUtility(IArchiveUploadPolicy, policy_name)()336 return getUtility(IArchiveUploadPolicy, policy_name)()
@@ -362,8 +340,7 @@
362 policies = [340 policies = [
363 BuildDaemonUploadPolicy,341 BuildDaemonUploadPolicy,
364 InsecureUploadPolicy,342 InsecureUploadPolicy,
365 SyncUploadPolicy, 343 SyncUploadPolicy]
366 SecurityUploadPolicy]
367 sm = getGlobalSiteManager()344 sm = getGlobalSiteManager()
368 for policy in policies:345 for policy in policies:
369 sm.registerUtility(346 sm.registerUtility(
370347
=== modified file 'lib/lp/soyuz/doc/soyuz-set-of-uploads.txt'
--- lib/lp/soyuz/doc/soyuz-set-of-uploads.txt 2010-08-06 14:29:36 +0000
+++ lib/lp/soyuz/doc/soyuz-set-of-uploads.txt 2010-08-25 19:32:53 +0000
@@ -319,22 +319,6 @@
319 Subject: bar_1.0-3_source.changes rejected319 Subject: bar_1.0-3_source.changes rejected
320 ...320 ...
321321
322Check the rejection (instead of failure) of an upload to a series or
323pocket which does not exist. We use the security upload policy for easier
324testing, as unsigned uploads are allowed.
325
326 >>> simulate_upload(
327 ... 'unstable_1.0-1', upload_policy="security", loglevel=logging.ERROR)
328 Rejected uploads: ['unstable_1.0-1']
329
330 >>> read_email()
331 To: Celso Providelo <celso.providelo@canonical.com>
332 Subject: unstable_1.0-1_source.changes rejected
333 ...
334 Rejected:
335 Unable to find distroseries: unstable
336 ...
337
338Force weird behavior with rfc2047 sentences containing '.' on322Force weird behavior with rfc2047 sentences containing '.' on
339bar_1.0-4, which caused bug # 41102.323bar_1.0-4, which caused bug # 41102.
340324
@@ -605,6 +589,9 @@
605589
606== Staged Security Uploads ==590== Staged Security Uploads ==
607591
592XXX: Salgado, 2010-08-25, bug=624078: This entire section should be removed
593but if you do so publish-distro.py (called further down) will hang.
594
608Perform a staged (source-first) security upload, simulating a run of595Perform a staged (source-first) security upload, simulating a run of
609the buildd-queuebuilder inbetween, to verify fix for bug 53437. To be596the buildd-queuebuilder inbetween, to verify fix for bug 53437. To be
610allowed a security upload, we need to use a released distroseries,597allowed a security upload, we need to use a released distroseries,
@@ -617,7 +604,7 @@
617 >>> ss = SectionSelection(distroseries=warty, section=devel)604 >>> ss = SectionSelection(distroseries=warty, section=devel)
618605
619 >>> simulate_upload(606 >>> simulate_upload(
620 ... 'baz_1.0-1', is_new=True, upload_policy="security",607 ... 'baz_1.0-1', is_new=True, upload_policy="anything",
621 ... series="warty-security", distro="ubuntu")608 ... series="warty-security", distro="ubuntu")
622609
623Check there's a SourcePackageRelease with no build.610Check there's a SourcePackageRelease with no build.
@@ -650,7 +637,7 @@
650Upload the i386 binary:637Upload the i386 binary:
651638
652 >>> simulate_upload(639 >>> simulate_upload(
653 ... 'baz_1.0-1_single_binary', is_new=True, upload_policy="security",640 ... 'baz_1.0-1_single_binary', is_new=True, upload_policy="anything",
654 ... distro="ubuntu", series="warty-security")641 ... distro="ubuntu", series="warty-security")
655642
656Should still just have one build, and it should now be FULLYBUILT.643Should still just have one build, and it should now be FULLYBUILT.