Merge lp:~maxb/launchpad/use-hashlib into lp:launchpad
- use-hashlib
- Merge into devel
Proposed by
Max Bowsher
| Status: | Merged |
|---|---|
| Approved by: | Graham Binns |
| Approved revision: | not available |
| Merged at revision: | not available |
| Proposed branch: | lp:~maxb/launchpad/use-hashlib |
| Merge into: | lp:launchpad |
| Diff against target: |
902 lines (+110/-178) 27 files modified
lib/canonical/base.py (+3/-3) lib/canonical/buildd/slave.py (+3/-3) lib/canonical/launchpad/database/emailaddress.py (+3/-2) lib/canonical/launchpad/database/temporaryblobstorage.py (+4/-3) lib/canonical/launchpad/helpers.py (+4/-3) lib/canonical/launchpad/mailman/monkeypatches/lphandler.py (+3/-2) lib/canonical/launchpad/scripts/logger.py (+4/-2) lib/canonical/launchpad/webapp/authentication.py (+3/-3) lib/canonical/launchpad/webapp/login.py (+5/-4) lib/canonical/launchpad/webapp/tests/test_encryptor.py (+10/-5) lib/canonical/librarian/client.py (+7/-6) lib/canonical/librarian/ftests/test_storage.py (+5/-4) lib/canonical/librarian/storage.py (+3/-4) lib/canonical/librarian/tests/test_storage.py (+0/-1) lib/canonical/librarian/utils.py (+3/-3) lib/lp/archivepublisher/library.py (+4/-2) lib/lp/archivepublisher/publishing.py (+8/-34) lib/lp/archivepublisher/tests/test_librarianwrapper.py (+2/-2) lib/lp/archivepublisher/tests/test_pool.py (+4/-4) lib/lp/archivepublisher/tests/test_publisher.py (+2/-66) lib/lp/archiveuploader/nascentuploadfile.py (+3/-4) lib/lp/services/mail/sendmail.py (+7/-4) lib/lp/soyuz/doc/soyuz-upload.txt.disabled (+2/-2) lib/lp/soyuz/scripts/ftpmaster.py (+2/-2) lib/lp/soyuz/scripts/gina/library.py (+5/-2) lib/lp/soyuz/scripts/queue.py (+3/-3) lib/lp/soyuz/scripts/tests/test_queue.py (+8/-5) |
| To merge this branch: | bzr merge lp:~maxb/launchpad/use-hashlib |
| Related bugs: |
| Reviewer | Review Type | Date Requested | Status |
|---|---|---|---|
| Graham Binns (community) | code | Approve | |
|
Review via email:
|
|||
Commit message
Switch from using sha and md5 to hashlib. Also use hashlib.sha256 instead of the python-apt implementation.
Description of the change
To post a comment you must log in.
Revision history for this message
| Max Bowsher (maxb) wrote : | # |
Revision history for this message
| Graham Binns (gmb) wrote : | # |
Hi Max,
Great branch, and I'm happy for it to land.
review:
Approve
(code)
Revision history for this message
| Max Bowsher (maxb) wrote : | # |
This somehow ended up with this in ec2test, despite working fine in plain old make check locally:
Error in test testAptSHA256 (lp.archivepubl
Traceback (most recent call last):
File "/usr/lib/
testMethod()
File "/var/launchpad
text, file = _getSHA256(
File "/var/launchpad
file = hashlib.
TypeError: sha256() argument 1 must be string or read-only buffer, not file
Back to "Work in progress" whilst I try to figure out why.
Revision history for this message
| Max Bowsher (maxb) wrote : | # |
" This test only runs on Ubuntu/hardy systems. "
Well, meh.
Revision history for this message
| Max Bowsher (maxb) wrote : | # |
It turns out the entire test is no longer relevant, because it tests for the presence of a bug in apt_pkg.sha256sum, which the first revision in this branch removes all uses of. Hence, the fix is to simply delete the test.
Incremental diff: just the deletion of the entire method lp.archivepubli
This is ready for re-review and landing now.
Revision history for this message
| Graham Binns (gmb) wrote : | # |
Approved after re-review.
review:
Approve
(code)
Preview Diff
[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
| 1 | === modified file 'lib/canonical/base.py' | |||
| 2 | --- lib/canonical/base.py 2009-06-25 05:30:52 +0000 | |||
| 3 | +++ lib/canonical/base.py 2010-02-10 00:32:23 +0000 | |||
| 4 | @@ -33,8 +33,8 @@ | |||
| 5 | 33 | This method is useful for shrinking sha1 and md5 hashes, but keeping | 33 | This method is useful for shrinking sha1 and md5 hashes, but keeping |
| 6 | 34 | them in simple ASCII suitable for URL's etc. | 34 | them in simple ASCII suitable for URL's etc. |
| 7 | 35 | 35 | ||
| 10 | 36 | >>> import sha, md5 | 36 | >>> import hashlib |
| 11 | 37 | >>> s = sha.new('foo').hexdigest() | 37 | >>> s = hashlib.sha1('foo').hexdigest() |
| 12 | 38 | >>> s | 38 | >>> s |
| 13 | 39 | '0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33' | 39 | '0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33' |
| 14 | 40 | >>> i = long(s, 16) | 40 | >>> i = long(s, 16) |
| 15 | @@ -42,7 +42,7 @@ | |||
| 16 | 42 | 68123873083688143418383284816464454849230703155L | 42 | 68123873083688143418383284816464454849230703155L |
| 17 | 43 | >>> base(i, 62) | 43 | >>> base(i, 62) |
| 18 | 44 | '1HyPQr2xj1nmnkQXBCJXUdQoy5l' | 44 | '1HyPQr2xj1nmnkQXBCJXUdQoy5l' |
| 20 | 45 | >>> base(int(md5.new('foo').hexdigest(), 16), 62) | 45 | >>> base(int(hashlib.md5('foo').hexdigest(), 16), 62) |
| 21 | 46 | '5fX649Stem9fET0lD46zVe' | 46 | '5fX649Stem9fET0lD46zVe' |
| 22 | 47 | 47 | ||
| 23 | 48 | A sha1 hash can be compressed to 27 characters or less | 48 | A sha1 hash can be compressed to 27 characters or less |
| 24 | 49 | 49 | ||
| 25 | === modified file 'lib/canonical/buildd/slave.py' | |||
| 26 | --- lib/canonical/buildd/slave.py 2010-01-14 02:36:36 +0000 | |||
| 27 | +++ lib/canonical/buildd/slave.py 2010-02-10 00:32:23 +0000 | |||
| 28 | @@ -8,9 +8,9 @@ | |||
| 29 | 8 | 8 | ||
| 30 | 9 | __metaclass__ = type | 9 | __metaclass__ = type |
| 31 | 10 | 10 | ||
| 32 | 11 | import hashlib | ||
| 33 | 11 | import os | 12 | import os |
| 34 | 12 | import re | 13 | import re |
| 35 | 13 | import sha | ||
| 36 | 14 | import urllib2 | 14 | import urllib2 |
| 37 | 15 | import xmlrpclib | 15 | import xmlrpclib |
| 38 | 16 | 16 | ||
| 39 | @@ -301,7 +301,7 @@ | |||
| 40 | 301 | else: | 301 | else: |
| 41 | 302 | of = open(self.cachePath(sha1sum), "w") | 302 | of = open(self.cachePath(sha1sum), "w") |
| 42 | 303 | # Upped for great justice to 256k | 303 | # Upped for great justice to 256k |
| 44 | 304 | check_sum = sha.sha() | 304 | check_sum = hashlib.sha1() |
| 45 | 305 | for chunk in iter(lambda: f.read(256*1024), ''): | 305 | for chunk in iter(lambda: f.read(256*1024), ''): |
| 46 | 306 | of.write(chunk) | 306 | of.write(chunk) |
| 47 | 307 | check_sum.update(chunk) | 307 | check_sum.update(chunk) |
| 48 | @@ -316,7 +316,7 @@ | |||
| 49 | 316 | 316 | ||
| 50 | 317 | def storeFile(self, content): | 317 | def storeFile(self, content): |
| 51 | 318 | """Take the provided content and store it in the file cache.""" | 318 | """Take the provided content and store it in the file cache.""" |
| 53 | 319 | sha1sum = sha.sha(content).hexdigest() | 319 | sha1sum = hashlib.sha1(content).hexdigest() |
| 54 | 320 | present, info = self.ensurePresent(sha1sum) | 320 | present, info = self.ensurePresent(sha1sum) |
| 55 | 321 | if present: | 321 | if present: |
| 56 | 322 | return sha1sum | 322 | return sha1sum |
| 57 | 323 | 323 | ||
| 58 | === modified file 'lib/canonical/launchpad/database/emailaddress.py' | |||
| 59 | --- lib/canonical/launchpad/database/emailaddress.py 2009-06-25 05:30:52 +0000 | |||
| 60 | +++ lib/canonical/launchpad/database/emailaddress.py 2010-02-10 00:32:23 +0000 | |||
| 61 | @@ -11,8 +11,9 @@ | |||
| 62 | 11 | 'UndeletableEmailAddress', | 11 | 'UndeletableEmailAddress', |
| 63 | 12 | ] | 12 | ] |
| 64 | 13 | 13 | ||
| 65 | 14 | |||
| 66 | 15 | import hashlib | ||
| 67 | 14 | import operator | 16 | import operator |
| 68 | 15 | import sha | ||
| 69 | 16 | 17 | ||
| 70 | 17 | from zope.interface import implements | 18 | from zope.interface import implements |
| 71 | 18 | 19 | ||
| 72 | @@ -80,7 +81,7 @@ | |||
| 73 | 80 | @property | 81 | @property |
| 74 | 81 | def rdf_sha1(self): | 82 | def rdf_sha1(self): |
| 75 | 82 | """See `IEmailAddress`.""" | 83 | """See `IEmailAddress`.""" |
| 77 | 83 | return sha.new('mailto:' + self.email).hexdigest().upper() | 84 | return hashlib.sha1('mailto:' + self.email).hexdigest().upper() |
| 78 | 84 | 85 | ||
| 79 | 85 | 86 | ||
| 80 | 86 | class EmailAddressSet: | 87 | class EmailAddressSet: |
| 81 | 87 | 88 | ||
| 82 | === modified file 'lib/canonical/launchpad/database/temporaryblobstorage.py' | |||
| 83 | --- lib/canonical/launchpad/database/temporaryblobstorage.py 2009-06-25 05:30:52 +0000 | |||
| 84 | +++ lib/canonical/launchpad/database/temporaryblobstorage.py 2010-02-10 00:32:23 +0000 | |||
| 85 | @@ -10,13 +10,14 @@ | |||
| 86 | 10 | 'TemporaryStorageManager', | 10 | 'TemporaryStorageManager', |
| 87 | 11 | ] | 11 | ] |
| 88 | 12 | 12 | ||
| 91 | 13 | from cStringIO import StringIO | 13 | |
| 90 | 14 | from datetime import timedelta, datetime | ||
| 92 | 15 | import random | 14 | import random |
| 93 | 16 | import sha | ||
| 94 | 17 | import time | 15 | import time |
| 95 | 18 | import thread | 16 | import thread |
| 96 | 19 | 17 | ||
| 97 | 18 | from cStringIO import StringIO | ||
| 98 | 19 | from datetime import timedelta, datetime | ||
| 99 | 20 | |||
| 100 | 20 | from pytz import utc | 21 | from pytz import utc |
| 101 | 21 | from sqlobject import StringCol, ForeignKey | 22 | from sqlobject import StringCol, ForeignKey |
| 102 | 22 | from zope.component import getUtility | 23 | from zope.component import getUtility |
| 103 | 23 | 24 | ||
| 104 | === modified file 'lib/canonical/launchpad/helpers.py' | |||
| 105 | --- lib/canonical/launchpad/helpers.py 2009-12-10 13:20:12 +0000 | |||
| 106 | +++ lib/canonical/launchpad/helpers.py 2010-02-10 00:32:23 +0000 | |||
| 107 | @@ -10,16 +10,17 @@ | |||
| 108 | 10 | 10 | ||
| 109 | 11 | __metaclass__ = type | 11 | __metaclass__ = type |
| 110 | 12 | 12 | ||
| 112 | 13 | import subprocess | 13 | import hashlib |
| 113 | 14 | import gettextpo | 14 | import gettextpo |
| 114 | 15 | import os | 15 | import os |
| 115 | 16 | import random | 16 | import random |
| 116 | 17 | import re | 17 | import re |
| 117 | 18 | import subprocess | ||
| 118 | 18 | import tarfile | 19 | import tarfile |
| 119 | 19 | import warnings | 20 | import warnings |
| 120 | 21 | |||
| 121 | 20 | from StringIO import StringIO | 22 | from StringIO import StringIO |
| 122 | 21 | from difflib import unified_diff | 23 | from difflib import unified_diff |
| 123 | 22 | import sha | ||
| 124 | 23 | 24 | ||
| 125 | 24 | from zope.component import getUtility | 25 | from zope.component import getUtility |
| 126 | 25 | from zope.security.interfaces import ForbiddenAttribute | 26 | from zope.security.interfaces import ForbiddenAttribute |
| 127 | @@ -462,7 +463,7 @@ | |||
| 128 | 462 | """ | 463 | """ |
| 129 | 463 | return '%s.msg' % ( | 464 | return '%s.msg' % ( |
| 130 | 464 | canonical.base.base( | 465 | canonical.base.base( |
| 132 | 465 | long(sha.new(message_id).hexdigest(), 16), 62)) | 466 | long(hashlib.sha1(message_id).hexdigest(), 16), 62)) |
| 133 | 466 | 467 | ||
| 134 | 467 | 468 | ||
| 135 | 468 | def intOrZero(value): | 469 | def intOrZero(value): |
| 136 | 469 | 470 | ||
| 137 | === modified file 'lib/canonical/launchpad/mailman/monkeypatches/lphandler.py' | |||
| 138 | --- lib/canonical/launchpad/mailman/monkeypatches/lphandler.py 2009-06-25 05:30:52 +0000 | |||
| 139 | +++ lib/canonical/launchpad/mailman/monkeypatches/lphandler.py 2010-02-10 00:32:23 +0000 | |||
| 140 | @@ -3,7 +3,8 @@ | |||
| 141 | 3 | 3 | ||
| 142 | 4 | """A global pipeline handler for determining Launchpad membership.""" | 4 | """A global pipeline handler for determining Launchpad membership.""" |
| 143 | 5 | 5 | ||
| 145 | 6 | import sha | 6 | |
| 146 | 7 | import hashlib | ||
| 147 | 7 | import xmlrpclib | 8 | import xmlrpclib |
| 148 | 8 | 9 | ||
| 149 | 9 | from Mailman import Errors | 10 | from Mailman import Errors |
| 150 | @@ -24,7 +25,7 @@ | |||
| 151 | 24 | secret = msg['x-launchpad-hash'] | 25 | secret = msg['x-launchpad-hash'] |
| 152 | 25 | message_id = msg['message-id'] | 26 | message_id = msg['message-id'] |
| 153 | 26 | if secret and message_id: | 27 | if secret and message_id: |
| 155 | 27 | hash = sha.new(mm_cfg.LAUNCHPAD_SHARED_SECRET) | 28 | hash = hashlib.sha1(mm_cfg.LAUNCHPAD_SHARED_SECRET) |
| 156 | 28 | hash.update(message_id) | 29 | hash.update(message_id) |
| 157 | 29 | if secret == hash.hexdigest(): | 30 | if secret == hash.hexdigest(): |
| 158 | 30 | # Since this message is coming from Launchpad, pre-approve it. | 31 | # Since this message is coming from Launchpad, pre-approve it. |
| 159 | 31 | 32 | ||
| 160 | === modified file 'lib/canonical/launchpad/scripts/logger.py' | |||
| 161 | --- lib/canonical/launchpad/scripts/logger.py 2009-10-16 18:07:01 +0000 | |||
| 162 | +++ lib/canonical/launchpad/scripts/logger.py 2010-02-10 00:32:23 +0000 | |||
| 163 | @@ -25,12 +25,14 @@ | |||
| 164 | 25 | 'DEBUG6', 'DEBUG7', 'DEBUG8', 'DEBUG9' | 25 | 'DEBUG6', 'DEBUG7', 'DEBUG8', 'DEBUG9' |
| 165 | 26 | ] | 26 | ] |
| 166 | 27 | 27 | ||
| 167 | 28 | |||
| 168 | 29 | import hashlib | ||
| 169 | 28 | import logging | 30 | import logging |
| 170 | 29 | import re | 31 | import re |
| 171 | 30 | import sha | ||
| 172 | 31 | import sys | 32 | import sys |
| 173 | 32 | import traceback | 33 | import traceback |
| 174 | 33 | import time | 34 | import time |
| 175 | 35 | |||
| 176 | 34 | from optparse import OptionParser | 36 | from optparse import OptionParser |
| 177 | 35 | from cStringIO import StringIO | 37 | from cStringIO import StringIO |
| 178 | 36 | from datetime import datetime, timedelta | 38 | from datetime import datetime, timedelta |
| 179 | @@ -168,7 +170,7 @@ | |||
| 180 | 168 | expiry = datetime.now().replace(tzinfo=utc) + timedelta(days=90) | 170 | expiry = datetime.now().replace(tzinfo=utc) + timedelta(days=90) |
| 181 | 169 | try: | 171 | try: |
| 182 | 170 | filename = base( | 172 | filename = base( |
| 184 | 171 | long(sha.new(traceback).hexdigest(),16), 62 | 173 | long(hashlib.sha1(traceback).hexdigest(),16), 62 |
| 185 | 172 | ) + '.txt' | 174 | ) + '.txt' |
| 186 | 173 | url = librarian.remoteAddFile( | 175 | url = librarian.remoteAddFile( |
| 187 | 174 | filename, len(traceback), StringIO(traceback), | 176 | filename, len(traceback), StringIO(traceback), |
| 188 | 175 | 177 | ||
| 189 | === modified file 'lib/canonical/launchpad/webapp/authentication.py' | |||
| 190 | --- lib/canonical/launchpad/webapp/authentication.py 2009-10-20 22:13:21 +0000 | |||
| 191 | +++ lib/canonical/launchpad/webapp/authentication.py 2010-02-10 00:32:23 +0000 | |||
| 192 | @@ -14,8 +14,8 @@ | |||
| 193 | 14 | 14 | ||
| 194 | 15 | 15 | ||
| 195 | 16 | import binascii | 16 | import binascii |
| 196 | 17 | import hashlib | ||
| 197 | 17 | import random | 18 | import random |
| 198 | 18 | import sha | ||
| 199 | 19 | 19 | ||
| 200 | 20 | from contrib.oauth import OAuthRequest | 20 | from contrib.oauth import OAuthRequest |
| 201 | 21 | 21 | ||
| 202 | @@ -178,7 +178,7 @@ | |||
| 203 | 178 | plaintext = str(plaintext) | 178 | plaintext = str(plaintext) |
| 204 | 179 | if salt is None: | 179 | if salt is None: |
| 205 | 180 | salt = self.generate_salt() | 180 | salt = self.generate_salt() |
| 207 | 181 | v = binascii.b2a_base64(sha.new(plaintext + salt).digest() + salt) | 181 | v = binascii.b2a_base64(hashlib.sha1(plaintext + salt).digest() + salt) |
| 208 | 182 | return v[:-1] | 182 | return v[:-1] |
| 209 | 183 | 183 | ||
| 210 | 184 | def validate(self, plaintext, encrypted): | 184 | def validate(self, plaintext, encrypted): |
| 211 | @@ -191,7 +191,7 @@ | |||
| 212 | 191 | return False | 191 | return False |
| 213 | 192 | salt = ref[20:] | 192 | salt = ref[20:] |
| 214 | 193 | v = binascii.b2a_base64( | 193 | v = binascii.b2a_base64( |
| 216 | 194 | sha.new(plaintext + salt).digest() + salt)[:-1] | 194 | hashlib.sha1(plaintext + salt).digest() + salt)[:-1] |
| 217 | 195 | pw1 = (v or '').strip() | 195 | pw1 = (v or '').strip() |
| 218 | 196 | pw2 = (encrypted or '').strip() | 196 | pw2 = (encrypted or '').strip() |
| 219 | 197 | return pw1 == pw2 | 197 | return pw1 == pw2 |
| 220 | 198 | 198 | ||
| 221 | === modified file 'lib/canonical/launchpad/webapp/login.py' | |||
| 222 | --- lib/canonical/launchpad/webapp/login.py 2010-01-14 13:25:34 +0000 | |||
| 223 | +++ lib/canonical/launchpad/webapp/login.py 2010-02-10 00:32:23 +0000 | |||
| 224 | @@ -6,10 +6,11 @@ | |||
| 225 | 6 | __metaclass__ = type | 6 | __metaclass__ = type |
| 226 | 7 | 7 | ||
| 227 | 8 | import cgi | 8 | import cgi |
| 228 | 9 | import hashlib | ||
| 229 | 10 | import random | ||
| 230 | 9 | import urllib | 11 | import urllib |
| 231 | 12 | |||
| 232 | 10 | from datetime import datetime, timedelta | 13 | from datetime import datetime, timedelta |
| 233 | 11 | import md5 | ||
| 234 | 12 | import random | ||
| 235 | 13 | 14 | ||
| 236 | 14 | from BeautifulSoup import UnicodeDammit | 15 | from BeautifulSoup import UnicodeDammit |
| 237 | 15 | 16 | ||
| 238 | @@ -162,7 +163,7 @@ | |||
| 239 | 162 | expected = self.request.form.get(self.captcha_hash) | 163 | expected = self.request.form.get(self.captcha_hash) |
| 240 | 163 | submitted = self.request.form.get(self.captcha_submission) | 164 | submitted = self.request.form.get(self.captcha_submission) |
| 241 | 164 | if expected is not None and submitted is not None: | 165 | if expected is not None and submitted is not None: |
| 243 | 165 | return md5.new(submitted).hexdigest() == expected | 166 | return hashlib.md5(submitted).hexdigest() == expected |
| 244 | 166 | return False | 167 | return False |
| 245 | 167 | 168 | ||
| 246 | 168 | @cachedproperty | 169 | @cachedproperty |
| 247 | @@ -182,7 +183,7 @@ | |||
| 248 | 182 | 183 | ||
| 249 | 183 | The hash is the value we put in the form for later comparison. | 184 | The hash is the value we put in the form for later comparison. |
| 250 | 184 | """ | 185 | """ |
| 252 | 185 | return md5.new(str(self.captcha_answer)).hexdigest() | 186 | return hashlib.md5(str(self.captcha_answer)).hexdigest() |
| 253 | 186 | 187 | ||
| 254 | 187 | @property | 188 | @property |
| 255 | 188 | def captcha_problem(self): | 189 | def captcha_problem(self): |
| 256 | 189 | 190 | ||
| 257 | === modified file 'lib/canonical/launchpad/webapp/tests/test_encryptor.py' | |||
| 258 | --- lib/canonical/launchpad/webapp/tests/test_encryptor.py 2009-06-25 05:30:52 +0000 | |||
| 259 | +++ lib/canonical/launchpad/webapp/tests/test_encryptor.py 2010-02-10 00:32:23 +0000 | |||
| 260 | @@ -3,15 +3,19 @@ | |||
| 261 | 3 | 3 | ||
| 262 | 4 | __metaclass__ = type | 4 | __metaclass__ = type |
| 263 | 5 | 5 | ||
| 264 | 6 | |||
| 265 | 7 | import binascii | ||
| 266 | 8 | import hashlib | ||
| 267 | 6 | import unittest | 9 | import unittest |
| 270 | 7 | import sha | 10 | |
| 269 | 8 | import binascii | ||
| 271 | 9 | from zope.component import getUtility | 11 | from zope.component import getUtility |
| 272 | 10 | from zope.app.testing import ztapi | 12 | from zope.app.testing import ztapi |
| 273 | 11 | from zope.app.testing.placelesssetup import PlacelessSetup | 13 | from zope.app.testing.placelesssetup import PlacelessSetup |
| 274 | 14 | |||
| 275 | 12 | from canonical.launchpad.webapp.authentication import SSHADigestEncryptor | 15 | from canonical.launchpad.webapp.authentication import SSHADigestEncryptor |
| 276 | 13 | from canonical.launchpad.interfaces import IPasswordEncryptor | 16 | from canonical.launchpad.interfaces import IPasswordEncryptor |
| 277 | 14 | 17 | ||
| 278 | 18 | |||
| 279 | 15 | class TestSSHADigestEncryptor(PlacelessSetup, unittest.TestCase): | 19 | class TestSSHADigestEncryptor(PlacelessSetup, unittest.TestCase): |
| 280 | 16 | 20 | ||
| 281 | 17 | def setUp(self): | 21 | def setUp(self): |
| 282 | @@ -26,7 +30,7 @@ | |||
| 283 | 26 | self.failIfEqual(encrypted1, encrypted2) | 30 | self.failIfEqual(encrypted1, encrypted2) |
| 284 | 27 | salt = encrypted1[20:] | 31 | salt = encrypted1[20:] |
| 285 | 28 | v = binascii.b2a_base64( | 32 | v = binascii.b2a_base64( |
| 287 | 29 | sha.new('motorhead' + salt).digest() + salt | 33 | hashlib.sha1('motorhead' + salt).digest() + salt |
| 288 | 30 | )[:-1] | 34 | )[:-1] |
| 289 | 31 | return (v == encrypted1) | 35 | return (v == encrypted1) |
| 290 | 32 | 36 | ||
| 291 | @@ -41,8 +45,9 @@ | |||
| 292 | 41 | encrypted2 = encryptor.encrypt(u'motorhead') | 45 | encrypted2 = encryptor.encrypt(u'motorhead') |
| 293 | 42 | self.failIfEqual(encrypted1, encrypted2) | 46 | self.failIfEqual(encrypted1, encrypted2) |
| 294 | 43 | salt = encrypted1[20:] | 47 | salt = encrypted1[20:] |
| 297 | 44 | v = binascii.b2a_base64(sha.new('motorhead' + salt).digest() + salt)[:-1] | 48 | v = binascii.b2a_base64( |
| 298 | 45 | return (v == encrypted1) | 49 | hashlib.sha1('motorhead' + salt).digest() + salt)[:-1] |
| 299 | 50 | return v == encrypted1 | ||
| 300 | 46 | 51 | ||
| 301 | 47 | def test_unicode_validate(self): | 52 | def test_unicode_validate(self): |
| 302 | 48 | encryptor = getUtility(IPasswordEncryptor) | 53 | encryptor = getUtility(IPasswordEncryptor) |
| 303 | 49 | 54 | ||
| 304 | === modified file 'lib/canonical/librarian/client.py' | |||
| 305 | --- lib/canonical/librarian/client.py 2010-01-11 18:06:23 +0000 | |||
| 306 | +++ lib/canonical/librarian/client.py 2010-02-10 00:32:23 +0000 | |||
| 307 | @@ -10,16 +10,17 @@ | |||
| 308 | 10 | 'RestrictedLibrarianClient', | 10 | 'RestrictedLibrarianClient', |
| 309 | 11 | ] | 11 | ] |
| 310 | 12 | 12 | ||
| 312 | 13 | import md5 | 13 | |
| 313 | 14 | import hashlib | ||
| 314 | 14 | import re | 15 | import re |
| 315 | 15 | import sha | ||
| 316 | 16 | import socket | 16 | import socket |
| 317 | 17 | from socket import SOCK_STREAM, AF_INET | ||
| 318 | 18 | from select import select | ||
| 319 | 19 | import time | 17 | import time |
| 320 | 20 | import threading | 18 | import threading |
| 321 | 21 | import urllib | 19 | import urllib |
| 322 | 22 | import urllib2 | 20 | import urllib2 |
| 323 | 21 | |||
| 324 | 22 | from select import select | ||
| 325 | 23 | from socket import SOCK_STREAM, AF_INET | ||
| 326 | 23 | from urlparse import urljoin | 24 | from urlparse import urljoin |
| 327 | 24 | 25 | ||
| 328 | 25 | from storm.store import Store | 26 | from storm.store import Store |
| 329 | @@ -130,8 +131,8 @@ | |||
| 330 | 130 | self._sendLine('') | 131 | self._sendLine('') |
| 331 | 131 | 132 | ||
| 332 | 132 | # Prepare to the upload the file | 133 | # Prepare to the upload the file |
| 335 | 133 | shaDigester = sha.sha() | 134 | shaDigester = hashlib.sha1() |
| 336 | 134 | md5Digester = md5.md5() | 135 | md5Digester = hashlib.md5() |
| 337 | 135 | bytesWritten = 0 | 136 | bytesWritten = 0 |
| 338 | 136 | 137 | ||
| 339 | 137 | # Read in and upload the file 64kb at a time, by using the two-arg | 138 | # Read in and upload the file 64kb at a time, by using the two-arg |
| 340 | 138 | 139 | ||
| 341 | === modified file 'lib/canonical/librarian/ftests/test_storage.py' | |||
| 342 | --- lib/canonical/librarian/ftests/test_storage.py 2009-06-25 05:30:52 +0000 | |||
| 343 | +++ lib/canonical/librarian/ftests/test_storage.py 2010-02-10 00:32:23 +0000 | |||
| 344 | @@ -1,7 +1,7 @@ | |||
| 345 | 1 | # Copyright 2009 Canonical Ltd. This software is licensed under the | 1 | # Copyright 2009 Canonical Ltd. This software is licensed under the |
| 346 | 2 | # GNU Affero General Public License version 3 (see the file LICENSE). | 2 | # GNU Affero General Public License version 3 (see the file LICENSE). |
| 347 | 3 | 3 | ||
| 349 | 4 | import sha | 4 | import hashlib |
| 350 | 5 | import shutil | 5 | import shutil |
| 351 | 6 | import tempfile | 6 | import tempfile |
| 352 | 7 | import unittest | 7 | import unittest |
| 353 | @@ -13,6 +13,7 @@ | |||
| 354 | 13 | from canonical.launchpad.database import LibraryFileContent, LibraryFileAlias | 13 | from canonical.launchpad.database import LibraryFileContent, LibraryFileAlias |
| 355 | 14 | from canonical.testing import LaunchpadZopelessLayer | 14 | from canonical.testing import LaunchpadZopelessLayer |
| 356 | 15 | 15 | ||
| 357 | 16 | |||
| 358 | 16 | class LibrarianStorageDBTests(unittest.TestCase): | 17 | class LibrarianStorageDBTests(unittest.TestCase): |
| 359 | 17 | layer = LaunchpadZopelessLayer | 18 | layer = LaunchpadZopelessLayer |
| 360 | 18 | 19 | ||
| 361 | @@ -26,7 +27,7 @@ | |||
| 362 | 26 | 27 | ||
| 363 | 27 | def test_addFile(self): | 28 | def test_addFile(self): |
| 364 | 28 | data = 'data ' * 50 | 29 | data = 'data ' * 50 |
| 366 | 29 | digest = sha.sha(data).hexdigest() | 30 | digest = hashlib.sha1(data).hexdigest() |
| 367 | 30 | newfile = self.storage.startAddFile('file1', len(data)) | 31 | newfile = self.storage.startAddFile('file1', len(data)) |
| 368 | 31 | newfile.srcDigest = digest | 32 | newfile.srcDigest = digest |
| 369 | 32 | newfile.append(data) | 33 | newfile.append(data) |
| 370 | @@ -36,7 +37,7 @@ | |||
| 371 | 36 | def test_addFiles_identical(self): | 37 | def test_addFiles_identical(self): |
| 372 | 37 | # Start adding two files with identical data | 38 | # Start adding two files with identical data |
| 373 | 38 | data = 'data ' * 5000 | 39 | data = 'data ' * 5000 |
| 375 | 39 | digest = sha.sha(data).hexdigest() | 40 | digest = hashlib.sha1(data).hexdigest() |
| 376 | 40 | newfile1 = self.storage.startAddFile('file1', len(data)) | 41 | newfile1 = self.storage.startAddFile('file1', len(data)) |
| 377 | 41 | newfile2 = self.storage.startAddFile('file2', len(data)) | 42 | newfile2 = self.storage.startAddFile('file2', len(data)) |
| 378 | 42 | newfile1.append(data) | 43 | newfile1.append(data) |
| 379 | @@ -63,7 +64,7 @@ | |||
| 380 | 63 | def test_alias(self): | 64 | def test_alias(self): |
| 381 | 64 | # Add a file (and so also add an alias) | 65 | # Add a file (and so also add an alias) |
| 382 | 65 | data = 'data ' * 50 | 66 | data = 'data ' * 50 |
| 384 | 66 | digest = sha.sha(data).hexdigest() | 67 | digest = hashlib.sha1(data).hexdigest() |
| 385 | 67 | newfile = self.storage.startAddFile('file1', len(data)) | 68 | newfile = self.storage.startAddFile('file1', len(data)) |
| 386 | 68 | newfile.mimetype = 'text/unknown' | 69 | newfile.mimetype = 'text/unknown' |
| 387 | 69 | newfile.append(data) | 70 | newfile.append(data) |
| 388 | 70 | 71 | ||
| 389 | === modified file 'lib/canonical/librarian/storage.py' | |||
| 390 | --- lib/canonical/librarian/storage.py 2009-07-17 00:26:05 +0000 | |||
| 391 | +++ lib/canonical/librarian/storage.py 2010-02-10 00:32:23 +0000 | |||
| 392 | @@ -4,9 +4,8 @@ | |||
| 393 | 4 | __metaclass__ = type | 4 | __metaclass__ = type |
| 394 | 5 | 5 | ||
| 395 | 6 | import os | 6 | import os |
| 396 | 7 | import md5 | ||
| 397 | 8 | import sha | ||
| 398 | 9 | import errno | 7 | import errno |
| 399 | 8 | import hashlib | ||
| 400 | 10 | import shutil | 9 | import shutil |
| 401 | 11 | import tempfile | 10 | import tempfile |
| 402 | 12 | 11 | ||
| 403 | @@ -89,8 +88,8 @@ | |||
| 404 | 89 | tmpfile, tmpfilepath = tempfile.mkstemp(dir=self.storage.incoming) | 88 | tmpfile, tmpfilepath = tempfile.mkstemp(dir=self.storage.incoming) |
| 405 | 90 | self.tmpfile = os.fdopen(tmpfile, 'w') | 89 | self.tmpfile = os.fdopen(tmpfile, 'w') |
| 406 | 91 | self.tmpfilepath = tmpfilepath | 90 | self.tmpfilepath = tmpfilepath |
| 409 | 92 | self.shaDigester = sha.new() | 91 | self.shaDigester = hashlib.sha1() |
| 410 | 93 | self.md5Digester = md5.new() | 92 | self.md5Digester = hashlib.md5() |
| 411 | 94 | 93 | ||
| 412 | 95 | def append(self, data): | 94 | def append(self, data): |
| 413 | 96 | self.tmpfile.write(data) | 95 | self.tmpfile.write(data) |
| 414 | 97 | 96 | ||
| 415 | === modified file 'lib/canonical/librarian/tests/test_storage.py' | |||
| 416 | --- lib/canonical/librarian/tests/test_storage.py 2009-07-19 04:41:14 +0000 | |||
| 417 | +++ lib/canonical/librarian/tests/test_storage.py 2010-02-10 00:32:23 +0000 | |||
| 418 | @@ -2,7 +2,6 @@ | |||
| 419 | 2 | # GNU Affero General Public License version 3 (see the file LICENSE). | 2 | # GNU Affero General Public License version 3 (see the file LICENSE). |
| 420 | 3 | 3 | ||
| 421 | 4 | import os | 4 | import os |
| 422 | 5 | import sha | ||
| 423 | 6 | import shutil | 5 | import shutil |
| 424 | 7 | import tempfile | 6 | import tempfile |
| 425 | 8 | import unittest | 7 | import unittest |
| 426 | 9 | 8 | ||
| 427 | === modified file 'lib/canonical/librarian/utils.py' | |||
| 428 | --- lib/canonical/librarian/utils.py 2009-06-25 05:30:52 +0000 | |||
| 429 | +++ lib/canonical/librarian/utils.py 2010-02-10 00:32:23 +0000 | |||
| 430 | @@ -2,7 +2,6 @@ | |||
| 431 | 2 | # GNU Affero General Public License version 3 (see the file LICENSE). | 2 | # GNU Affero General Public License version 3 (see the file LICENSE). |
| 432 | 3 | 3 | ||
| 433 | 4 | __metaclass__ = type | 4 | __metaclass__ = type |
| 434 | 5 | |||
| 435 | 6 | __all__ = [ | 5 | __all__ = [ |
| 436 | 7 | 'copy_and_close', | 6 | 'copy_and_close', |
| 437 | 8 | 'filechunks', | 7 | 'filechunks', |
| 438 | @@ -10,7 +9,8 @@ | |||
| 439 | 10 | 'sha1_from_path', | 9 | 'sha1_from_path', |
| 440 | 11 | ] | 10 | ] |
| 441 | 12 | 11 | ||
| 443 | 13 | import sha | 12 | |
| 444 | 13 | import hashlib | ||
| 445 | 14 | 14 | ||
| 446 | 15 | MEGABYTE = 1024*1024 | 15 | MEGABYTE = 1024*1024 |
| 447 | 16 | 16 | ||
| 448 | @@ -39,7 +39,7 @@ | |||
| 449 | 39 | def sha1_from_path(path): | 39 | def sha1_from_path(path): |
| 450 | 40 | """Return the hexdigest SHA1 for the contents of the path.""" | 40 | """Return the hexdigest SHA1 for the contents of the path.""" |
| 451 | 41 | the_file = open(path) | 41 | the_file = open(path) |
| 453 | 42 | the_hash = sha.new() | 42 | the_hash = hashlib.sha1() |
| 454 | 43 | 43 | ||
| 455 | 44 | for chunk in filechunks(the_file): | 44 | for chunk in filechunks(the_file): |
| 456 | 45 | the_hash.update(chunk) | 45 | the_hash.update(chunk) |
| 457 | 46 | 46 | ||
| 458 | === modified file 'lib/lp/archivepublisher/library.py' | |||
| 459 | --- lib/lp/archivepublisher/library.py 2009-06-24 23:28:16 +0000 | |||
| 460 | +++ lib/lp/archivepublisher/library.py 2010-02-10 00:32:23 +0000 | |||
| 461 | @@ -113,7 +113,9 @@ | |||
| 462 | 113 | return os.link(path, archive) | 113 | return os.link(path, archive) |
| 463 | 114 | 114 | ||
| 464 | 115 | if __name__ == '__main__': | 115 | if __name__ == '__main__': |
| 466 | 116 | import os, sys, sha | 116 | import hashlib |
| 467 | 117 | import os | ||
| 468 | 118 | import sys | ||
| 469 | 117 | 119 | ||
| 470 | 118 | lib = Librarian('localhost', 9090, 8000, "/tmp/cache") | 120 | lib = Librarian('localhost', 9090, 8000, "/tmp/cache") |
| 471 | 119 | 121 | ||
| 472 | @@ -124,7 +126,7 @@ | |||
| 473 | 124 | lib.upload_port) | 126 | lib.upload_port) |
| 474 | 125 | fileobj = open(name, 'rb') | 127 | fileobj = open(name, 'rb') |
| 475 | 126 | size = os.stat(name).st_size | 128 | size = os.stat(name).st_size |
| 477 | 127 | digest = sha.sha(open(name, 'rb').read()).hexdigest() | 129 | digest = hashlib.sha1(open(name, 'rb').read()).hexdigest() |
| 478 | 128 | 130 | ||
| 479 | 129 | fileid, filealias = lib.addFile(name, size, fileobj, | 131 | fileid, filealias = lib.addFile(name, size, fileobj, |
| 480 | 130 | contentType='test/test', | 132 | contentType='test/test', |
| 481 | 131 | 133 | ||
| 482 | === modified file 'lib/lp/archivepublisher/publishing.py' | |||
| 483 | --- lib/lp/archivepublisher/publishing.py 2009-12-14 17:32:06 +0000 | |||
| 484 | +++ lib/lp/archivepublisher/publishing.py 2010-02-10 00:32:23 +0000 | |||
| 485 | @@ -9,12 +9,11 @@ | |||
| 486 | 9 | 9 | ||
| 487 | 10 | __metaclass__ = type | 10 | __metaclass__ = type |
| 488 | 11 | 11 | ||
| 491 | 12 | import apt_pkg | 12 | import hashlib |
| 490 | 13 | from datetime import datetime | ||
| 492 | 14 | import logging | 13 | import logging |
| 493 | 15 | from md5 import md5 | ||
| 494 | 16 | import os | 14 | import os |
| 496 | 17 | from sha import sha | 15 | |
| 497 | 16 | from datetime import datetime | ||
| 498 | 18 | 17 | ||
| 499 | 19 | from zope.component import getUtility | 18 | from zope.component import getUtility |
| 500 | 20 | 19 | ||
| 501 | @@ -59,23 +58,6 @@ | |||
| 502 | 59 | Architecture: %s | 58 | Architecture: %s |
| 503 | 60 | """ | 59 | """ |
| 504 | 61 | 60 | ||
| 505 | 62 | class sha256: | ||
| 506 | 63 | """Encapsulates apt_pkg.sha256sum as expected by publishing. | ||
| 507 | 64 | |||
| 508 | 65 | It implements '__init__' and 'hexdigest' methods from PEP-247, which are | ||
| 509 | 66 | the only ones required in soyuz-publishing-system. | ||
| 510 | 67 | |||
| 511 | 68 | It's a work around for broken Crypto.Hash.SHA256. See further information | ||
| 512 | 69 | in bug #131503. | ||
| 513 | 70 | """ | ||
| 514 | 71 | def __init__(self, content): | ||
| 515 | 72 | self._sum = apt_pkg.sha256sum(content) | ||
| 516 | 73 | |||
| 517 | 74 | def hexdigest(self): | ||
| 518 | 75 | """Return the hexdigest produced by apt_pkg.sha256sum.""" | ||
| 519 | 76 | return self._sum | ||
| 520 | 77 | |||
| 521 | 78 | |||
| 522 | 79 | def reorder_components(components): | 61 | def reorder_components(components): |
| 523 | 80 | """Return a list of the components provided. | 62 | """Return a list of the components provided. |
| 524 | 81 | 63 | ||
| 525 | @@ -518,13 +500,13 @@ | |||
| 526 | 518 | f.write("MD5Sum:\n") | 500 | f.write("MD5Sum:\n") |
| 527 | 519 | all_files = sorted(list(all_files), key=os.path.dirname) | 501 | all_files = sorted(list(all_files), key=os.path.dirname) |
| 528 | 520 | for file_name in all_files: | 502 | for file_name in all_files: |
| 530 | 521 | self._writeSumLine(full_name, f, file_name, md5) | 503 | self._writeSumLine(full_name, f, file_name, hashlib.md5) |
| 531 | 522 | f.write("SHA1:\n") | 504 | f.write("SHA1:\n") |
| 532 | 523 | for file_name in all_files: | 505 | for file_name in all_files: |
| 534 | 524 | self._writeSumLine(full_name, f, file_name, sha) | 506 | self._writeSumLine(full_name, f, file_name, hashlib.sha1) |
| 535 | 525 | f.write("SHA256:\n") | 507 | f.write("SHA256:\n") |
| 536 | 526 | for file_name in all_files: | 508 | for file_name in all_files: |
| 538 | 527 | self._writeSumLine(full_name, f, file_name, sha256) | 509 | self._writeSumLine(full_name, f, file_name, hashlib.sha256) |
| 539 | 528 | 510 | ||
| 540 | 529 | f.close() | 511 | f.close() |
| 541 | 530 | 512 | ||
| 542 | @@ -607,16 +589,8 @@ | |||
| 543 | 607 | 589 | ||
| 544 | 608 | in_file = open(full_name, 'r') | 590 | in_file = open(full_name, 'r') |
| 545 | 609 | try: | 591 | try: |
| 556 | 610 | # XXX cprov 20080704 bug=243630,269014: Workaround for hardy's | 592 | contents = in_file.read() |
| 557 | 611 | # python-apt. If it receives a file object as an argument instead | 593 | length = len(contents) |
| 548 | 612 | # of the file contents as a string, it will generate the correct | ||
| 549 | 613 | # SHA256. | ||
| 550 | 614 | if sum_form == sha256: | ||
| 551 | 615 | contents = in_file | ||
| 552 | 616 | length = os.stat(full_name).st_size | ||
| 553 | 617 | else: | ||
| 554 | 618 | contents = in_file.read() | ||
| 555 | 619 | length = len(contents) | ||
| 558 | 620 | checksum = sum_form(contents).hexdigest() | 594 | checksum = sum_form(contents).hexdigest() |
| 559 | 621 | finally: | 595 | finally: |
| 560 | 622 | in_file.close() | 596 | in_file.close() |
| 561 | 623 | 597 | ||
| 562 | === modified file 'lib/lp/archivepublisher/tests/test_librarianwrapper.py' | |||
| 563 | --- lib/lp/archivepublisher/tests/test_librarianwrapper.py 2009-06-24 23:28:16 +0000 | |||
| 564 | +++ lib/lp/archivepublisher/tests/test_librarianwrapper.py 2010-02-10 00:32:23 +0000 | |||
| 565 | @@ -5,9 +5,9 @@ | |||
| 566 | 5 | 5 | ||
| 567 | 6 | __metaclass__ = type | 6 | __metaclass__ = type |
| 568 | 7 | 7 | ||
| 569 | 8 | import hashlib | ||
| 570 | 8 | import os | 9 | import os |
| 571 | 9 | import shutil | 10 | import shutil |
| 572 | 10 | import sha | ||
| 573 | 11 | import sys | 11 | import sys |
| 574 | 12 | import unittest | 12 | import unittest |
| 575 | 13 | 13 | ||
| 576 | @@ -47,7 +47,7 @@ | |||
| 577 | 47 | 47 | ||
| 578 | 48 | fileobj = open(path, 'rb') | 48 | fileobj = open(path, 'rb') |
| 579 | 49 | size = os.stat(path).st_size | 49 | size = os.stat(path).st_size |
| 581 | 50 | digest = sha.sha(open(path, 'rb').read()).hexdigest() | 50 | digest = hashlib.sha1(open(path, 'rb').read()).hexdigest() |
| 582 | 51 | 51 | ||
| 583 | 52 | ## Use Fake Librarian class | 52 | ## Use Fake Librarian class |
| 584 | 53 | uploader = FakeUploadClient() | 53 | uploader = FakeUploadClient() |
| 585 | 54 | 54 | ||
| 586 | === modified file 'lib/lp/archivepublisher/tests/test_pool.py' | |||
| 587 | --- lib/lp/archivepublisher/tests/test_pool.py 2009-06-24 23:28:16 +0000 | |||
| 588 | +++ lib/lp/archivepublisher/tests/test_pool.py 2010-02-10 00:32:23 +0000 | |||
| 589 | @@ -5,13 +5,13 @@ | |||
| 590 | 5 | 5 | ||
| 591 | 6 | __metaclass__ = type | 6 | __metaclass__ = type |
| 592 | 7 | 7 | ||
| 594 | 8 | 8 | import hashlib | |
| 595 | 9 | import os | 9 | import os |
| 596 | 10 | import sha | ||
| 597 | 11 | import sys | 10 | import sys |
| 598 | 12 | import shutil | 11 | import shutil |
| 599 | 12 | import unittest | ||
| 600 | 13 | |||
| 601 | 13 | from tempfile import mkdtemp | 14 | from tempfile import mkdtemp |
| 602 | 14 | import unittest | ||
| 603 | 15 | 15 | ||
| 604 | 16 | from lp.archivepublisher.tests.util import FakeLogger | 16 | from lp.archivepublisher.tests.util import FakeLogger |
| 605 | 17 | from lp.archivepublisher.diskpool import DiskPool, poolify | 17 | from lp.archivepublisher.diskpool import DiskPool, poolify |
| 606 | @@ -44,7 +44,7 @@ | |||
| 607 | 44 | def addToPool(self, component): | 44 | def addToPool(self, component): |
| 608 | 45 | return self.pool.addFile( | 45 | return self.pool.addFile( |
| 609 | 46 | component, self.sourcename, self.filename, | 46 | component, self.sourcename, self.filename, |
| 611 | 47 | sha.sha(self.contents).hexdigest(), MockFile(self.contents)) | 47 | hashlib.sha1(self.contents).hexdigest(), MockFile(self.contents)) |
| 612 | 48 | 48 | ||
| 613 | 49 | def removeFromPool(self, component): | 49 | def removeFromPool(self, component): |
| 614 | 50 | return self.pool.removeFile(component, self.sourcename, self.filename) | 50 | return self.pool.removeFile(component, self.sourcename, self.filename) |
| 615 | 51 | 51 | ||
| 616 | === modified file 'lib/lp/archivepublisher/tests/test_publisher.py' | |||
| 617 | --- lib/lp/archivepublisher/tests/test_publisher.py 2009-12-21 18:05:27 +0000 | |||
| 618 | +++ lib/lp/archivepublisher/tests/test_publisher.py 2010-02-10 00:32:23 +0000 | |||
| 619 | @@ -8,6 +8,7 @@ | |||
| 620 | 8 | 8 | ||
| 621 | 9 | import bz2 | 9 | import bz2 |
| 622 | 10 | import gzip | 10 | import gzip |
| 623 | 11 | import hashlib | ||
| 624 | 11 | import os | 12 | import os |
| 625 | 12 | import shutil | 13 | import shutil |
| 626 | 13 | import stat | 14 | import stat |
| 627 | @@ -20,8 +21,7 @@ | |||
| 628 | 20 | 21 | ||
| 629 | 21 | from lp.archivepublisher.config import getPubConfig | 22 | from lp.archivepublisher.config import getPubConfig |
| 630 | 22 | from lp.archivepublisher.diskpool import DiskPool | 23 | from lp.archivepublisher.diskpool import DiskPool |
| 633 | 23 | from lp.archivepublisher.publishing import ( | 24 | from lp.archivepublisher.publishing import Publisher, getPublisher |
| 632 | 24 | Publisher, getPublisher, sha256) | ||
| 634 | 25 | from canonical.config import config | 25 | from canonical.config import config |
| 635 | 26 | from canonical.database.constants import UTC_NOW | 26 | from canonical.database.constants import UTC_NOW |
| 636 | 27 | from canonical.launchpad.ftests.keys_for_tests import gpgkeysdir | 27 | from canonical.launchpad.ftests.keys_for_tests import gpgkeysdir |
| 637 | @@ -745,70 +745,6 @@ | |||
| 638 | 745 | self.assertReleaseFileRequested( | 745 | self.assertReleaseFileRequested( |
| 639 | 746 | publisher, 'breezy-autotest', component, dist) | 746 | publisher, 'breezy-autotest', component, dist) |
| 640 | 747 | 747 | ||
| 641 | 748 | def testAptSHA256(self): | ||
| 642 | 749 | """Test issues with python-apt in Ubuntu/hardy. | ||
| 643 | 750 | |||
| 644 | 751 | This test only runs on Ubuntu/hardy systems. | ||
| 645 | 752 | |||
| 646 | 753 | The version of python-apt in Ubuntu/hardy has problems with | ||
| 647 | 754 | contents containing '\0' character. | ||
| 648 | 755 | |||
| 649 | 756 | The documented workaround for it is passing the original | ||
| 650 | 757 | file-descriptor to apt_pkg.sha256sum(), instead of its contents. | ||
| 651 | 758 | |||
| 652 | 759 | The python-apt version in Ubuntu/Intrepid has a fix for this issue, | ||
| 653 | 760 | but it already has many other features that makes a backport | ||
| 654 | 761 | practically unfeasible. That's mainly why this 'bug' is documented | ||
| 655 | 762 | as a LP test, the current code was modified to cope with it. | ||
| 656 | 763 | |||
| 657 | 764 | Once the issue with python-apt is gone, either by having a backport | ||
| 658 | 765 | available in hardy or a production upgrade, this test will fail. At | ||
| 659 | 766 | that point we will be able to revert the affected code and remove | ||
| 660 | 767 | this test, restoring the balance of the force. | ||
| 661 | 768 | |||
| 662 | 769 | See https://bugs.edge.launchpad.net/soyuz/+bug/243630 and | ||
| 663 | 770 | https://bugs.edge.launchpad.net/soyuz/+bug/269014. | ||
| 664 | 771 | """ | ||
| 665 | 772 | # XXX cprov 20090218 bug-279248: when hardy's apt gets fixed by a | ||
| 666 | 773 | # SRU, this test will fail in PQM/Buildbot. Then we should change | ||
| 667 | 774 | # the actual code for passing file descriptors instead of text to | ||
| 668 | 775 | # apt (it will perform better this way) and obviously remove this | ||
| 669 | 776 | # test. | ||
| 670 | 777 | |||
| 671 | 778 | # Skip this test if it's not being run on Ubuntu/hardy. | ||
| 672 | 779 | lsb_info = get_lsb_information() | ||
| 673 | 780 | if (lsb_info.get('ID') != 'Ubuntu' or | ||
| 674 | 781 | lsb_info.get('CODENAME') != 'hardy'): | ||
| 675 | 782 | return | ||
| 676 | 783 | |||
| 677 | 784 | def _getSHA256(content): | ||
| 678 | 785 | """Return checksums for the given content. | ||
| 679 | 786 | |||
| 680 | 787 | Return a tuple containing the checksum corresponding to the | ||
| 681 | 788 | given content (as string) and a file containing the same string. | ||
| 682 | 789 | """ | ||
| 683 | 790 | # Write the given content in a tempfile. | ||
| 684 | 791 | test_filepath = tempfile.mktemp() | ||
| 685 | 792 | test_file = open(test_filepath, 'w') | ||
| 686 | 793 | test_file.write(content) | ||
| 687 | 794 | test_file.close() | ||
| 688 | 795 | # Generate the checksums for the two sources. | ||
| 689 | 796 | text = sha256(content).hexdigest() | ||
| 690 | 797 | file = sha256(open(test_filepath)).hexdigest() | ||
| 691 | 798 | # Remove the tempfile. | ||
| 692 | 799 | os.unlink(test_filepath) | ||
| 693 | 800 | return text, file | ||
| 694 | 801 | |||
| 695 | 802 | # Apt does the right thing for ordinary strings, both, file and text | ||
| 696 | 803 | # checksums are identical. | ||
| 697 | 804 | text, file = _getSHA256("foobar") | ||
| 698 | 805 | self.assertEqual(text, file) | ||
| 699 | 806 | |||
| 700 | 807 | # On the other hand, there is a mismatch for strings containing '\0' | ||
| 701 | 808 | text, file = _getSHA256("foo\0bar") | ||
| 702 | 809 | self.assertNotEqual( | ||
| 703 | 810 | text, file, "Python-apt no longer creates bad SHA256 sums.") | ||
| 704 | 811 | |||
| 705 | 812 | def _getReleaseFileOrigin(self, contents): | 748 | def _getReleaseFileOrigin(self, contents): |
| 706 | 813 | origin_header = 'Origin: ' | 749 | origin_header = 'Origin: ' |
| 707 | 814 | [origin_line] = [ | 750 | [origin_line] = [ |
| 708 | 815 | 751 | ||
| 709 | === modified file 'lib/lp/archiveuploader/nascentuploadfile.py' | |||
| 710 | --- lib/lp/archiveuploader/nascentuploadfile.py 2009-12-14 13:49:03 +0000 | |||
| 711 | +++ lib/lp/archiveuploader/nascentuploadfile.py 2010-02-10 00:32:23 +0000 | |||
| 712 | @@ -21,9 +21,8 @@ | |||
| 713 | 21 | 21 | ||
| 714 | 22 | import apt_inst | 22 | import apt_inst |
| 715 | 23 | import apt_pkg | 23 | import apt_pkg |
| 716 | 24 | import hashlib | ||
| 717 | 24 | import os | 25 | import os |
| 718 | 25 | import md5 | ||
| 719 | 26 | import sha | ||
| 720 | 27 | import subprocess | 26 | import subprocess |
| 721 | 28 | import sys | 27 | import sys |
| 722 | 29 | import time | 28 | import time |
| 723 | @@ -214,8 +213,8 @@ | |||
| 724 | 214 | 213 | ||
| 725 | 215 | # Read in the file and compute its md5 and sha1 checksums and remember | 214 | # Read in the file and compute its md5 and sha1 checksums and remember |
| 726 | 216 | # the size of the file as read-in. | 215 | # the size of the file as read-in. |
| 729 | 217 | digest = md5.md5() | 216 | digest = hashlib.md5() |
| 730 | 218 | sha_cksum = sha.sha() | 217 | sha_cksum = hashlib.sha1() |
| 731 | 219 | ckfile = open(self.filepath, "r") | 218 | ckfile = open(self.filepath, "r") |
| 732 | 220 | size = 0 | 219 | size = 0 |
| 733 | 221 | for chunk in filechunks(ckfile): | 220 | for chunk in filechunks(ckfile): |
| 734 | 222 | 221 | ||
| 735 | === modified file 'lib/lp/services/mail/sendmail.py' | |||
| 736 | --- lib/lp/services/mail/sendmail.py 2009-07-23 18:34:25 +0000 | |||
| 737 | +++ lib/lp/services/mail/sendmail.py 2010-02-10 00:32:23 +0000 | |||
| 738 | @@ -22,11 +22,14 @@ | |||
| 739 | 22 | 'simple_sendmail', | 22 | 'simple_sendmail', |
| 740 | 23 | 'simple_sendmail_from_person', | 23 | 'simple_sendmail_from_person', |
| 741 | 24 | 'raw_sendmail', | 24 | 'raw_sendmail', |
| 743 | 25 | 'validate_message'] | 25 | 'validate_message', |
| 744 | 26 | ] | ||
| 745 | 27 | |||
| 746 | 28 | |||
| 747 | 29 | import hashlib | ||
| 748 | 30 | import sets | ||
| 749 | 26 | 31 | ||
| 750 | 27 | from binascii import b2a_qp | 32 | from binascii import b2a_qp |
| 751 | 28 | import sha | ||
| 752 | 29 | import sets | ||
| 753 | 30 | from email.Encoders import encode_base64 | 33 | from email.Encoders import encode_base64 |
| 754 | 31 | from email.Utils import getaddresses, make_msgid, formatdate, formataddr | 34 | from email.Utils import getaddresses, make_msgid, formatdate, formataddr |
| 755 | 32 | from email.Message import Message | 35 | from email.Message import Message |
| 756 | @@ -375,7 +378,7 @@ | |||
| 757 | 375 | # helps security, but still exposes us to a replay attack; we consider the | 378 | # helps security, but still exposes us to a replay attack; we consider the |
| 758 | 376 | # risk low. | 379 | # risk low. |
| 759 | 377 | del message['X-Launchpad-Hash'] | 380 | del message['X-Launchpad-Hash'] |
| 761 | 378 | hash = sha.new(config.mailman.shared_secret) | 381 | hash = hashlib.sha1(config.mailman.shared_secret) |
| 762 | 379 | hash.update(str(message['message-id'])) | 382 | hash.update(str(message['message-id'])) |
| 763 | 380 | message['X-Launchpad-Hash'] = hash.hexdigest() | 383 | message['X-Launchpad-Hash'] = hash.hexdigest() |
| 764 | 381 | 384 | ||
| 765 | 382 | 385 | ||
| 766 | === modified file 'lib/lp/soyuz/doc/soyuz-upload.txt.disabled' | |||
| 767 | --- lib/lp/soyuz/doc/soyuz-upload.txt.disabled 2009-05-13 14:05:27 +0000 | |||
| 768 | +++ lib/lp/soyuz/doc/soyuz-upload.txt.disabled 2010-02-10 00:32:23 +0000 | |||
| 769 | @@ -144,9 +144,9 @@ | |||
| 770 | 144 | each FTP session. Below we ensure that, and also that the content | 144 | each FTP session. Below we ensure that, and also that the content |
| 771 | 145 | of these files match the uploaded ones. | 145 | of these files match the uploaded ones. |
| 772 | 146 | 146 | ||
| 774 | 147 | >>> import md5 | 147 | >>> import hashlib |
| 775 | 148 | >>> def get_md5(filename): | 148 | >>> def get_md5(filename): |
| 777 | 149 | ... return md5.new(open(filename).read()).digest() | 149 | ... return hashlib.md5(open(filename).read()).digest() |
| 778 | 150 | 150 | ||
| 779 | 151 | >>> def get_upload_dir(num, dir=incoming_dir): | 151 | >>> def get_upload_dir(num, dir=incoming_dir): |
| 780 | 152 | ... """Return the path to the upload, if found in the dir.""" | 152 | ... """Return the path to the upload, if found in the dir.""" |
| 781 | 153 | 153 | ||
| 782 | === modified file 'lib/lp/soyuz/scripts/ftpmaster.py' | |||
| 783 | --- lib/lp/soyuz/scripts/ftpmaster.py 2009-12-13 11:55:40 +0000 | |||
| 784 | +++ lib/lp/soyuz/scripts/ftpmaster.py 2010-02-10 00:32:23 +0000 | |||
| 785 | @@ -21,7 +21,7 @@ | |||
| 786 | 21 | 21 | ||
| 787 | 22 | import apt_pkg | 22 | import apt_pkg |
| 788 | 23 | import commands | 23 | import commands |
| 790 | 24 | import md5 | 24 | import hashlib |
| 791 | 25 | import os | 25 | import os |
| 792 | 26 | import stat | 26 | import stat |
| 793 | 27 | import sys | 27 | import sys |
| 794 | @@ -836,7 +836,7 @@ | |||
| 795 | 836 | @classmethod | 836 | @classmethod |
| 796 | 837 | def generateMD5Sum(self, filename): | 837 | def generateMD5Sum(self, filename): |
| 797 | 838 | file_handle = open(filename) | 838 | file_handle = open(filename) |
| 799 | 839 | md5sum = md5.md5(file_handle.read()).hexdigest() | 839 | md5sum = hashlib.md5(file_handle.read()).hexdigest() |
| 800 | 840 | file_handle.close() | 840 | file_handle.close() |
| 801 | 841 | return md5sum | 841 | return md5sum |
| 802 | 842 | 842 | ||
| 803 | 843 | 843 | ||
| 804 | === modified file 'lib/lp/soyuz/scripts/gina/library.py' | |||
| 805 | --- lib/lp/soyuz/scripts/gina/library.py 2009-06-25 04:06:00 +0000 | |||
| 806 | +++ lib/lp/soyuz/scripts/gina/library.py 2010-02-10 00:32:23 +0000 | |||
| 807 | @@ -5,12 +5,15 @@ | |||
| 808 | 5 | 5 | ||
| 809 | 6 | __metaclass__ = type | 6 | __metaclass__ = type |
| 810 | 7 | 7 | ||
| 811 | 8 | |||
| 812 | 9 | import hashlib | ||
| 813 | 8 | import os | 10 | import os |
| 814 | 9 | import sha | ||
| 815 | 10 | 11 | ||
| 816 | 11 | from zope.component import getUtility | 12 | from zope.component import getUtility |
| 817 | 12 | 13 | ||
| 818 | 13 | from canonical.launchpad.interfaces.librarian import ILibraryFileAliasSet | 14 | from canonical.launchpad.interfaces.librarian import ILibraryFileAliasSet |
| 819 | 15 | |||
| 820 | 16 | |||
| 821 | 14 | def _libType(fname): | 17 | def _libType(fname): |
| 822 | 15 | if fname.endswith(".dsc"): | 18 | if fname.endswith(".dsc"): |
| 823 | 16 | return "text/x-debian-source-package" | 19 | return "text/x-debian-source-package" |
| 824 | @@ -41,7 +44,7 @@ | |||
| 825 | 41 | def checkLibraryForFile(path, filename): | 44 | def checkLibraryForFile(path, filename): |
| 826 | 42 | fullpath = os.path.join(path, filename) | 45 | fullpath = os.path.join(path, filename) |
| 827 | 43 | assert os.path.exists(fullpath) | 46 | assert os.path.exists(fullpath) |
| 829 | 44 | digester = sha.sha() | 47 | digester = hashlib.sha1() |
| 830 | 45 | openfile = open(fullpath, "r") | 48 | openfile = open(fullpath, "r") |
| 831 | 46 | for chunk in iter(lambda: openfile.read(1024*4), ''): | 49 | for chunk in iter(lambda: openfile.read(1024*4), ''): |
| 832 | 47 | digester.update(chunk) | 50 | digester.update(chunk) |
| 833 | 48 | 51 | ||
| 834 | === modified file 'lib/lp/soyuz/scripts/queue.py' | |||
| 835 | --- lib/lp/soyuz/scripts/queue.py 2009-08-28 06:39:38 +0000 | |||
| 836 | +++ lib/lp/soyuz/scripts/queue.py 2010-02-10 00:32:23 +0000 | |||
| 837 | @@ -9,7 +9,6 @@ | |||
| 838 | 9 | # as Launchpad contains lots of queues. | 9 | # as Launchpad contains lots of queues. |
| 839 | 10 | 10 | ||
| 840 | 11 | __metaclass__ = type | 11 | __metaclass__ = type |
| 841 | 12 | |||
| 842 | 13 | __all__ = [ | 12 | __all__ = [ |
| 843 | 14 | 'CommandRunner', | 13 | 'CommandRunner', |
| 844 | 15 | 'CommandRunnerError', | 14 | 'CommandRunnerError', |
| 845 | @@ -17,11 +16,12 @@ | |||
| 846 | 17 | 'name_queue_map' | 16 | 'name_queue_map' |
| 847 | 18 | ] | 17 | ] |
| 848 | 19 | 18 | ||
| 849 | 19 | |||
| 850 | 20 | import errno | 20 | import errno |
| 851 | 21 | import hashlib | ||
| 852 | 21 | import pytz | 22 | import pytz |
| 853 | 22 | 23 | ||
| 854 | 23 | from datetime import datetime | 24 | from datetime import datetime |
| 855 | 24 | from sha import sha | ||
| 856 | 25 | 25 | ||
| 857 | 26 | from zope.component import getUtility | 26 | from zope.component import getUtility |
| 858 | 27 | 27 | ||
| 859 | @@ -420,7 +420,7 @@ | |||
| 860 | 420 | libfile.close() | 420 | libfile.close() |
| 861 | 421 | else: | 421 | else: |
| 862 | 422 | # Check sha against existing file (bug #67014) | 422 | # Check sha against existing file (bug #67014) |
| 864 | 423 | existing_sha = sha() | 423 | existing_sha = hashlib.sha1() |
| 865 | 424 | for chunk in filechunks(existing_file): | 424 | for chunk in filechunks(existing_file): |
| 866 | 425 | existing_sha.update(chunk) | 425 | existing_sha.update(chunk) |
| 867 | 426 | existing_file.close() | 426 | existing_file.close() |
| 868 | 427 | 427 | ||
| 869 | === modified file 'lib/lp/soyuz/scripts/tests/test_queue.py' | |||
| 870 | --- lib/lp/soyuz/scripts/tests/test_queue.py 2009-12-13 11:55:40 +0000 | |||
| 871 | +++ lib/lp/soyuz/scripts/tests/test_queue.py 2010-02-10 00:32:23 +0000 | |||
| 872 | @@ -4,14 +4,17 @@ | |||
| 873 | 4 | """queue tool base class tests.""" | 4 | """queue tool base class tests.""" |
| 874 | 5 | 5 | ||
| 875 | 6 | __metaclass__ = type | 6 | __metaclass__ = type |
| 879 | 7 | 7 | __all__ = [ | |
| 880 | 8 | __all__ = ['upload_bar_source'] | 8 | 'upload_bar_source', |
| 881 | 9 | 9 | ] | |
| 882 | 10 | |||
| 883 | 11 | |||
| 884 | 12 | import hashlib | ||
| 885 | 10 | import os | 13 | import os |
| 886 | 11 | import shutil | 14 | import shutil |
| 887 | 12 | import tempfile | 15 | import tempfile |
| 888 | 16 | |||
| 889 | 13 | from unittest import TestCase, TestLoader | 17 | from unittest import TestCase, TestLoader |
| 890 | 14 | from sha import sha | ||
| 891 | 15 | 18 | ||
| 892 | 16 | from zope.component import getUtility | 19 | from zope.component import getUtility |
| 893 | 17 | from zope.security.proxy import removeSecurityProxy | 20 | from zope.security.proxy import removeSecurityProxy |
| 894 | @@ -929,7 +932,7 @@ | |||
| 895 | 929 | 932 | ||
| 896 | 930 | def _getsha1(self, filename): | 933 | def _getsha1(self, filename): |
| 897 | 931 | """Return a sha1 hex digest of a file""" | 934 | """Return a sha1 hex digest of a file""" |
| 899 | 932 | file_sha = sha() | 935 | file_sha = hashlib.sha1() |
| 900 | 933 | opened_file = open(filename,"r") | 936 | opened_file = open(filename,"r") |
| 901 | 934 | for chunk in filechunks(opened_file): | 937 | for chunk in filechunks(opened_file): |
| 902 | 935 | file_sha.update(chunk) | 938 | file_sha.update(chunk) |
Python 2.5 introduced the hashlib module as the new home of hash functions.
Python 2.6 deprecated the old md5 and sha modules.
To help in clearing the way for a future migration to Python 2.6, here is a branch substituting the use of hashlib for md5 and sha. It also replaces the usage of python-apt's sha256 function (formerly necessary since Python 2.4 offered no sha256 implementation) with hashlib.sha256, allowing a removal of compatibility code.
These changes were originally authored by Barry during the Python 2.5/2.6 sprint, and left on a branch including Python-2.6-only changes at the end of the sprint. I've disentangled them for landing on mainline.